Low-level service software forms the foundation upon which reliable systems are built. Yet, this crucial software is the most over-looked component of the system. No matter how well conceived and well executed the higher-level application software, the system will never be reliable without robust system services. These services include inter-process communications, event reporting, system timing, software health verification, and non-volatile memory management. As software systems become increasingly complex, well-designed system services are critical to rapidly develop reliable software systems.

This paper discusses common system services and explores techniques for implementing them. It explains three architectural approaches for implementing services in the context of a real-time operating system and provides concrete examples of each. The paper covers design issues such as mutual exclusion, interrupt safety, reentrancy, timing and race-conditions, and blocking and non-blocking services. It also explores the dangers that the service designer faces. Well-designed system services promote modularity, flexibility, reliability, and portability. Poorly implemented services lead to instability, unreliability, insidious bugs, and code bloat.