March 14, 2007

Middleware: The last roadblock for distributed systems--Part II

Most systems share only a fraction of the information required to develop a truly distributed application. Why is that? The information is there. The network is fast, cheap, and reliable. Unfortunately, today, getting the data---the right data at the right time---isn't that simple. Enter Data Distribution Service (DDS).

By Dr. Stan Schneider, Real-Time Innovations, Inc.

Although it may be tough, solving the networked real-time challenge is critical. This challenge is addressed by middleware.

Middleware, the class of software that serves distributed applications by delivering data, is a layer between the ubiquitous network stack and the user application. The stack provides fundamental access to the network hardware and low-level routing and connection management. The application software, a set of custom-written modules, implements the various parts of the particular system. The middleware delivers data to the application software modules by using the underlying stack. The middleware's fundamental job is to find the right data, know where to send it, and deliver it to the right place at the right time.

So, what makes middleware real time? The easy answer would be that middleware is real time if it can always process requests in a sufficiently-short deterministic timeframe . Unfortunately, this is rarely possible or even definable. Most designers cannot assume a reliable or strictly time-deterministic underlying network transport, as most real-time systems must operate without these luxuries. Today's networks are fast; the transport usually can succeed in delivering data on time if properly managed.

The key to distributed real-time middleware is to optimize those last two words: "properly managed." The trick is to put only the exact needed data on the wire at the right time, without violating any of the transport limits. That sounds easy, but in reality, that requires control over many "Quality of Service" (QoS) parameters, including reliability, bandwidth, deadlines, fault, discovery, and resource limits. None of the popular enterprise middleware products, or the embedded messaging systems addresses these constraints. So, the lack of a reliable, capable real-time middleware is the roadblock to complex distributed systems.

The rise of data-centric thought
Most systems built today are connected to a network. However, most are not designed to flexibly deliver data to support truly distributed applications. They are instead designed around clients and servers, application interfaces, and other code-centric concepts. Addressing the middleware roadblock requires more than adding timing QoS to current designs. It requires a change in thought.

The real change is from code-centric or architecture-centric thinking to data-centric design. Instead of configuring clients and servers, or building data-access objects and invoking remote methods, data-centric design implies that the developer directly controls information exchange. Data-centric developers don't write or specify code until very late in the process. They first build a "data dictionary" that defines who needs what data. Then they answer the information questions: How fast is the data coming? Does it need to be reliable? Do I need to store it? Where is it coming from? What happens if a node fails?

With this information in hand, the next task is to map the information flow. Publish-subscribe middleware is the key enabling technology for data-centric design. In contrast to the central server with many clients model of enterprise middleware, publish-subscribe nodes simply subscribe to data they need and publish information they produce. Thus, the data dictionary map is directly translatable to publishers and subscribers. The middleware does the rest: messages pass directly between the communicating nodes. The fundamental communications model implies both discovery--what data should be sent where, and delivery--when and how to send it.

This design should be familiar; it mirrors time-critical information-delivery systems in everyday life. All mass-media communications, including television, radio, magazines, and newspapers, are fundamentally publish-subscribe technologies. Publish-subscribe systems are good at distributing large quantities of time-critical information quickly, even in the presence of unreliable delivery mechanisms.

With direct access to data, system integration is orders-of-magnitude easier. Designers have always built interface specifications that detail which information flows between system components. With a publish-subscribe information bus, the interface is the design; interface specifications can essentially be directly implemented as in Figure 4. This data-centric design technique eliminates numerous intermediate steps, and with them all the overhead and opportunity for error they entrain.