Posts Tagged ‘Distributed computing’

Tickled Pink

April 22, 2013 Leave a comment

Since I’m a fan of DDS, I was tickled pink to see this newsletter appear in my mailbox:


Since the above graphic is a bitmap, clicking on the links won’t work. In case you do want to read the linked-to pages, here they are:

Our Stack

April 10, 2013 Leave a comment

The figure below shows a layered view of the latest distributed system product that I’m working on. Customer teams compose their applications by writing their own system-specific Crumbs and linking them with our pre-written, pre-tested Crumbs. In the ideal case, customers don’t have to write a single line of Crumb code. They simply compose, compile, link, configure, and deploy an amalgamation of our off-the-shelf Crumbs as a set of application components that meets their needs.

Note that we are using C++11 to build the system. Also note the third party, open source libraries that we are building upon. Except for Poco, Crumb developers don’t directly use the OpenDDS or ACE/TAO APIs. Our Crumb “Tray” serves as a wrapper/facade that hides the complexity those inter-process communication facilities.

My role on the development team is as a Libs team “Crumb” designer/writer. If I gave you anymore product views or disclosed anything more concrete, then I’d either get fired or I’d have to kill you, or both.

What are you currently working on?

Our Stack

Message-Centric Vs. Data-Centric

August 13, 2012 Leave a comment

The slide below, plagiarized from a recent webinar presented by RTI Inc’s CEO Stan Schneider, shows the evolution of distributed system middleware over the years.

At first, I couldn’t understand the difference between the message-centric pub-sub (MCPS) and data-centric pub-sub (DCPS) patterns. I thought the difference between them was trivial, superficial, and unimportant. However, as Stan’s webinar unfolded, I sloowly started to “get it“.

In MCPS, application tier messages are opaque to to the middleware (MW). The separation of concerns between the app and MW tiers is clean and simple:

In DCPS systems, app tier messages are transparent to the MW tier – which blurs the line between the two layers and violates the “ideal” separation of concerns tenet of layered software system design. Because of this difference, the term “message” is superceded in DCPS-based technologies (like the OMG‘s DDS) by the term “topic“. The entity formerly known as a “message” is now defined as a topic sample.

Unlike MCPS MW, DCPS MW supports being “told” by the app tier pub-sub components which Quality Of Service (QoS) parameters are important to each of them. For example, a publisher can “promise” to send topic samples at a minimum rate and/or whether it will use a best-effort UDP-like or reliable TCP-like protocol for transport. On the receive side, a subscriber can tell the MW that it only wants to see every third topic sample and/or only those samples in which certain data-field-filtering criteria are met. DCPS MW technologies like DDS support a rich set of QoS parameters that are usually hard-coded and frozen into MCPS MW – if they’re supported at all.

With smart, QoS-aware DCPS MW, app components tend to be leaner and take less time to develop because the tedious logic that implements the QoS functionality is pushed down into the MW. The app simply specifies these behaviors to the MW during launch and it gets notified by the MW during operation when QoS requirements aren’t being, or can’t be, met.

The cost of switching from an MCPS to a DCPS-based distributed system design approach is the increased upfront, one-time, learning curve (or more likely, the “unlearning” curve).

Distributed Functions, Objects, And Data

July 17, 2011 2 comments

During a discussion on, the following distributed system communication architectural “styles” came up:

I felt the need to draw a picture of them, so here it is:

The DF and DO styles are point-to-point, client-server oriented. Client functions invoke functions and object methods invoke object methods on remotely located servers.

The DD style is many-to-many, publisher-subscriber oriented. A publisher can be considered a sort of server and subscribers can be considered clients. The biggest difference is that instead of being client-triggered, communication is server-triggered in DD systems. When new data is available, it is published out onto the net for all subscribers to consume. The components in a DD system are more loosely coupled than those in DF and DO systems in that publishers don’t need to know anything (no handles or method signatures) about subscribers or vice versa – data is king. Nevertheless, there are applications where each of the three types excel over the other two.

Too Much Pressure

The figure below models an N node, “Enterprise Bus” (EB) connected, software-centric system.

Even though the “shared” EB supplies a raw, theoretical bandwidth of 1 Gbps to its connected user set, contention for frequent but temporary control of the bus among its clients increases as the number of message publisher (pubber) clients increases. Thus, the total effective bandwidth of the system as a whole decreases below the raw 1 Gbps rate as its utilization increases.

In the example, instead of accommodating up to 10, 100 Mbps clients, the increasing “back pressure” exerted by the bus under the duress imposed by multiple, “talky” (you know, wife-like) clients will throttle the per-pubber message send rate to well below 100 Mbps.

I recently found this out the hard way – through time-consuming testing and making sense out of the “unexpected” results I was getting. Bummer.

Horse And Buggy

March 21, 2011 10 comments

Werner Vogels is’s Chief Technology Officer and author of the blog “All Things Distributed – Building Scalable And Robust Distributed Systems“. I just couldn’t stop myself from laughing when he announced that fellow distributed-systems guru Steve Vinoski had just joined Twitter:

Specifically, the last sentence put me in stitches. As you can see from the book cover below, Steve is a CORBA expert and way back in the dark ages he was a CORBA advocate. However, he’s moved into the 21st century and left that horse and buggy behind – unlike others who cling to their horse whips and ostracize those who point out the obvious.

Our Stack

November 18, 2010 Leave a comment

The model below shows “our stack”. It’s a layered representation of the real-time, distributed software system that two peers and I are building for our next product line. Of course, I’ve intentionally abstracted away a boatload of details so as not to give away the keys to the store.

The code that we’ve written so far (the stuff in the funky green layers) is up to 50K SLOC of C++ code and we guesstimate that the system will stabilize at around 250K SLOC. In addition to this computationally intensive back end, our system also includes a Java/PHP/HTML display front end, but I haven’t been involved in that area of development. We’ve been continuously refactoring our “App Support Libs” layer as we’ve found turds in it via testing the multi-threaded, domain-specific app components that use them.

The figure below illustrates the flexibility and scalability of our architecture. From our growing pool of application layer components, a large set of product instantiations can be created and deployed on multiple  hardware configurations. In the ideal case, to minimize end-to-end latency, all of the functionality would be allocated to a single processor node like the top instantiation in the figure. However, at customer sites where processing loads can exceed the computing capacity of a single node, voila, a multi-node configuration can be seamlessly instantiated without any source code changes – just XML file configuration parameter changes. In addition, as the bottom instantiation shows, a fault tolerant, auto-failover, dual channel version of the system can be deployed for high availability, safety-critical applications.

Are you working on something as fun as this?

%d bloggers like this: