The Angle Bracket Tax
The following C++14 code fragment represents a general message layout along with a specific instantiation of that message:
Side note: Why won’t a C++98/03 compiler accept the above code?
Assume that we are “required” to send thousands of these X-Y position messages per second between two computers over a finite bandwidth communication link:
There are many ways we can convert the representation of the message in memory into a serial stream of bytes for transmittal over the communication link, but let’s compare a simple binary representation against an XML equivalent:
The tradeoff is simple: human readability for performance. Even though the XML version is self-describing and readable to a human being, it is 6.5 times larger than the tight, fixed-size, binary format. In addition, the source code required to serialize/deserialize (i.e. marshal/unmarshal) the XML version is computationally denser than the code to implement the same functionality for the fixed-size, binary representation. In the software industry, this tradeoff is affectionately known as “the angle bracket tax” that must be payed for using XML in the critical paths of your system.
If your system requires high rates of throughput and low end-to-end latency for streaming data over a network, you may have no choice but to use a binary format to send/receive messages. After all, what good is it to have human readable messages if the system doesn’t work due to overflowing queues and lost messages?
Side note:
Uniform initialisation was introduced to C++11. It’s very lovely, but can be confusing when mixed with old code. Prior to C++11, the correct invocation would be
Message current_position = {4, 56101, 723674};
Also, the int32_t type was standardised only then, although available on several platforms as a vendor specific implementation.
You da man Guy. Thanks!
How about using (binary) serialization, like one of these:
https://kentonv.github.io/capnproto/news/2014-06-17-capnproto-flatbuffers-sbe.html
Human readability is another matter, but should help with performance 😉
Two words: OMG DDS 🙂
Googled, found OMG’s Data Distribution Service (DDS), looks interesting, thanks! 🙂 Out of curiosity, would you happen to know how does it fare in comparison with the aforementioned serialization formats?
Two more words: msgpack python
mttpd, I think it’s like comparing apples to oranges. DDS is an industrial strength, full blown, distributed communication subsystem that uses binary serialization/deserialization (CDR format) while protobufs, Boost.Serialize, etc, are simply binary serialization/deserialization implementations. DDS is standardized and has many vendors who supply implementations that are interoperable with each other.
There are open source versions of DDS that you can play with: OpenDDS and OpenSplice. Also check out my “hands on” article about my experience playing with RTI’s stellar implementation of DDS: http://bulldozer00.com/dds-hands-on/
Cheers!