Archive
Cry Babies
On Oct. 6, the US Air Force awarded the Raytheon/Saab team a phase-one, four year, $19M, fixed-price contract to deliver 3 long range ground-to-air surveillance radars by 2018. The other two contractors vying for the award were Lockheed Martin (LM) and Nothrup Grumman (NG).
The contract award was expected in early September, but it was pushed back until Oct. 6. That could have been to ensure everything was handled properly to head off a potential protest from either Lockheed Martin or Northrop Grumman.
The total 3DELRR (3 Dimension Expeditionary Long Range Radar) contract value is estimated at $71M, and it requires the Raytheon-led team to produce an additional 3 radars. In the long run, the Air Force plans to procure 29 more radars for a grand total of 35 sensor systems. The total contract effort could span decades and possibly net $1B for the team. In addition, since “exportability” features were baked into the design UP FRONT, Raytheon will have the opportunity to sell many more radars to US allies all over the world without having to be concerned about US foreign technology transfer restrictions.
Since we’re not talkin’ chump change here, you can infer that the losing competitors were not at all happy with the Air Force’s decision – despite the delay to ensure a fair evaluation. But of course, as is standard practice with big government contracts, both losers filed formal protests of the award with the US government’s GAO (General Accounting Office) after they were formally debriefed on why they lost. NG did not publicly specify the grounds for its protest, but LM proclaimed that their offering was “the most affordable and capable solution.”
The need to allow for contract award protests is obvious: to ensure that no hanky-panky occurred behind the scenes during the decision-making process. However, since the chances of being successful are small and the protest process is normally a huge waste of time and money for all involved parties, you would think that controls would be in place to prevent every single big ticket award to be protested willy-nilly. But alas, no matter what controls are in effect, a desperate loser can always find a cadre of clever lawyers to skirt the rules.
Abstract Decomposition And Concrete Composition
On the left we have the process of abstract decomposition, and on the right we have the process of concrete composition:
Note that during the concrete composition from parts to final system on the right, we gracefully transition through two stable, intermediate forms. Some people and communities, especially those obsessed with “velocity” and “time-to-market“, would say “bollocks” to all those value-subtracting, intermediate steps. We no need no stinking intermediate forms:
The Trees And The Forest
As a result of an online Twitter exchange with Mr. Jon Quigley, I was able to purchase a copy of his and Kim Pries’s book, “Project Management Of Complex And Embedded Systems“. In exchange for a half-price deal, I promised to blog a review of the book and, thus, this is it.
As indicated by the book title, the subject matter is all about the methods and tools commonly used by program/project managers for orchestrating large, capital-intensive, multi-disciplined, product development endeavors. Specifically, the content focuses on how the automotive industry successfully manages the development and production of products composed of thousands of electro-mechanical parts and hundreds of networked processors, some of which run safety-critical software. Even though we tend to take them for granted, when you think about it, an automobile is an extremely complex distributed system requiring lots of coordinated mental, physical, and automated, labor to produce.
The book provides comprehensive, yet introductory, coverage of the myriad of tools and processes used in the world of big project management. It’s more of a broad, sweeping, reference book than a detailed step-by-step prescription for executing a specific set of processes. It’s jam packed with lots of useful lists, figures, tables, and graphs. The end of each chapter even includes a specific “war story” experienced by one or both of the authors over their long careers.
As a long time software developer of complex embedded systems in the aerospace and defense industry, much of the book’s subject matter is familiar to me. RFPs, SOWs, WBSs, EVM, BOMs, V&V, SRRs, PDRs, CDRs, TRRs, FMEA, staged-gate phases, prime-subcontractor relationships, master schedules, multi-level approvals, quality metrics, docu-centric information exchanges, etc, are amongst the methods used to facilitate, focus, constrain, and guide end-to-end system development. Many of the chapter-ending war stories tickled my funny bone too!
For the types of projects Mssrs. Pries and Quigley target in the book, kicking off a project at sprint 0 with a self-organizing team of eight cross-functional developers and a primed product backlog of user stories just doesn’t cut it. So, if you’re a young, naive, cloistered software developer or scrum master or product owner who belittles all “traditional“, rigorous, non-agile processes, I highly recommend this book. It will give you a glimpse into a whole different world and broaden your horizons – perhaps allowing you to see both the trees and the forest.
A Fascinating Conversation
“A Conversation with Bjarne Stroustrup, Carl Hewitt, and Dave Ungar” is the most fascinating technical video I’ve seen in years. The primary focus of the discussion was how to write applications that efficiently leverage multicore processors. Because of the diversity of views, the video is so engrossing that I watched it three times. I also downloaded the MP3 podcast of the discussion and saved it to a USB stick for the drive to/from work.
When pure physics started limiting the enormous CPU clock speed gains being achieved in the 90s, vertical scaling came to an abrupt halt and horizontal scaling kicked into gear. The number of cores per processor started going up, and it still is today.
There was much discussion over the difference between a “lock” and “synchronization“. As the figure below illustrates, main memory is physically shared between the cores in a multicore processor. Thus, even if your programming language shields this fact from you by supporting high level, task-based, message passing instead of low-level cooperative threading with locks, some physical form of under-the-hood memory synchronization must be performed in order for the cores to communicate with each other through shared memory without data races.
Here is my layman’s take on each participant’s view of the solution to the multicore efficiency issue:
- Hewitt: We need new, revolutionary, actor-based programming languages that abandon the traditional, sequential Von Neumann model. The current crop of languages just don’t cut it as the number of cores per processor keeps steadily increasing.
- Ungar: We need incoherent, unsynchronized hardware memory architectures with background cache error correction. Build a reliable system out of unreliable parts.
- Stroustrup: Revolutions happen much less than people seem to think. We need to build up and experiment with efficient concurrency abstractions in layered libraries that increasingly hide locks and core-to-memory synchronization details from programmers (the C++ threads-to-tasks-to-“next?” approach).
Hopelessly Irreducible
Some of the parody accounts on Twitter are hilariously creative. Take “InfoSec Taylor Swift” for example:
The key word in Ms. Swift’s aphorism is “irreducible“. Being one of those things that isn’t objectively measurable (it’s funny how many things in real life are unmeasurable), one man’s “irreducible” is another man’s “reducible“.
I’ve discovered that many people, especially penny-obsessed managers, are so quick to assume and accept irreducibility in a complex system that they don’t even attempt to reduce its complexity:
- It takes time (which directly maps into expense) and hard mental labor to unravel complexity.
- In macho org cultures, irreducible complexity is often seen as intelligent sophistication, a badge of honor, a step toward lofty guru status.
Conversely, it is easier and less time consuming (which again directly maps into expense) to concoct an irreducible complex monolith than it is to thoughtfully build a reducible complex system of smoothly interacting parts.
The #noprograms Revolution
I’ve been developing software in the aerospace and defense industry for 30+ years. Because of the sizes of, and innate System-of-Systems nature of, the solutions in this domain, the processes used to develop them are by necessity, heavyweight. Oh sure, agile/lean techniques can be used effectively in the teeny-tiny-small deep down in the bowels of some of these programs, but to assume “scaling agile” will work in a domain with hundreds of programmers from multiple contractors working on the same distributed system is naive at best, and downright disingenuous at worst.
The most ridiculous examples of inapplicable development practices to the aerospace and defense industry are #noestimates” and “#noprojects“. The “#noprojects” cult is so far off base that they don’t even acknowledge the existence of “programs” – which are a way of organizing a large set of inter-dependent “#noprojects” into, well, a larger grained mission-critical entity worth tens of millions of dollars. It’s like before there were classes and namespaces(packages) added to a programming language to accommodate increasing software system size, there were only “functions” for organizing the structure of a program – and having functions as your only organizing mechanism doesn’t scale well to large software systems.
Hell, the #noprojects people are akin to #noclasses and #nonamespaces people, of which there are thankfully, #nomembers. Really hardcore #noprojects people are perhaps even more loony. They are equivalent to a hypothetical #nofunctionsallowed crowd that demands monolithic, straight-line code only. Line number 1 straight to line number 5,000,000 – and you can’t use loops or “ifs” because they don’t add value and #nofunctionsallowed is all about “maximizing the amount of work not done“.
But hey, if you don’t need any “projects” to build a successful product from the gecko, then by extrapolation you don’t need any “programs” for organizing your projects. Hence, the inimitable BD00 proposes to “scale up” the “#noprojects” movement. We’ll start a revolutionary #noprograms movement, or equivalently, #justcode . We’ll leapfrog both the #noestimates and #noprojects movements at once. W00t!
Live Instructor, Or Live Code?
I’m currently taking an in-house course on Java programming. The material is being presented in standard lecture fashion where the instructor sequentially walks through a series of static powerpoint slides and explains what the code snippets on each slide are doing.
In parallel to the in-house class, I’m also taking an online Java video course taught by Paul Deitel in the comfort of my home. The 36 hour course is available to me through my pricey (but outstanding) safaribooksonline subscription:
Other than in the lesson video intros (e.g. the one above), you never see Mr. Deitel on the screen. Like many educational videos you can watch for free on YouTube and Kahn Academy, you see code and a yellow circle highlighter that Paul dynamically steers around the screen as he explains what the code is doing.
Even though the standard “live instructor” teaching method enables the student to ask a physically-present instructor questions in real-time, I find the video “live code” method much more flexible and effective for the following reasons:
- You can see the instructor dynamically interacting with the IDE (in this case Eclipse).
- You can learn at your own speed by replaying lesson snippets that you didn’t fully grasp the first time. (I’m not so smart, so I use this feature all the time).
- The instructor can compile/run the code in front of your eyes and connect the code with its output by switching between the console and the code editor as needed.
- The instructor can intentionally insert bugs in the code and step through the debugger.
- The instructor can switch between the code and detailed online documentation as needed.
Regarding this particular course, I’m thrilled with Mr. Deitel’s work. He’s very thorough, shows great empathy for the student, and he dispassionately points out the differences between Java and C++ as they emerge throughout his code examples.
As I learn more details about Java, I’m confirming what everybody has been saying about the language since its inception: it is a smoother and easier to learn language than C++. However, I (still) don’t like:
- Centralized garbage collection (I prefer C++’s RAII idiom and smart pointers for localized garbage collection).
- The fact that all methods are virtual by default.
- The fact that every user-defined reference type inherits from class Object.
- The fact that a whole other layer of software, the JVM, sits between user code and the operating system.
- The fact that Java” steers” programmers toward designing deep, complicated inheritance hierarchies regardless of whether that style fits the application.
Regarding the last point, the mental steerage toward a pure OO programming style is subliminally catalyzed in the way that the JDK library packages are designed and the proliferation of large frameworks in the Java community. The obsession with deep hierarchies reminds me of Microsoft’s horrendously complicated old MFC framework for writing Windows GUI applications.
I’ve personally seen the effects of a fixed OO mindset in Java programmers who subsequently learn and write C++ production code in my domain: distributed real-time event systems that are naturally “flat“. The code ends up being overly complicated, hard to understand, and very few methods in the deep trees end up being called. The end result is that the “smoothness” of the language is nullified, developer time is wasted on unnecessary code, and the code is hard to maintain.
Design Disaster
In an attempt to gain an understanding of the software design he was carrying around in his head, I sat down with a colleague and started talking face to face with him. To facilitate the conversation, I started sketching my emergent understanding of his design in my notebook. As you can see, by the time we finished talking, 20 minutes later, I ran out of ink and I wasn’t much better off than before we started the conversation:
If I had a five year old son, I would proudly magnetize my sketch on the fridge right next to his drawings.
Nasty, Non-linear, Inter-variable Couplings
To illustrate the difference between analytical thinking and systems thinking (which some people think are identical), Jamshid Gharajedaghi presents these two figures in his wonderful book: “Systems Thinking: Managing Chaos and Complexity“.
Anyone who has been through high school has been exposed to equations of the type on the left. To discover the impact of one variable, say x1, on the system output, y, you simply vary its value while keeping the values of the other (so-called) independent variables fixed. But what about equations of the type on the right? Every time you attempt to vary the value of one variable to discover its effect on system behavior, you unwittingly “disturb” the values of each of the other variables… which in turn disturbs the variable you’re trying to directly control… which in turn disturbs the set of variables yet again… ad infinitum. It’s called the law of unintended consequences.
Nice and tidy equations of the type on the left are applicable to, and only to, problem modeling in the natural sciences in which the players are time, energy, and mindless chunks of matter. Intractable sets of equations of the type on the right, unsolvable messes in which every variable is correlated which every other variable, are applicable to socio-technical systems where the most influential players have minds of their own. System thinkers focus on the coupling and dynamic interactions between the variables in the system to understand emergent behaviors. Analytical thinkers assume away (either consciously or unconsciously) the nasty, non-linear, inter-variable couplings and thus form an erroneous understanding of the underlying causes of system behavior. Welcome to the guild of business management.
The Angle Bracket Tax
The following C++14 code fragment represents a general message layout along with a specific instantiation of that message:
Side note: Why won’t a C++98/03 compiler accept the above code?
Assume that we are “required” to send thousands of these X-Y position messages per second between two computers over a finite bandwidth communication link:
There are many ways we can convert the representation of the message in memory into a serial stream of bytes for transmittal over the communication link, but let’s compare a simple binary representation against an XML equivalent:
The tradeoff is simple: human readability for performance. Even though the XML version is self-describing and readable to a human being, it is 6.5 times larger than the tight, fixed-size, binary format. In addition, the source code required to serialize/deserialize (i.e. marshal/unmarshal) the XML version is computationally denser than the code to implement the same functionality for the fixed-size, binary representation. In the software industry, this tradeoff is affectionately known as “the angle bracket tax” that must be payed for using XML in the critical paths of your system.
If your system requires high rates of throughput and low end-to-end latency for streaming data over a network, you may have no choice but to use a binary format to send/receive messages. After all, what good is it to have human readable messages if the system doesn’t work due to overflowing queues and lost messages?
Why am I here?
WTF?
Meh!
D'oh!
My BTC Address
Bulldozer00's Blog 