Archive
Exploring Processor Loading
Assume that we have a data-centric, real-time product that: sucks in N raw samples/sec, does some fancy proprietary processing on the input stream, and outputs N value-added measurements/sec. Also assume that for N, the processor is 100% loaded and the load is equally consumed (33.3%) by three interconnected pipeline processes that crunch the data stream.
Next, assume that a new, emerging market demands a system that can handle 3*N input samples per second. The obvious solution is to employ a processor that is 3 times as fast as the legacy processor. Alternatively, (if the nature of the application allows it to be done) the input data stream can be split into thirds , the pipeline can be cloned into three parallel channels allocated to 3 processors, and the output streams can be aggregated together before final output. Both the distributor and the aggregator can be allocated to a fourth system processor or their own processors. The hardware costs would roughly quadruple, the system configuration and control logic would increase in complexity, but the product would theoretically solve the market’s problem and produce a new revenue stream for the org. Instead of four separate processor boxes, a single multi-core (>= 4 CPUs) box may do the trick.
We’re not done yet. Now assume that in the current system, process #1 consumes 80% of the processor load and, because of input sample interdependence, the input stream cannot be split into 3 parallel streams. D’oh! What do we do now?
One approach is to dive into the algorithmic details of the P1 CPU hog and explore parallelization options for the beast. Assume that we are lucky and we discover that we are able to divide and conquer the P1 oinker into 5 equi-hungry sub-algorithms as shown below. In this case, assuming that we can allocate each process to its own CPU (multi-core or separate boxes), then we may be done solving the problem at the application layer. No? 
Do you detect any major conceptual holes in this blarticle?
Unforgettable
Like telling telling someone: “You’re the worst project manager I’ve ever worked for!“, being told “You can’t lead anything!” is pretty unforgettable. I was the transmitter of the former subjective sentence, and the receiver of the latter subjective sentence – by someone who thinks it’s objective, of course. As time ticks by, I’m glad that I’m becoming more and more comfortable accepting opinions that I don’t agree with, especially the infallible opinions of those in positions of authority.
Watercooler Whining
Regardless of whether they work for a world class org or a brutal and oppressive CCH bureaucracy, I assert that most DICs “discuss” among themselves what they think is wrong with their org at all layers in the caste system. The difference is that in CCH abominations, the discussions are confined to the local environment and well out of earshot of the BMs in charge (In CCHs, BMs, as opposed to PHORS, are always in charge). As soon as the whiff of cologne and bright beams of light emitted by an approaching self-important BM is detected by the “whining” DICs, all dialog stops and the malcontents disperse as if someone let loose an SBD stanker.
It doesn’t take Sherlock’s genius to realize that most of the water-cooler discussions ‘tween DICs are self-serving and myopic BS stories about how they are “victims” and how they have been “wronged” by other fellow DICs and disconnected BMs. However, some, just maybe some complaints are about legit, systemically baked-in problems that, if competently addressed, would improve corpo performance. In most cases, the DICs don’t know how to solve the org issue or they don’t have the authority and clout to try out their solutions.
“The day soldiers stop bringing you their problems is the day you have stopped leading them. They have either lost confidence that you can help or concluded that you don’t care. Either case is a failure of leadership.” – Karl Popper
I’d like to mangle Popper’s brilliant quote with:
“If you haven’t setup and maintained your corpo culture so that your soldiers feel comfortable bringing their problems to you, or you have done so but have continuously ignored their concerns by doin’ nuthin’ of substance to help them, they have either lost confidence that you can help or concluded that you don’t care. Either case is a failure of leadership.” – bulldozer00.
Of course, I just make stuff up and I’m not fit to lead anybody, so don’t pay attention to anything I say.
(Dys)functional Managers
IMHO, “functional” engineering managers (e.g. software, hardware, systems, test, etc) should be charged with: developing their people, removing obstacles to their progress, ensuring that tools and training are available, and streamlining bloated processes so that their people can work more efficiently and produce higher quality work outputs. Abdicating these responsibilities makes these dudes (dys)functional bozo managers in my (and maybe only my) eyes.
It really blows my mind when (dys)functional managers are allowed to anoint themselves “chief architect” over and above individual product team functional leads. It’s doubly annoying and counterproductive to an org when these BMs don’t work hands-on with any of the org’s products day-to-day, and they haven’t done any technical design work in this millenium. If I was their next level manager (and not a BM myself so that I could actually see the problem), I’d, as textbook clone managers love to say, “aggressively address” the BM problem by making it crystal clear what their real job is. I’d follow up by periodically polling the BM’s people directly to evaluate how well the BM is performing. Of course, I’m not fit to lead anyone, so you should totally ignore what I say :^)
Get The Hell Out Of There!
When a highly esteemed project manager starts a project kickoff meeting with something like: “Our objective is to develop the cheapest product and get it out to the customer as quickly as possible to minimize the financial risk to the company“, and nobody in attendance (including you) bats an eyelash or points out the fact that the proposed approach conflicts with the company’s core values, my advice to you is to do as the title of this post says: “get the hell out of there (you spineless moe-foe)!”. Conjure up your communication skills and back out of the project – in a politically correct way, of course. If you get handcuffed into the job via externally imposed coercion or guilt inducing torture techniques (a.k.a corpo waterboarding), then, then, then……. good luck sucka! I’ll see you in hell.
The bitterness of poor system performance remains long after the sweetness of low prices and prompt delivery are forgotten. – Jerry Lim
Application Infrastructure
The most recent C++ application that I wrote is named “ADSBsim” The figure below shows some source code level metrics that characterize the program. The metrics are presented in two categories: global and infrastructure. Infrastructure code includes all of the low level, non-application layer logic. For this application, the ratio of infrastructure code to total code is 1725/2784*100 = 62%. Thus, over half of the application is comprised of unglamorous infrastructure scaffolding.
Unlike the application layer logic, which doesn’t get neglected up front, the amount of infrastructure code to be developed is hardly ever known to any degree of certainty at the beginning of a new project. Thus, in addition to the crappy guesstimates that you usually give for the application layer, you should add an equivalent amount of effort to cover the well-hidden infrastructure logic. Instead of multiplying your guesstimate by the classic factor of “2” (rule of thumb) to accommodate uncertainty, you should consider multiplying it by “4” to get a half-way reasonable result. If your org managers mandate schedules from above and always ignore your input, then never mind the advice in this post. You’re hosed no matter WTF you do :^)
BTW, I initially estimated 2 months to complete the ADSBsim project. It ended up taking 4 months instead of the 8 recommended by the technique above. One could interpret this as successfully finishing the project well under budget and within schedule. On the other hand, if one “thought” that it should’ve only taken two months to complete, then my performance can be interpreted as being horrendously below par.
Conflict Aversion And Cultures Of Fear
With no scientific backing or personal credentials to provide me with any semblance of credibility, I assert that conflict aversion and cultures of fear go together like hand and glove; Jenny and Forrest; peas and carrots; peanut butter and chocolate.
In an org that operates in accordance with a culture of fear, inter-personal and inter-group conflicts are avoided at all costs because of the fear of post-conflict consequences. If a culture of fear doesn’t already exist, all it takes is one or two publicly visible rebukes of a conflict initiator to snap a “culture of fear” into place. Common forms of rebuke are: peek-a-boo visits, compensation ceilings, withholding of career development opportunities, placement on a formal performance improvement plan (affectionately called a “PIP”), and covert persecution. The closer to home that a conflict initiator treads to a hairball problem that is eroding performance of the whole, the more severe the rebuke.
In a culture of fear, because there’s no sane incentive for motivating well-meaning people to point out emergent org problems that everybody already knows about, nobody does nuthin’ of substance until there’s a crisis. When a crisis inevitably manifests because of problem neglect, conflict aversion temporarily goes out the window because real feelings and passions bubble to the surface. Under the duress of a crisis, the conflicts that do emerge in a normally conflict averse org are much more explosive and damaging than those that occur in a continuously conflict-accepting org. Thus, when the crisis passes, the left over socio-communication system infrastructure wreckage breeds poorer future performance and a regression back into – you guessed it – the same old, same old, conflict averse way of operation. Bummer.
Machine Age Thinking, Systems Age Thinking
In Ackoff’s Best, Mr. Russell Ackoff states the following
…Machine-Age thinking: (1) decomposition of that which is to be explained, (2) explanation of the behavior or properties of the parts taken separately, and (3) aggregating these explanations into an explanation of the whole. This third step, of course, is synthesis.
The figure below models the classical machine age, mechanistic thinking process described by Ackoff. The problem with this antiquated method of yesteryear is that it doesn’t work very well for systems of any appreciable complexity – especially large socio-technical systems (every one of which is mind-boggingly complex). During the decomposition phase, the interactions between the parts that animate the “thing to be explained” are lost in the freakin’ ether. Even more importantly, the external environment in which the “thing to be explained” lives and interacts is nowhere to be found. This is a huge mistake because the containing environment always has a profound effect on the behavior of the system as a whole.
Mr. Ackoff professes that the antidote to mechanistic thinking is……. system thinking (duh!):
In the systems approach there are also three steps:
1. Identify a containing whole (system) of which the thing to be explained is a part.
2. Explain the behavior or properties of the containing whole.
3. Then explain the behavior or properties of the thing to be explained in terms of its role(s) or function(s) within its containing whole.
Note that in this sequence, synthesis precedes analysis.
The figure below graphically depicts the systems thinking process. Note that the relationships between the “thing to be explained” and its containing whole are first class citizens in this mode of thinking.
One of the primary reasons why we seek to understand systems is so that we can diagnose and solve problems that arise within established systems; or to design new systems to solve problems that need to be controlled or ameliorated. By applying the wrong thinking style to a system problem, the cure often ends up being worse than the disease. D’oh!
Growth And Development
Two of my favorite bureaucracy-busting systems thinkers, Russell Ackoff and John Warfield, died this year. I’ll miss them both because whenever I’ve read something written by these guys, I always learned something new.
In a tribute to Mr. Ackoff, I’m currently reading Ackoff’s Best: His Classic Writings On Management. One of his off-the-beaten-path insights that he’s passed on to me is the independence between growth and development. Growth is an increase in size or wealth, while development is an increase in capability or competence as a result of learning. According to Ackoff, dumb-ass corpocracies automatically and unquestioningly equate growth with development, and that’s why the vast majority of orgs are obsessed with growth at all costs. Of course, growth and development can, and often do, reinforce each other, but neither is necessary for the other.
Ackoff points out that a system can experience growth without development, and vice versa. A heap of rubbish, or equivalently, a bureaucracy, can grow indefinitely without developing, and artists can develop without growing. If an undeveloped company or company is showered with money, it becomes richer but not more developed. On the other hand, if a well developed company or country is suddenly deprived of wealth, it doesn’t become less developed.
What is your org doing? Growing? Developing? Both? Neither?
Abstraction
Jeff Atwood, of “Coding Horror” fame, once something like “If our code didn’t use abstractions, it would be a convoluted mess“. As software projects get larger and larger, using more and more abstraction technologies is the key to creating robust and maintainable code.
Using C++ as an example language, the figure below shows the advances in abstraction technologies that have taken place over the years. Each step up the chain was designed to make large scale, domain-specific application development easier and more manageable.
The relentless advances in software technology designed to keep complexity in check is a double-edged sword. Unless one learns and practices using the new abstraction techniques in a sandbox, haphazardly incorporating them into the code can do more damage than good.
One issue is that when young developers are hired into a growing company to maintain legacy code that doesn’t incorporate the newer complexity-busting language features, they become accustomed to the old and unmaintainable style that is encrusted in the code. Because of schedule pressure and no company time allocated to experiment with and learn new language features, they shoe horn in changes without employing any of the features that would reduce the technical debt incurred over years of growing the software without any periodic refactoring. The problem is exacerbated by not having a set of regression tests in place to ensure that nothing gets broken by any major refactoring effort. Bummer.
Why am I here?
WTF?
Meh!
D'oh!
My BTC Address
Bulldozer00's Blog 