Archive
Leverage Point
In this terrific systems article pointed out to me by Byron Davies, Donella Meadows states:
Physical structure is crucial in a system, but the leverage point is in proper design in the first place. After the structure is built, the leverage is in understanding its limitations and bottlenecks and refraining from fluctuations or expansions that strain its capacity.
The first sentence doesn’t tell me anything new, but the second one does. Many systems, especially big software systems foisted upon maintenance teams after they’re hatched to the customer, are not thoroughly understood by many, if any, of the original members of the development team. Upon release, the system “works” (and it may be stable). Hurray!
In the post delivery phase, as the (always) unheralded maintenance team starts adding new features without understanding the system’s limitations and bottlenecks, the structural and behavioral integrity of the beast starts surely degrading over time. Scarily, the rate of degradation is not constant; it’s more akin to an exponential trajectory. It doesn’t matter how pristine the original design is, it will undoubtedly start it’s march toward becoming an unlovable “big ball of mud“.
So, how can one slow the rate of degradation in the integrity of a big system that will continuously be modified throughout its future lifetime? The answer is nothing profound and doesn’t require highly skilled specialists or consultants. It’s called PAYGO.
In the PAYGO process, a set of lightweight but understandable and useful multi-level information artifacts that record the essence of the system are developed and co-evolved with the system software. They must be lightweight so that they are easily constructable, navigable, and accessible. They must be useful or post-delivery builders won’t employ them as guidance and they’ll plow ahead without understanding the global ramifications of their local changes. They must be multi-level so that different stakeholder group types, not just builders, can understand them. They must be co-evolved so that they stay in synch with the real system and they don’t devolve into an incorrect and useless heap of misguidance. Challenging, no?
Of course, if builders, and especially front line managers, don’t know how to, or don’t care to, follow a PAYGO-like process, then they deserve what they get. D’oh!
Waiting For Management Guidance
In “The Design Of Design“, Fred Brooks laments that he used to be able to track new developments in the entire field of software engineering – which was born in 1968. In the present age, because of massive innovation, expansion, and deep specialization requiring steep learning curves, he realizes that there’s no hope of anyone being super human enough to keep up anymore.
Since I used to try to keep abreast of all developments in the field, I felt the same way as Fred. Now, I have a strategy that keeps me from staying awake 24×7 reading books, papers, newsletters, blogs, and articles. I filter the tsunami of information down by trying to track developments only in my domain of focus; distributed and embedded real-time systems. I occasionally look into the fast moving enterprise IT technology space for cross-domain applicability of ideas, but it’s all I can do to keep abreast of my area while simultaneously doing some real application work that adds value to my company‘s products.
How about you? Do you find yourself overwhelmed by the rate and amount of information being created and disseminated in your field? Do you care about and pursue personal exploration and discovery, or do you just punch the clock and wait for the infallible geniuses in management to guide and train you in the new technologies that could keep you and your org viable? If it’s the latter, then you may (as the left portion of the figure below shows) remain stuck in the stagnant and boring “Waiting For Management Guidance” state forever. If you’re dwelling in this sad and potentially infinite state, there is still hope in the form of an “epiphany” of understanding – big daddy ain’t gonna help you grow personally or professionally.
Ideally, one never wastes any time in the spirit-sucking “Waiting For Management Guidance” state. The “Exploring And Discovering” state, which is a natural gift to all human beings, would be transitioned to right out of the box. When we are born, we actually do enter this state right out of the box – so to speak. As soon as we start going to school, the institutional indoctrination starts and before we can say WTF?, we’re well on our way to the “Waiting For Management Guidance” state.
The Answer To A Burning Question
Ever since Fred Brooks hatched his legendary “The Mythical Man Month” over 20 years ago, he’s been on my hero/mentor list. His latest insightful work, “The Design Of Design“, is just as good as TMMM. Of course, since my views on software engineering (if it can be called engineering) are heavily influenced by his experiences as shared through his writing, I’m totally biased and unobjective (but….. aren’t we all to some extent?).
I pre-ordered TDOD as soon as I heard about its impending release and I received it from Amazon last week. Unlike most books, which I mildly speed read, I’ve been savoring this one slowly. As expected, I’ve been discovering and extracting a treasure trove of personally valuable fieldstones from TDOD at a feverish pace.
Fred opens up one of his chapters with this brilliant quote:
“A meeting is a refuge from the dreariness of labor and the loneliness of thought.” – Bernard Baruch
I think it’s brilliant because it answers a burning question that I haven’t been able to self-answer for a long time in one short sentence:
Why do managers spend the vast majority of their time in meetings?
Thanks to Fred and Bernie, I now know why 🙂
Conceptual Integrity
Like in his previous work, “The Mythical Man Month“, in “The Design Of Design“, Fred Brooks remains steadfast to the assertion that creating and maintaining “conceptual integrity” is the key to successful and enduring designs. Being a long time believer in this tenet, I’ve always been baffled by the success of Linux under the free-for-all open source model of software development. With thousands of people making changes and additions, even under Linus Torvalds benevolent dictatorship, how in the world could the product’s conceptual integrity defy the second law of thermodynamics under the onslaught of such a chaotic development process?
Fred comes through with the answers:
- A unifying functional specification is in place: UNIX.
- An overall design structure, initially created by Torvalds, exists.
- The builders are also the users – there are no middlemen to screw up requirements interpretation between the users and builders.
If you extend the reasoning of number 3, it aligns with why most of the open source successes are tools used by software developers and not applications used by the average person. Some applications have achieved moderate success, but not on the scale of Linux and other application development tools.
Underbid And Overpromise
As usual, I don’t get it. I don’t get the underbid-overpromise epidemic that’s been left untreated for ages. Proposal teams, under persistent pressure from executives to win contracts from customers, and isolated from hearing negative feedback by unintegrated program execution and product development teams, perpetually underbid on price/delivery and over-promise on product features and performance. This unquestioned underbid-overpromise industry worst practice has been entrenched in mediocracies since the dawn of the cover-your-ass, ironclad contract. The undiscussable but real tendency to, uh, “exaggerate” an org’s potential to deliver is baked into the system. That’s because competitors and customers are willing co-conspirators in this cycle of woe. The stalemate ensures that there’s no incentive for changing the busted system. As the saying goes; “if we can’t fix it, it ain’t broke!“. D’oh!
If a company actually could take the high road and submit more realistic proposals to customers, they’d go out of business because non-individual customers (i.e. dysfunctional org bureaucracies where no one takes responsibility for outcomes) choose the lowest bidder 99.99999% of the time. I said “actually could” in the previous sentence because most companies “can’t“. That’s because most are so poorly managed that they don’t know what or where their real costs are. Unrecorded overtime, vague and generic work breakdown structures, inscrutable processes, and wrongly charged time all guarantee that the corpo head sheds don’t have a clue where their major cost sinks are. Bummer.
Perverted Inversion
In simplistic terms, material wealth in the form of profits is created through the delivery of products and/or services that provide some sort of perceived value to a set of customers. The figure below illustrates the priorities, from highest on the top to lowest on the bottom, of all successful product-oriented startup businesses.
Since products create the wealth that sustains a business, they receive top billing. The care and feeding of the golden geese via a supportive product development group is next in importance. At the bottom, and deservedly so, is the management of the business. Nevertheless, the fact that it’s on the priority list at all means that managing the business is important – but not as important as the product-centric activities.
Without any facts or any background research to back it up (cuz I like to make stuff up), I assert that most entrepreneurs hate doing the business management “stuff”. Some despise the mundane activities of running a business so much that their negligence can cause the fledgling enterprise to fail just as quick as launching the business without any product or service to sell. Having said that, hopefully you’ll agree that any business priority list without the product portfolio perched at the top is a sure path to annihilation. The other two arrangements of the lower priority activities (shown below) can also work, but maybe not as effectively as the initial proposed stack.
As a business thrives and grows larger, a strange perverted inversion occurs to those who lose their way (but thankfully, not all do). The business management function bubbles to the surface of the priority stack (WTF?). This happens ubiquitously across the land because hot shot, fat headed, generic managers who don’t know squat about the org’s specific product portfolio are chaufeurred in to grow the business. These immodest dudes, thinking of themselves as Godsends from MBA city, put themselves and what they do at the top of the priority stack to…… enrich themselves no matter what. Of course, these wall street stooges perform this magic act while espousing that the product set and those that create it are forever the org’s “most valuable asset“.
Post-startup businesses can survive with the perverted priority stack in place, but they usually muddle along with the rest of the herd and they aren’t exhilarating or engaging places to work. Do you work for one?
At a certain age institutional minds close up; they live on their intellectual fat. – William Lyon Phelps
Makin’ Stuff Up
As I’ve said many times before: “I like to make stuff up“.
Two radically different ways of labeling people (I know, I know – you’re above passing judgment on people) who make stuff up are: liar and artist. After all, makin’ stuff up means that the “stuff” is new in some way. In the case of a liar, the spoken or written words assert that something exists when it doesn’t, or an event happened when it didn’t, or vice versa. The same idea holds for the case of an artist, except there are more choices of media for expression.
Now that you know what I am, what are you?
Architectural Choices
Assume that a software-intensive system you’re tasked with designing is big and complex. Because of this, you freely decide to partition the beast into two clearly delineated layers; a value-added application layer on top and an unwanted overhead, but essential, support layer on the bottom.
Now assume that a support layer for a different application layer already exists in your org – the investment has been made, it has been tested, and it “works” for those applications it’s been integrated with. As the left portion of the figure below shows, assume that the pre-existing support layer doesn’t match the structure of your yet-to-be-developed application layer. They just don’t naturally fit together. Would you:
- try to “bend” the support layer to fit your application layer (top right portion of the figure)?
- try to redesign and gronk your application layer to jam-fit with the support layer (bottom right portion of the figure)?
- ditch the support layer entirely and develop (from scratch) a more fitting one?
- purchase an externally developed support layer that was designed specifically to fit applications that fall into the same category as yours?
After contemplating long term maintenance costs, my choice is number 4. Let support layer experts who know what they’re doing shoulder the maintenance burden associated with the arcane details of the support layer. Numbers 1 through 3 are technically riskier and financially costlier because, unless your head is screwed on backwards, you should be tasking your programmers with designing and developing application layer functionality that is (supposedly) your bread winner – not mundane and low value-added infrastructure functionality.
Golf Outfit
For the past 15 or so years, I’ve been going on a yearly golf trip with a group of good friends. This year’s “golf camp 2010” starts next Saturday, 4/10/10. In preparation for the upcoming debacle, I e-mailed the group a picture of the top third of my planned opening day outfit:
I asked who wanted to be in my foursome and I received these two answers:
- “You look pretty”
- “I would, but I heard that you had three others already.”
I’m having trouble deciding which shorts to wear. I was thinking “Hot Dog”, but “Disco Balls” seems to better match my hair style. What do you think?
Bah Hum BUG
Note: For those readers not familiar with c++ programming, you still might get some value out of this nerd-noid blarticle by skipping to the last paragraph.
The other day, a colleague called me over to help him find a bug in some c++ code. The bug was reported in from the field by system testers. Since the critter was causing the system to crash (which was better than not crashing and propagating bad “logical” results downstream) a slew of people, including the customer, were “waiting” for it to be fixed so the testers could continue testing. My friend had localized the bug to the following 3 simple lines:
After extracting the three lines of code out of the production program and wrapping it in a simple test driver, he found that after line 3 was executed, “val” was being set to “1” as expected. However, the “int” pointer, p_msg, was not being incremented as assumed by the downstream code – which subsequently crashed the system. WTF?
After figuring it out by looking up the c++ operator precedence rules, we recreated the crime scene in the diagram below. The left scenario at the bottom of the figure below was what we initially thought the code was doing, but the actual behavior on the right was what we observed. Not only was the pointer not incremented, but the value in msg[0] was being incremented – a double freakin’ whammy.
Before analyzing the precedence rules, we thought that there was a bug in the compiler (yeah, right). However, after thinking about it for a while, we understood the behavior. Line 3 was:
- extracting the value in msg[0] (which is “1”)
- assigning it to “val”
- incrementing the value in msg[0]
Changing the third line to “int val = *p_msg++” solved the problem. Because of the operator precedence rules, the new behavior is what was originally intended:
- extract the value in msg[0] (which is “1”)
- assign it to “val”
- increment the pointer to point to the value in msg[1]
A simple “const” qualifier placed at the beginning of line 2 would have caused the compiler to barf on the code in line 3: “you cannot assign to a variable that is const“. The bug would’ve been squashed before making it out into the field.
It’s great to be brought down to earth and occasionally being humbled by experiences like these; especially when you’re not the author of the bug 🙂 Plus, after-the-fact fire fighting is cherished by institutions over successful prevention. After all, how can you reward someone for a problem that didn’t occur because of his/her action? Even worse, most institutions actively thwart the application of prevention techniques by imposing Draconian schedules upon those doing the work.
The world is full of willing people, some willing to work, the rest willing to let them. – Robert Frost
My contortion of this quote is:
The world is full of willing people, some willing to work, the rest willing to manage them while simultaneously pressuring them into doing a poor job. – Bulldozer00
Why am I here?
WTF?
Meh!
D'oh!
My BTC Address
Bulldozer00's Blog 