Archive
Pay As You Go
An age old and recurring source of contention in software-intensive system development is the issue of deciding how much time to spend coding and how much time to spend writing documentation artifacts. The figure below shows three patterns of development: BDUF, CADOB, and PAYGO.
Prior to the agile “revolution“, most orgs spent a lot of time generating software documentation during the front end of a project. The thinking was that if you diligently mapped out and physically recorded your design beforehand, the subsequent coding, integration, and test phases would proceed smoothly and without a hitch. Bzzzt! BDUF (bee-duff) didn’t work out so well. Religiously following the BDUF method (a.k.a. the waterfall method) often led to massive schedule and cost overruns along with crappy and bug infested software. Bummer.
In search of higher quality and lower cost results, a well meaning group of experts conceived of the idea of “agile” software development. These agile proponents, and the legions of programmers soon to follow, pointed to the publicly visible crappy BDUF results and started evangelizing minimal documentation up front. However, since the vast majority of programmers aren’t good at writing anything but code, these legions of programmers internalized the agile advice to the extreme; turning the dials to “10”, as Kent Beck would say. Citing the agile luminaries, massive numbers of programmers recoiled at any request for up front documentation. They happily started coding away, often leading to an unmaintainable shish-CADOB (Crappy Architecture and Design Out Back). Bozo managers, exclusively measured on schedule and cost performance by equally unenlightened corpocratic executives, jumped on this new silver bullet train. Bzzzt! Extreme agility hasn’t worked very well either. The extremist wing of the agilista party has in effect regressed back to the dark ages of hack and fix programming, hatching impressive disasters on par with the BDUF crews. In extreme agile projects where documentation is still required by customers, a set of hastily prepared, incorrect, and unusable design/user/maintenance artifacts (a.k.a. camouflage) is often produced at the tail end of the project. Boo hoo, and WAAAAGH!
As the previously presented figure illustrates, a third, hybrid pattern of software-intensive system development can be called PAYGO. In the PAYGO method, the coding/test and artifact-creation activities are interlaced and closely coupled throughout the development process. If done correctly, progressively less project time is spent “updating” the document set and more time is spent coding, integrating, and testing. More importantly, the code and documentation are diligently kept in synch with each other.
An important key to success in the PAYGO method is to keep the content of the document artifact set at a high enough level of abstraction “above” the source code so that it doesn’t need to be annoyingly changed with every little code change. A second key enabler to PAYGO success is the ability and (more importantly) the will to write usable technical documentation. Sadly, because the barriers to adoption are so high, I can’t imagine the PAYGO method being embraced now or in the future. Personally, I try to do it covertly, under the radar. But hey, don’t listen to me because I don’t have any credentials, I like to make stuff up, and I’ve been told by infallible and important people that I’m not fit to lead 🙂
The only way to learn how to write is by wrote.
Structure And The “ilities”
In nature, structure is an enabler or disabler of functional behavior. No hands – no grasping, no legs – no walking, no lungs – no living. Adding new functional components to a system enables new behavior and subtracting components disables behavior. Changing the arrangement of an existing system’s components and how they interconnect can also trade-off qualities of behavior, affectionately called the “ilities“. Thus, changes in structure effect changes in behavior.
The figure below shows a few examples of a change to an “ility” due to a change in structure. Given the structure on the left, the refactored structure on the right leads to an increase in the “ility” listed under the new structure. However, in moving from left to right, a trade-off has been made for the gain in the desired “ility”. For the monolithic->modular case, a decrease in end-to-end response-ability due to added box-to-box delay has been traded off. For the monolithic->redundant case, a decrease in buyability due to the added purchase cost of the duplicate component has been introduced. For the no feedback->feedback case, an increase in complexity has been effected due to the added interfaces. For the bowl->bottle example, a decrease in fill-ability has occurred because of the decreased diameter of the fill interface port.
The plea of this story is: “to increase your aware-ability of the law of unintended consequences”. What you don’t know CAN hurt you. When you are bound and determined to institute what you think is a “can’t lose” change to a system that you own and/or control, make an effort to discover and uncover the ilities that will be sacrificed for those that you are attempting to instill in the system. This is especially true for socio-technical systems (do you know of any system that isn’t a socio-technical system?) where the influence on system behavior by the technical components is always dwarfed by the influence of the components that are comprised of groups of diverse individuals.
Islands Of Sanity
Via InformIT: Safari Books Online – 0201700735 – The C++ Programming Language, Special Edition, I snipped this quote from Bjarne Stroustrup, the creator of the C++ programming language:
“AT&T Bell Laboratories made a major contribution to this by allowing me to share drafts of revised versions of the C++ reference manual with implementers and users. Because many of these people work for companies that could be seen as competing with AT&T, the significance of this contribution should not be underestimated. A less enlightened company could have caused major problems of language fragmentation simply by doing nothing.”
There are always islands of sanity in the massive sea of corpocratic insanity. AT&T’s behavior at that time during the historical development of C++ showed that they were one of those islands. Is AT&T still one of those rare anomalies today? I don’t have a clue.
Another Bjarne quote from the book is no less intriguing:
“In the early years, there was no C++ paper design; design, documentation, and implementation went on simultaneously. There was no “C++ project” either, or a “C++ design committee.” Throughout, C++ evolved to cope with problems encountered by users and as a result of discussions between my friends, my colleagues, and me.”
WTF? Direct communication with users? And how can it be possible that no PMI trained generic project manager or big cheese executive was involved to lead Mr. Stroustrup to success? Bjarne should’ve been fired for not following the infallible, proven, repeatable, and continuously improving, corpo product development process. No?
Dynamic “To Do” List
While making an iterative pass over the pages in my wiki space, I stumbled upon the goals for 2009 that I recorded on my “Dynamic To-Do List” Page:
- Increase my question to statement ratio.
- Don’t comment on stuff that I don’t know anything about!
- Don’t give unsolicited opinions as much; especially when I know it will piss off someone with a higher rank.
- Listen without immediately formulating refutations and alternative views: resist the urge to “auto-reject”. Listen to understand, not to criticize.
- Ask “How can I help?” more often, especially to younger folks.
Stacks And Icebergs
The picture below attempts to communicate the explosive growth in software infrastructure complexity that has taken place over the past few decades. The growth in the “stack” has been driven by the need for bigger, more capable, and more complex software-intensive systems required to solve commensurately growing complex social problems.
In the beginning there was relatively simple, “fixed” function hardware circuitry. Then, along came the programmable CPU (Central Processing Unit). Next, the need to program these CPUs to do “good things” led to the creation of “application software”. Moving forward, “operating system software” entered the picture to separate the arcane details and complexity of controlling the hardware from the application-specific problem solving software. Next, in order to keep up with the pace of growing application software size , the capability for application designers to spawn application tasks (same address space) and processes (separate address spaces) was designed into the operating system software. As the need to support geographically disperse, distributed, and heterogeneous systems appeared, “communication middleware software” was developed. On and on we go as the arrow of time lurches forward.
As hardware complexity and capability continues to grow rapidly, the complexity and size of the software stack also grows rapidly in an attempt to keep pace. The ability of the human mind (which takes eons to evolve compared to the rate of technology change) to comprehend and manage this complexity has been overwhelmed by the pace of advancement in hardware and software infrastructure technology growth.
Thus, in order to appear “infallibly in control” and to avoid the hard work of “understanding” (which requires diligent study and knowledge acquisition), bozo managers tend to trivialize the development of software-intensive systems. To self-medicate against the pain of personal growth and development, these jokers tend to think of computer systems as simple black boxes. They camouflage their incompetence by pretending to have a “high level” of understanding. Because of this aversion to real work, these dudes have no problem committing their corpocracies to ridiculous and unattainable schedules in order to win “the business”. Have you ever heard the phrase “aggressive schedule”?
“You have to know a lot to be of help. It’s slow and tedious. You don’t have to know much to cause harm. It’s fast and instinctive.” – Rudolph Starkermann
Metrics Dilemma
“Not everything that counts can be counted. Not everything that can be counted counts.” – Albert Einstein.
When I started my current programming project, I decided to track and publicly record (via the internal company Wiki) the program’s source code growth. The list below shows the historical rise in SLOC (Source Lines Of Code) up until 10/08/09.
10/08/09: Files= 33, Classes=12, funcs=96, Lines Code=1890
10/07/09: Files= 31, Classes=12, funcs=89, Lines Code=1774
10/06/09: Files= 33, Classes=14, funcs=86, Lines Code=1683
10/01/09: Files= 33, Classes=14, funcs=83, Lines Code=1627
09/30/09: Files= 31, Classes=14, funcs=74, Lines Code=1240
09/29/09: Files= 28, Classes=13, funcs=67, Lines Code=1112
09/28/09: Files= 28, Classes=13, funcs=66, Lines Code=1004
09/25/09: Files= 28, Classes=14, funcs=57, Lines Code=847
09/24/09: Files= 28, Classes=14, funcs=53, Lines Code=780
09/23/09: Files= 28, Classes=14, funcs=50, Lines Code=728
09/22/09: Files= 28, Classes=14, funcs=48, Lines Code=652
09/21/09: Files= 26, Classes=10, funcs=35, Lines Code=536
09/15/09: Files= 26, Classes=10, funcs=29, Lines Code=398
The fact that I know that I’m tracking and publicizing SLOC growth is having a strange and negative effect on the way that I’m writing the code. As I iteratively add code, test it, and reflect on its low level physical design, I’m finding that I’m resisting the desire to remove code that I discover (after-the-fact) is not needed. I’m also tending to suppress the desire to replace unnecessarily bloated code blocks with more efficient segments comprised of fewer lines of code.
Hmm, so what’s happening here? I think that my subconscious mind is telling me two things:
- A drop in SLOC size from one day to the next is a bad thing – it could be perceived by corpo STSJs (Status Takers and Schedule Jockeys) as negative progress.
- If I spend time refactoring the existing code to enhance future maintainability and reduce size, it’ll put me behind schedule because that time could be better spent adding new code.
The moral of this story is that the “best practice” of tracking metrics, like everything in life, has two sides.
Endless Loop
Are you a player in an endless loop like this:
If so, are you the pontiff? A cardinal? A bishop? A “mass”? If you ARE a player in this endless loop, do you aspire to play a different role, and why? Do you want to be the selector and promulgator of the next “j” proclamation? Do you fancy the bling that the cardinals, bishops, and pontiff adorn themselves in? Do you want to change the content of one or more of the loop steps, especially number 9?
No BS, From BS
You certainly know what the first occurrence of “BS” in the title of this blarticle means, but the second occurrence stands for “Bjarne Stroustrup”. BS, the second one of course, is the original creator of the C++ programming language and one of my “mentors from afar” (it’s a good idea to latch on to mentors from afar because unless your extremely lucky, there’s a paucity of mentors “a-near”).
I just finished reading “The Design And Evolution of C++” by BS. If you do a lot C++ programming, then this book is a must read. BS gives a deeply personal account of the development of the C++ language from the very first time he realized that he needed a new programming tool in 1979, to the start of the formal standardization process in 1994. BS recounts the BS (the first one, of course) that he slogged through, and the thinking processes that he used, while deciding upon which features to include in C++ and which ones to exclude. The technical details and chronology of development of C++ are interesting, but the book is also filled with insightful and sage advice. Here’s a sampling of passages that rang my bell:
“Language design is not just design from first principles, but an art that requires experience, experiments, and sound engineering trade-offs.”
“Many C++ design decisions have their roots in my dislike for forcing people to do things in some particular way. In history, some of the worst disasters have been caused by idealists trying to force people into ‘doing what is good for them'”.
“Had it not been for the insights of members of Bell Labs, the insulation from political nonsense, the design of C++ would have been compromised by fashions, special interest groups and its implementation bogged down in a bureaucratic quagmire.”
“You don’t get a useful language by accepting every feature that makes life better for someone.”
“Theory itself is never sufficient justification for adding or removing a feature.”
“Standardization before genuine experience has been gained is abhorrent.”
“I find it more painful to listen to complaints from users than to listen to complaints from language lawyers.”
“The C++ Programming Language (book) was written with the fierce determination not to preach any particular technique.”
“No programming language can be understood by considering principles and generalizations only; concrete examples are essential. However, looking at the details without an overall picture to fit them into is a way of getting seriously lost.”
For an ivory tower trained Ph.D., BS is pretty down to earth and empathic toward his customers/users, no? Hopefully, you can now understand why the title of this blarticle is what it is.
Motivility
In one of the Vital Smarts crew’s books (I forget which one, and I’m too lazy-ass to look it up) they mention motivation and ability as two important metrics that leaders can leverage to help people improve performance. To make things simple, but hopefully not simplistic, I’ve constructed a “Leader’s Action Table” (LAT) below using a binary “Present” or “Absent” value for each of the motivility attributes.
Since, by definition, a leader is pro-active and he/she cares about people and performance (both), he/she will take the time and effort to get to know his/her people well. The leader can then use the simple, two attribute , four action LAT to help his/her people grow and develop.
With bozo managers, the story is much different. Even if they stopped thinking about themselves and their careers long enough to consider the individual needs of their people in terms of the two motivility attributes, those bozeltines would get it back-asswards and hose everything up – of course. Instead of a LAT, they’d wield the BAT shown below. BATter up!
Do you think the LAT could be useful? What would your LAT look like? Are there any important attributes that you think are missing from the table? Should one or either of the motivility attributes be multi-valued instead of binary? Meh?
“Half of the harm that is done in this world is due to people who want to feel important. They do not mean to do harm… They are absorbed in the endless struggle to think well of themselves.” – T. S. Eliot
Don’t Be Late!
The software-intensive products that I get paid to specify, design, build, and test involve the real-time processing of continuous streams of raw input data samples. The sample streams are “scrubbed and crunched” in order to generate higher-level, human interpretable, value-added, output information and event streams. As the external stream flows into the product, some samples are discarded because of noise and others are manipulated with a combination standard and proprietary mathematical algorithms. Important events are detected by monitoring various characteristics of the intermediate and final output streams. All this processing must take place fast enough so that the input stream rate doesn’t overwhelm the rate at which outputs can be produced by the product; the product must operate in “real-time”.
The human users on the output side of our products need to be able to evaluate the output information quickly in order to make important and timely command and control decisions that affect the physical well-being of hundreds of people. Thus, latency is one of the most important performance metrics used by our customers to evaluate the acceptability of our products. Forget about bells and whistles, exotic features, and entertaining graphical interfaces, we’re talking serious stuff here. Accuracy and timeliness of output are king.
Latency (or equivalently, response time) is the time it takes from an input sample or group of related samples to traverse the transformational processing path from the point of entry to the point of egress through the software-dominated product “box” or set of interconnected boxes. Low latency is good and high latency is bad. If product latency exceeds a time threshold that makes the output effectively unusable to our customers , the product is unacceptable. In some applications, failure of the absolute worst case latency to stay below the threshold can be the deal breaker (hard real time) and in other applications the average latency must not exceed the threshold xx percent of the time where xx is often greater than 95% (soft real time).
Latency is one of those funky, hard-to-measure-until-the-product-is-done, “non-functional” requirements. If you don’t respect its power to make or break your wonderful product from the start of design throughout the entire development effort, you’ll get what you deserve after all the time and money has been spent – lots of rework, stress, and angst. So, if you work on real-time systems, don’t be late!
Why am I here?
WTF?
Meh!
D'oh!
My BTC Address
Bulldozer00's Blog 