Archive
Convex, Not Linear
For a large, complex software system that can be represented via an instantiation of the above template, three levels of testing are required to be performed prior to fielding a system: unit, integration, and system. Unit testing shakes out some (hopefully most) of the defects in the local, concretely visible, micro-behavior, of each of possibly thousands of units. Integration testing unearths the emergent behaviors, both intended and unintended, of each individual subsystem assembly. System level, end-to-end, testing is designed to do the same for the intended/unintended behaviors of the whole shebang.
Now that the context has been set for this post, let’s put our myopic glasses on and zero in on the activity of unit testing. Unit testing is unarguably a “best practice“. However, just because it’s a best practice, does it mean we should, as one famous software character has hinted, strive to “turn the dials up to 10” on unit testing (or any other best practice?).
Check out these utterly subjective, unbacked-by-data, convex, ROI graphs:
If you believe these graphs hold true, then you would think that at some point during unit testing, you’d get more bang for the buck by investing your time in some other value-added task – such as writing another software unit or defining and running “higher level” integration and/or system tests on your existing bag of units.
Now, check out these utterly subjective, unbacked-by-data, linear, “turn the dials up to 10“, ROI graphs:
People who have drank the whole pitcher of unit-testing koolaid tend to believe that the linear model holds true. These miss-the-forest-for-the-trees people have no qualms about requiring their developers to achieve arbitrarily high unit test coverage percentages (80%, 85%, 90%, 100%) whilst simultaneously espousing the conflicting need to reduce cost and delivery time.
Given a fixed amount of time for unit + integration + system testing in a finite resource environment, how should that time be allocated amongst the three levels of testing? I don’t have any definitive answer, but when schedules get tight, as so often happens in the real world, something has gotta give. If you believe the convex unit testing model, then lowering unit test coverage mandates should be higher on your list of potential sacrificial lambs than cutting integration or system test time – where the emergent intended, and more importantly, unintended, behaviors are made visible.
Like Big Requirements Up Front (BRUF) and its dear sibling, Big Design Up Front (BDUF), perhaps we should add “Big Unit Testing Tragedy” (BUTT) to our bag of dysfunctional practices. But hell, it won’t be done. The linear thinkers seem to be in charge.
Note: Jim Coplien has been eloquently articulating the wastefulness of too much unit testing for years. This post is simply a similar take on his position:
Beware Of Micro-Fragmentation
While watching Neal Ford’s terrific “Agile Engineering Practices” video series, I paid close attention to the segment in which he interactively demonstrated the technique of Test Driven Development (TDD). At the end of his well-orchestrated example, which was to design/write/test code that determines whether an integer is a perfect number, Mr. Ford presented the following side-by-side summary comparison of the resulting “traditional” Code Before Test (CBT) and “agile” TDD designs.
As expected from any good agilista soldier, Mr. Ford extolled the virtues of the TDD derived design on the right without mentioning any downside whatsoever. However, right off the bat, I judged (and still do) that the compact, cohesive, code-all-in-close-proximity CBT design on the left is more readable, understandable, and maintainable than the micro-fragmented TDD design on the right. If the atomic code in the CBT isPerfect() method on the left ended up spanning much more space than shown, I may have ended up agreeing with Neal’s final assessment that the TDD result is better – in this specific case. But I (and hopefully you) don’t subscribe to this, typical-of-agile-zealots, 100% true, assertion:
The downside of TDD (to which there are, amazingly, none according to those who dwell in the TDD cathedral), is eloquently put by Jim Coplien in his classic “Why Most Unit Testing Is Waste” paper:
If you find your testers (or yourself) splitting up functions to support the testing process, you’re destroying your system architecture and code comprehension along with it. Test at a coarser level of granularity. – Jim Coplien
As best I can, I try to avoid being an absolutist. Thus, if you think the TDD generated code structure on the right is “better” than the integrated code on the left, then kudos to you, my friend. The only point I’m trying to make, especially to younger and less experienced software engineers, is this: every decision is a tradeoff. When it comes to your intimate, personal, work habits, don’t blindly accept what any expert says at face value – especially “agile” experts.
Challenging the Priesthood
It seems to have taken awhile, but people are finally speaking out strongly against the papal infallibility of the TDD high priesthood. Michał Bartyzel, Andrew Binstock, and Jim Coplien (who actually has been speaking out against it for years) are a few of these blasphemous heretics.
Any of the “driven” techniques can work, but to insinuate that they are the “only way(s)” to build superior products is arrogant, hubristic, and plain stupid. In mentally challenging, knowledge-intensive work, people have, and always will have, different ways of creating beautiful products.
When I speak with adherents of test-driven development (TDD) in particular, there is a seeming non-comprehension that truly excellent, reliable code was ever developed prior to the advent of this one practice. I sense their view that the long history of code that put man on the moon, ran phone switches, airline reservation systems, and electric grids was all the result of luck or unique talents, rather than a function of careful discipline and development rigor. – Andrew Binstock
“…these promises were not supported by unambiguous and verifiable data in early years of TDD. The enthusiastic reaction to TDD came first and then some (selective?) measurements were made to verify its promises. – Michał Bartyzel
“If you find your testers (or yourself) splitting up functions to support the testing process, you’re destroying your system architecture and code comprehension along with it. Test at a coarser level of granularity.” – Jim Coplien
Personally, I start sketching out quite a bit of a design upfront (OMG!!!!!) in “bent” UML (OMG, OMG!!!!!) prior to writing a single line of code. I then start writing the code and I subsequently write/run some selective unit tests on that code. In summary, I dynamically build/test the code base toward the coarse, “upfront” design as I go. Of course, according to the high priests of TDD, I’m unprofessional and I obviously produce inferior designs and bug-ridden, unmaintainable code. Gosh, it sux to be me.
For more details on how I develop software, check out this four year old post: PAYGO II. And no, I’m not promoting it as better than your personal process or, gasp, the unassailable holy grail of software development… TDD! No books, magazine articles, conference talks, or two-day certification courses are planned. It’s simply better for me, and perhaps only me.
Hurry! Hurry!
Lots of smart and sincere software development folks like Ron Jeffries, Jim Coplien, Scott Ambler, Bob Marshall, Adam Yuret, etc. have recently been lamenting the dumbing-down and commercialization of the “agile” brand. Since I get e-mails like the one below on a regular basis, I can deeply relate to their misery.
Hurry! Hurry! After just 2 days of effort and a measly 1300 beaners of “investment“, you’ll be fully prepared to lead your next software development project into the promised land of “under budget, on schedule, exceeds expectations“.
Whoo Hoo! My new SCRUM Master certificate is here! My new SCRUM Master certificate is here!
Why am I here?
WTF?
Meh!
D'oh!
My BTC Address
Bulldozer00's Blog 