"The Mythical Man Month" towers over the world of professional software development. I can't think of a single other book that has been so frequently cited in other books, articles, papers, and among peers as this one. I've been aware of it for a really long time, but never got around to reading it. Why? Well, while I knew it was an important book, it was also a book related to programming, and whenever i had the chance to read this book or pick up a new language or algorithm, I'd always pick the one that let me build cooler things.
That seems to be changing now. The older I get, and the deeper my experience with software development grows, the more I appreciate that programming is actually only a fraction of what I do. In some ways it's very different from how I envisioned my career as a child; back then I worried that I would hurt my hands by typing for eight or nine hours straight. Now, a day can easily go by when I don't code at all. (Not nearly as frequently in my current job as at previous places, but the principle is the same: programming is just a piece of what I produce.)
One of the most significant items I have begun to appreciate is the difference between what I called "professional" software development and the other kind. Many people are first introduced to this world through personal programming. They build a tool or a toy, controlling every aspect of it, for an audience of one. They can experiment, play, change it on a whim, and have nobody but themselves to blame if it doesn't work as expected.
By contrast, professional programming is always done for someone else. Early on, I had assumed that the biggest challenge would be successfully building the program. Now I realize that the biggest challenge often comes from determining what the customer wants. Few situations are more frustrating than completing a feature, showing it off, and then hearing, "Oh, that's not what I meant!" Communication is incredibly important, and is a prerequisite for all the other stages of development. It has taken me years, but I finally understand that time I spend in discussion instead of in programming is NOT wasted time. It is a crucial, ongoing ingredient, and doing it well will not only save me weeks of crazed overtime, but is absolutely necessary to every complete a project.
Wow, tangent. Sorry about that. It isn't totally tangential, though. You may already be able to sense some of the frustrations that I have encountered in my journey as a dude who gets paid to write software. I find that I'm growing my skill set outward, not just deeper, and striving to increase my abilities in areas other than Java and C++. So it is that, when I have the choice between reading The Mythical Man Month and learning Python, for the first time in my life I'm picking up TMMM.
My quick summary: this book is incredible. It is also insanely depressing. It is mordantly funny. It is prescient enough to be frightening. It is written specifically for software MANAGERS, not developers, but I think these are lessons that everyone in a software organization need to absorb... upper management can get the most mileage out of its suggestions, but the rank and file can and should understand when they need to apply pressure in order to stave off potential disaster.
The truly incredible thing about the book is how timeless it is. As Brooks reminds us, we're in a world where hardware CPU capacity is continually doubling every few years. We have gone from vacuum tubes to transistors to microchips. The Cold War has been won, we've put a man on the moon, and elected a black man as President since inventing the computer... and yet, in fifty years, software projects are still run in much the same manner and continue to fail for the same reasons, over and over again. Brooks wrote this book in 1975, about an experience in the mid-1960's. You would think that in 40 years we would have figured out how to do this right. It amazes me that this book, which was written before I was even born, still seems to describe situations today.
Why is making software so hard? Brooks does a phenomenal and elegant job of explaining why. He is an extremely talented writer, with one of the best voices I have come across in the literature: kind, wry, self-deprecating, bright and committed. He describes early on how software engineering is so different from other endeavors, and in so doing he perfectly nails why I'm so attracted to it as a discipline. I recognized a kindred spirit as he enumerated the joys of working with a medium of pure thought-stuff, utterly malleable to the creator's whim, the delight of crafting puzzles, the way new challenges arrive every day and one never needs repeat oneself.
And, when he laid it out like that, I could also see why the problems we face are endemic to the field. Making software is inherently hard; it isn't just that we've had a really bad string of luck. Its difficulty comes from the very things I love about it: perfect malleability means constant change. Dealing in thought-stuff within a single mind is delightful, but combining multiple minds risks everything in translation. He also points out that programmers are, by nature, optimistic. The field attracts people who can enthusiastically believe that they can solve any problem, who are always looking towards solutions; as a result, we have a tendency to subconsciously assume that everything will go as according to plan, that there will be no (or at least few) bugs.
Brooks then proceeds to do something rather remarkable: after establishing his bona fides in this peculiar field of endeavor, he lays out a very rational explanation of the problem that (to me) seems to transcend it. The book's title summarizes his main point: we persist in assuming that men and months are interchangeable. If one man can do a job in three months, we believe that we can do it in one month with three men. Such math feels very tidy, and is perfectly applicable when you are building discrete repeated manufactured things, such as cars or chairs or televisions. Software isn't just harder than those things; it's another kind of thing altogether, and a different set of rules apply. Using actual data, his own experiences, and very rational arguments, he shows how adding more men to a project actually DECREASES productivity and makes the schedule worse. The reasons why are simple: the new men must be trained, so they are not effective at first; furthermore, the previously effective members of the team must then spend time in training rather than building the system; the system must be split apart into pieces so that each man has a discrete piece to work on; and as the number of workers increases, so do the number of dependencies and communication channels between them. Communication needs grow exponentially, slowing down the productive work of writing code as the need for synchronization goes up.
So, the central argument is brilliant, and fully matches what I've experienced to date. The one-sentence summary Brooks provides is, "Adding more manpower to a late project will make it later." What's horrible is that adding more people is everyone's immediate instinct once it becomes clear that a project is behind schedule. There are no good choices in this situation, but Brooks points out that we at least know that this solution is counter-productive. The schedule WILL slip, and you must manage it as best as you can.
That said, "The Mythical Man Month" is about much more than this admittedly important argument. It isn't all doom and gloom; he offers some great insight into how to build great software. His most important argument on this front is the importance of conceptual unity: any successful piece of software must provide a clear, coherent, and unified presence. Such software helps at all stages. When building, every engineer will understand how their piece fits into the whole, and will feel comfortable moving between subsystems. When fixing bugs and adding new features, a clean architecture that was also designed for flexibility and extensibility will allow people to keep that conceptual unity in mind as they make their changes, thus staving off entropy and keeping the creeping kudzu of cruft from polluting the system. When maintaining software, an engineer who can envision the whole will be more confident and effective than one who must tiptoe around disparate and confusing structures built by many hands. Finally, some of the quality will shine through to the end user, whether that is another programmer or your grandmother: they will sense and appreciate the elegance of the system.
Again, I think that time has proven the immortality of Brooks's statements. People used to say that all the interesting programs which could be written by a single person had already been written, and that only large teams will be able to write the good programs of the future. That may be true; however, the main point Brooks is making is that a single architect must come up with a vision for the system, and work diligently to promulgate and maintain that vision. It's still possible that many hands might be put to the task of building the cathedral, but there should only be one architect, one mind at the top. Brooks unapologetically speaks in favor of an aristocracy, with a handful of supremely talented individuals laying out the direction while the masses of workers follow their lead.
My first reaction was awe. My second was anger. I still remembered the frustration I felt in my first post-graduation job; as a software developer, I worked under an [extremely talented] architect. However, the culture at this company was for the architect to provide super-detailed plans. They would not just write the interfaces, but essentially provide prose explanations of how each method should be implemented and even suggest private variable names and helper functions. Needless to say, this made things really easy, but I didn't WANT them to be easy; I wanted to be involved in a creative act, not merely act as a transcriber from English into Java.
Yet, as Brooks pointed out, my experience doesn't approach the ideal. He argues that the constraints imposed by an architect ought not hamper creativity; they can actually encourage it. When someone has a limitless universe of possibilities before them, they will spend a lot of time pondering and tentatively trying possibilities, and feel their job is done when they have made a selection. When forced to work within boundaries, however, the question changes from "What should I do?" to "How can I make this happen?" The second question is much more interesting to a software developer. It becomes a challenge, and they can turn their analytical skills upon it. Developers can consider what algorithms are appropriate to use, think about what performance boosts they can get, look at opportunities to re-use and share code, and otherwise make it the best that they can.
Brooks advocates a clear line of demarcation between the architect and the developer. Much like the line between church and state, it is meant to protect both parties. The lowly developer cannot ruin the conceptual unity of the architect's vast system. At the same time, the haughty architect cannot write the developer's code, nor mandate how he is to achieve his goals. Now, Brooks DOES strongly encourage communication between the two, which is absolutely essential. The developer should make suggestions to the architect on how to improve the architecture; perhaps the developer has realized that, if the interface changes slightly, he will be able to produce a much faster implementation. The architect is free to take or discard the suggestion, but he must treat it with respect. Similarly, the architect should always have an idea for how to implement a given component; this is why it's important to have an architect who is experienced at development, and why the architect-developer relationship can be a very important mentoring one. In this situation as well, though, the developer must be free to accept or reject the architect's suggestion. As long as he fulfills his task, he should be the master of his domain.
(And now, many paragraphs into this post, I finally realize that I've been exclusively using male gendered pronouns the whole time. Sorry about that! For the record, there are many very talented female developers and architects out there. This is a bad habit of mine and one I should break. But I probably won't today.)
Anyways: I found it very interesting that Brooks used the imagery of cathedrals and architects in describing his ideal system. I'll need to re-read it, but now I'm wondering if Eric Raymond's "The Cathedral and the Bazaar" explicitly challenges him. Raymond's overall thesis is clearly in opposition... he describes the conventional wisdom that software systems should be built like cathedrals, with top-down vision and control by a few brilliant men living serenely in ivory towers; and contrasts that with the open-source model of software development, which is a babbling bazaar: constantly in chaos, with people always arriving and leaving, no master plan, no central goal, and yet capable of creating amazing and wonderful things. Raymond very persuasively argues that the bazaar model can produce software of notably higher quality than the vaunted cathedral model.
So who is right? I'm still thinking this through, but I'm now thinking that Raymond's bazaar model may be a bit too neat. Quick, think of the best open-source software you know of. The ones that spring to my mind are Linux, Python, Apache, Perl, GNU, the World Wide Web. Now, when you look at that list, what immediately springs to mind? For me it is Linus Torvalds, Guido von Rossum, the Apache Group, Larry Wall, Richard Stallman, and Tim Berners-Lee. Is it a coincidence that the most successful open source projects, many of which have received contributions from thousands upon thousands of individual programmers, nevertheless are tightly associated with the talented INDIVIDUALS (or very small group, in the case of Apache) who invented them and shepherded their growth? It seems to me that the open source movement offers enormous benefits, but it does not break Brooks' essential rule that the best software requires an individual, unified vision driving it. Without that vision, things fall apart and software grows too large, complicated, and ungainly, as is the case with the vast majority of open source projects one can find on Freshmeat or Sourceforge.
[Consider open-source web browsers as a case study. For years, the Mozilla Foundation plodded along, putting out a product that had way more features than Internet Explorer, but never took off. Even its fans, including yours truly, complained that it was too sprawling, too slow and bloated, and not snappy enough. It wasn't until Blake Ross came along and started from scratch with Firefox that things took off. What did Blake do? He got rid of all the baggage and history that had accumulated over the years - the Bazaar - and laid out a clear, simple, and compelling vision of what the browser could be - he made blueprints for a Cathedral. Once all the inhabitants of the bazaar saw how beautiful his creation was, they cheerfully abandoned it, and devoted their considerable skills and energy to furthering his vision. The true mark of success on an open source project is to marry the clarity and structure of the cathedral with the energy and creativity of the bazaar.]
So, is Brooks right that limiting the choices available to a coder actually enhances their creativity? I suspect that he may be right. His argument reminds me of one of my favorite anecdotes: a high-school teacher taught a playwriting unit. Students would divide into groups, get a selection of props from the teacher, then write and perform a one-act play. They were free to use any, all, or none of the props. However, something interesting happened. Whenever a group's prop set included a toy pistol, their play invariably incorporated a murder. The groups without a pistol made an incredibly wide variety of plays - contemporary farces, historical re-enactments, surreal nonsense, parodies of popular culture, etc. - and the groups with the choice to use a pistol were far less creative, consistently turning out murder stories. I can't argue that there's a direct line to be drawn here, but I'll absolutely agree that being forced to focus on a narrower universe of possibilities can make it much easier to draw inspiration.
Some more random thoughts about the book:
It's really interesting to read a book with such an open and consistent religious perspective. It isn't a religious book, of course, but Brooks cheerfully and regularly draws examples from the Bible and invokes God and His creation. This is a field that is dominated by secularists - Larry Wall is the only other open Christian at the top of the field who springs to my mind - so such statements really jumped out at me.
I found myself getting really nostalgic for manuals while reading this. Brooks consistently talks about the significance of the manual, which has importance similar to that of the system architecture, and he makes similar recommendations about it - the best manuals, he says, flow from just one or two pens. It got me to thinking about how great software manuals were back in the 80's and early 90's. Sometimes I would enjoy reading a game's manual as much as playing the actual game (I'm thinking here of "Machiavelli".), and my favorite franchises (Hero's Quest, Ultima, Space Quest, Civilization, etc.) all had absolutely fantastic manuals. Our ancient IBM-compatible 8088 computer had a nice big thick user's manual that came in two three-ring binders, and described pretty much every aspect of using the computer and its MS-DOS operating system.
Today, of course, we deal with dregs. You're lucky to get any printed material with a new game, and most often it is just a list of keyboard shortcuts and a reference to the in-game tutorial. There's no back-story, no mythos, no artwork, no lost journals, nothing to whet your appetite while you wait for the 12 5 1/4" floppy disks to install. And commercial system manuals are even worse. Most only go so far as describing how to install, which they can and do fit on a single page. Now, you could certainly claim that the printed material has mainly moved into other media, either digitally installed locally or available online. But that isn't true... just look at the Windows Help sometimes. It is the most useless piece of crud that I have come across, filled with marketing messages and providing absolutely no help in discovering how to use features or how to fix simple problems. The whole notion of the technical manual seems to have been lost, and people are expected to either silently suffer or root for themselves among the slop of self-help internet forums.
Not to say that manuals were always wonderful things, of course. I was stunned to read that the manual for a particular IBM system covered six feet of shelf space. Yikes! That would be a wonderful resource in hypertext, but seems practically useless otherwise.
Which brings up another thing. My copy of TMMM included some later material published by Brooks, including his seminal essay on software engineering productivity called "No Silver Bullet". This paper, which is probably more influential among software developers (as opposed to managers) than TMMM, argues that the biggest gains in software productivity have already (as of 1985) been made, and that for the next decade (that is, until 1995), no single innovation will provide a similarly huge boost in productivity. He enumerates past innovations that have brought about phenomenal factors-of-ten gains in productivity: digital microprocessors, time-sharing systems, high-level languages (meaning PL/1 and its ilk). He points out that, with these wins in hand, the "accidental" time-consuming aspects of software engineering that take up time but don't contribute to the product (such as waiting for a seat at the terminal, transcribing punch cards by hand, etc.) have virtually disappeared, and that the remaining things that take up our time are almost entirely due to the difficulty of making software: the time spent in designing systems, building prototypes, writing code, testing that code, and making changes.
It isn't an entirely pessimistic essay - he does identify some things that he expects will improve productivity - but the overall message is clear: things won't be getting much easier any time soon.
For the most part I agree with him, and I'd say that his original prediction (covering 1985-1995) was utterly proven. The one innovation since then that may have changed the game, though, is the Internet. It's pretty amazing to see how this has changed things in my own lifetime. When I was first programming as a child, if I ran into a problem when programming, I would have to pull out my QBASIC manual. If it provided no help, I could try and hunt down a book at the library to shed some light on my situation. In practice, though, I'd be reduced to trial and error, methodically changing every individual variable I could think of and re-testing the result until it worked.
And now, when I get stuck, the first thing I think to do is Google. I'd say that 90% of the time, I have my answer within about 30 seconds. Google helps me find information that is hard to categorize, like "What options does Windows Mobile provide for rotating PNG images?" More importantly, it helps me decipher the amazingly cryptic error messages I can encounter during my day. "How do I fix 'Error -1671: Bad Data' in the Motorola App Signing tool?" Importantly, what the internet does is connect my mind with that of millions of other people. Motorola will not tell me what that error means; but someone else will have run across it before, and will have spent hours figuring it out, and been generous enough to recount their findings online. Thanks to the Internet and Google, I can spend 30 seconds fixing the problem instead of duplicating the hours of lost time and frustration.
THAT is where you get a potential order-of-magnitude boost in productivity. When you can think outside your own head, when you can leverage billions of hours of software development experience from around the globe, some of the essential problems of development - research, discovering capabilities, solving errors - become far easier.
It also makes everything move much faster. I have no data to back this up, but I suspect that if there was no Internet today, software projects would have much longer schedules, and we'd see longer product life cycles between major releases. The fact that everything is at our fingertips leads people to assume that problems can be easily solved. Seeing the competition and instantly getting feedback about your own product leads to a feeling of constantly being in a race, with no time to spare. I guess this is analogous to the idea of "The Rate Race" from the 1950's: we have an incredibly potent new tool, but everyone's expectations shift to accommodate that tool, and as a result it can feel like we're in the same place where we began.
Closing the loop: I'm convinced that's why companies no longer put appreciable effort into their manuals. They know that their users will be able to find answers to their questions online, so why bother? They'll do the absolute minimum to keep their corporate customers happy, and save costs by leaving everyone else to fend for themselves. In the same way that games' bugginess exploded once companies gained the ability to deliver patches after shipping, so manual quality has crashed once companies' users gained the ability to research information online.
Another thing I noticed was that TMMM focuses almost entirely on writing software for your own company or for a particular client. It completely leaves aside the practice of writing software for end-users, for the mass market. As Brooks explains in the book's final chapter, this is because in the 1960's, there WAS no mass market. When you bought a computer, you paid $2 million, and you expected to get all your custom-written software. The exciting and dramatic and largely unexpected growth in low-cost personal computers has created an entire industry that now dominates its traditional counterpart. Now, I'm guessing that a majority of developers are still in IT (within a company) or B2B (writing software for other businesses), but all of the glamour and excitement rests with the minority of us lucky people who get to write programs that will be used by actual people. So, how does this change things?
My immediate response is, "Not very much." The biggest impact is probably on scheduling. Back in Brooks' day, if you bought an IBM computer, you would be getting IBM software. If that software was late, then that was very unfortunate for you and IBM, but you didn't have a lot of choice in the matter. Today, if a company isn't delivering a product or is putting out a bad one, then people will swiftly choose a rival's who is more modern, better priced, or less buggy. Because of this, there is a relentless pressure for constant improvement and continual reinvention - and, as a side effect, frequent public releases. If you drop a date, it affects the company's bottom line and the status of your enterprise. Those of us who thrive in that high-pressure high-thrills environment love it. Those who don't probably long for the days that Brooks describes, when a product could be in development for five years and not be considered a failure.
Other than Brooks-Raymond, the face-off that I found myself thinking about most was Brooks-Ambler. Brooks's proposals for software engineering, both in its large themes (pristine architecture, extremely detailed documentation) and its minutiae (quiet offices, secretaries) flies in the face of Agile, the ascendant trend in software project management. I tend to be a fan of the ideas of Agile, and also found myself becoming a fan of Brooks. How could I possibly reconcile the two?
Fortunately, I found that I could disagree with Brooks without feeling bad. As I said before, he is a very humble and generous man, and my edition of this book included a whole chapter where he looks back at the book after 20 years and describes everything he got wrong and where his mind had changed. As he apologetically describes, much of his book is built on the assumption that software will be built using the Waterfall method - that is, a sequential system whereby you first define requirements, the draw up the design, then write the code, then test the code, and finally deliver. We've known for decades now that the waterfall model produces horrible results, and Brooks cheerfully concedes the point.
In other cases, Brooks almost seems to anticipate Agile. He describes an organizational system built on what he calls the "Surgical team" that bears a resemblance to the small teams used in Agile. He dismisses the Surgical team as unsuitable for large-scale projects, but recent years have provided evidence that small teams can be successfully scaled up into large organizations.
Throughout the book, one of the things I appreciated most was Brooks' devotion to evidence. Anyone can make an argument, but it is very rare to see actual data about what is effective and what is not. Whether comparing lines of code per year or technical support calls in the months after release, his evidence is very compelling, even forty years after it was collected. I think that Ambler's excellent work in quantifying Agile effectiveness plays a similar role. Either man will give the thumbs-up to whatever method is proven in practice to work the best.
As for myself: I'm sure this will change, but right now, I'm leaning towards following Brooks in the very early stages of a project and Ambler for the rest. I'm now more convinced than ever about the importance of a brilliant, senior software architect to lay out the groundwork and provide the map of the cathedral. Once the architect is confident in his plans, a core team must be brought on who share the vision and can see the way forward. Once actual work starts, the real world takes over, and Agile is your best chance of surviving. People must be prepared to throw away code, to regularly deliver working systems, to incorporate change after change into the project. While I agree with Agile's argument that such change must be embraced, I think that Brooks is right that only the best architectures can successfully adapt to those changes with any sort of grace.
I found that for me, the most depressing part of this book wasn't the description of late software projects, nor late-night programming, nor the chronically inferior status afforded to technical staff. No, it was his description of entropy. Occurring towards the end of the original TMMM manuscript, he recounted a story that I know all too well: software systems start out fresh and fun and effective and powerful. Over time, bugs are fixed, features are added, the vision gets blurred and then lost. The system grows brittle. It can no longer incorporate change, everyone is afraid to touch it, and the effort of any individual change becomes prohibitively high. At this point, there is no alternative to sweeping it all away and rewriting it from scratch.
Why do things have to be this way? Why must software be subject to the same laws of entropy that bedevil our physical world? Why do things always fall apart? Brooks, with his religious perspective, has an answer for us, one that nobody wants to hear but that many will grudgingly agree to. It is because all software is written by humans. And humans are imperfect. We are fallen creatures, doing our best to achieve a pure vision that can never be realized within our world, doomed to constantly fall short.
Yeah... a downer, huh? Still, there's hope. After all, someone will need to re-build that system. Someone will get to add the cool new feature. Someone will have the brilliant idea that makes a product special. Software's curse is also its joy, a fortunate fall if you will. The fact that there is no perfect software means that there will always be better software. Keep striving! We'll never get there, but the future will always be better than the past.
I love my job.
No comments:
Post a Comment