What we've lost in the push to agile software development, and how to get it back

 The process of software design often begins on a messy whiteboard that doesn't prepare anyone for anything, according to design advocate Simon Brown. It's time for more upfront thinking.

In the age of agile, too many software designers are afraid to over-design their applications upfront. As a result, many software teams have abandoned architectural thinking, up front design, documentation, diagramming, and modelling. "In many cases this is a knee-jerk reaction to the heavy bloated processes of times past, and in others it's a misinterpretation and misapplication of the Agile Manifesto."

That's the word from Simon Brown, author of Software Architecture for Developers, who urged, in a compelling talk at the Yow! conference, that more thinking about applications be moved up to the whiteboard phase of software creation. Incidentally, he eschews whiteboards, noting they often result in confusing or unintelligible sketchings. "Tragically, as an industry, we've stopped teaching [software design]. And if you go and ask people on your team 'how do you design software?', they'll stumble around for a bit, and say, 'well, we use a whiteboard.' Do you get code off a whiteboard? What are you using the whiteboard for? They'll say, 'we're drawing pictures, boxes, and lines.'" 

Brown is a tireless advocate of richer upfront design planning, including the use of tooling such as those employing unified modeling language (UML), which helps provide standardization and process-driven approaches to architectural design. "Richer design diagrams lead to richer design discussions," he states. Team members and business partners should not have to ask questions such as "what does that arrow mean?" "Is that a Java application?" or "is that a monolithic application or a set of microservices," he says. Rather, discussions should focus on the functions and services being delivered to the business.

"The thing nobody talks about is you have to do design to get version 1," Brown says. "You have to put some foundations in place to give you a sufficient starting point to iterate, and evolve on top of. And that's what we're missing."

Many software design teams keep upfront design to a minimum, assuming details will be fleshed out in an agile process as things move along. Brown says this is misplaced thinking, and design teams should incorporate more information into their upfront designs, including the type of technology and languages that are being proposed. "During my travels, I have been given every excuse you can possibly imagine for why teams should not do upfront design," he says. Some of his favorite excuses even include the question, "are we allowed to do upfront design?" Other responses include "we don't do upfront design because we do XP [extreme programming]," and "we're agile. It's not expected in agile."

This thinking "comes from all the literature about agile that says 'there is no big design up front,'" Brown says. "But people miss out the word 'big,' and they go, 'oh great, we don't need to do design now.'" It's also important to consider that "the people who put together the Agile Manifesto have a ton of experience. We don't likely have that same level of experience. If you look around most scenes now, they're staffed with relatively young people. If we took these super-experienced agile people out of software, and put them into a world they're not familiar with, would they still talk about doing 'just experiment and refactor?'" 

Plus, he adds, the manifesto itself, on its principles page, "talks about design and architecture, and says continuous attention to technical excellence and good design enhances agility." The following components should be part of an upfront design:

• Significant architectural drivers (requirements, quality attributes, constraints)
• Context and scope of what is being built
• Significant design decisions (technology, modularity, etc.)
• A way to communicate your technical vision to other people

Brown advocates a simplified method for charting out applications he calls the C4 model in upfront design: context, containers, components, and code. It brings in and identifies specific technologies that are proposed for the architecture -- such as platforms, programming languages and standards. "It's basically a set of hierarchical diagrams that allow you to tell different stories to different audiences." 

While "big design" upfront goes too far in excluding engagement and input down the line, the goal is "to do enough upfront design, enough upfront thinking, to put a starting point in place, and set a general direction," he says. "Maybe not the perfect direction, maybe not the final direction, but at least some vague notion of direction that we as a team can then follow."