Extreme Programming for millennials and gen-Z

Building software is hard. And the more I learn about the history of our industry the more I realise it has always been. In 1968 at the first NATO Software Engineering Conference, some attendees coined the term Software Crisis to refer to the difficulty of writing useful and efficient computer programs in the required time. In 1972 Edsger Dijkstra mentions it in his ACM Turing Award lecture.

The major cause of the software crisis is that the machines have become several orders of magnitude more powerful! To put it quite bluntly: as long as there were no machines, programming was no problem at all; when we had a few weak computers, programming became a mild problem, and now we have gigantic computers, programming has become an equally gigantic problem. —  Edsger Dijkstra

People stopped talking about the Software Crisis at the end of the 1980s because it is psychologically extremely difficult to remain in crisis mode for more than two decades. But the Software Crisis never really ended.

And suddenly it was the 90s. And the Internet was booming with digital products that required a shorter time to market. Which implied shorter life cycles and requirements that changed constantly.

Disaster

Most software projects ended up in disaster. And most still do today. Because nothing has really changed. I'm sure you're familiar with some disastrous situations:

• Delivery fails. The Product Manager goes into the office on the deadline day expecting a release only to hear the bad news from you: The feature isn't ready. You estimated poorly. When you're asked for another estimation, you think two more weeks will be enough. And of course, your estimation is wrong. Again.
• The project gets cancelled. If the feature has been dealyed too many times, someone in charge of the budget will eventually get it cancelled. Both their money and your time wasted.
• Cost changes. Maybe you end up delivering that feature but the cost of maintaining it is so high that it would be wiser to delete everything and start over.
• Bugs. Sometimes you end up delivering the feature but it has so many bugs that nobody ends up using your software.
• Business misunderstanding. When Engineering is far from the Business, you don't even understand what problems the business needs you to solve. So you end up developing software that doesn't solve any problem whatsoever.
• Business changes. Maybe you do understand the business but it takes you so long to implement a solution that by the time the software is ready, the business has changed completely.
• Features that don't add value. Sometimes you develop awesome, beautiful and fun features. And you deploy them really fast. But no value is added to anyone.
• Team saying goodbye. You and people from your team are so burned up that people start leaving.

But why do software projects fail? No, seriusly. Why? I know that you've been in some of these situations. But how could you possibly avoid them? Back in the 90s Kent Beck asked himself the same question. And after years of experimentation he published Extreme Programming explained.

The mother of Agile

Extreme Programming is the mother of what some people today call "Agile methodologies". Actually that is not entirely true because other similar initiatives emerged in the 90s: Rapid Application Development, Feature Driven Development, Pragmatic Programming or Adaptive Software Developmet. But these are not methodologies. Not really. Do you know why when someone talks about implementing agile methodologies, everything sounds so empty and stale? Because Agile methodologies can't be implemented. They do not exist. There is nothing to implement. Agile is not about methodologies. It's about culture. Some Engineers realised this in the 90s because their life was full of methodologies. And they despised them. Extreme Programming appeared as a cultural change. A different way of creating software reducing risk and avoiding waste.

Extreme Programming proposes adopting certain values, some principles and some good practices to plan, develop, test, communicate and design. None of these values, principles or practices are written in stone. Although most of them have stood the test of time. I'm going to skip the practices because you can find whole books about each of them. Take a look at them here. But I do want to stop and take a look at the values and principles.

Values

• Communication. The cause of a problem can be traced back in time to someone not talking to someone else about something important.
• Simplicity. This is my favorite. The simpler your socio-technical system is, the easier the communication.
• Feedback. You need to understand what happens inside the system so you can make further decisions and take action. This is the same as working in an empirical way.
• Courage. The other values enable you to take actions with high risk and high returns. This value by itself is useless and leads to toxic behaviours. That's why courage is at the top of the pyramid.

Principles

• Rapid Feedback. The time between an action and the its feedback is essential for learning. The fastest you receive feedback, the fastest you can understand it and the fastest you can take that knowledge into accout to make decisions.
• Assume simplicity. Not all problems are easy. Some are complicated. But that doesn't mean they can't be solved in a simple way. If you think you've reached the simplest solution, you haven't.
• Incremental changes. If you do small changes you're going to be more certain about what the received feedback means. It is a way of getting rid noise.
• Respect. Respect your team members and make them feel valued. Respect the user, the client and your manager. Follow the golden rule.
• Quality work. Do not forget excellence. Quality should never be negotiated. Negotiate scope or time. Never quality. I've met people who focus only on quality ignoring the other principles. Quality by itself is useless.

Not that simple

But why do software projects fail? No, seriously. Why? What does Extreme Programming actually solve that other methodologies don't? Because, at first sight, it seems just a ligthweight methodology. Another set of rules.

We, humans, always want something to lead to something else. But the plan almost never goes as expected. We have a tendency towards causal determinism: each thing that happens is caused by something that happened before. And this is true but we usually ignore complexity: no event has just one cause. Our minds have an overdeveloped sense of cause and effect. That's why causal determinism allows us to design, plan and estimate how long is going to take us to put code in production. But because our linear minds are so bad at understanding complexity, we are always headed towards disaster when we ignore it.

Engineering teams are complex social systems who create and maintain complex software systems. And our first approach to avoid chaos is to create a system of rules. A methodology. But a complex system can only be controlled by a more complex system. So the methodologies we create end up being more complex than the system itself. Because mathematically it makes sense. Bureaucracy and dark agile make sense when thinking about control. There is no other way of controlling a combination of complex systems.

If you take a look at the values and principles of Extreme Programming again, you can observe that all of them are there to help you navigate complexity instead of trying to tame it. This is not something only Kent Beck realised. The headline of The Nature of Software Development by Ron Jeffries is Keep It Simple, Make It Valuable, Build It Piece by Piece. Keep it simple because adding more complexity to an already complex system always makes it worse. Favoring simplicity has its trade offs but at a socio-technical system level the benefits always outweigh the disadvantages. Make it valuable is very tight to rapid feedback. How else would you know if what you are creating actually ads value? And build it piece by piece is another way of talking about incremental changes.

The Mythical Man Month

But this is not something Software Engineers only realised in the 1990s. Frederick Brooks tells the story of how IBM built OS/360 in the 1960. In his book The Mythical Man Month he talks about the lessons learned while building such a big software project.

Adding people to a late software project makes it later. - Frederick Brooks

Brooks realised that most Software Engineers are optimists. They assume everything will go just ok. The reason is we ignore complexity. No only in software but also in our social environment. We forget that creating something in a team is complex. The interactions between people can't be ignored. And the team doesn't have to be large for it to be complex. Large numbers and complexity are not tied together. Take for example the Three body problem.

It's the people, dummy

Edsger Dijkstra thought the problems were computers. But powerful computers only allow us to build more powerful software. And powerful software needs to be built in teams. Building software, specially within a team, is more about the people than about the software itself. The solution and the processes have to take into account the complexity of the social structure. Most problems in Software Engineering arise because we, people, are complex and impredictable.

Extreme Programming is a good place to start. If you find it's principles and values too simple is because they are. Don't underestimate how complex can a situation get when you add just one new practice. And don't try to tame it. Controlling complexity is unmanageable. Navigate it.