Wehrmacht, Elon Musk and Unix Software Development

A look at the power of iterative approaches to solving complex problems in software development, economics, engineering and war.

18 min readApr 23, 2022

A TOPIC which has fascinated me for a long time is how different organizations can achieve vastly different outcomes and success despite having access to similar amount of resources at their disposal. For instance how could Tesla outcompete huge automakers in making long range electric cars? How could a startup like SpaceX design a rocket such as Falcon 9 outcompeting far more experienced competitors such as Boeing and ULA, Roscosmos and Arianespace with much more experience and deeper pockets?

Why was Unix so successful compared to the long forgotten Multics despite having a tiny team and very little resources at the their disposal?

Why did Japanese automakers in the 1980s and 1990s outcompete their American counterparts, which had larger markets, more experience and deeper pockets?

I have chosen to frame all these different experience through the Unix software development philosophy of “worse is better” which emphasis these points:

Simplicity — The design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface.
Correctness — The design should be correct in all observable aspects. It is slightly better to be simple than correct.
Consistency — The design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either complexity or inconsistency in the implementation.
Completeness — The design must cover as many important situations as is practical. All reasonably expected cases should be covered. Completeness can be sacrificed in favor of any other quality.

I will cover a variety of seemingly unrelated topics and show how these can be related to this “worse is better” philosophy.

I will begin with a story about the German army during WW2, the Wehrmacht. Despite the fact that the Wehrmacht was in many ways weaker than the French army it demolished the French in record time. Time and again superior allied forces would get routed by numerically inferior German forces.

Many of the same lessons are in fact observed today when observing how Ukrainian forces are beating numerically superior Russian military.

The Success of The German Wehrmacht

Serval years ago I read a book by two Swedish authors Michael Tamelander and Niklas Zetterling about D-day. One of the things they studied where the assumption that the Allied forces were superior to the Germans because the Allies won WW2. On closer examination this was not true, and neither was it true during WW1. In both wars the Axis powers were defeated due superior production capacity, manpower and resources of the allied powers. The book gives a surprising number of examples how of German forces would win battles despite surprisingly large odds against them.

To understand the superiority of the German forces on the battlefield, the authors look at how military training was done in Germany and how it was done in America and Britain. Allied soldiers were given problems to solve in the classroom. These were usually cases that involved tactics and strategy. The soldiers were given an hour to figure out the optimum solution to a problem on the battlefield. If after the hour had passed, they gave a sub optimal answer there were told to go back and think about the problem longer.

I think this is a good analogy to the MIT software engineering philosophy of doing “the right thing” as described Richard Gabriel. The MIT approach emphasize correctness above all else. We could rephrase this type of thinking on the battlefield as an extreme emphasis on getting the correct or optimal solution in each battlefield scenario.

When German soldiers were presented with the same kind of problems in the classroom they were given a mere 7 minutes to solve the same problem. 7 minutes! That is orders of magnitude shorter time. How could their teachers expect them to solve these problems in such a short time?

Simple! They follow the UNIX philosophy of “worse is better”. Their solution didn’t have to be any good. It just had to be adequate. But there was one thing that is more important than anything. They came up with some solution in less than seven minutes. Spending more time than seven minutes was a failure.

The results of this difference in philosophy was catastrophic for Allied forces. By the time Allied forces had pondered the situation on the ground properly the realities had changed too much for their solution to be applicable. Germans soldiers, on the other hand were very dynamic and always adapting to the current situation.

What Germans were doing is at the heart of iterative design. You make a choice which may not be optimal, but then always use feedback about outcomes to quickly change your tactics. That is different from a large upfront design approach where everything is analyzed in detail to come up with an optimal solution.

With an iterative approach we only care be less wrong on each iteration. Every iteration should be an improvement on the previous one. A battle is a series of iterations driven by feedback. Each time you make a decision on the battlefield it will have consequences. After observing those consequences you make the next decision.

Russian forces fighting in Ukraine today are facing much the same problem. Central command is moving soldiers into planned positions. It seems apt to quote Mike Tyson here:

Everyone has a plan until they get punched in the mouth
— Mike Tyson

What Tyson observed in the ring holds true in battle: The most elaborate plans can quickly fall apart as you face the enemy punching you in the face. In such situation it is not the quality of your carefully laid plans which matters, but your ability to react to challenges as they appear.

Let us jump to an entirely different topic to see the same forces at play.

Evolution vs Intelligent Design in Biology

I believe evolution presents the ideas around iteration and feedback in their purest form. It is counterintuitive that a process such as evolution is able to produce systems which are far more complex and advance than anything intelligent design has been able to do. By intelligent design I mean an intelligent human being or teams of humans creating a system or machine. Evolution is not intelligent. Evolution is not able to look ahead and see potential problems with a particular design. Essentially evolution churns out a lot of half-baked ideas, which are incremental modifications of existing solutions. Then it sees what sticks.

The reason why this works so well is because of the feedback system. People who don’t understand evolution very frequently overlooked this crucial part. If evolution was just random changes to DNA, one would never have been able to produce any organism.

The same ideas pop up in various software development philosophies. Let us explore the “worse is better” ideas further.

The Cathedral and the Bazaar Open Source Development

Open source guru Eric S Raymond coined the term cathedral and the bazaar. Like evolution he showed that by using a lot of mediocre software developers, coupled with frequent releases, a lot of users and lots of feedback, higher-quality software could be built than by a small team of exceptional developers. This team of intelligent and extremely capable developers represents intelligent design in the analogy.

When you designed the system, no matter how well you have thought about it you are likely to get some things wrong. If you spend a long time thinking and planning the solution there is a high chance that you veered off in the wrong direction. Evolution and the bazaar mode of development might not create very good solutions at each iteration. However each iteration is very short and feedback is given quickly so development can be adjusted before any problems compound themselves.

Humans have all kinds of design flaws which exist because evolution cannot look far ahead. Evolutions is always making small adjustments to what it has. That is why we have ended up with stupid designs like the cables going into the sensors of our eyes being connected at the front so that we get a blind spot. It is also why our spine is poorly suited for walking on two legs. However there’s little evolution can do about that because we started as four-legged creatures and were adapted from that.

We could compare this with software development by considering developing a computer game and we have the code for a four legged animal roaming around in our game world. Then we are given the task to add two legged humanoids to the game. There are two ways we could go about this. We could create the code for the two legged creature from scratch with some careful planning or just start hacking the code for the four legged creature until we get what we want. Hacking requires less sophistication, while creating the code from scratch requires developers with more skill.

We could actually see how something very much like this played out in real life with the Tanenbaum–Torvalds debate about Microkernels vs Monolithic Kernels. Or just look at HURD vs Linux. The ragtag group of hackers of varying quality managed to create a better product than elites designing the HURD microkernel.

Market Economy vs Planned Economy

No person or government bureau plans how many toilet paper rolls, bread or cars should be produced in market economies. Figuring out how much should be produced of each item in a modern economy is an extremely complicated problem. And yet the problem is being solved by a surprisingly simple process.

If more people start demanding a product then increase the price.
If fewer people start demanding it then decrease the price.

The first rule spurs companies to try to make more of the product because they can make more money. The second induce them to cut production. Like evolution a set of very simply rules are used to produce a very complex system.

Today we are conditioned to think that a market economy is superior to a planned economy. It is hard to appreciate how much sense planned economy makes intuitively. With a planned economy one can think about the economy and what should be produced for years ahead. For a while this actually works. Countries with planned economies were able to industrialize and grow their industrial production of an amazing rates for a number of years. The reason a planned economy was able to channel much higher levels of resources into building factories and avoid duplication of effort. In a market economy lots of companies are spending resources to make almost the same product. At the face of it that seems very wasteful.

However planned economies ultimately fails to a large degree because of the information problem. You can read about Joseph E. Stieglitz and Paul Krugman among others write about this. As economies get more advanced, and there are more products to decide production levels for, the number of variables in your equations explode. Solving the equation or even getting the information needed to input into the equations become an impossible task. The result is the wrong level of production for all kinds of things. The Soviet Union was constantly plagued by overproduction of certain items and severe shortage of other random products.

This problem is perhaps not that different from the one faced by the software architects trying to gather all kinds of information about requirements of the software system that they will design. Like the Soviet planners, they will ultimately fail in getting all the information correct. Market economies and get stuff wrong all the time as well, but they get feedback quickly based on real usage to adjust quickly.

Although I should point out that there are many cases where a market based approach doesn’t work because the feedback system itself is broken.

Toyota Production System vs Ford

There are a lot of differences between the Toyota and Ford production systems, but I will concentrate on those parts relevant to this discussion. The Ford approach to mass production is similar to a planned economy in that somebody at the top tries to figure out all the details of the operation. There are few local decisions as in a free market economy. Like intelligent design one makes a large upfront design where one optimizes the utilization of every machine.

This is done by e.g. placing a large buffer of parts at each machine in case there is a failure at one of the preceding machines at the assembly line. Thus the assembly line can keep going even when other machines fail.

The Toyota production system is more similar to evolution and the market economy. Failure is celebrated because it gives valuable feedback which will help us improve. Batch sizes between machines are made small so that if any machine fails, big chunks of the assembly line is brought to a halt. That brings focus to problems. This is coupled with encouragement to regular workers to help make things work again. This is continuous improvement. So essentially you get a manufacturing system which is not set in stone but which improves through multiple iterations with lots of feedback.

This is analogous to the Cathedral vs the Bazaar in the mediocre developers are used actively not just the elite programmers. Workers are helping making the assembly line work better. It is not just about the specialists and the engineers as with the Ford production line.

The Toyota production system is part of a larger trend that existed among Japanese Automakers referred to as lean or agile. These ideas would later inspired a number of different ideas around how to organize software development.

Scrum vs Waterfall in Software Development

Scrum is one of many agile methods of software development inspired by lean manufacturing among Japanese automakers. These methods are often contrasted with what is called Waterfall method of development. The Waterfall approach means large upfront design and planning. A bit like allied forces in WW2 would plan an attack or how Russian generals today would plan attacks in Ukraine, rather than adapting to the situation on the battlefield as the evolve.

Scrum its most basic level this is really just about the length of one iteration in the development process. In Evolution or Scrum each iteration is really small, so in a given time frame one gets a lot of feedback to steer the development. Intelligent design or Waterfall method on the other hand, has very long iterations so that in the same time frame very little feedback is given to steer the development. It is hard to avoid going in the wrong direction. The result of this in the case of the Waterfall method one ends up making far too many features. Research shows 50% of the features delivered are never used. This increases the costs and time of development unnecessarily. With Scrum on the other hand customer keeps getting new versions of the software all the time until they can say I’m satisfied with the results. This makes it very easy to avoid creating functionality that the users don’t need.

Ironically Scrum looks on paper much more wasteful than the Waterfall method. The reason is that since Scrum requires a fully working product at each iteration one has to tear up and rebuilds a lot of parts later.

The Marshmallow Challenge

Peter Skillman came up with a design exercise called the Marshmallow Challenge. The goal was simple:

In eighteen minutes, teams of four must build the tallest freestanding structure out of 20 sticks of spaghetti, one yard of tape, one yard of string, and one marshmallow. The marshmallow has to be on top.

It is harder than it sounds. Among the interesting results from this was:

Recent MBA graduates did worst at this exercise. Among the best performers were kindergarden graduates.

The reason for this was that the kids used the principle of iterative development, while the MBA students focused on developing and executing the one true solution. So when they finally put the marshmallow on, the top the structure would collapse because they had not properly understood how to create the structure. The kids however started putting on the marshmallow very early on and learned from mistakes.

In other words the MBA students were using “intelligent design” or acting like Allied soldiers in WW2, while the kids were using an evolutionary strategy and acting more like the Wehrmacht or Toyota.

User Interface Design

I would like to share my experience with designing user interfaces for computer software. I find that developers often think they have a much better idea of how users think than they actually do. When some thought is being put into designing user interfaces it often involves people who are experts in the field arguing over what is best. These discussions can draw out for very long time, and are IMHO a total waste of time. After a lot of discussions back and forth they agree and build a system and then release it to the users. It turns out the users do not understand it. By that time it is too late. It was very expensive to build the system and too expensive to start changing it now.

Make cheap prototypes of user interface instead of building complete solution from scratch. Use feedback to drive design of final solution.

I find it better to start testing with users using mockups. Instead of arguing over 2 solutions it would often be faster to just test each solution on a few users. By having lots of iterations and testing one gets the benefits of evolution over intelligent design in that the massive amounts of feedback allows one to quickly home in on the right approach.

Another important aspect of evolution, which I have not mentioned thus far is the importance of large populations and a lot of variance. Natural selection is of no use if all individuals in the population are genetically equal. Nor is it useful if there are only a couple of individuals. If one only has one design to test it’s much harder to evolve it than if you have multiple designs. With multiple designs you can pick out the ones that work best and have them “mate” with each other.

This is not very different from how I have read that the design process at Apple works. They start with 10 prototypes for products and where one of the goals is that there should be a large variations in those prototypes. Then these prototypes are evolved and the bad ones are discarded through several iterations. At each iteration the remaining prototypes are developed further.

Unfortunately this way working does not seem to appeal to managers. Ironically they seem to think the same way as Soviets planners: that by throwing out the free market of ideas you avoid duplication of effort and can thus progress faster. They think it is wasteful that a lot of engineers are working on different solutions to the same problem. However, this is exactly why open source software has been so successful because there is a free market of ideas, which are constantly competing against each other. Ironically the open source world is much more like a market economy with respect to product development than commercial software development, despite not being driven forward by a profit motive.

SpaceX Approach To Rocket Design

NASA is building the Space Launch System (SLS) in a very incremental manner. They make one piece at a time and test it thoroughly. Everything is planned in the smallest detail.

SpaceX in contrast is forging ahead quickly using an iterative approach. Instead of building complete parts they make lots of prototypes of their Starship rocket which they launch and see what happens. It is what we call a hardware-rich approach.

Yet we can see many relations to other approaches discussed earlier. NASA is following the meticulous MIT style approach where making every part correct is paramount.

Elon Musk in contrast is forging ahead with a “worse is better” approach. It doesn’t matter that the first Starship versions are flawed in some way. He builds them quickly so he can launch them and gain feedback to power the whole development process.

A former NASA employee Dan Rasky joined SpaceX. One of his first interactions with Elon Musk was when determining how to make their PICA (Phenolic Impregnated Carbon Ablator) heat shield for the Dragon capsule.

The issue they were considering was whether to buy PICA or manufacture it in-house. Such a decision would have taken months of back and forth according to Dan Rasky. Every possible problem and benefit would be examined in detail. Elon Musk instead basically had one meeting and when Dan Rasky recommended manufacturing in-house then a decision to do that was made on the spot.

The SpaceX approach was to being building smaller prototypes of the PICA factory rather than doing a lengthy theoretical analysis. Lessons from each prototype would be fed into the next larger model until they had a full working factory.

A problem with a length upfront design is that once you being building the full solution and discover a problem it may get too costly to change direction.

Developers Favoring Dynamic Typing Over Static Typing

Steve Yegge once wrote a blog entry about developers framed as conservatives vs liberals. While the original article if hard anymore JL Gray has a good summary:

“Conservative” programming views:

Software should aim to be bug free before it launches.
Programmers should be protected from errors.
Programmers have difficulty learning new syntax.
Production code must be safety-checked by a compiler.
Data stores must adhere to a well-defined, published schema.
Public interfaces should be rigorously modeled.
Production systems should never have dangerous or risky back-doors.
If there is ANY doubt as to the safety of a component, it cannot be allowed in production
Fast is better than slow.

“Liberal” programming views:

Bugs are not a big deal.
Programmers are only newbies for a little while.
Programmers figure stuff out amazingly fast when their jobs depend on it.
Succinctness is power.
Rigid schemas limit flexibility and slow down development.
Public interfaces should above all else be simple, backward-compatible, and future-compatible.
System flexibility can mean the difference between you getting the customer (or contract) vs. your competitor nabbing it instead.
Companies should take risks, embrace progress, and fiercely resist ossification.
Premature optimization is the root of all evil.

I like to reframe some of these ideas into incremental vs iterative software development. The “conservative” developer is tool heavy: They use statically typed languages so an advance compiler can catch lots of programmer mistakes. They tend to love using complex IDEs and tools to aid development. One relies on tools to a large extent help make the software correct.

The “liberal” developer may prefer to a more organic approach with interactive coding in a REPL (read-evaluate-print-loop) environment using a dynamically typed language. Tools are simple. Emphasis is on moving fast and iterating. The REPL environment facilitates quick feedback on code changes. It is a bit of the SpaceX philosophy: Move fast and blow up things until it all works.

I am not sure I think the labels “liberal” and “conservative” are ideal here. In the politically tribal society the English-speaking online world lives in today that might simply further tribalism.

Steve Yegge’s point was that software developers are all somewhere between these extremes. Personally I am in the iterative development camp. I suppose it would be strange if I wasn’t since I make such a big deal out of the advantages of an iterative approach. It is easy to misunderstand what that means. Just because you begin hacking a solution doesn’t mean you are an iterative oriented guy. The worst of both worlds IMHO is when people incrementally build a final solution without any planning.

When I hack together a solution, I tend to treat that as a prototype. The lessons learned from that I might use to sketch out what the system should look like. Those lessons are then fed into a second design. Even this design may end up as just a prototype.

If you don’t have iteration steps with refinements you will easily just end up with an incredibly messy and bad design. Real iteration will almost always involve throwing away work. If you are keeping around everything you code, that might be an indication that you are not really working in an iterative fashion.

Remember that increments and iterations are not the same thing. With increments you are building the final solution one piece at a time. With iterations you are building the whole solution on each iteration but it could be a flawed or simplistic version.

An analogy with UI design could illuminate the difference: With incremental design you code up a whole dialog or panel used in your application. You finish it completely before moving onto the next panel or screen.

An iterative approach might instead mean making a paper mockup of your whole application to test it. It is cheap but it is not really a finished product. The next version may contain all the user interface coded but without any coloring, styling or correct font choices. The emphasis on iterations is to have something that mimics the full solution at all stages.

Conclusion

Evolution, market economy, Wehrmacht “worse is better” — all derive their advantage by using small iterations, each guided feedback from the “real world” of the previous iteration. Intelligent design, water fall method, “the right way” etc on the other hand, have long iterations, and thus little feedback.

I don’t aim to say that intelligent design is always worse or inferior. This story is a caution against the thought that there is always a shortcut that avoids duplicated effort. The thought that if we just think hard about it we can just build the product right away without creating prototypes that we have to throw away later.