But there are systems which have to actually operate in the real world and which must react to the “mutabilty of life itself,” and they have done so with great success. Some of these systems have been in operation for more than forty years and are accessed by millions of people each and every day. The changes in the details of the system have increased in complexity and volume and are still increasing. But the toolkits used to develop these systems were superior to those game designers are forced to use, and facing real life instead of imitating it, in my opinion, increases the intensity of the designer and programmer and unleashes levels of creativity that are absent from computer games. Designing a large-scale, complex system to support a national institution that affects the daily lives of millions of people in very important ways is a real kick and far exceeds the enjoyment one gets from developing games. Perhaps then the situation is evolutionary. Perhaps the designers and programmers who can do the hard and rewarding work of serving the real world seek out that challenge leaving those who can’t to find work in design and programming of games.

But then I am probably wrong.

I write as someone with software development experience. Your discussion of business processes in software development only applies for those s/w development projects where the required tasks can be readily defined in advance — ie, for those projects which are either very simple or, if not simple, have been undertaken before. Since most contemporary software development projects involve activities which are novel, this task-decomposition phase of project planning is simply impossible to be undertaken accurately or with any great confidence. Even for the design of a new system of the same TYPE as previously completed systems (eg, when say Microsoft writes a new operating system), where the requirements are likely to be articulated accurately, it is usually not possible, even for experienced project planners and developers, to forecast in any detail what tasks will need to be done, in what order, how long these tasks will take to complete, and what other tasks they will impact.

This failure has almost nothing to do with the motivations or incentives or alleged novelty-seeking personalities of the project planners or their development staffs. It has ALL to do with the fact that large-scale software development of novel applications is one of the most difficult activities known to man. Building large-scale software is NOT like building houses (since clients cannot specify their own requirements accurately, and developers cannot estimate the resources needed to fulfil such requirements accurately), and it is NOT like manufacturing widgets with mass-production processes (since projects are almost always one-off, novel, customized, and must usually cope with unique legacy environments). No good is done anybody by persisting with these inappropriate analogies.

For more on the challenges and complexities of large-scale software development, I suggest reading the various reports completed in recent years by the British Computer Society, the Royal Society of Engineering and the UK Department of Trade and Industry (by Cliff and Bullock) on the inherent complexity of large-scale software systems. The last report is here:

http://www.foresight.gov.uk/OurWork/CompletedProjects/IIS/Docs/ComplexityandEmergentBehaviour.asp

One of the things I like about agent-based emergence is that you can end up with something I’ve come to think of as “inefficient efficiencies”. Meaning that if you were to give each agent an energy budget to spend on carrying out its ruleset, you’d find that there would be much more efficient ways to build the structure or system that they end up building. In an emergent system, there is a lot of repetition of action, a lot of excess or junk activity. But on the other hand, sometimes building a system which accomplishes the same task in the most efficient, top-down, designed manner is surprisingly difficult or time-consuming. Moreover, it’s inflexible. Now you have a very efficient system for doing one thing, whereas the emergent approach, you can have different outcomes if you just change the ruleset of the agents slightly. Maybe not optimal outcomes, or outcomes that have anything to do with your needs, however.

Does this resemble (for example) the way that Smith talks about markets and human agency? Yes, certainly. And I don’t think it’s an either/or sort of thing, that either resoundingly endorses laissez-faire minarchism or demonstrates its flaws. The key insight I get, though, is that efficiency per se is very much the wrong objective in any design, at any level, for any purpose. And yet it was the absolute obsession of high-modernist designers and political philosophers (both left and right).

The Deming approach requires a certain understanding of the process in which all workers are engaged. From the standpoint of a software designer, it can all be conceptulized in terms of a workflow. Person A has tasks 1,2,3 Person B with function X has tasks 4,5. Task 4 requires 2, etc on down the line. The willingness of software engineers to absorb the details of workflows in business is highly limited – and the ability of managers of such processes to communicate their complexity and possible efficiencies and chokepoints, etc. is also highly limited. Furthermore the willingness for software engineers and business managers to deal with each other on a long-term basis is slight. Both see their time more profitably spent elsewhere. This is especially costly to the Deming scenario, and so what generally happens is a Hammer implementation.

This does damage to the process by obviating some complexity. So you actually have more complex and new systems shoved into a process which is actually dumbed down because of the communications limitations of managers and engineers. Then what ends up happening is that those employees in the process chain make up the difference by partially working with, and partially working around the new system in ways they have no incentive to explain, and often in ways their managers are unaware of. The outputs of these systems as well as their inputs may be manipulated because those processes to which information technology is applied are rarely end to end systems. So there is a complex adaptive dynamism at work and in the end describing these systems are like describing the flow of molecules of water down a stream. They are chaotic, somewhat random, extraordinarily complex, yet at a macro level comprehensible and predictable.

Once a system becomes predictable at a macro level it gets frozen into place as the practice of the business with all of the inefficiencies and mysteries locked in.

And then the boss changes and all of the internal stresses on the system lose particular incentives.

Systems engineers generally don’t have patience for such matters as conventions of using three employees to do a job that one person with an improved system could do. In the process of conceptualizing the system they are responsible for improving through information technology, they will be introduced to more information about the process than those people we call ‘functional people’ aka staff. This generally causes tension and a resistance for staffers to be forthcoming about those skills which be rendered obsolete.

Furthermore systems engineers have incentives to build generally applicable systems. Laziness is good quality, that is to say that re-usable systems which might be applicable to mulitple businesses and their processes are always seen as more valuable to one-off custom systems.

Business managers generally don’t have patience or tolerance for such systems which make transparent the operations of their staff. There are often common sense improvements that can be made which exposes them or their superiors to an embarrassment.

In general, business process improvement / re-engineering projects and deliverables benefit most from being transparent and self-explanatory. You want God powers immediately and you want control over as many aspects as possible. However once these aspects of the system are established you want them to respond to the changing nature of the business. There is a fundamental conflict between thoroughness, time to implement and adaptability. Pick two.

Gaming environments on the other hand tend to be hermetic, even in sandboxes. There are certain aspects of the system which gives it a specific feel which must be maintained in order to keep it interesting and within a genre. The value of a game is in having a play value that keeps someone progressing towards a God level which must be entertainingly hidden. Emergent behavior in games is entirely desirable. In business systems it is discourage.

Regarding the theme that complexity inevitably becomes intractable, this has historically been true, but I think has been rendered untrue in many cases by the development of refactoring skills. Linux, for example, has been able to avoid the “heat death” that tended to pull down previous operating systems by aggressive refactoring.

Maybe the methods for doing this have been discovered / invented recently, or maybe it has always been possible. In the commercial environments where I worked most of my life, there were never time or resources to do it, something we (but not those who set priorities) recognized as a serious problem.

Refactoring engages the sort of craft ethos that leads to spending “excessive” amounts of time on making wooden furniture that’s beautiful as well as functional, etc.

Regarding the growth of baroque (or even rococo?) complexity to avoid the experience of game play as mechanical: I’m haven’t played these games, so I really can’t comment from the inside. But my guess is none of the games are “deep” in the sense that Go or one of the major Shakespeare plays is — where out of the same finite material one can keep unfolding new possibilities and then exploring them.

I don’t see that this is inevitable, or even that some genius is required to create games that are deep in this way. Sim City and the Sims approach being deep, in different ways. I think Will Wright is a very smart guy, but not Shakespeare and probably not someone who could invent Go. I think he wanted to create games that offered these possibilities, and for some reason other game designers didn’t, for reasons that aren’t clear to me. His games were extremely successful, but didn’t spawn a genre.

There is a problem here: We’ve never created games that are deep and that can still sustain a very large set of players without undergoing periodic crises, and sometimes crashing so hard they take centuries to regain their momentum (think of the middle ages).

I do think we might be able to build and manage such games though, and if we can then refactoring is the key. The various fixes to the financial system (e.g. SEC, FDIC, etc.) are somewhere between patches and refactoring. The growth of financial reporting standards and norms that severely punish attempts to evade them are pretty much refactoring of business culture.

However just as with software we’ll always face assertions that we “can’t afford” to constrain innovation, spend time on reporting, accept the risks of being transparent, etc. etc. I have no idea what it will take for us to collectively decide that we can’t afford to skip these forms of organizational hygiene.

One possibility (and here I am being utopian) is that game culture could lead. If players can take a big enough role in (re)defining the game play, a craft-oriented community will form that balances pragmatism and sophisticated analysis of failures and possible fixes. Over time a workable culture and ethos of social engineering could crystallize.

A thoroughly admirable post. As one who knows a thing or two about system software, I can tell you that your aesthetic understanding of irreducible complexity in large software systems, including but not limited to virtual worlds, is quite accurate. (I don’t think any technical expertise is required, for example, to read Foote and Yoder’s Big Ball of Mud essay.)

It would certainly be interesting to hear how you apply this line of thinking to the massively-multiplayer virtual world known as “Washington.” You know, the one in which one “manipulates procedural outcomes.” I suspect that, as usual, our results are very different.