Planning as Statetime Manipulation

"All happy families are alike; each unhappy family is unhappy in its own way."

[1,1] [1,0] [0,1] [0,0]

Successful large technical systems represent a bounded region of a near-infinite space of possible configurations.

Unlike purely mechanical systems, which are defined purely by a limited set of physical dimensions, the dimensionality of LTSs is highly abstract, often consisting of intricate couplings between sub-systems and nuanced relationships between both the human and artificial agents who perform actions (who have agency) in the context of the broader system.

Making large-scale changes to these systems is the responsibility of the organizations which steward their evolution. Traditionally, this is done by conceptualizing a vision, developing a strategy to achieve that vision, breaking that strategy down into a plan, and then executing the plan.

The words we use to describe this activity, which is the core activity that all system developers (and therefore all modern leaders) must excel at, are anchored in skeuomorphism - they are loanwords from a prior era in which the largest systems (armies, cities, factories) were primarily physical entities.

In our cybernetic age, though, these words are mere analogies.

Modern system design is not just about manipulating space; it is about manipulating state - and that means that the most critical aspect of system design is figuring out how to represent a given system's state. What is the right way to conceptualize it? What set of words, diagrams and quantitative metrics should be used to describe the space? How to balance accuracy with simplicity? How to evaluate different representations?

Put another way, the first step of manipulating state is making a map.

Maps are an ancient tool for representing physical space; schematics and blueprints are a more modern tool for doing the same. The tool for representing state space hasn't really been developed yet - in most organizations, it's most commonly represented in Excel, often for the purpose of financial reporting and budgeting. But I think that the more appropriate tool is probably a variant of the ontology system we developed at Palantir, and the process of constructing an ontology is akin to cartography.

Mathematically, this map, or ontology, is effectively a manifold - a low dimensional representation of the high dimensional space. By definition, this description is lossy - the map is not the territory, all models are wrong etc. Pragmatically, though, some models are better than others - they can be manipulated more productively.

In this endeavor, it's fascinating how many words we use derive from this notion of a map. Vision is about describing a particular region of state-space in enough detail to communicate it broadly. Strategy is often described as "charting a course" to that future vision and avoiding barriers. A milestone, which literally derives from a physical stone placed along a one-dimensional road or path, represents a discrete position within state-space. A single team's roadmap may represents a vector connecting multiple milestones while the organization's roadmap is a zoomed out version representing all of the vectors actively being manipulated by the organization.

These analogies are helpful, but they miss the key point. Unlike a traditional map, which represents 2D projection of our 3D globe, our map is purely imagined. It does not exist in any absolute form. Of course, this is loosely true in cartography as well. The NYC subway map is designed to show subways and warps the region for aesthetic and practical purposes. But spatial cartography is always anchored in physical reality - the territory is at some philosophical level observable, while ontological cartography is a more unbounded exercise (see also Cartographic Grounds: Projecting the Landscape Imaginary for a whimsical overview of cartographic creativity).

To conclude, our modern software systems are highly dimensional; but our mechanisms for understanding complex systems have evolved based on our need to understand lower-dimensional, predominantly physical systems.  

This doesn't make software easier or harder to design and manage - it makes it different. Hardware is hard because doing it is hard; software is hard because defining it is hard. Software has eaten the world, but our ability to conceptualize and visualize these cybernetic human-computer symbiotes has lagged our ability to build them. At some visceral level, we do not understand what we have created. This is not uncommon; the dance between theory and practice is rarely linear.

But the conclusion is clear - in an era where theory has gone out of fashion in science, it's more critical than ever in business. The idea of a map only gets you to the starting line - creating the right map is the foundation of everything else in a modern business.