Planning as Statetime Manipulation

"All happy families are alike; each unhappy family is unhappy in its own way."

[1,1] [1,0] [0,1] [0,0]

Successful large technical systems represent a bounded region of a near-infinite space of possible configurations.

Unlike purely mechanical systems, which are defined purely by a limited set of physical dimensions, the dimensionality of LTSs is highly abstract, often consisting of intricate couplings between sub-systems and nuanced relationships between both the human and artificial agents who perform actions (who have agency) in the context of the broader system.

Making large-scale changes to these systems is the responsibility of the organizations which steward their evolution. Traditionally, this is done by conceptualizing a vision, developing a strategy to achieve that vision, breaking that strategy down into a plan, and then executing the plan.

The words we use to describe this activity, which is the core activity that all system developers (and therefore all modern leaders) must excel at, are anchored in skeuomorphism - they are loanwords from a prior era in which the largest systems (armies, cities, factories) were primarily physical entities.

In our cybernetic age, though, these words are mere analogies.

Modern system design is not just about manipulating space; it is about manipulating state - and that means that the most critical aspect of system design is figuring out how to represent a given system's state. What is the right way to conceptualize it? What set of words, diagrams and quantitative metrics should be used to describe the space? How to balance accuracy with simplicity? How to evaluate different representations?

Put another way, the first step of manipulating state is making a map.

Maps are an ancient tool for representing physical space; schematics and blueprints are a more modern tool for doing the same. The tool for representing state space hasn't really been developed yet - in most organizations, it's most commonly represented in Excel, often for the purpose of financial reporting and budgeting. But I think that the more appropriate tool is probably a variant of the ontology system we developed at Palantir, and the process of constructing an ontology is akin to cartography.

Mathematically, this map, or ontology, is effectively a manifold - a low dimensional representation of the high dimensional space. By definition, this description is lossy - the map is not the territory, all models are wrong etc. Pragmatically, though, some models are better than others - they can be manipulated more productively.

In this endeavor, it's fascinating how many words we use derive from this notion of a map. Vision is about describing a particular region of state-space in enough detail to communicate it broadly. Strategy is often described as "charting a course" to that future vision and avoiding barriers. A milestone, which literally derives from a physical stone placed along a one-dimensional road or path, represents a discrete position within state-space. A single team's roadmap may represents a vector connecting multiple milestones while the organization's roadmap is a zoomed out version representing all of the vectors actively being manipulated by the organization.

These analogies are helpful, but they miss the key point. Unlike a traditional map, which represents 2D projection of our 3D globe, our map is purely imagined. It does not exist in any absolute form. Of course, this is loosely true in cartography as well. The NYC subway map is designed to show subways and warps the region for aesthetic and practical purposes. But spatial cartography is always anchored in physical reality - the territory is at some philosophical level observable, while ontological cartography is a more unbounded exercise (see also Cartographic Grounds: Projecting the Landscape Imaginary for a whimsical overview of cartographic creativity).

To conclude, our modern software systems are highly dimensional; but our mechanisms for understanding complex systems have evolved based on our need to understand lower-dimensional, predominantly physical systems.  

This doesn't make software easier or harder to design and manage - it makes it different. Hardware is hard because doing it is hard; software is hard because defining it is hard. Software has eaten the world, but our ability to conceptualize and visualize these cybernetic human-computer symbiotes has lagged our ability to build them. At some visceral level, we do not understand what we have created. This is not uncommon; the dance between theory and practice is rarely linear.

But the conclusion is clear - in an era where theory has gone out of fashion in science, it's more critical than ever in business. The idea of a map only gets you to the starting line - creating the right map is the foundation of everything else in a modern business.

Anti-agglomeration Policies

 The entire developed world is facing an extremely acute housing crisis; as agglomeration effects drive more global talent into a smaller number of international supercities, local incentives bias towards increasing real estate values. These incentives are in direct conflict with a clear national and international interest in increasing the rate of technological development, as real estate extracts an increasing amount of value from the innovation created via agglomeration.

Historically, technology development was widely distributed - the unique geographic features of distinct regions and the high cost of transportation allowed individual cities to leverage their unique natural environment to facilitate the development of localized supply chains and innovation hubs. Toledo was a glass city, producing windows and windshields for Detroit; Pittsburgh's access to iron and coal made it a natural place where steel could be produced.

However, with the rise of container transportation, global air travel and widespread internet access, goods, people and information are now able to move across space much more quickly. As a result, the physical barriers to agglomeration have been largely eliminated - the natural result is a drive towards consolidation around a smaller number of winning cities.

This effect is particularly acute in America, a country accustomed to boomtowns and their inevitable busts. But it is not limited to America. Innovation-driven capitalism naturally optimizes for agglomeration as density creates the human connections which fuel the development of new ideas and creation of new products and services.

One obvious solution is to examine the spatial distribution of major government functions. The United States is a federal republic; our government does not need to be geographically localized in Washington. By intentionally picking new locations for critical government agencies, policymakers could redirect their employees to communities which need investment. Much like universities can create geographic gravity wells around college towns, moving large swaths of the federal government into states would create similar hubs - helping agglomeration revitalize cities across the country that have been left behind by our increasingly globalized logistics systems and airborne travel networks.

It should be easy to organize a dedicated lobbying effort to rally Congressional support to eliminate all executive presence in Washington, DC. The Department of Agriculture could move to Des Moines; HHS could move to Cleveland; the Department of Energy could move to Houston; the Department of the Interior could move to Bozeman. Put each agency close to a city which already has an affinity for that kind of work, but which has lost population over the past 50 years. Critically, this should be extremely popular within the legislative branch - with the exception of Virginia and Maryland, every state could be a winner of a meaningful part of the ever-expanding federal bureaucracy.

Cyclists, Drivers, Runners and Pedestrians

One challenge for urban designers is that cyclists exist in an awkward superposition state between pedestrians and drivers - bikes are larger than people but smaller than cars; cycling is faster than walking and slower than driving.

Across the road network, therefore, we've generally accepted that dedicated bike lanes are an appropriate compromise - they create dedicated space for bikers ranging from 10 mph to 25 mph at the cost of ~6 horizontal feet.

The rise of electric scooters and bikes, however, necessitate a similar segmentation on mixed-use trails - because electric assistance increases the median speed of a bike has increased from ~10-15 mph to 15-20mph, the gap between both walkers and runners and cyclists has become much larger. Especially on highly-trafficked trails, situations in which two cyclists at different speeds are trying to pass pedestrians is becoming increasingly common.

The solution is to split off a packed gravel running and walking trail from an asphalt-based cycling trail. It should be well-graded to support strollers and ideally isolated from the biking trail as much as possible - e.g. by putting a pedestrian trail on one side of a river and a bike trail on the other side.

Leaving Palantir

Most well-known technology companies achieved their initial success by creating a singular breakout product which dominated an emerging category. Each story is different, but the pattern is similar.

But even after twelve years, the uniqueness of Palantir’s story continues to fascinate me – the lack of explicit hierarchy inherent in the company’s design creates a superposition of possible company structures, each only accessible from a different vantage point. And the symbiosis between us and our customers makes untangling what we do from what they do especially challenging.

Studying Palantir has been something of a hobby of mine over this past decade, and over the past few months, as I’ve started to feel my time here coming to an end, I’ve tried my best to distill my reflections down into something transmissible.

In my time here, I’ve been right about many small things, but I was wrong about one big thing – programs, not products or platforms, are the defining concept at the heart of Palantir.

Shrink-wrapped SaaS products may be good businesses, but they are a bad way to shape the world predicated as they are on an end of history worldview which implies that zero-sum optimization and financial engineering, not positive-sum growth and technology development, will be the defining activity of our generation.

Cloud-based computing platforms are also good businesses, but their optimism is indeterminate in nature, with a teleology defined by their users; a platform is a tool, but it has no intrinsic purpose. Great platforms are always a consequence of successful products, and their value is always defined by the next generation of products built on top of them; they are supporting actors solidifying known patterns into a strong foundation, not prime movers pushing on the ceiling into the unknown.

Programs – living cybernetic systems consisting of data, logic, and action – are the things that Palantir has learned how to build, at extreme cost. Programs are n-of-one entities; instances which are members of a class, but with unique identities, designed to be different and therefore to differentiate. We have collectively built these programs as an output of a tightly integrated business model – our best BD work combines elements of strategy and technical consulting, and our best PD work intentionally blurs the line between traditional software development and a classic services approach.

Producing programs is what Palantir was designed (has evolved?) to do.

But perhaps our work here has an even higher purpose, bordering on the spiritual – to challenge those of us who pursue it, and prepare us for our next adventures. Why did PayPal and General Magic, not Google and Facebook, spawn Silicon Valley mafias? Where are the Snowflake and Databricks founder mafias? Why did my brothers and I work for ABL, Tesla and Palantir, not Boeing, Ford, and Microsoft?

Culture, not technology, may end up being the main thing these institutions end up contributing to the world; people may turn out to be their most enduring legacy.

Palantir has a very special culture – of agency in the face of bureaucracy, of engagement with the world, of curiosity about how things work coupled with an optimistic belief that the future can be better, in concrete and achievable ways, than the present. A community of pragmatists, programmers, and philosophers. It’s a quintessentially American culture that transcends American geography, a melting pot of ideas and people whose whole is greater than the sum of its parts.

We are, and have been for some time, in the early stages of going supernova – spreading this unique culture across the broader industry and society, and I’m excited to join the diaspora.

Since I joined Palantir twelve years ago, there’s never been a bad time to join, and there’s never been a good time to leave; my work here isn’t complete, partially because it isn’t the kind of thing that is completable. The beauty of building a mission-driven company is that even as each campaign ends, the movement itself regroups to focus on the work ahead.

And yet the time has come to leave - to peddle my wares on the open market of late-stage capitalism and sink my teeth into a new challenge. I'm stepping onto the well-trodden path from software to hardware, joining an exquisitely capital-intensive space company and hoping to help them build an integrated hardware + software product strategy, something I've always been fascinated by theoretically; but as they say - in theory, there's no difference between theory in practice, but in practice, there is!

Perhaps I can even bring an ontology into my new board room, and into our new operations centers - the most forward-deployed engineer.

VLR!