The Legibility Tradeoff

Kartik is a 2014 blogging resident visiting us from his home turf at akkartik.name.

I am fascinated by organizations as a technology for agency transfer — getting people to follow some plan outside of their selves. We’re not yet very good at building such agency transformers; our organizations get gamed, taken over, taken advantage of, treated as externalities, captured by minority interests, ground down to gridlock, etc. But we’ve been getting better at it, finding better ways to influence others than the coercion and threat of violence that we started out with. In this post I want to survey the progress we’ve made, and suggest that there’s still wisdom to be milked from the old saw of “don’t micromanage, delegate.”

Delegation is, at root, about tolerating illegibility in means in exchange for greater control/legibility of ends. Backing away from micromanagement allows one to see the forest for the trees. You may think that our ability to delegate requires language, but it started far earlier, around the time our ancestors first turned a stone into a tool. In fact, all our ways to bend the environment to our will lie on this spectrum fellow-resident Kevin Simler shared earlier this year, from building things to cultivating and domesticating flora and fauna, all the way to persuading our peers:

Sometimes we assemble things to further our will, and sometimes we enter into delicate negotiations. It seems weird at first to consider those two classes of activities together — “furthering our will” seems too tenuous a connection, the mindset required to merge them too instrumental and sociopathic. In western culture mixing these categories is usually unflattering — like calling someone a tool, or calling a machine temperamental. But there’s something here worth pushing past the discomfort.

Look again at the spectrum. How would you decide where to put something? The first distinction that comes to my mind is that there’s more agency on the right. But agency is a complicated subject. Smart minds disagree about where to position the line dividing the inanimate from the animate, like Bruno Latour’s famous example of the speed bump. A simpler, more objective test for our purposes is the level of push-back on our will.

On the left it’s easier to have fine-grained control, to micromanage, to predict precisely what the consequences of your action will be. On the right most of the stimuli are internal, the process of evolution is non-linear and hard to predict. It’s doing its own thing, all you can do is try to nudge it one way or another and wait, often without any certainty about consequences.

Another way to view the spectrum is as an answer to a question: how likely were the ancients pray to it? If you plot old gods on this spectrum they cluster to the right — nature, wind, fire, rain and so on. Phrased like this, it seems obvious why: there’s no reason to pray to something you can instrumentally control. What’s more, all the gods I could think of toward the left of the spectrum are older gods, from a time when humans had less instrumental control. A particularly cool case study is the evolution of the Egyptian goddess Bastet from fiery lioness to rattle-wielding pussycat over the millennium it took to domesticate cats.

When things bite back

Back to agency transfer. Given this spectrum, it’s tempting to conclude that we are forced to engineer or micromanage the inorganic, the shiny, the straight lines, the flat chromed surfaces. And that we can only train or manage or delegate to the organic, the realm of leaves, flowers, thorns, bark, fur, tooth, claw. But if you look closely, even the inanimate can push back on your will. And it does so in relatively simple and instructive ways, allowing us to take some lessons onward to the complexities of people in organizations.

Say you want to design a kettle. You might decide that you care about the following five characteristics:

• weight: It shouldn’t be too heavy to carry
• capacity: As much water as a person is likely to want at one sitting.
• strength: Enough to hold the water capacity without collapsing.
• conduction: When you stick it over a fire the water inside eventually boils.
• insulation: There’s a way to pick it up without getting scalded.

Each of these characteristics can be reduced down to a number (in appropriate units). Under this scheme, one possible way to visualize a kettle is as a list of five numbers. Or alternatively as a pentagon.

This view loses a lot of information but also helps in some ways. For one, you can visualize many of the differences between two designs at a glance:

With this visualization you can also imagine a “habitable region of kettles” — anything outside this region is guaranteed to be a bad kettle that nobody wants for anything (except maybe art).

You can also imagine regions of impossibility. For example, given the materials we have access to, there’s no way to make kettles that are really strong and really light. But such contradictions take us beyond the scope of the pentagon visualization (into the territory of Sam Penrose’s talk at Refactor camp 2012, and further to Christopher Alexander’s work that inspired it). The point is, even with simple kettles in a tiny space of possibilities, we start to see some constraints pushing back on our will to create a kettle.

***

But that’s an idealized sort of experiment. There’s far more that goes into kettles.

This wide variety comes from a wide variety of constraints. You need kettles to be easily mass-produced, so you have to know where you’ll source the raw materials from, ensure you’ll have a steady supply. You might need them to look a certain way so they signal the sorts of things that your target market care to signal. Put all that together and you end up with a lot more variables. Instead of five regions on that diagram, you might end up with twenty or five hundred. Pentagons or polygons won’t do now. We have to wrestle directly with the fact that we’re dealing with many more dimensions than we can visualize.

Difficulty visualizing isn’t just an abstract problem. One of the things that’s really hard to intuitively understand about dimensionality is that as you add dimensions the number of nooks and crannies in your space explodes. As evidence, consider that as the number of dimensions increases, a sphere takes up less and less space inside the smallest cube that can fit it. As you add dimensions, hypercubes turn into these spiky hedgehog-like shapes, with less and less space in the center where a sphere can use it.

If that’s hard to imagine, trying to visualize a cubical planet can be quite effective at providing a visceral sense for just how spiky even a regular three-dimensional cube truly is. Scale helps where dimensionality fails.

So increasing dimensions quickly increases the number of nooks and crannies where our kettle design can fall into some contradiction or unanticipated consequence. The curse of dimensionality pushes back on our will to create a kettle.

***

Constraints and dimensionality are just two waypoints in the vast gulf between legible things that don’t have agency and creatures with an illegible mind of their own. A book called Why Things Bite Back is filled with examples of technology having unforeseen consequences, consequences that we can’t help but anthropomorphize as mischief or malice. Building a simple kettle isn’t entirely mechanistic, and building shiny inanimate things can seem increasingly organic. Perhaps the regions of Kevin’s spectrum are better viewed as concentric circles of progressively harder challenges to agency, from easy cases of building and assembling, through cultivating and domesticating, and finally the ‘wild’ terrain of negotiating. All three kingdoms of things, creatures and groups exhibit examples of these regions. Across all three kingdoms, the number of situations where you need to negotiate far exceed those where you can get by with engineering.

All our thousands of years of tool-building intuition are starting to rub up against fundamental limits. To make progress we may well need to go against all our instincts.

Delegating to the inorganic

If designing a kettle like a tool creates these inevitable eddies of secondary consequences, what’s the alternative? Since things bite back primarily by expanding the amount of work beyond our capacities, the solution might be to take on less work. Think about what it would take to grow a plant or even a single cell if you had to arrange every last molecule just so. That’s a problem far more complex than any kettle. And yet we grow things all the time. We do so by arranging the environment to nudge them in the right direction, and by applying selection pressure to weed out undesirable characteristics. We don’t micromanage how things grow, we delegate to them.

Growing kettles instead of designing them would obviate our worries about pentagons, constraints and alternatives. But how would it work? Imagine if you could easily create new random kettles in large numbers. You already have a list of desirable characteristics that you can score each of them on. If you selectively mix the characteristics of only desirable kettles you would be in effect breeding them, and repeated actions of selection over time would cultivate a wild crop of kettles into something more tame. In software we embody such thought exercises in genetic algorithms, which enable us to rapidly simulate multitudes of say kettle designs without having to deal with any real kettles. It’s a well-known idea. It isn’t the solution to our organizational problems in itself, but it’s indicative of some fruitful directions to explore.

Why does breeding kettles work where designing them doesn’t? It relinquishes control over the precise details of construction and focuses control over the desired outcome. It renders the ‘why’ more legible by allowing the ‘how’ to grow illegible. Let us now leave kettles behind and consider some other examples where focusing on the ‘why’ yields good results. That should increase confidence that we’re on the right track.

As our first example, consider chess. Under the selection pressures of increased competition by ever-smarter opponents, players have been forced to replace straightforward attacks on the opposing king with ever more indirect methods. This evolution makes it hard to anticipate say which opening moves will be in fashion a year from now, because the focus on the goal has forced us to relax constraints on the means.

Another example. Classical mathematics deals with simple curves where you can plot an equation point by point — quite direct. But we’ve also more recently learned about strange attractors, where the curve gradually emerges from randomly performing a repeated computation. Illegible (randomized) computation, legible (deterministic) result.

A more practical example from our everyday experience is the modern notion of a free market. It took us decades after the creation of free markets to articulate their key benefit: efficiently transmitting information through an economy. Had we tried to solve this problem by micromanaging the ‘how’, our solution would have involved a lot more plumbing: rules governing what people are required to disclose, forms to fill out showing what they disclosed, ad infinitum. It turns out we don’t need any of that. It turns out all we need is a single place for buyers and sellers to meet and trade bids. Then let them do their thing.

As we’ve discovered limitations of the free market, our attempts to regulate it haven’t shown similar sophistication. They’re instead blunt patches, simplistic rules and forms 10-K. We haven’t paused to wonder if an unanticipated secondary effect perhaps requires a more subtle change to the rules that affects the environment and the selection pressures. For example, there’s lots of regulation affecting how property is assessed for its value at times of taxation. But an elegant alternative to all the regulation is to allow people to give any assessment — and then force them to sell if they receive a matching bid.

Growing better government

Unfortunately this idea of giving up agency in the small is swimming against current fashions in the realm of government. There’s been a trend in the US recently of law-makers taking away the powers of discretion of the enforcers with legislation like the three-strikes rule and various mandatory sentencing laws. In effect they replace human beings with rules too stupid to recognize exigencies in individual cases, rules more stupid than humans that are inevitably far more susceptible to gaming by humans. Just returning discretion to the humans in the system would be an easy way to stem the tide of ever more knotty secondary consequences that are starting to arise. And if you’re concerned about bias in your people, say the problem of prosecutorial overreach, a new option has just become available to increase accountability: surveillance. It’s worth consideration since it plays to our strengths in this era, and leans into the direction technology is taking anyway.

Such solutions are hard to countenance because they almost inevitably lead to uncanny-valley effects, causing a new rule to seem deeply unnatural and alien. Forcing you to sell a property in some situations feels weird because it impinges on our basic notions of ownership. But then our notions of ownership are already being rendered obsolete by licensing rules, end-user agreements and so on. Even a simple auction that requires you to stand by your bid is a precedent to this idea. So while we need to address uncanny-valley effects head on, we needn’t fear them completely. There’s a shift underway; leaning into it might give us more room for maneuver.

To make these considerations more concrete, I’ll end with a little fictionalized example of a legal system that makes the ‘why’ more legible by letting the ‘how’ grow illegible. It isn’t a serious proposal. In fact it’s deliberately framed as a naive caricature. As you read it, ask yourself not if it is feasible as a coherent whole (it isn’t!) but whether some of its elements might bear consideration and working into our world.

The utopian participatory panopticon

Imagine a world where we still elect representatives and they still gather to decide on the great questions of the land. However the result of these gatherings isn’t to generate new rules or laws or amendments. Instead, they meet to pick cases that everyone should pay attention to.

The cases are generated by roving judges who roam the land making judgments and deciding claims. When you as a citizen feel an injustice has been done to you, you call the nearest judges or go to the nearest judge station to receive summary judgement. In the old days they were called “police”. Since then their job has come to be dominated not by detection — that’s easy in a world of ubiquitous surveillance — but by Judgement and Sanctioning (‘sentencing’ in the old days).

The powers of discretion given judges are vast. Their jobs come with a stable life free of wants — and the knowledge that their actions are scrutinized far more closely than those of regular citizens. When their decisions are unpopular, they get appealed by actions of Protest. Protest requires Investment: donations made to the global commons. If you want to bump up the queue of appeals you need to make a larger Investment to get yourself noticed. Investments are denominated by your means, and don’t have to be monetary. If your time is worth a lot, coming down in person to appeal a parking ticket is an Investment, something the judge will take into account in your case, along with every other last detail of your circumstances. Convincing others to Protest on your behalf is also a sign of Investment.

Children learn about all this in school. In history they learn of a dark age when people were ruled by rules, and when even the science fiction involved insufficiently-radical ideas like being ruled by a single monolithic, dispassionate Algorithm. That seems self-evidently silly to them all. Of course a single Algorithm is a terrible idea, being a single point of failure. And besides, who gets to modify it? Who watches the watchers? The only reasonable answer to both questions is: everyone. Instead of an Algorithm, they have an ocean of Big Data that everyone is free to apply heuristics to. And it is Good.

The seat of government is marked by a giant sculpture of a frail Lady Liberty weighed down by the devil, reminding citizens of the temptations of government, and of their duty to Invest in the best causes, to police those who govern. Voyeurism is a non-problem; the real problem is maintaining a sense of meaning around the inevitable tedium of watching and analyzing.

***

Like I said, I’m sure there are many angles here that I haven’t considered, so don’t go agitating for change just yet. Consider it a thought-provoking exercise. We currently leave individuals too unpoliced at the top or center of our organizations, and take away too much discretion from the fringes. Understanding the patterns by which such blind spots emerge in our collective consciousness will help us to create a wiser society out of the imperfect human clay we have at our disposal.

Thanks to Sam Bhagwat, Rebecca Allard and Venkatesh Rao for reading early drafts of this post.