The Future of Compromise

Whether it is in stopping quarrels between children or in deciding any of the thousand issues that come up in a large household, Anita can always make up her mind and keep things moving. A family such as ours must have a strong, capable leader.
(Strong, capable tyrant, I said under my breath.)
-Robert A. Heinlein, Friday

Getting things done involves a strong dose of leading with a vision, and ignoring those that disagree. When such leaders are given the reins, the forward progress can sometimes, post-hoc, justify trampling others. Of course, when men do this, it’s called leadership, but when women do it, even when they are doing the same things, the research shows that it’s likely to be referred to more negatively . On the other hand, once given the reins, a rising tide can lift all boats . Successful leaders ensure that enough of the progress is towards shared goals, so that the rising tide compensates the trampled masses. But it doesn’t always work out.

The key difference between leaders seen as heroes after the fact and those seen as villains is the post-hoc consensus that what they accomplished was good. (Gender stops mattering in retrospect.) The tension between disagreement now and perceptions in the future illuminates the essence of how democracies fail — but also how politics can promote wider success. I think this dynamic shows deep reasons that compromise can be reached, that decisions are not impossible, and that politics doesn’t need to destroy our ability to move forward.

Of course, the US may still be royally screwed.

Leadership and Shared Goals

In my earlier posts, I’ve assumed that decision makers have coherent goals, albeit perhaps underspecified ones. In doing so, I have noted that within organizations, the goals are less coherent. The earlier assumption was that coherence between stakeholders is possible, but ugly reality intrudes on the earlier soi-disant theory. Most of the time, different groups with different interests aren’t being aligned from above. This leads to controversy and fighting — which can, but needn’t, undermine the possibility of progress.

It seems even prior agreement about priorities fall by the wayside if the outcome isn’t beneficial. If the tide of globalization hurts people, few of them are utilitarian enough to care that the benefits to others greatly outweigh those costs. But if the benefits of a policy are widespread enough, (and recognized as such,) few will protest after the fact that other groups benefited more than they did.

In my previous post, I discussed Wilson’s Bureaucracy and how organizations deal with goals — but now I want to move towards this other side of organizational culture, where there is no alignment or consensus.

If everyone wants the same thing, the differences are only about tactics. A company controlled by shareholders that only seek profit falls nicely into this category. A startup where the founders’ vision conflicts with maximizing short term profitability, though, has deeper strategic disagreements. Even worse are public organizations of the sort Wilson discusses, where the will of the public can dictate conflicting goals, and the goals of regulators and bureaucrats may not align with either. But worst of all is politics, where we can start.

Politics

I don’t usually like talking about politics, so don’t worry — just this once, politics isn’t about politics, it’s about policy.

Impossibility, and the Median Voter.

In the real world, we don’t see many shared goals in politics. That’s because the central political question is always how we make decisions when people disagree. Why is it so hard to decide? To explain, imagine I’m designing a voting system (a pastime that mathematicians and cryptographers have largely stolen from political scientists.) I want a few simple things to be true;

• If everyone agrees that one alternative is better than another, the worse alternative can’t win. (This is called “Unanimity”)
• Adding choices that no-one likes shouldn’t change the result. (This is called “Independence of Irrelevant Alternatives”)
• We aren’t just picking whatever a particular person wants. (This is called “Non-Dictatorship”)

What Arrow showed, and won a Nobel prize for, was that (with two or more voters, and three or more possible alternatives,) you can’t make all three of these things true. (My favorite theorem to mispronounce, the Gibbard–Satterthwaite theorem, extended this proof even further.) If you want to make reasonable choices, you can’t have a democracy, you need that “strong, capable leader” I mentioned earlier instead.

This is frequently interpreted in a way that is overly cynical; voting systems cannot work. What these theorems actually show is that, given some crazy set of preferences, one of these conditions can always be violated. Most of the time, people’s preferences are, if not aligned, are at least compatible. And even if we’re using the simple “majority rules” system, given some simple assumptions, the Median Voter Theorem says we’ll end up OK. The median voter theorem, as the New York Times explains it, says that “when it comes to the big issues, voters at the midpoint usually get the policies, if not always the exact outcomes, they want.” And despite the simplifications needed for this to apply, it seemed to hold up quite well in practice. Well, at least until the 1960s.

The key problem with the median voter theorem is that it assumes we can place all our preferences neatly in a single line. As long as you’re OK with an American Democrats-Republicans split, that works. If 49% of the public is Republican, and 51% is Democrat, obviously the Democrats win. But if 46% are far-right, 44% are far-left, and the remaining 10% are centrists, then the two parties will converge to capture the center, ignoring the strong preferences of their base in order to win 51% of the vote.

When libertarians and other electorally irrelevant third party groups complain that the two parties are basically the same, it’s a product of exactly this dynamic. But as soon as people’s preferences get more complex, this breaks down.

Wikipedia’s 2-dimensional political spectrum

If you remember your high school math, you find the media by throwing out the pair of lowest and highest values from your data until you have 1 value left — the median. (If it’s even, we average those points.) Voters can be treated this way in a simple 2-party system; you’ll support whichever party your views are closest to, so the parties can ignore anyone that isn’t near or at the median. But this requires the parties to figure out who is to the left, and who is to the right— meaning we need the values to have an order from lowest to highest.

But you can’t put points on our 2-dimensional spectrum in a single unambiguous order; is (1,3) bigger than (3,1)? What about (2,2)? It turns out that there is something special about medians that doesn’t apply to higher dimensions. And this leads to logrolling — the practice of exchanging favors, especially in politics by reciprocal voting for each other’s proposed legislation. But before we get there, I want to talk about how powerful the simple idea of the mean voter can be.

Simple models and predicting the future

It turns out that the simple logic of the median voter theorem can be used to great advantage when trying to predict even complex behaviors in chaotic systems. For instance, Bruce Bueno de Mesquita has built a career around using models inspired by this approach to make impressive predictions about international affairs — from predicting the future leader of Iran 5 years in advance, to the way Beijing would handle the handover of Hong Kong a decade in advance. To forestall the inevitable rejoinders, I agree that his claims are a bit overstated, but I want to discuss the methodology, which has been adapted and used by others. (Footnote: Full disclosure: the linked dissertation is from a graduate of my current program.)

The key inputs into the model are the preferences and relative influence of the different groups involved in the decision. For example, in the case of predicting the election of Rafsanjani, the influence and preferences of 27 different groups was assessed by a panel of experts. The assumption is that the different parties would interact in specific ways dictated by rationality and game theory. The way that the communication and recursive estimation of the position of each party occurs is complex, but in the end, the model boils down to a question of how the different parties arrive at a compromise on an issue.

The key simplification, however, is that the preferences are on a single scale — in the case of Iran, from Hard-liners on the right, to Reformers on the left. This is essentially the same tune used in American politics, with the key transposed from Christian to Islamic, then shifted a couple octaves.

Impasses and Logrolling

The problem of dimensionality presented earlier isn’t necessarily a death-blow to using the median voter theorem to resolve disputes, for two reasons. The first, which is illustrated in the example above, is that for many issues the point being considered is so critical that everyone truly does line up on a single axis. The second is logrolling.

If I’m arguing about the menu for a dinner, and I prefer potatoes to squash, and would like fruit soup as an appetizer, and my wife prefers squash to potatoes, but hates fruit soup, we can trade; we’ll have potatoes and vegetable soup. As explained in the Buchanan and Tullock’s classic “The Calculus of Consent,” this is exactly what happens in Congress, or any similar system.

The senators from Idaho care deeply about potato policy, and those from California care about policies that affect fruits and nuts grown there — but neither cares much about the other’s issues. When a key vote on potato taxes comes up, the senators from Idaho are willing to give up their choice of soup — and/or their votes on almost any other legislation — in order to get their way. And behavior like this is a significant problem for methods like Mesquita’s. As he says, it’s “less likely to be effective in situations [where] cooperative side deals like logrolls or vote trading shape outcomes.”

According to some political scientists, logrolling is, or was, the grease that allows the smooth functioning of a political system where critical disagreements are common — because there are still points of agreement. I’m not qualified to judge whether they are right, but the explanation seems to have some merit. And as Jake Sherman at Politico put it, we live in “an era when back-room dealing is often seen by party hard-liners as treason.” And you might think this makes Mesquita’s predictive models more applicable, but as we’ll see, it won’t.

Smashing the Patriarchy

Feminists were not the only people who rebelled against the system. But change in organizations and cultures is slow. As Scott Alexander put it, “Systems are hard. Institutions are hard.” If you have an agenda, you need a strategy for getting it done, and a plan for executing it.

A 1971 article entitled “The Tyranny of Structurelessness” noted that “During the years in which the women’s liberation movement has been taking shape, a great emphasis has been placed on what are called leaderless, structureless groups as the main — if not sole — organizational form of the movement.” The drawbacks to this were clear at the time, and I laid them out in a different context when discussing corporate structure.

Feminists were powerful as long as their issue was central; the group could push for what they wanted purely on the basis of owning the attention of a decent percentage of voters. At some point, when some key aims had been accomplished, however, despite commanding a huge following, they ended up marginalized; the Equal Rights Amendment never passed, and the movement was outmaneuvered. They never ended up well organized enough or again powerful enough to recover.

In order for an interest group to push their agenda forward, they need the either be powerful, or smart. An example of the former is how labor unions use strikes; the business cannot function without workers, and by creating a new constraint on the system, they can render it inoperable. This gives them leverage even when they aren’t striking. The latter mode is where I want to focus first; using the system against itself.

Exit, Voice and Coalition-Building

In my last post, I noted an annoyance that agents insist on being all agent-y, wrecking the plans of the people in charge. Now, I’ll take the opposite side.

As an agent, especially one without power to veto, nor the ability to create systemic rules, you are playing a different type of game, and your options differ. The classic approach to this is the trichotomy of exit, voice, and loyalty, which, if you’re not familiar with you should read at least the linked summary of before continuing — and thanks to Olya Oliker for telling me I needed to read it. My contention, once you’re familiar with those terms, is that there are a panoply of options in-between voice and loyalty which I’ll call manipulations.

If you want the rest of the system to pay attention to you, “voice” says that you can shout, and the people in charge will listen; that’s what happened to the feminists agenda in the 1960s. But that has a limited effect unless you’re really loud, or already have a platform. “Exit” says that if the system is broken beyond repair, letting it die is the alternative — and I’ll get back to that point. For now, the alternative to voice and loyalty is not exit, but building coalitions and trading favors to change things.

Mesquita’s insight here is critical if you’re figuring out how to do this, and it is where the chess of organizational manipulation gets started. If you don’t have an advantage, you either need to capture the center, or you need to develop your position and gain enough freedom so that you can trade pieces until you get to your endgame. Of course, all of these strategies are complex enough to require clear goals and well-managed organizations.

Capturing the center is a well-respected political strategy. If your position is in the center of the Overton window, and looks like the reasonable compromise, then you only need to ensure that the decisions being made are public enough, and one-dimensional enough, that the median voter result happens. You don’t need to be strategic about your use of power, you just need to prevent someone else from undermining or outmaneuvering you. When you’re on track to win, you need the decision to stay one-dimensional; keep the fight public and the rules of engagement simple.

Much of the time, though, groups have goals that are idiosyncratic or orthogonal to the rest of the organization or the public. That’s when the chess game gets interesting. A chess novice doesn’t become a master because they are great at building end-games, where they let their advantage play itself out. Instead the mid-game, with all its complexity, is where great players shine. It is for this reason that special interest groups thrive on backroom deals, trading favors, and obscure rules and rule changes that make navigating the system harder for others. They push to break Mesquita’s model, instead of ensuring its applicability.

And that’s where manipulation, midway between loyalty and voice, is used. But it too has limits.

Holmström’s theorem and the limits of manipulation

As I mentioned briefly in my last post, “Holmström’s theorem shows that when people are carving a fixed pie, it’s impossible to achieve a stable game-theoretic equilibrium and be efficient too, unless you ignore the budget constraints.” This is closely related to Arrow’s theorem, but pays attention to agents a bit more; each one can’t do any better than the current level simply by changing their vote — if there was extra pie, it would be taken already. Instead, the only ways to benefit are to maneuver to steal pie from other people, or to make more pie.

Corresponding to these two modes, manipulation by agents takes one of two forms; selfish, or beneficial. Selfish maneuvering is always equivalent to taking cake from someone else, on some dimension. If the axes are fixed, and the end point is closer to your position now than it was, then the change will be zero-sum. (This is assuming the earlier position was stable. If not, you could move the position from somewhere no-one likes closer to everyone’s preferences.)

The better form of maneuvering is making more pie. This is transforming the axes or relaxing constraints — because in a transformed system, perhaps it’s possible to move closer to everyone’s goals. I noted in the earlier post that alignment of goals is one way to make more pie; you don’t need anything fancier than working together towards an agreed upon outcome. If the goal is profit, that relaxes the budget constraint, and more is available to everyone. Unfortunately, it seems we’ve eliminated that option now that the outcomes are exactly what is being contested. But there is still a way out of this dilemma.

There’s No Exit

When we’re talking about corporations, exit is a third alternative. Once enough people exit, the organization is marginalized, its market share drops, and the inevitable competition picks away at whatever remains. This is not true of government organizations — the only way to replace a government organization, short of armed revolution, is from inside of government. (Of course, you can try starting over on a boat, or man-made island, or just going to Mars — but that doesn’t change the system anywhere else.)

Many social movements, Feminism among them, have widely shared dissatisfaction with the status quo, but no clear shared vision for what the future will look like instead. Do we want mandatory government intervention to equalize wages, or legal recourse for those marginalized? Do we want equality of opportunity, or equality of outcome? The underspecification of goals is a malady that affects the structureless even more than it affects the structured. A dictatorship, on the other hand, has it easy.

The problem is that in self-organizing systems, complexity and chaos is the natural result of benign neglect. Breaking things is even easier than creating bad metrics. To finish the earlier Scott Alexander quote, “If your goal is to replace the current systems with better ones, then destroying the current system is 1% of the work, and building the better ones is 99% of it.” If there are goals other than a Tyler-Durden-esque smashing of the system in order to let anarchy reign, you need to be incredibly sure your goals are clearly specified.

And this is where I mention Trump, the elephant-nominee in the room. His goals are unclear, and he’s strategically refusing to be coherent long enough to change that. That means he’s building a coalition with shared dissatisfaction, without needing to clarify where he’s headed. He’s alternatively interpreted as a chaos candidate, or a wannabe dictator. But given my thoughts above, this should make perfect sense as an outcome of a powerful coalition without a goal; either the complex system gets broken, or it gets controlled from the top-down. Neither outcome is a good one.

A Ray of Hope

A candidate whose position is unclear can only triumph over a field of opponents because their ideological positions can’t be confined to one dimension; the median Republican primary voter got screwed. The current situation isn’t inevitable; the law of large numbers can lead to the mean voter theorem, instead of, as Eliezer Yudkowsy puts it, “caus[ing] all democracies to make similar mistakes.” The problem remains, however, that the spectrum isn’t defined clearly; we’re adrift in a multidimensional sea of options. But picking between options is exactly what some people I know have spend a couple decades thinking about, and they have some ideas.

As I alluded to earlier, Holmström’s theorem does leave room for breaking the trap, but it involves finding cooperative solutions despite disagreeing on goals. Rob Lempert, a professor of mine, has had success getting both sides to agree on, of all things, how to react to climate change. The trick is being clever dealing with multi-dimensionality, and work he did with Steven Popper, a member of my dissertation committee, has shown a way around the problem of cooperating despite disparate goals. It’s incredibly interesting math, but I’ll try to avoid getting too deep into the math, so I won’t do it justice.

Everyone wants something different, but that doesn’t mean we can never agree on anything. The trick is in having a reliable enough system model that we can make decisions; if we have that, the internal model principle says we can guarantee our outcome matches what we want. Of course, “what we want,” i.e. the outcome, is exactly what we’re debating — and we don’t necessarily agree on the inputs either.

This is where I think Robust Decision Making (RDM) comes in. It essentially uses simulation to explore the space of potential outcomes from different possible inputs and decisions, instead of looking for optimal outcomes using a specific metric. By analyzing many plausible options, and considering many different and disputed inputs, the space of plausible future conditions can be generated. This allows the generation of many metrics, and exploration and discussion of that space, instead of choosing the optimum value.

Based on these discussions, analysis can sometimes identify decision options where the divergence between goals is minor, or nonexistent. (It also lets decision makers deals with uncertain inputs, as opposed to contentious ones.) You might note that Arrow’s theorem and its extensions prove that this isn’t always possible — and that’s OK as long as our preferences aren’t pathological. But an example could make this clearer.

Droughts and Compromise

California is in the middle of a drought. (Yes, it still is, despite hope that El Niño would fix everything. Like I said in my last post, sometimes staying indoors doesn’t help.) The types of projects that would mitigate this are expensive (billions of dollars), and the impact depends on many uncertain and disputed factors, like climate change, future agricultural and demographic changes, and always uncertain weather.

The question of what to do about it is huge, uncertain, and subject to a clear disagreement; Southern California and Northern California are both hoping for more water, but it’s not coming anytime soon. The problem is that they have only one state budget, but the two halves disagree on where more water is needed. (Of course, city dwellers, farmers, and environmentalists also have their own coalitions and goals)

The best option for providing water is unclear, and disagreements in goals is real. Even a basic analysis finds there are some scenarios where critical water shortfalls will need to be addressed. Of course, there are many options that will help one group, but not others. For example, desalinization plants will help wherever they are located, but not elsewhere. And the environmental impacts of such plants leads to NIBYism. This leads to a question, not of deep uncertainty, which is where the “Robust” part of RDM is incredibly powerful, but of cooperative decision making.

For my current purposes, what RDM does is discover or create options that groups with partially incompatible goals can agree on. Another key part of the cooperative component of RDM is how to actually involve decision makers, and get them to agree — critical, but not central to my current point

Some options, it turns out, are helpful in almost any situation, no matter what your goal is, as long as it’s not pathologically opposed to the other goals, nor insisting on keeping the game zero sum. In this case, water pipes connecting northern and southern California are one game-theoretic cooperative strategy. If it turns out that we need more water in the north than in the south, we can send it — and vice-versa.

This does NOT require us to keep the options in a single dimension. It also doesn’t tell us, given despite different preference, what to do optimally for everyone — because as we mentioned earlier, no such decision system can exist. These pipes probably won’t solve the problem on their own, and in fact are not optimal given anyone’s preferences. But despite being suboptimal, they are a robust decision, because they can be implemented now, are acceptable to all parties, and they keep almost all other options open. More importantly given the earlier impossibility results is that they are a pareto improvement — everyone gains, because they mitigate the effects of a water shortage. This can give us a huge advantage in considering how to approach the problem, keeping the decision non-zero sum.

The potato farmers in Idaho and the fruit growers in California both need access to water for farming. They may also disagree about whether climate change is occurring, or man-made. But that doesn’t mean they can’t find common ground. (And we don’t even need to convince anyone that they are wrong.) In this case, the Snake River brings water from the farmland of Idaho all the way to the Columbia River, which goes right through Portland, into the Pacific. People have proposed building a pipeline from the Columbia River down to northern California, at significant cost. One outcome of this would be to let water be shared between those Idaho farmers, and the ones in California. I haven’t done the types of analysis needed to evaluate this, but I know this almost certainly isn’t the best option. And despite that, it may be a robust option, one that helps, and that people can agree on.

Compromise, Elsewhere

The specific mechanics of how RDM is used makes it inapplicable to many decisions, (especially where you can’t apply the internal model principle,) but I think the general mode is still a useful one. We can allow for people to disagree, and still find positive-sum compromises on key questions. When different groups with different desires come together, abandoning optimality in favor of compromise seems like a good way to get around the fact that otherwise, it’s an impossible dilemma.

The requirements to do so, however, are twofold; groups with clear goals, and a willingness to transform zero-sum games like politics into positive sum ones. The alternative, if we’re looking to change things, is breaking the system instead — and it should be obvious by now that, far from being a solution, powerful coalitions with unclear goals are really just a way pulling others into a negative sum hell.

Here’s to hoping we don’t end up there.