Why Habit Formation is Hard

Recently, I moved from Las Vegas to Seattle. In the process I realized that activities like moving belongings and getting a new driver’s license are not the hardest part. The difficulty of moving habits is much higher. About 80% of the cost of a move, I suspect, is the cost of moving habits. We lose months of time in the run-up to a move and after.

An example is your gym routine. It’s possibly the most important habit in your life. But it is surprisingly hard to “move” from one context to another.

In my case, I signed up for a gym very similar to the one I used to go to in Vegas. It has similar facilities and a similar range of equipment, trainers and programs. Like my old Vegas gym, my Seattle gym is about a mile and a half from home. The membership cost is about the same.

Yet, it’s been more than a month and I still haven’t found my rhythm. By contrast, when I joined the Vegas gym, it took me less than a week to settle into a great routine.

Why is this?

The Structure of Habits

Let’s start with a definition.

A habit is a stable, repeatable pattern of behavior that involves minimal meta-cognition, and achieves predictable results within a particular local range of conditions, defined as a combination of a cognitive context and a physical context.

In other words, it is a predictable behavior you can execute without thinking too much about it, so long as you are in a particular state of mind and in the right place/time for it.

We can borrow a term from mathematics and call the “local range of conditions” the region of attraction. Think of it like the gravitational region around a planet. Within a certain distance, and a certain range of velocities, small asteroids that enter the region will be captured into orbit. 

So every instance of trying to execute a “gym workout” is like an asteroid flying by a planet. Getting captured into orbit is like successfully finishing a workout.

A large region of attraction makes for a stable and robust habit. A small one makes for a fragile, easily derailed habit.

From the definition, we can infer that every habit is actually two intertwined habits. There is a habit of thought and a coupled habit of action. 

• A habit of thought is a set of coupled patterns of thought and a practiced ability to switch among them appropriately and effectively.
• A habit of action is a learned pattern of physical behavior involving sensory processing and physical movements.

Both are context-dependent. The former is dependent on your immediate state of mind, the latter is dependent on your immediate environment.

Some habits, like switching between brain-dump mode and edit-mode in writing, are almost entirely habits of mind. As you mature at the writing game, you become better at switching between the two modes at the right time. Inexperienced writers often get writers’ block primarily because they stay too long in one or the other mode and get frustrated. The action component is trivial (the physical behavior in both modes is typing or working with pen and paper; switching at most involves taking a short break or changing locations).

Other habits, like your bedtime ritual (end-of-day chores, brushing your teeth, changing into pajamas, and whatever else you do), are almost entirely habits of action.

Most complex habits are a mix of non-trivial thought and action components. Exercising is an example of a complex habit. For some of us, being outside of a narrow range of moods makes it impossible to work out, and expanding that range is hard work. For others, the mood is never a problem, but even a slight change in physical conditions derails the intention to work out.

Learning and Habits

Habits that have been successfully acquired involve very low meta-cognition, but getting there involves plenty of meta-cognition: learning the habit.

Almost all this learning has to do with increasing the size of the region of attraction where the habit “works.”  We can distinguish two phases of learning a habit.

1. Doing it right for the very first time (10%)
2. Expanding the region of attraction till it is large enough to be worthwhile (90%)

As we’ve discussed before, beginners typically learn (and get anxiously attached to) one way of doing something. For example, in a video game, you might learn exactly one fragile way to pass a level. In a level of a vertical shooter I used to play a few years ago, that “one way” involved starting in the bottom right and shooting the aliens in a very specific sequence.

This “one way” is obviously fragile, since if initial conditions are not exactly right, your behavior will be derailed. This is why you need the second phase: gradually expanding the region of attraction by introducing small contextual variations — both mental and physical.

In terms of the idea of a region of attraction, the one way of a beginner can be thought of as a single point. Any deviation from ideal conditions will derail the habit.

So if the behavior is “mounting a bicycle” and you can initially do it only when you are in a confident and energetic mood and are starting on a hill, headed down, you might expand the domain of your “bicycle mounting” habit by learning how to do it in a variety of moods and energy levels, and in all sorts of conditions.

Eventually, you hit diminishing returns. We all have different stopping points when we give up refining a habit. In mounting a bike, most of us give up once we can start on any reasonable grade, and learn the standing-astride and running mounts. Circus performers might try and learn how to mount easily from both sides, from behind, facing backwards and on extreme grades you’d never encounter on normal bike-paths.

This tells you that a habit is also an algorithm that has been gradually evolved to handle a sufficiently large range of conditions to make it worthwhile. Your investment in learning is wasted if the region of attraction does not grow sufficiently large.

The shape of the learning curve for a given habit (either new, or ported) in a given context depends on the complexity of cognitive and physical environments.

As a first-order approximation, you could say that things you learn about the physical context require very few repetitions. Once you’ve done laundry in a new apartment once, you’re pretty much done learning. The problem is that there are a lot of such things in a given physical environment. You might have to learn a hundred little things, each involving 1-3 repetitions.

Learning mental things usually involves more repetitions but fewer discrete elements. For example, when you go from high school to college, your study habits have to change because you are at a different level of the education game. It might take studying for several dozen quizzes or midterms before you settle into your new study habits.

Jokes aside, each kind of learning can go on indefinitely. People generally stop or declare a temporary detente once they start to derive a net positive “return on investment” for their learning efforts around one habit (or group of habits), and direct their limited learning energies to another front with higher returns.

As a result, our behavioral personality is always a set of habit-fronts at various stages of evolution. Some are stable, some are being learned (and bleeding red), some are being ported, some are atrophying. Some are in a state of diminishing returns.

Habits and Personality

Depending on whether they are primarily internally focused or externally focused, people tend to be better at either the mental or physical components of habit formation. It seems to be a zero-sum trade-off. I’ve never met anyone who was equally good at both. It’s like being right or left-handed.

I am pretty lousy at forming physical habits. I am much better at forming habits of thought. There are different sources of difficulty for the two kinds.

Habits of action are difficult to acquire because they require processing or memorizing a large amount of arbitrary information. To learn to get from point home to gym without too much thinking or a GPS, you need to learn various routes, turnings, traffic conditions, construction conditions, hacks, shortcuts, and so forth. Much of this habituation to the arbitrariness of a context is not portable. It is also not efficiently compressible, so it is energy intensive.

Responding to traffic signs is a kind of physical learning that ports within a country. But learning the quirks of a specific city (for example, here in Seattle, one rule of thumb is “avoid Mercer Avenue during rush hour”) is not very helpful when you move to a different city with different quirks.

Habits of thought are difficult to acquire for a different reason, though they might seem superficially easier to acquire. Your mind goes with you from context to context, so anything you learn about yourself, like whether you are more of a lark or owl goes with you everywhere. But learning about your mind is also fundamentally harder than learning something like “Maple Street is one-way east to west.” Since identity hangups, biases, demons and shadows all reside inside your head, every single useful true thing you learn about yourself comes at a 10x cost.

So you may not pay “porting costs” each time you move to a different physical or cognitive context (new city, new level of the mental game via a promotion or role change), but you pay more upfront, and the learning curve is much longer.

For those who like their finance metaphors, habits of action are op-ex heavy, habits of thought are cap-ex heavy.

Habits versus Addictions and Aversions

Healthy habits are the ones which are delivering a return and have an appropriate region of attraction. When the region of attraction for a habit expands where it is potentially harmful, you have an addiction. Addictions can form once a habit is generating a predictable “profit” after the learning curve has been traversed.

Once you get used to the “profit” and come to expect it, the motivational structure for the learned behavior can change. You no longer exercise because you recognize the benefits. You exercise because you are addicted to (say) the “Runner’s High.” You might turn into an extreme, obsessive runner and ruin your knees.

Habits can also turn into aversions. This happens in two ways. Either the habit becomes a displacement behavior for another activity (for example, cleaning the apartment to avoid working on your thesis), or the habit itself can develop a convoluted region of attraction to avoid certain painful regimes.

So you might develop a habit of using humor to steer conversations away from uncomfortable topics and develop a (justified) reputation as a great conversationalist.

Many behaviors are ambiguous, in that they are healthy habits, addictions or aversions depending on the context. These are the behaviors around which developing a strong sense of narrative rationality is very important.

Porting a Habit

Porting a habit to a new context is hard because the learned elements that don’t port well cause a shrinkage of the size of the region of attraction. Sometimes the shrinkage is so dramatic that an expert behavior becomes completely useless in a new context, or at least “unprofitable,” to use our finance metaphor.

One way to think about porting a habit is to think of it as recompiling a program on a new computer. Depending on how big the context difference, you may need to do anything from tweak a few settings to rewrite the entire program to compile on the new computer.

For the gym example I started with, the basic context-dependent algorithm is the same in both cases:

1. Change into gym clothes
2. Go to gym
3. Workout
4. Shower and change
5. Move on to next activity

But this is deceptively simple. There is a whole lot of context dependency that is not captured here. For my Vegas to Seattle example, here are some key ones:

1. In Vegas, the weather is always suitable for any mode of transport, in Seattle, the weather varies in non-trivial ways that affect whether you can walk or ride a bike or drive.
2. In Seattle, driving the 1.5 miles is much harder because it is urban driving with a lot of pedestrians, bicycles, traffic lights and one-way streets. In Vegas, I had an easy suburban route with just one left turn and no pedestrians. I have to be 3x as alert to drive to the gym in Seattle. So I am sufficiently alert less often.
3. In Vegas, parking was not a concern. It was always available and always free. In Seattle, depending on the time of day, day of week and season of the year, my gym has different rules about parking and how much I pay. I have to get the parking ticket validated at the front desk. So the algorithm has a whole branch of logic having to do with parking decisions that didn’t exist in Vegas.
4. In Vegas, transitioning to/from the next activity was easy. I could make it up as I went along. In Seattle, if I plan to work at a coffee shop after the workout, I have to plan differently depending on whether I drove, walked or took the bus.

This is just a small subset of the differences. If I wrote out the pseudocode for “go to gym” for Vegas and Seattle, I suspect, the latter program would be at least 5x as long. Much of the added complexity is because my living situation in Seattle has a good deal more physical complexity in the environment, since I live close to the city center rather than in a suburb.

As an aside, the closer you live to the center of a city, the more computationally demanding all habits become. This is one reason where you live is such a useful variable for personality typing.

Acquisition versus Porting

You can quantify the difficulty of acquiring or porting a habit loosely by assessing the physical and cognitive context complexity. It is easier to assess the latter (porting) because you only have to assess a pair of differences between before/after contexts. As mathematicians know, the “differential” or “variational” version of any kind of problem is generally easier than the absolute version. There is even a well-developed theory of how to do it well (it’s called perturbation analysis).

Another reason porting is easier than acquiring a new behavior from scratch is that a lot of learning is illegible and unconscious. You have to build up a robust algorithm in your head that you don’t fully understand. Say a habit algorithm has a conscious component C that you could write down, and an unconscious component U that you are not really aware of. So moving the habit from one context to the other is about moving C to C1 and U to U1.

In my experience, the unconscious component generally ports far more easily. But when it breaks, it is also much harder to debug.

It’s like the algorithm for every habit has two components: a piece for which you have the source code, and a piece for which you only have the compiled binary code. The former nearly always needs some modification when you change contexts, while the latter usually ports without any effort. But when the latter fails to port, you’re in serious trouble.

Complex ports can be understood as ports where C1 and U1 are both functions of both C and U rather than C1 being a function of C and U1 of U.  In other words, you may have to consciously process previously unconscious components of behavior, and vice-versa. This takes a higher level of self-awareness.

In a way, context-switching and simply expanding the region of attraction in one context don’t differ much. So ports are simply disconnected expansions of the region of attraction. You are adding a planet, so you can capture asteroids around two centers.

Meta-Habits

A meta-habit is a habit you use to port or acquire other habits. For example, dive off the deep end, is a specific learning strategy (immersion) that can jumpstart any new habit. If you know how to handle the extreme disorientation, chaos and anxiety it induces, and if you know when it is safe to do so.

In the prototypical example of swimming, diving off the deep end is actually a terrible way to learn unless there is a teacher around. You will almost certainly drown if you aren’t close enough to the edge to thrash your way to something secure within a few seconds.

I don’t know if there has been much research into meta-habits, but here are some I am aware of, some of which I even practice.

1. Diving off the deep end
2. Confidence building with small wins
3. Not perturbing behaviors along too many dimensions at once (however, the naive experimentation idea that you should tweak only one variable at a time is misguided and inefficient when there are many dimensions).
4. Gradualism (push yourself only a tiny bit extra with each attempt to expand the region of attraction)
5. “Exercise to failure” (keep pushing yourself in one direction until you cannot handle a particular case, a matter of finding the boundary of the region of attraction)
6. Never make a decision when depressed, especially a quit/persist decision
7. Fail fast — not in the product development sense, but in the sense of quickly putting a scratch or dent on your pristine learning effort (remember the lowering of anxiety you felt when you first put a scratch on a new car?)
8. Shut up when learning a physical habit (verbalization slows down acquisition of tacit knowledge — if you have a teacher who talks too much during teaching of a physical behavior like swinging a tennis racket, find a new teacher: you need periods of silent repetition between being given instructions and suggestions)

Curiously, I am much better at incorporating some of these meta-habits into my teaching than into my learning.

Variety Reduction as Anti-habit Formation

You can come at habit formation from the other direction. Instead of trying to learn a habit to handle a sufficiently large region of attraction to make it worthwhile, you can change the environment to simplify the habit acquisition or porting problem.

You do this primarily by reducing variety and introducing homogeneity in both cognitive and physical contexts. This depends on your resources. If you are rich enough to hire a chauffeur you can forget about the complexities of driving. If you’re powerful enough that a lot of people want to work with you, you can pick and choose people who adapt to you, so you can keep your thinking the same.

“Simplifying” your life, far from being a monkish thing to do, is actually a behavior practiced most often by the rich. Monks (and sour-grapes foxes) simplify in a different way: by giving up desires, either mindfully or via sour-grapes rationalization.

For most of us, variety reduction is a limited option. We make up rueful aphorisms about the human condition that reflect this, like if you don’t get what you like, you are forced to like what you get. Shaw’s famous line,  “The reasonable man adapts himself to the world; the unreasonable one persists in trying to adapt the world to himself” speaks to the relatively higher difficulty of changing the environment to reduce variety. It is an option that is dependent on your power.

But suppression of contextual variety to make habit formation cheaper comes at a price. It reduces your general ability to learn. This is perhaps the primary reason power leads to its own downfall. By making variety-reduction and homogenization an attractive alternative to the harder problems of habit formation and porting, power causes learning abilities to atrophy.

Organizational Habit Formation

Organizational habit formation is surprisingly similar to habit formation in individuals. We know it as economies of scale and scope.  The primary difference is that since organizations are typically far more powerful than individuals, the option of reducing variety and introducing homogeneity is more available to them.

Organizations become bad at learning and dealing with circumstances that cannot be homogenized away.

This is the reason industrial age organizations are associated with homogeneity and “seeing like a state.”

A major challenge in designing post-industrial organizations is to build into them the ability to accommodate variety and the discipline to not use the variety-reduction/homogenization option even when they have the power to do so. I’ve called this problem the “economies of variety” problem that I am still trying to figure out.

Thanks to Kartik Agaram and Jason Morton for useful discussions that informed this post.