My feed continues to tempt me with irresistible rabbit trails. This is a transcript of Yascha Mounk’s weekly podcast The Good Fight, this time a conversation with C. Thi Nguyen, philosophy professor at the University of Utah, who just wrote a book about metrics and their use. Note to paid newsletter producers: Mounk very generously puts his paywall 41 minutes into the read! And even though his interviews are always high quality, this one totally sucked me in — I bought Nguyen’s book right away once I finished. The conversation is dense and rich, so be forewarned that I’ll be generous with my excerpts.
Mounk starts with a fundamental dilemma of metrics: without them you often don’t know whether your work is actually achieving anything, but with them you risk shifting your attention from the purpose of the process to maximizing its efficiency.
We need some metrics in order to be able to have productive processes, to be relatively efficient, to make sure that we actually are achieving something. When you’re trying to bake bread for your neighborhood, whether at the end of this process you have two loaves of bread and most people starve or a hundred loaves of bread and most people can have breakfast makes an obvious difference.
On the other hand, there are all of these different areas in which the metrics we use end up either misdirecting our productive processes or robbing us of the joy that activity should actually entail. In the most straightforward form, in some forms of central planning, you might end up with 100 loaves of bread, but they’re actually inedible. That’s not well captured in the metrics. As long as the 100 loaves of bread come out of the oven, it doesn’t matter that nobody will actually be able to eat them.
Nguyen replies that confronted with the dilemma, people mostly fall into one of two camps, both mistaken.
There are two fantasies that people have when they approach metrics. One fantasy is that the metric captures everything that’s important and we just need to optimize for it. The other fantasy is one that I had lived in in the past, which is that these things are just wholly terrible. They’re evil. They miss everything that’s important. We should just get rid of them and enter some kind of non-metricized utopia.
He goes on to identify the true usefulness of metrics, an idea taken from Theodore Porter: by draining information of nuance and context, they make it portable:
Again, I think the obvious example is letter grades. Everyone agrees what an A means, a B means, a C means, approximately. So we can all collect information into the same bucket and it aggregates instantly. Porter’s insight here is that the very thing that makes metrics socially powerful is the design process that removes context and nuance—the thing that makes it travel. He calls this the portability theory.
Porter also emphasizes that it is exactly this which creates an inescapable tension, a tension people mostly resolve by joining one of the two camps. But instead we should embrace the tension, use a metric for its designed purpose while being vigilant about the collateral damage its use might cause.
He has this beautiful moment, I think you in particular would like this, where he says something like, “information is a specific thing, a way, a kind of human understanding that’s been prepared to travel to distant strangers and be understood in different contexts”. I find that so profound, and I think that stabs right at the heart of this inescapable tension. The reason metrics have social power is that they’re de-nuanced and that’s not something that we can hope to get around.
Instead, we should expect that there’s this constant trade-off and the things that are most legible at scale will be the least nuanced and we’re gonna have to use those to interact with each other. But also, we have to be constantly aware that they’ve achieved that kind of social centrality precisely because they’re de-nuancing. Then it becomes hard because that’s exactly the terms that everyone can understand instantly. Again, as Porter points out, that was the design goal. We achieved the design goal.
The conversation continues with a long examination, well worth reading — the whole conversation is, so I’ll stop saying that — of how this tension plays out with academic letters of recommendation. Other examples:
Nguyen: That’s a superb question. So it seems really doubtful that any metric is good or bad across the board. It’s highly contextual. Here’s one of my favorite examples. I think a lot of us know that BMI, Body Mass Index, is a terrible way to manage your health if people just try to make their BMI go down.But that’s not what BMI was built for. BMI was built as a large-scale public health measure that was like a kind of litmus test. If across an entire nation, BMI suddenly jumps five points suddenly, we know that something has happened.
Mounk: Or if it goes down a lot, which is not the problem today in the United States. If BMI suddenly craters, well, perhaps we’ve just discovered GLP-1, but most likely it’s because there’s a famine going on. So it’s really useful at that level.
Nguyen: A standard example for me is what we actually care about in a lot of cases is intellectual and emotional maturity, but that doesn’t have legal objectivity. Eighteen years of age does have legal objectivity, so even if that’s imprecise, we use that.
And this one from Mounk regarding one of my favorite topics, how we die:
I recently had Atul Gawande on the podcast, and he was talking about a very similar topic, which is that we are great at extending the lifespan by a few months at the end of people’s lives. But, first, most healthcare spending is on those few months of life. Second, the quality of life that people have in that time tends to be terrible. They are constantly in the hospital, and they are in significant pain. You are drawing out the process of dying, often much longer than patients would want it to be.That is downstream from the fact that the metric of “is this patient alive?” or “is this patient dead?” is pretty easy to measure, despite some hard cases. The metric of “was it a good death or a bad death?” is much harder. On the whole, did life go better for living those extra eight weeks in pain, half-conscious in the hospital, or not? That is incredibly difficult to assess.
There’s more of this, but then the conversation suddenly shifts in a way I did not see coming:
Part of the reason I wrote this book was that I ran into a puzzle that I did not know if anyone else was interested in, but I could not stop thinking about. I had written a bunch of stuff about games, and I had written that stuff because I had gotten frustrated with attempts to praise games as a kind of cinema, where all people talked about were the cutscenes, the dialogue, and the graphics. They did not talk about freedom or the quality of action. I really wanted to talk about how it felt, how rich the decisions were, and how interesting the decisions were.
One of the most interesting things I found was this moment from the great German board game designer Reiner Knizia, where he says, “The most important part of my game designer toolbox is the scoring system, because it sets the player’s desires.” I was a game player, and I thought that was exactly right. I was also a philosopher, and I thought, my God, that is so true and so deep, and so weird to put that so starkly.
What a game designer is doing is describing an alternate self with alternate desires and alternate abilities. [Emphasis added]
Wow! Still thinking about that one.
Then we’re on to rock climbing.
Rock climbing tuned me into something else, and it did so because of the scoring system. It told me to climb harder climbs inside a ranking system. Every rock climb has a community-established difficulty rating, and the internal scoring system is to climb harder routes. Interestingly, to climb harder routes I had to learn how to control and perceive my body. Over the course of that, I learned that movement was beautiful, and that subtle movement was incredibly beautiful, often in a way very similar to the beauty I found in philosophy, this kind of fine-grained subtlety, but also really different. I got that because of the rock-climbing scoring system.
After about five years, that scoring system stopped being useful to me, partially because I am just a mediocre climber with poor athletic ability. When I plateaued, the scoring system was good as long as I could keep improving. But at another stage of my life, as a parent and academic who is not that athletic, it started to destroy the joy of climbing to stay within the same scoring system. So I created an alternate goal set for myself. I started aiming to climb moderate routes as elegantly as I could, and I found again what I loved.
My gosh, have I been there. And am I wary of that danger in any new pursuit I’m considering! I’m learning the basics of chess right now and having an immersive, enjoyable experience — but right and left I see metrics that can track my “progress”, each one tempting me to focus on it and forget why I started this project in the first place.
(Basically I’d like to understand better how to learn something deeply and thoroughly, and I’ve been looking for a domain where practicing it would be a long and enjoyable journey for me. It’s not to become a grandmaster, or even to achieve a particularly high ranking. It’s not to vanquish opponents. It’s not to impress anyone. It’s not to spend time with other people. All those things — rankings, playing to win, impressing myself with my progress, playing humans rather than bots — may be critical elements of the experience. I just need to be vigilant that when they are brought into the mix none of them substitutes itself for the original purpose.)
A bit further on I was stopped dead by this:
A lot of this started because, in two different places, I saw the same distinction. When I was in graduate school, my advisor, Barbara Herman, a Kantian ethicist, once said in a graduate seminar, after I said something, “You’re failing to distinguish between a goal and a purpose.” I replied that those were just the same word. She said, “No. When you have your friends over for a night of cards, the goal is to win, and the purpose is to have fun.” You know, deeply, that unless you are a very strange person, if you lose but had a great time in that context, it was a good evening. It was not a wasted evening.
That distinction is extremely important, and games illuminate something fascinating about us. Our purpose is often to have fun, or to relax, but the only way to get there is by hyper-focusing on a goal. I climb to clear my head, and I cannot clear my head by trying to clear my head. I clear my head by climbing. The same is true of fishing. One reason I fish is that staring intensely at the surface of the water, searching for rising trout, creates a meditative stance that I cannot reach directly.
We need very clear goals to get us into mental states we cannot otherwise approach. What games are, in a sense, are pre-packaged mental states. They say, “Do this,” and suddenly you are hyper-focused on balance. “Do this,” and suddenly you are focused on complex logical interplay, or geometric patterns in chess. They are pre-packaged.
The distinction between a goal and a purpose. Often the only way to achieve a purpose is to pursue a goal. Pursuing a goal can get us into mental states we can’t otherwise achieve. Yikes!
OK, last excerpt, and I promise I have still only scratched the surface of this conversation.
Mounk: One of the things that really struck me is a student who, I think, watched one of your videos. I do not think it was a student of yours. She realized, in a very typical story, that she was a high-achieving student who worked really hard and was successful at gaming all of the metrics that life threw at her, and that she was unhappy because that was not what she actually wanted to do.
I think she wrote on the background of her phone something like, “Is this the game you want to be playing?” It is a kind of self-admonition, a reminder to always ask that question. It is not a simplistic version of this idea, which is very popular, that says, “Stop playing the game. Reject all the metrics. Just go hang out with your friends on a beach.” That is not the question she is asking.
What she is saying is more interesting. Life is going to consist of playing games, not just fun games on a game night with friends, but games like wanting to achieve at some endeavor. I want to become a professional philosopher, and that will involve publishing in certain places and jumping through a certain number of hoops. Even if it is inevitable that you will have to play certain kinds of games in life, you need to make sure that you do not get so caught up in a particular game and its rules that you end up doing an activity you do not actually want to be doing. So the right move is not to stop playing games. It is to keep reflecting on whether you are still playing the game that you actually want to be playing.
Remember that you’re playing a game. Remember why you’re playing the game. Continually reflect on whether you are still playing the game you actually want to be playing.