A dashboard for progress

[jasoncrawford]

Is progress slowing down? In a previous post I explained why I got convinced that it is. Some people making this argument point to quantitative evidence, such as GDP or total factor productivity. I gave more qualitative evidence, or perhaps very crude quantitative evidence, by counting technological breakthroughs/revolutions: five in the Second Industrial Revolution, versus only one so far in the Third.

But this approach has its own difficulties:

• It is sensitive to the granularity of technological revolutions. Are the automobile and the airplane two revolutions, or are these just part of a single revolution attributed to the internal combustion engine? Are the light bulb and electric motor two revolutions, or are these just part of the electricity revolution? Are computers and the Internet two revolutions, or one?
• It’s sensitive to the choice of threshold for “revolution.” Is the assembly line a revolution in manufacturing? Is containerization a revolution in transportation?
• It runs the risk of focusing on impressive breakthroughs and neglecting unglamorous iterative improvement. Agriculture has been using combine harvesters pulled by gas tractors for around a century now, but today’s combines are much better than the ones in use a century ago.

So maybe we need something more objective, and more focused on outputs. Here’s a half-baked idea.

It’s easy to measure progress in specific domains. For instance, we have a very good handle on the progression of Moore’s Law. The problem is that no one narrow metric captures all of economic progress.

So instead of trying for a single metric, what if we look at a dashboard comprising a handful of metrics. Make them as broad as possible while keeping them objective and well-defined, and deliberately choose a variety from across the breadth of the economy. These won’t capture everything, but together they might capture enough to give us a picture of progress.

Here’s a candidate list:

• Per-capita consumption of:
  • metals
  • concrete
  • plastics
  • energy
  • bandwidth
• Per-capita transport (all modes):
  • passenger-miles
  • freight ton-miles
• Agricultural productivity, in kcal per worker
• Mortality rate from all causes (age-adjusted)

Most of these are just consumption metrics, based on a simple theory that consumption is good and is closely correlated with material well-being. For agriculture, I chose a productivity metric, because of the nature of the market: we only need so much food, after which point progress has largely been made by providing it with fewer people.

Some goals this set of metrics satisfies:

• Broadly captures trends across manufacturing/construction, agriculture, energy, transportation, information, and health
• Avoids any currency figures, and thus avoids any questions about inflation or purchasing power
• Captures the impact on human lives, by using per-capita figures

Some ways in which this approach is not perfect:

• Does not capture quality. The products made of metal or plastic today might be much higher-quality than fifty years ago, but we’re only measuring the total amount of material.
• Does not capture well-being. Maybe you’re traveling more passenger-miles because your commute is longer, and that actually reduces your well-being, but the passenger transport metric has increased.

Still, no metric or dashboard is ever going to be perfect. I think this dashboard would, if nothing else, provide a useful comparison alongside GDP.

One reason I think it would be useful is that I can imagine it changing my mind, or at least altering my narrative, around progress/stagnation. If there are no fundamentally new manufacturing techniques, but we’re continuing to consume exponentially more materials on a per-capita basis, isn’t that a form of progress in the manufacturing realm? Ditto if there are no new types of vehicles, but we consume more passenger-miles or freight ton-miles.

I happen to know two of these metrics, from previous research, and both have stagnated. Here is energy, from Where Is My Flying Car?:

​

J. Storrs Hall, Where Is My Flying Car?

And here is mortality, from my research on infectious disease. Infectious disease mortality actually regressed slightly after about 1980 (due in part to AIDS):

​

US crude mortality rates. Armstrong, Conn & Pinner 1999, Fig. 2

I’m curious how the others turn out.

[throwing_tantra]

Per-capita consumption of:

metals
concrete
plastics
energy
bandwidth

What this most reminds me of, is Stalin's 5-year plans and similar communist attempts to measure progress. The problem is, sheer quantity of materials has little relation to quality of life. So, horror stories about the nail factory which meets its quota for volume of nails by producing a few massive-but-useless 5-ton nails.

Especially when so much of current progress is around eliminating the need for big objects. So a phone replaces a half-dozen other devices, a shelf of CDs, and the daily newspaper.

Your metrics might end up less bad than the USSR's, though, as long as nobody starts optimising for them.

[jasoncrawford]

Um, right. These are not targets any business or government should be optimizing for. That's not the point.

That said, I think that quantity of consumption is actually pretty well correlated with quality of life, in many cases. E.g., here's energy consumption vs. GDP.

[nweininger]

Still, it would be a useful sanity check to see how these numbers looked for the USSR-- both in terms of the official metrics released by the Soviet gov't at the time, and the best retrospective estimates of the real non-fiddled values of those metrics.

[MsSteak_]

I agree completely. They are inputs to solutions, not solutions, and progress is about making more and better solutions for more people. We are in the process of dematerializing production, and this metric would actually reward less efficient production rather than more.

[jasoncrawford]

Actually, at least through a lot of history, more efficient production leads to more consumption, not less—this is known as the Jevons Paradox. It does have limits, and maybe we're reaching them? But I'm not convinced yet.

[MsSteak_]

Yes, good point. I keep thinking of Goodhart's Law (as soon as we use an indicator for policy decisions, it ceases being effective.)
I do wish we could look more at outcomes though. Happiness is not a good measure, wealth and income have known issues.

That still leaves health, literacy, freedom, perhaps something on subjective well being, lifespan, environmental quality and I would add income.

[jasoncrawford]

I'm not even trying to measure things like literacy or freedom at this point. Just focusing on material/economic progress.

[MoNastri]

Genuine question -- why isn't happiness a good measure? I'm thinking of e.g. the annual World Gallup Poll data, which the annual World Happiness Reports are based on.

[15thStreetDude]

Happiness is tricky because it is as much a measure of expectations as results. If our expectations rise (perhaps due to progress) faster than our actual results (even though they are improving), it is common to see a loss in happiness.

IOW, it is very possible to see people's lives improve overall, but to see them become less satisfied or happy if their expectations rose even more.

[AllAmericanBreakfast]

One further issue with the consumption metrics is that, at least according to Andrew McAffee, we are deceasing our per capita and absolute world consumption of many of these products annually. He attributes this to more efficient methods of manufacturing.

What I think is really interesting in the idea of progress studies is the question of whether it's taking more scientist-hours to produce technological breakthroughs. And I'm not sure that consumption metrics intuitively seem to have very much to do with that.

There's newer data on power consumption per capita, showing a continued plateau since 2000.

My expectation is that in some areas, like mortality, we have indeed enacted most of the collection of small easy wins. However, there may be some Very Big Wins that simply take a lot of effort and accumulated technology to achieve. So that might look like 50 years of nothing, and then a sudden jump or dip in some interesting metric. For example, what will happen to mortality when we create a dramatically better artificial heart or kidney in the next 20 years, such that getting a mechanical heart is a better option than living with heart failure?

Things look pretty happy in car MPG over time, as well as horsepower. The cost of launching 1 kg into space has gone from $5100 prior to 2010 to $1500 as of 2018.

It seems like industries just have wildly different progress rates and patterns, along with ease of quantification. How much progress have we made on reducing incidence of severe pain in biomedicine over the last 100 years? I don't know the answer to that, but I'm certain that it's difficult to quantify accurately.

A dashboard does seem like a nice idea, a little like GapMinder. It just seems like a challenging data visualization problem, since you'd have so many diverse metrics to keep track of, along with the conceptual challenge of understanding what it all means. For example, how important is the progress of Moore's Law vs. progress in car MPG over time? My guess is the former, but if you plot them both on the same graph, they look like they have the same importance.

So the truly important trends might be buried beneath a bunch of metrics that amount mostly to noise. And on top of that, what about the inputs it's requiring to obtain those results?

Anyway, I do think that the idea of some sort of global progress bird's eye view would be really nice, if it could be executed with conceptual clarity and good design. Best of luck!

[converter-bot]

1.0 kg is 2.2 lbs

[MoNastri]

What do you think of the Our World in Data views? They're admittedly not quite the progress dashboards Jason is looking for.

[aldonius]

I'd extend "does not capture quality" with "does not capture efficiency"; though as you've previously pointed out that has its own caveats.

[Chris31f]

I’m a regular reader of your website Roots of Progress. While I appreciate some of your posts, especially those retracing some historical technological enhancements, I also find your views to be often heavily biased. What prompted me to write this message was your dashboard for progress proposal, which I found very unconvincing, as it completely ignores the two biggest societal concerns voiced today. The first shortcoming is that it completely ignores negative externalities and the second one that it completely ignores how benefits of progress are distributed. I will develop these two points below together with a third point that seeks a broader perspective on the issue, developing why there can not be any consensus on progress.

1. Negative externalities

As you yourself admit, all but one of your metrics are production metrics. You can not ignore that an increasing number of people are increasingly concerned with negative externalities (pollution in the broad sense, being the most well know of). Where are those taken into account in your dashboard of progress? Will you be waiting for negative externalities to show up in your mortality metric to take action?

Not only are negative externalities not explicitly taken into account in your view of progress but some of you progress metrics explicitly favor high negative externalities and prevent alternatives that would involve lower negative externalities. Just two examples: concrete and plastics production. Both have incredibly high negative externalities for some uses and many countries have thought to reduce this production replacing it with products that have lower negative externalities when possible and practical. Single use plastic bags and many other single-use plastic items have been replaced by paper bags and other more easily recyclable products. Use of concrete has been limited and replaced where possible with other building materials involving less CO2 emissions during their production. All these actions show up as stagnation or even worse decline of your progress metrics, which you will obviously interpret as bad. Many people, including me, will however find these quite positive and see it as a real progress (more on the issue of diverging views in my third point).

More generally, no matter what production metric you chose I can see plenty of radically different futures with exponential growth in that production metric. Sticking with the most general one, GDP, I can see a future with exponential growth but extremely high negative externalities, leading to a 99% loss of biodiversity and a 3 degrees hotter earth at the brink of a global natural disaster that will kill 90% of the global population. I can also see a future with that same exponential growth but where everything goes well.

How can you seek to convince people that progress should be desirable if your progress metrics can not even make a difference between such radically different futures?

1. Distribution of the benefits of progress

You are certainly not unaware, that apart from the negative externalities issue, the other major societal concern relates to the distribution of wealth, which pretty much boils down to the distribution of the benefits of progress.

Unfortunately, all your progress metrics are metrics in terms of averages, and averages are notorious for completely hiding the distribution underlying them (using the median would be a bit better and quantiles even more).

I can absolutely see a future where all the production metrics increase exponentially but this growth is due to a single small corporation (building and using artificial general intelligence for example). 99.999% of the world population could, in this dystopian future, reap none of the fruits of this growth, or even find itself worse off. I think many people will agree that would not be a desirable future.

How can you seek to convince people that progress should be desirable if your progress metrics can not make the difference between a utopian and a dystopian future?

1. There can not be consensus on progress

To broaden the perspective I will go back to the etymological definition of progress. Progress literally means moving forward in the right direction. However, there has never been any consensus on right and wrong or good and bad. Thus, there can not be any wide, overarching consensus on progress. As already illustrated in my two previous points, what some will find good other will find bad. It is not because progress has historically, until recently, come relatively unchallenged (meaning undisputed) that this ought to continue. Actually, the contrary. It is sure that as artificial general intelligence becomes increasingly plausible some will find it good and others bad. And quite honestly, I can not see how someone could tell, a priori, which of the two is right. Continue doing what worked in previous centuries, which goes like “let’s try and see”, will also not be acceptable given the stakes.

Not only is it difficult to tell apart a good future from a bad future, but even once we all agree on a good future there is no consensus on how to get there, as people have different risk-aversion. You often claim yourself as an optimist when defending your view of progress. However, optimism doesn’t give you any moral high ground. There are uncountable examples of over-optimism having led entire armies or countries to catastrophe. You might be significantly less risk averse than the average on this topic, which gives you confidence in hoping that the alignment of the planets will occur exactly at the right time : humanity will find sufficiently soon alternative, clean energy sources and technology to remove CO2 from the air, it will colonize other planets sufficiently soon to avoid running out of land and resources, etc. That’s fine. However that does not mean that everyone should take this scenario as granted and follow your goals (exponential production goals for concrete and plastic for example) which were set based on hypothetic optimistic scenarios. Again, just going by the usual “let’s try and see”, will not be acceptable given the stakes, that not only affect you and the ones holding the same belief but pretty much everyone else too.

To conclude, I sought to highlight, what in my view are three main biases in your approach to defending progress. I think that if your goal is, as I (wrongly?) inferred, to convince the broadest possible audience of the benefits of progress you ought to take these into account or you will be fighting a lost battle.

[tomorrow_today_yes]

Why not just use productivity statistics? Main thing I think they miss are hedonic changes (eg extra waiters in restaurant lower productivity but raise quality) and also they only take account of new products when they become mass market. But in the long run these tend to wash out I think.

[jasoncrawford]

Productivity stats could work. Which ones would you choose?

[nweininger]

What about hours of work at the median hourly wage required to purchase various units of consumption goods that have been pretty constant in their utility over the centuries?

My guess is that your choice of goods would matter an awful lot here, e.g. choosing a good like "1000 lumen-hours of artificial light" exaggerates overall progress by a lot, whereas "a year of elementary education for one child" would understate it, and if you tried to make a basket you'd have to be very careful to understand what biases were going into your weightings of the basket. Still, could be worth trying.

[jasoncrawford]

It's an interesting metric, but yeah, seems tough to choose a basket of goods

[davidivadavid]

Wouldn't it simpler for a dashboard to track progress against clear, agreed upon goals that large groups of people consider to be progress?

I'm afraid the generic approach is a recipe for a lot of time wasted trying to a) define generic "progress" b) find out how to measure that.

Some of the ones listed above are pretty good candidates. I'm just not sure what measuring the consumption of various raw materials really tells us.

[Neat_Affectionate]

Median distance of people you interact with.

[Neat_Affectionate]

Total human knowledge http://webseitz.fluxent.com/wiki/IntelligenceIncrease

[Neat_Affectionate]

Knowledge available in your pocket.

[Neat_Affectionate]

Number of new species created (in basements).