The Heritage Foundation Economic Freedom Index is bad

If you enjoy this post, check out my (free) book https://deponysum.com/2020/03/30/something-to-read-in-quarantine-essays-2018-to-2020/

TLDR: The problem with freedom indexes is this. They are aggregates of separate variables, some of which indicate small government, others of which indicate what would uncontroversially be regarded as ‘good government’. It is the good government indicators, not the small government indicators, which actually correlate with GDP per capita- the measure that the Heritage foundation uses to tout the success of their metric.

You may have seen various indexes of economic freedom floating around. These typically claim to show that “economic freedom” is good for the economy. The practical use of these indexes is to argue in favour of limited government on the basis that economic freedom is correlated with prosperity. However these indexes are conceptually flawed and basically try to win by shifting the goal posts, bundling together things everyone agrees are good, with the actually controversial small government criteria in a way that produces misleading results.

To see why, let us suppose that I create the crystal-health (TM) index of crystal-healthy living. It’s scored on the basis of:

1. Number of hours you spend exercising a day

2. Caloric intake

3. Work-life balance

4. Number of crystal-health (TM) products purchased

5. Amount of time you spend each day meditating in a circle of crystal-health (TM) products.

If you have a high score on this test you are deemed to have a crystal-healthy lifestyle. Now I can guarantee that having a crystal-healthy lifestyle will be positively correlated with actual health outcomes. I can also guarantee that this tells us absolutely nothing about whether or not there’s any real merit to magic crystals. The problem is that the scoring overlaps with a number of things that, definitely, are good for you- exercise, a balanced diet and relaxation for example. These are bundled in with things that have no effect (purchasing magic crystals). The overall bundle will correlate with healthiness, but that’s not really evidence in favour of the specifically magic crystal parts of the bundle.

This is the methodological problem with measures like the Heritage Foundation’s Index of economic freedom. Incontrovertibly good things like an independent judiciary- things that basically just indicate whether a state is running well or not- are thrown in with more controversial things like low government spending. Whatever your political viewpoint it’s totally unsurprising this bundle would correlate with economic growth, because so many parts of it are obviously good things supported by everyone such as:

-Government integrity

-Judicial Effectiveness

-Fiscal health

But this doesn’t prove that the distinctively small governent parts of the bundle are good for economic growth. Parts like:

-Tax burden

-Government spending

-Labor ‘freedom’

Maybe these latter parts of the index do have a real relationship with economic health, or maybe they are like the magic crystal bits of the crystal-health index- superfluous additions included for ideological reasons.

These economic freedom indexes are often used to argue for small government policies, on the grounds that the highest ranked countries usually have high GDP. To expose the inconsistencies between the variables that are included in the bundle of “Economic freedom” I thought I’d run a few correlations using the Heritage foundation’s own data.  Now it’s important to be clear that these correlations do not prove anything of significance about what makes for good or bad economic management- for that we need very strong controls, Granger causation etc. However they will serve to demonstrate the underlying problem with getting a grab bag of things everyone likes, a grab bag of things you like because you support small government, mixing them together and calling it an economic freedom index.

I have only included sub-scales which are measured in a way I can reproduce, either by looking up publicly available data, or by accessing third party independent reports. This is because, to be blunt, I do not trust scales produced by the subjective judgements of Heritage foundation employees anymore than I would trust Dracula with the blood bank. It’s not that I think that they’re dishonest, it’s just that I don’t think they’re disinterested. Scales that involve subjective or qualitative assessments are fine, so long as those assessments have been done by third parties.

Probably the most serious criticism of my post here is that I’ve chosen to correlate the various sub-scales with log GDP per capita rather than using some other measure. In some ways this is not ideal, we could have chosen, for example, growth as our point of comparison. But unadjusted growth has its own issues- e.g. growth tends to slow down in richer countries and many other disparities. I’ve seen how messy these debates can get, for example there’s a whole argument in the effects of redistribution literature on this. I really didn’t want to get bogged down trying to construct an ideal point of comparison- say growth adjusted for GDP with numerous other tweaks included. This isn’t how the economic freedom index is sold, or validated, and it’s not like people are relying on the index in the main because it has undergone these sophisticated tests.

So I decided log GDP per capita (henceforth just “GDP”) is the closest thing to a single fair metric. What ultimately clinched it was that the Heritage institute itself often boasts about the correlation of the freedom index with GDP, so it seemed like a natural and fair comparison because they use it all the time, and this is a response within the framework of their own research.

SUBSCALES MEASURING THINGS EVERYONE LIKES

Judicial effectiveness: Everyone likes judicial effectiveness, from democratic socialists to small government advocates. The scale is based on survey data and third party opinion. Unsurprisingly it correlates 0.6 with GDP per capita, a very respectable relationship.

Government integrity: Government integrity is also universally liked. Government integrity scores in this report are taken from a variety of survey data. No one will be surprised that it correlates 0.65 with per capita GDP.

Fiscal Health: While it is true that small government advocates are more fixated on balanced budgets than the left, no one actually likes running large deficits. Basically this scale tells us that countries that have recently run deficits are likely to have somewhat smaller GDPs (A small correlation at 0.22). This may be partly because many countries use deficit spending counter-cyclically during periods of economic downturn.

Business freedom: Wait a moment! Surely not everyone likes business freedom? Surely this is a small vs big government issue issue? In general yes, but given the way the authors choose to measure it, almost every single criteria is unequivocally a good thing. For example, there is no one on the planet who wants it to take more days to setup a business, get a licence or obtain electricity. Maybe one criteria out of 13 marks a genuine ideological point of contention (number of procedures required to get a licence) and this is very debatable. Given that this is basically a measure not of the scope of bureaucratic procedures, but their efficiency, and a very specific suite of procedures at that, it is totally unsurprising that this criteria correlates 0.66 with GDP- a strong relationship.

Now we turn attention to the bits of the index that divide small government advocates from everyone else.

SUB-SCALES MEASURING THINGS SMALL GOVERNMENT ADVOCATES LIKE AND OTHER PEOPLE DON’T LIKE

Tax Burden: Some people like high taxes for the services they allow, some people don’t.  There is no practically or statistically significant correlation between low taxes as indicated by this scale and GDP according to the Heritage Index’s own data. In fact it’s non-significantly negative (-0.08).

Government spending: Obviously a good measure of small v large government.  The correlation between GDP and this score is somewhat negative (-0.34), suggesting that more government spending as measured by this scale is associated with a bigger economy per person. In other words, this sub-scale actually somewhat anti-predicts economic success, not a great result for the Heritage foundation.

Labor Freedom: The right likes it, the left hates it and would dispute the “freedom” bit (do you enjoy your freedom to be fired?) I could probably quibble with how the heritage foundation define this measure on a few points but I’ve chosen not to because I don’t want to be accused of special pleading. Labor freedom as measured by the Heritage foundation has a small positive correlation with GDP (0.28), good news for the Heritage foundation I suppose, but it’s only about 8% of the variance.

SUB-SCALES EXCLUDED AS THEY DEPEND ON THE SUBJECTIVE OPINIONS OF HERITAGE FOUNDATION EMPLOYEES

Property rights: Entirely graded on by Heritage foundation employees. Worse, the criteria arguably bunch together a bunch of potentially independent things.

Investment freedom: same as property rights.

Monetary Freedom: Honestly I wouldn’t have been terribly against including this one, as it is mostly scored on objective grounds, but I’m sticking to my own criteria. The subjective part is the way they measure price controls.

Trade Freedom: Trade freedom is a bit of an ideologically weird issue that transcends distinctions between people who normally vote for small or large government. I wouldn’t be especially perturbed if it had a large positive, negative or no correlation with GDP. It’s subjectively graded in part by Heritage foundation employees through the assessment of non-tariff barriers so I haven’t included it.

Cronbach’s alpha

I thought I’d check the quantitative consistency of the scale using Cronbach’s alpha, a measure of the internal consistency of a scale. Cronbach’s Alpha for the scale with subjectively scored items removed is .68, which according to trusty Wikipedia is “Questionable”.

Conclusions

The Heritage foundation economic freedom index is a bundle of often unrelated criteria. Excluding criteria graded on the basis of Heritage foundation employee opinions, those criteria on which pro and anti big government thinkers are likely to agree are good things are correlated strongly with GDP. However those criteria which small government proponents are likely to think are important, and big government supporters disagree, do not correlate positively with GDP. The only exception is labor freedom, but the correlation is small and there are other problems with the labor scale.

As a result of the unrelatededness of the criteria, the scale is inchoate. Many of the sub-scales either do not correlate with each other, or even correlate negatively with each other, suggesting that there is no one, real coherent construct being measured by “economic freedom”. This is shown in the scale’s Cronbach alpha score of .68.

You could always just say “By economic freedom I mean that which the economic freedom index measures.” But if so, “Economic freedom” has very little internal unity, and its parts predict different, sometimes contrary things. The whole thing seems very misleading, whether intentionally so or not.

Just for fun: DePony Sum’s Index of Small Government

Just as an exercise I took the objectively measurable variables which are measures of small government and added them together (this is how the freedom index does it- no fancy maths here). Those variables are labor freedom, government expenditure and taxation rate. I then correlated the resulting sum with GDP.

The result is actually slightly negative, -0.15. It seems this index won’t be demonstrating a link between small government and prosperity any time soon.

Also just for fun: DePony Sum’s Index of Good Government

Inversely, what would happen if we just used the measures which everyone- small government advocates or not- regard as measures of good government. To recap these are judicial effectiveness, government integrity, fiscal health and “business freedom” (because of the peculiar way “business freedom” is measured and defined in this data.) This index fares much better as a predictor of economic prosperity, correlating 0.61 with GDP per capita.

The use of this scale in serious economics

The 2001 Index of Economic freedom alone was cited 1425 times, and an edition is produced every year. The combined number of scholar citations for all EFI editions is probably in the tens of thousands. The EFI has been cited approvingly in distinguished journals such as Economic Policy, Journal of Development economics and The Journal of Law and Economics. It has been cited in such highly regarded non-economics journals as The Journal of Accounting research. The majority of these citations treat the index positively, at least implicitly, and some even treat it as authoritative, including it as a variable in regressions and other statistical analyses.

To my mind, the fact that this scale has been cited as authoritative in major journals is a scandal of the name of economics. This stuff doesn’t belong in an undergraduate journal. It’s a mess.

We’ve honestly used a pretty light touch here. I’ve pretty much accepted that the scales are internally well constructed representations of what they purport to measure, despite many many sub-scales being themselves constituted by an odd mishmash of variables. I could have been a lot meaner.

No one likes to think of themselves as engaging in ad hom argument in academia, but at the end of the day, it really should have been obvious that scepticism is needed when the index was called the “Economic Freedom Index” and created by an organisation called “The Heritage Foundation”. You shouldn’t take my word for this of course- I’m just as bad as they are but from the opposite end. However, it doesn’t have to be a matter of trust. If you are a social scientist of any sort and take your profession seriously, please read through the methodology of the Freedom Index and at least look at the first order correlations in the data before you cite it, let alone use it as a variable in your regressions. This is simple minimal due diligence. Like a good doctor facing a pharmaceutical representative, use common sense.