The Heritage Foundation Economic Freedom Index is bad

Related image

TLDR: The problem with freedom indexes is this. They are aggregates of separate variables, some of which indicate small government, others of which indicate what would uncontroversially be regarded as ‘good government’. It is the good government indicators, not the small government indicators, which actually correlate with GDP per capita- the measure that the Heritage foundation uses to tout the success of their metric.

You may have seen various indexes of economic freedom floating around. These typically claim to show that “economic freedom” is good for the economy. The practical use of these indexes is to argue in favour of limited government on the basis that economic freedom is correlated with prosperity. However these indexes are conceptually flawed and basically try to win by shifting the goal posts, bundling together things everyone agrees are good, with the actually controversial small government criteria in a way that produces misleading results.

To see why, let us suppose that I create the crystal-health (TM) index of crystal-healthy living. It’s scored on the basis of:

1. Number of hours you spend exercising a day

2. Caloric intake

3. Work-life balance

4. Number of crystal-health (TM) products purchased

5. Amount of time you spend each day meditating in a circle of crystal-health (TM) products.

If you have a high score on this test you are deemed to have a crystal-healthy lifestyle. Now I can guarantee that having a crystal-healthy lifestyle will be positively correlated with actual health outcomes. I can also guarantee that this tells us absolutely nothing about whether or not there’s any real merit to magic crystals. The problem is that the scoring overlaps with a number of things that, definitely, are good for you- exercise, a balanced diet and relaxation for example. These are bundled in with things that have no effect (purchasing magic crystals). The overall bundle will correlate with healthiness, but that’s not really evidence in favour of the specifically magic crystal parts of the bundle.

This is the methodological problem with measures like the Heritage Foundation’s Index of economic freedom. Incontrovertibly good things like an independent judiciary- things that basically just indicate whether a state is running well or not- are thrown in with more controversial things like low government spending. Whatever your political viewpoint it’s totally unsurprising this bundle would correlate with economic growth, because so many parts of it are obviously good things supported by everyone such as:

-Government integrity

-Judicial Effectiveness

-Fiscal health

But this doesn’t prove that the distinctively small governent parts of the bundle are good for economic growth. Parts like:

-Tax burden

-Government spending

-Labor ‘freedom’

Maybe these latter parts of the index do have a real relationship with economic health, or maybe they are like the magic crystal bits of the crystal-health index- superfluous additions included for ideological reasons.

These economic freedom indexes are often used to argue for small government policies, on the grounds that the highest ranked countries usually have high GDP. To expose the inconsistencies between the variables that are included in the bundle of “Economic freedom” I thought I’d run a few correlations using the Heritage foundation’s own data.  Now it’s important to be clear that these correlations do not prove anything of significance about what makes for good or bad economic management- for that we need very strong controls, Granger causation etc. However they will serve to demonstrate the underlying problem with getting a grab bag of things everyone likes, a grab bag of things you like because you support small government, mixing them together and calling it an economic freedom index.

I have only included sub-scales which are measured in a way I can reproduce, either by looking up publicly available data, or by accessing third party independent reports. This is because, to be blunt, I do not trust scales produced by the subjective judgements of Heritage foundation employees anymore than I would trust Dracula with the blood bank. It’s not that I think that they’re dishonest, it’s just that I don’t think they’re disinterested. Scales that involve subjective or qualitative assessments are fine, so long as those assessments have been done by third parties.

Probably the most serious criticism of my post here is that I’ve chosen to correlate the various sub-scales with log GDP per capita rather than using some other measure. In some ways this is not ideal, we could have chosen, for example, growth as our point of comparison. But unadjusted growth has its own issues- e.g. growth tends to slow down in richer countries and many other disparities. I’ve seen how messy these debates can get, for example there’s a whole argument in the effects of redistribution literature on this. I really didn’t want to get bogged down trying to construct an ideal point of comparison- say growth adjusted for GDP with numerous other tweaks included. This isn’t how the economic freedom index is sold, or validated, and it’s not like people are relying on the index in the main because it has undergone these sophisticated tests.

So I decided log GDP per capita (henceforth just “GDP”) is the closest thing to a single fair metric. What ultimately clinched it was that the Heritage institute itself often boasts about the correlation of the freedom index with GDP, so it seemed like a natural and fair comparison because they use it all the time, and this is a response within the framework of their own research.

SUBSCALES MEASURING THINGS EVERYONE LIKES

Judicial effectiveness: Everyone likes judicial effectiveness, from democratic socialists to small government advocates. The scale is based on survey data and third party opinion. Unsurprisingly it correlates 0.6 with GDP per capita, a very respectable relationship.

Government integrity: Government integrity is also universally liked. Government integrity scores in this report are taken from a variety of survey data. No one will be surprised that it correlates 0.65 with per capita GDP.

Fiscal Health: While it is true that small government advocates are more fixated on balanced budgets than the left, no one actually likes running large deficits. Basically this scale tells us that countries that have recently run deficits are likely to have somewhat smaller GDPs (A small correlation at 0.22). This may be partly because many countries use deficit spending counter-cyclically during periods of economic downturn.

Business freedom: Wait a moment! Surely not everyone likes business freedom? Surely this is a small vs big government issue issue? In general yes, but given the way the authors choose to measure it, almost every single criteria is unequivocally a good thing. For example, there is no one on the planet who wants it to take more days to setup a business, get a licence or obtain electricity. Maybe one criteria out of 13 marks a genuine ideological point of contention (number of procedures required to get a licence) and this is very debatable. Given that this is basically a measure not of the scope of bureaucratic procedures, but their efficiency, and a very specific suite of procedures at that, it is totally unsurprising that this criteria correlates 0.66 with GDP- a strong relationship.

Now we turn attention to the bits of the index that divide small government advocates from everyone else.

SUB-SCALES MEASURING THINGS SMALL GOVERNMENT ADVOCATES LIKE AND OTHER PEOPLE DON’T LIKE

Tax Burden: Some people like high taxes for the services they allow, some people don’t.  There is no practically or statistically significant correlation between low taxes as indicated by this scale and GDP according to the Heritage Index’s own data. In fact it’s non-significantly negative (-0.08).

Government spending: Obviously a good measure of small v large government.  The correlation between GDP and this score is somewhat negative (-0.34), suggesting that more government spending as measured by this scale is associated with a bigger economy per person. In other words, this sub-scale actually somewhat anti-predicts economic success, not a great result for the Heritage foundation.

Labor Freedom: The right likes it, the left hates it and would dispute the “freedom” bit (do you enjoy your freedom to be fired?) I could probably quibble with how the heritage foundation define this measure on a few points but I’ve chosen not to because I don’t want to be accused of special pleading. Labor freedom as measured by the Heritage foundation has a small positive correlation with GDP (0.28), good news for the Heritage foundation I suppose, but it’s only about 8% of the variance.

SUB-SCALES EXCLUDED AS THEY DEPEND ON THE SUBJECTIVE OPINIONS OF HERITAGE FOUNDATION EMPLOYEES

Property rights: Entirely graded on by Heritage foundation employees. Worse, the criteria arguably bunch together a bunch of potentially independent things.

Investment freedom: same as property rights.

Monetary Freedom: Honestly I wouldn’t have been terribly against including this one, as it is mostly scored on objective grounds, but I’m sticking to my own criteria. The subjective part is the way they measure price controls.

Trade Freedom: Trade freedom is a bit of an ideologically weird issue that transcends distinctions between people who normally vote for small or large government. I wouldn’t be especially perturbed if it had a large positive, negative or no correlation with GDP. It’s subjectively graded in part by Heritage foundation employees through the assessment of non-tariff barriers so I haven’t included it.

Cronbach’s alpha

I thought I’d check the quantitative consistency of the scale using Cronbach’s alpha, a measure of the internal consistency of a scale. Cronbach’s Alpha for the scale with subjectively scored items removed is .68, which according to trusty Wikipedia is “Questionable”.

Conclusions

The Heritage foundation economic freedom index is a bundle of often unrelated criteria. Excluding criteria graded on the basis of Heritage foundation employee opinions, those criteria on which pro and anti big government thinkers are likely to agree are good things are correlated strongly with GDP. However those criteria which small government proponents are likely to think are important, and big government supporters disagree, do not correlate positively with GDP. The only exception is labor freedom, but the correlation is small and there are other problems with the labor scale.

As a result of the unrelatededness of the criteria, the scale is inchoate. Many of the sub-scales either do not correlate with each other, or even correlate negatively with each other, suggesting that there is no one, real coherent construct being measured by “economic freedom”. This is shown in the scale’s Cronbach alpha score of .68.

You could always just say “By economic freedom I mean that which the economic freedom index measures.” But if so, “Economic freedom” has very little internal unity, and its parts predict different, sometimes contrary things. The whole thing seems very misleading, whether intentionally so or not.

Just for fun: DePony Sum’s Index of Small Government

Just as an exercise I took the objectively measurable variables which are measures of small government and added them together (this is how the freedom index does it- no fancy maths here). Those variables are labor freedom, government expenditure and taxation rate. I then correlated the resulting sum with GDP.

The result is actually slightly negative, -0.15. It seems this index won’t be demonstrating a link between small government and prosperity any time soon.

Also just for fun: DePony Sum’s Index of Good Government

Inversely, what would happen if we just used the measures which everyone- small government advocates or not- regard as measures of good government. To recap these are judicial effectiveness, government integrity, fiscal health and “business freedom” (because of the peculiar way “business freedom” is measured and defined in this data.) This index fares much better as a predictor of economic prosperity, correlating 0.61 with GDP per capita.

The use of this scale in serious economics

The 2001 Index of Economic freedom alone was cited 1425 times, and an edition is produced every year. The combined number of scholar citations for all EFI editions is probably in the tens of thousands. The EFI has been cited approvingly in distinguished journals such as Economic Policy, Journal of Development economics and The Journal of Law and Economics. It has been cited in such highly regarded non-economics journals as The Journal of Accounting research. The majority of these citations treat the index positively, at least implicitly, and some even treat it as authoritative, including it as a variable in regressions and other statistical analyses.

To my mind, the fact that this scale has been cited as authoritative in major journals is a scandal of the name of economics. This stuff doesn’t belong in an undergraduate journal. It’s a mess.

We’ve honestly used a pretty light touch here. I’ve pretty much accepted that the scales are internally well constructed representations of what they purport to measure, despite many many sub-scales being themselves constituted by an odd mishmash of variables. I could have been a lot meaner.

No one likes to think of themselves as engaging in ad hom argument in academia, but at the end of the day, it really should have been obvious that scepticism is needed when the index was called the “Economic Freedom Index” and created by an organisation called “The Heritage Foundation”. You shouldn’t take my word for this of course- I’m just as bad as they are but from the opposite end. However, it doesn’t have to be a matter of trust. If you are a social scientist of any sort and take your profession seriously, please read through the methodology of the Freedom Index and at least look at the first order correlations in the data before you cite it, let alone use it as a variable in your regressions. This is simple minimal due diligence. Like a good doctor facing a pharmaceutical representative, use common sense.

5 thoughts on “The Heritage Foundation Economic Freedom Index is bad

  1. So which countries are incorrectly ranked on the list in your opinion? Will you publish your own list of economic freedom that corrects the Heritage Foundation’s inaccuracies, and properly ranks the countries? Will your list, for example, rank Cuba above Canada, or some other drastic difference?

    Like

    1. I object to the whole notion of an economic freedom ranking, at least as a tool of rigorous social science. More specific variables should be used to investigate particular issues. Some of those more specific rankings will be drastically different from the economic freedom index.

      For example, let us suppose we compose a size of government index by summing together the tax burden index with the government spending index, we now find that Sweden, Denmark, Norway, Finland and other social democracies score lowest, because they have large tax burdens and governments. This is the complete opposite of the normal ranking Economic Freedom Index, where these countries are close to the top, largely because, as noted in the article, many of the variables in the freedom index test for well-functioning governments, and the Nordic countries generally have extremely well-functioning governments.

      Like

      1. It seems plausible that small governments are more observable and controllable by their citizens, but indeed, why use size as a proxy for quality? Does the Heritage index account for decentralization below the national level? New England states might end up with a similar total amount of government as Nordic countries, but their local governments are highly accountable (e.g. the famous town meetings).

        Like

  2. >no one actually likes running large deficits.

    MMT is the latest meme in lefty economics and it’s all about how running huge deficits is a great thing…

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s