This week the ITU has published its latest ICT Development Index. I don't want to write about that now - except to state that like so many other similar exercises calling this an "index" is perpetrating a fraud.
The concept of an "index number" was developed to find a way to relate different prices and quantities in different time periods. The founder of econometrics Irving Fisher analysed said "For those who have made any attempt to penetrate their mysteries, index numbers seem to have
a perennial fascination." This may not be the case for my readers, but the survey article I took the quote from provides plenty of detail on how intricate is the process of developing index numbers for their use in analysing time series data.
The ITU's IDI is not such an index. It is an attempt to make comparisons across countries at one point in time. Indeed the construction of the index guarantees that the change in the index number from one time period to the next for an individual economy has no meaning. The only temporal comparison that can be made is of the rank.
This is because the final index number is composed as the weighted sum of a three sub-indices each in turn based on a number of indicators. The data for the indicators themselves are also first manipulated in a kind of standardisation process.
The report states that "The indicator weights were chosen based on the principal components analysis (PCA) results. The access and use sub-indices were given equal weight (40 per cent each). The skills sub-index was given less weight (20 per cent), since it is based on proxy indicators." Figure 2.2 provides a table of the actual weights used and it is hard to discern from this exactly what role the principal component analysis played.
Most significantly there is no objective test by which it is possible to determine if the IDI measures anything, nor if the value of the IDI has any purposeful predictive power. Indeed, like most indices of this kind (I'm thinking here of the Global Innovation Index) the composition of the index is heavily theory laden. There is nothing inherently wrong in a theory laden index if that index can then be compared to some other observable - because it then works as a test of theory. But if there is no such observable the index runs the risk of becoming part of a circular argument in support of the theory.
But I didn't come here to discuss the IDI - I need to do more maths before I reach any conclusions.
What I did come here to do was to pass comment on analysis in the report that purports to claim that competition in telecommunications markets has a statistically significant impact on reducing prices in telecommunications. I have serious concerns about the methodology employed.
(My own simple working paper on this reached a conclusion that competition is not a significant factor in price reductions).
My two concerns are to do with the model employed and the goodness of fit. Both fixed broadband and mobile market data are modelled. In both cases a simple linear model of prices is developed. This is highly unlikely to be the appropriate functional form for the relationship between prices and the relevant variables - including GNI per capita, industry concentration (HHI), urbanisation and a regulatory variable. At the very least theory would suggest that the effect of a change in concentration would be proportional to the HHI - not a linear composition.
In both cases the modelling claims that all the variables are statistically significant - though competition is identified as explaining only 5% of the variation in prices. However the R-squared for the two models are 0.408 and 0.409. The report claims that such a value of correlation means the models have "medium explanatory power" based on the range of possible values being zero to one.
This is simply rubbish. The reality is that such a low value means that more than half the variability in prices is due to factors not included in the model. One of those at least will be declining costs of technology due to local scale economies and global experience effects. The consequence of adding other variables or changing the functional form so that the explanatory power of the model increases will affect the statistical validity of all the variables.
It is, quite frankly, embarrassing to see a major international organisation publish such a poorly constructed piece of econometric modelling.
No comments:
Post a Comment