Fine survey, poor inference

In a seminar on decentralization yesterday, a team from the World Bank presented their survey result done with Gajah Mada University. The "most comprehensive study of the impact of decentralization on the quality of public service to date" concluded that "autonomy leads to improved public services". Well...

I'm not against decentralization. In fact, I like the idea of decentralization. But I have problem with the inference drawn by the research team -- or rather, on the way the team drew their conclusion. Look at their questionnaire here. One question to -- supposedly -- capture households' perception on decentralization, i.e. "local autonomy" is

"Local economy is currently undergoing since 2001. In your opinion, how is the quality service [sic] of [...] in the district/municipality compare to year 2000?" The items for [...] are school, puskemas (health clinic, head of village, sub-district office, and district/municipality office. The options for responses are 1) Worse, 2) Same, 3) Better, and 4) Don't know.

Now, from that kind of question, do you dare to make an inference about the effect of autonomy on public services? I would not.

Try another one:

"In you opinion, how is the quality of [...] program implementation in this district/municipality in 2001 compare with [sic!] year 2000?" The items include for example "poor people empowering" and the response options are 1) Decreasing, 2) Constant, 3) Increasing, and 4) Don't know.

Again, can this kind of question lead you to conclude that the autonomy or decentralization lead to improvement or disimprovement?

Probably yes, if you can find some way to isolate any possible other factors that might determine the change in "poor people empowering" etc. Just because many respondents said that "poor people empoering" is "increasing" is no justification to conclude that it was due to autonomy policy or decentralization, even though you picked up the relevant periods of time.

This is the same with the former question above. Just because majority of respondents said that the school quality is better in 2001 compared to 2000, you can't just conclude that it was because the autonomy/decentralization.

All that was presented yesterday was data frequency and percentages. No parametric treatment whatsoever. So, I was very surprised with such a strong conclusion.

It strikes me that people can rush to make inference that way. That is a dangerous approach to data.