There was an interesting story on the news last night and in this morning's paper. A study found that voucher students in Milwaukee test lower than non-voucher students.
Like many issues that are both statistical and political, there's a fair amount of posturing surrounding this one but let's try to focus on the data.
On this particular test, the average scores of voucher students were clearly lower than those of Milwaukee students overall. That's not open to debate. The question is, what does it mean? Voucher critics want to present this as evidence that vouchers don't work (i.e. a causal link between vouchers and performance). Voucher fans present a list of flaws in the study to refute the causal claims of the critics.
If the test scores were reversed, you'd probably read opposite comments by the same people. Voucher critics would say what was wrong with the study and voucher fans would claim the study "proves" vouchers work.
If you don't believe me, check out this article or this one. These came out today and cover different studies potentially implying that vouchers "work". The arguments aren't quite the opposite of yesterday, but they're close.
Most of those quoted in these articles miss the most important statistical point. Consider this statement from the first article: "Currently, only low-income students in Milwaukee can receive a voucher from the state."
Even though I've heard Milwaukee's voucher program called a grand "experiment" of some sort, it is not really an experiment at all. It's merely an observational study and observational studies cannot determine causality.
A few researchers quoted in these stories talk about the need to establish performance baselines and track long-term progress (or lack thereof). That would certainly help, but it still wouldn't be an experiment.
For an experiment, you must have random assignment to treatment groups. For the Milwaukee voucher program, this would mean random selection of voucher recipients. As it is now, voucher recipients must have low income AND must apply for the program.
These are both confounding variables. Many researchers believe that low income students generally perform worse than middle and high income students. If correct, that would skew voucher recipients' scores downward. On the other hand, parents who apply for the vouchers are potentially more concerned about their child's education and researchers also claim that parental concern and involvement leads to better school performance. Thus, "must apply" could skew recipients' scores upward.
Random assignment of vouchers is the only way to overcome these confounding variables (and likely other variables that I can't think of). Unfortunately, I doubt that random assignment would be politically tenable.
Even if the political issues surrounding random assignment could be overcome, you'd still risk placebo effect. There's no way I can think of to double-blind an experiment of this sort but I'm open to suggestions.
No comments:
Post a Comment