Why Narrow Definitions of "Evidence-Based" Are Useless

Why Narrow Definitions of "Evidence-Based" Are Useless

Narrowly-defined evidence standards for "what works" in K-12 education actually prove what doesn't work...

After re-reading the US Department of Ed's "What Works Clearinghouse" Research and Education handbook -- the more I read, the less convinced I am. Any study that is not a randomized controlled trial (RCT) is only considered as evidence "with reservations." Quite the opposite for me: I hold any study which was conducted as an RCT "with reservations." It appears that they have hijacked the term "evidence-based" to mean that RCTs are the only real evidence. Looks like a classic case of intellectual monopolism.

Because what I've also been wondering lately is whether the inadequacies of education research are the result of a bad model or bad science. Is it the rational-scientific model which is itself at fault, or is it the poor application of that model? Or both?

Maybe the problem is simply in failing to take all the important variables into account. Maybe a properly conducted RCT could provide "strong evidence" that hungry kids don't do as well in school subjects as well-fed ones. Do you want to run that RCT? I sure as hell don't. How many of the "What Works Clearinghouse"-vetted studies took into account the hungry kids? The kids who have some other pressing life issue on their mind? Or is it the case that, by definition, studies which use RCTs or other so-called "evidence-based" criteria deliberately exclude some of the most important factors yet encourage us to overlook them, focusing instead on finding some magic solution which will somehow overcome these overlooked issues?

If an intervention works for 60% of a student population instead of 40%, isn't this also proof that it doesn't work for the other 40%? Why make a decision based on an RCT to try a new program which has been proven not to work on a significant proportion of the population? Why even conceive of it in these black-and-white terms?

So I'm voting that it's a bad model. The model is bad because randomization is bad. It makes us look for 'one size fits all' solutions when we should be looking for 'solutions for everybody.' Isn't randomization the very opposite of customization? Randomizing assumes that it is possible to control for every important variable; it assumes that the "subjects" can be reduced to creatures whose behavior can be whittled down to one pivot point (the "treatment") on which success and failure will hinge. Somehow, this strikes me on some deep level as a form of brutality -- should we be stripping everything else meaningful out of the activity even if we could?

It also assumes that outside factors can be controlled to the point where they will not interfere with the outcome. Is this really how we want our children to live? I'm picturing a caring grandparent reading a story to a child, or a peer reading something that the child can't, or an older sibling, making a light go on: suddenly now the child wants to learn to read; the child understands something s/he didn't before. Or maybe the light has been on for some time, thanks to a parent or other caring adult who has been reading to the child since infancy.

I haven't read all of the so-called "evidence-based" studies, but I'm willing to bet -- er, hypothesize -- that none of the studies has any mention of any of these people in terms of the actual reportage of results. If gains are made, all credit goes to the treatment/reading program. It is hard to imagine a more willful distortion of reality.

Complex systems theory tells us that even the best RCT can't account for an entire complex system, which will change over time as its components evolve. Trying to hold randomized subjects "constant" during a treatment time period is pure folly in this view. Then again, perhaps that's what the caring people are trying to tell us through their actions, however antithetical these actions might be to pristine experimental designs.

In the meantime, the only thing that such a narrow definition of "evidence-based" proves is that a narrow definition doesn't work. It's an impediment to improving education. The savvier researchers are using broader definitions and producing more useful results...

SK/JS on the Web