Wednesday, August 03, 2005
Science Under Review
There's been a good bit of noise recently about the need to reconsider how clinical trials get conducted and published. A study in this week's BMJ describes a series of statistical tests which, the authors claim, can detect fabricated data in trials.
This is a very interesting paper to me; the methodology is fascinating, and the idea compelling. But alas, I think it is crap. Yeah, that's right - I. Think. It. Is. Crap. The theory that you could detect likely patterns of fabrication is probably valid, but only the first time, and only for people who fabricate like you do. The preference to round made-up numbers to the nearest 5 or 0 is common, but so is the knowledge that people are more likely to trust odd-numbered figures than even ones. Which takes precedent? Can you analyze for both?
Also, what happens after this paper, when fabricators change their methods? I agree with the beginning thesis - that reviewers need access to basic data elements, and that there should be more lookout for fraud - but don't think that there's an easy plug-and-chug solution like this. Still, it's fascinating.
This is a very interesting paper to me; the methodology is fascinating, and the idea compelling. But alas, I think it is crap. Yeah, that's right - I. Think. It. Is. Crap. The theory that you could detect likely patterns of fabrication is probably valid, but only the first time, and only for people who fabricate like you do. The preference to round made-up numbers to the nearest 5 or 0 is common, but so is the knowledge that people are more likely to trust odd-numbered figures than even ones. Which takes precedent? Can you analyze for both?
Also, what happens after this paper, when fabricators change their methods? I agree with the beginning thesis - that reviewers need access to basic data elements, and that there should be more lookout for fraud - but don't think that there's an easy plug-and-chug solution like this. Still, it's fascinating.
