Help - Search - Members - Calendar
Full Version: Why Most Published Research FindingsAre False
Unexplained Mysteries Discussion Forums > Science > Science & Technology
truethat


This article is quite interesting to me. Especially the part regarding bias. I'm curious if any of our science folks on here have ever seen any bias in studies?


Btw this is from PLoS Medicine a peer reviewed open access journal.


http://medicine.plosjournals.org/perlserv/...020124&ct=1


Summary

QUOTE
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.


Here's another article that explains the above


http://www.marginalrevolution.com/marginal...ost_publis.html

Here is a criticism of the paper

http://www.omninerd.com/articles/Science_Not_So_Sloppy




Fluffybunny
The entire idea is that peer reviewed data is just that; it is looked at by many many folks in different fields to see what bias there is and what may be being missed if anything. Science allows for correction, that is the beauty of the scientific method, and it works. It isnt a religion where one person sets canon and that goes; it is an effort that is from a community and is always challanged and added to, and if needed, changed.

Any one person will have bias no matter how hard they try not to, that is just they way it is. having a thousand people facing the same problem washes out that bias and allows for objective points of view and many many opinion. The scientific method works, and when you get involved and see it in action it is a pretty impressive thing to see.
truethat


I thought this paper was interesting in the way it discusses bias in layspeak. Fluffy, while your assertion that the thousands wash out, what about when the thousands all share the same bias? Couldn't that have an adverse effect as well in washing out the alternative view points?

Here's the article its long but interesting.

http://www.uow.edu.au/arts/sts/bmartin/pubs/92prom.html#fn0
Sthenno
Having just completed two modules at MA level in research methods, I have concluded that it is all indeed biased. Take statistics: if you have some results which don't match the general pattern, you essentially disregard them. Bless the poor bloke that had to explain this to a bunch of radical geographers, that debate went on for weeks!
Fluffybunny
QUOTE (truethat @ Feb 15 2008, 12:32 PM) *
I thought this paper was interesting in the way it discusses bias in layspeak. Fluffy, while your assertion that the thousands wash out, what about when the thousands all share the same bias? Couldn't that have an adverse effect as well in washing out the alternative view points?

Here's the article its long but interesting.

http://www.uow.edu.au/arts/sts/bmartin/pubs/92prom.html#fn0

I can see where you are going with this, but you are missing the point. Evolution and other huge studies have so many people looking at it from all different fields and different points of view that it keeps the facts, factual and people honest. The scientific method works very well that way, and we see it time and time again on big scales and on littles scales showing how the process can be used to find the answer to difficult questions. Science self corrects and always moves forward.

Washing out alternative view points? No, those viewpoints are to be represented by their own work, which gets peer reviewed and stands or falls on its own merits.

I love when you attempt to point out some flaw in a study or some idea in a journal that is incorrect; that in itself shows (in a small way) how a part the process works. You are trying to point out what you see as being incorrect. That is a part of the process. Not as cheeky as you get sometimes, but the concept is there. Unfortunately that is a bit difficult as neither you or I can always do that effectively as we may not have the background to be able to fully understand the argument being made. I may only have a 4 year degree in a given field of study, and a limited field internship to understand a particular area of science for example; so I would have to really be on my toes when I tell a person who has a Doctorate in multiple fields of study and 25 years of field research that I think they are blowing smoke when they make a statement about their lifes work that they have been dealing with day in and day out for 10 years. Not saying that I may not be right, but I better have my research done correctly, otherwise I am going to look like a fool. Who knows, I might be having a good day and make a name for myself...it could happen.

That being said, the power of 500 people looking at the same work can pool the life experience and knowledge of those 500 people and be able to present arguments effectively and accurately that the presenter would have to respond to. It goes back and forth and back and forth. It isnt as if the concepts we have now that are taken for granted(gravity for example) were accepted in a short period of time, nor would they be dropped or altered if the need be.

Point at the mistakes or the hoaxes or whatever you may choose, but that in itself proves the point that the process works.

You presented a paper the other day that made some extreme claims as if it were representative of all sciences opinion on that subject, that is not the case. It was one persons opinion, and from what I read not completely accepted anywhere else.

Try to understand how the process works and where the papers you are pulling from are in that process. Anyone can present papers to be peer reviewed in journals, it isnt difficult. They can be mad as a hatter and still get printed. That does not mean that the ideas they put forth mean anything of substance or will be accepted. There are wild ideas out there in acadamia, they are not all accepted. That takes a lot of work and a lot of research by a lot of people in a lot of different fields before it is even considered to be any kind of acceptable theory.

truethat
Excuse me but do not presume to tell me how I think. I am not interested in discussing Evolution in this thread. Nor another thread. I'd like to keep it ON TOPIC if you don't mind.

Additionally I opened the thread up to the scientists on this site to weigh in on their opinion.

Additionally I posted CRITICISM to the original article that I posted.

I'm interested in this thread in discussing the heirarchy and "business end" of science.

But I see once again any criticism of Science is somehow a giant taboo that is not allowed thus leading to your post in trying to shut me down.

I would appreciate if the topic of Evolution was not discussed in this thread. I'm not interested in it in the slightest here.

Thanks.
Fluffybunny
QUOTE
But I see once again any criticism of Science is somehow a giant taboo that is not allowed thus leading to your post in trying to shut me down



Forget I ever tried to have an actual exchange with you, or explain something that I felt was important to the topic. As you were...
truethat
QUOTE (Fluffybunny @ Feb 15 2008, 09:27 PM) *
Forget I ever tried to have an actual exchange with you, or explain something that I felt was important to the topic. As you were...



Thanks thumbsup.gif


Now the second article I posted I found very interesting. Unlike you I'm not talking about bias so much in the way you seem to take it, that the person has a preset agenda or theory that they are trying to push. But I do think that bias in the field of funding and who is paying for the study is interesting. You can make a test prove anything. I can't find it online but I've heard that Dominoe sugar is the one who paid for the original studies that proved that saccharin caused cancer in lab rats.

crtbud
Well without proof, I'd take that hearsay with a grain of... sugar.
truethat
QUOTE (crtbud @ Feb 15 2008, 10:48 PM) *
Well without proof, I'd take that hearsay with a grain of... sugar.



LOL Great one! No that's just an example. What I'd really like to look at in this thread is if any of this stuff has merit or is it someone being overzealous regarding a few cases.
capeo
QUOTE (truethat @ Feb 15 2008, 04:34 PM) *
Thanks thumbsup.gif


Now the second article I posted I found very interesting. Unlike you I'm not talking about bias so much in the way you seem to take it, that the person has a preset agenda or theory that they are trying to push. But I do think that bias in the field of funding and who is paying for the study is interesting. You can make a test prove anything. I can't find it online but I've heard that Dominoe sugar is the one who paid for the original studies that proved that saccharin caused cancer in lab rats.


I don't know about the Domino sugar thing but I will say the Pharmacuetical, Nutrition and other such largely corporate sciences can be very, very dirty. Large corporations want the results they want no matter what. Medical fields are plagued by the statistical anomolies that can come up in test trials. No amount of control can always cover ever eventuality. It's not like trying to arrive at a direct conclusion, it really trying to arrive at the correct infulence of which there can be many. That's why I can see a maybe 50% or more rate of error when reviewed down the road. I don't by apply bayesian statistical approaches across fields that aren't based inherently in statistical findings. These models just don't hold up in harder sciences. That said I could see at least 25%-40% of ALL scientific published papers to be incorrect. That wouldn't surprise me. Bear in mind peer review is to insure that things are done correctly not do the experiments and see if the same conclusion is reached. There are thousands and thousands of papers published every year that get refuted. That's the self correction of science.
truethat


I've posted about this elsewhere but I do remember a friend of mine in the early 90s being able to get her hands on an AIDS cocktail for her brother that was being "held back" for copyright reasons. In other words money.
questionmark
Well, it would be quite a scandal if MOST of the published research was false.... and I fail to see how a few casuistic cases demonstrate (or even hint) that. There are several ten thousand studies published every month.

Where there is a grain of truth in this is that there are scientists for sale to corporate interest, or the whores as they are known in academia. But that does not deny the validity of scientific research. And that is the research that gets ripped apart constantly by peer review... only that nobody, except scientists, seem to be interested in that.

Now, if you say that so called scientific research that is published in consumer publications.... those are infomercials, and have mostly been paid for by an interested party.No matter if pro sugar or anti sacharine, cholesterol killer or link from soda to heart disease proving that brand B is healthy.

But as I said before: Proving that research is faulty does not prove that the contrary is right by default.



truethat
No one is saying that it means that questionmark. I am just examining the reality of science as opposed to the sacred ideology of the perfect peer review system. In my opinion when people are involved there's always the possibility for bias.
questionmark
QUOTE (truethat @ Feb 16 2008, 02:06 AM) *
No one is saying that it means that questionmark. I am just examining the reality of science as opposed to the sacred ideology of the perfect peer review system. In my opinion when people are involved there's always the possibility for bias.


But that Ioanidis article is part of peer review ... that is how it works: including someone studying how many of the "findings" are pure crap.

frogfish
Medical journals, such as Nature Cell or Nature Cancer are very unbiased. Peer reviews on papers are thorough and harsh. You'd be surprised.

Researchers take great measures to remove as much bias as possible.
This is a "lo-fi" version of our main content. To view the full version with more information, formatting and images, please click here.
Invision Power Board © 2001-2008 Invision Power Services, Inc.