New Page 1

A special report on the objectivity and reliability of the reporting of biomedical research from The World Association of Medical editors conference-Barcelona, Sep.2001. as reported in The Economist; Sep. 13 2001.

Some major findings from the conference state that:

Some drug companies ensure that only favourable results of their drug trials are published. They may also suppress publication of unfavourable results.

Sometimes up to 76% of researchers listed as authors of a paper may have had nothing realistic to do with the paper at all.

At times, the contributing editors of a research report may disagree about almost everything in the paper, such as its key findings, weaknesses, implications and future prospects, but this will not be acknowledged anywhere in the report.

This kind of censorship can be attributed, at times, to the actions of the author who stood to gain most credit from a publication actually erasing weaknesses or disagreements stated by co contributors

An investigator examining these kind of problems in publication of research found; 'suppressed opinion, censored criticism and serious bias among contributors

The situation is getting worse and there are no clear solutions to improve it

Some of the most powerful words in print today are those in journals of biomedical research, for they can literally wield the power of life or death over future patients. If published clinical trials establish a drug's safety and efficacy, doctors will eventually prescribe it. If not it will be rlegated to the laboratory's hazardous waste basket. Ensuring the accurate reporting of such trials is therefore a matter of some importance.

Unfortunately, maintaining the integrity of this sort of science seems to be getting ever more difficult. To select and revise promising papers, journal editors have long relied on the process of peer review, in which a manuscript is sent out for criticism to several experts in the field. If peer review is to work, each party must fulfill its side of the bargain. First, authors genuinely have done the things they claim to have done and written the documents they claim to have written. Second, reviewers must show they are able to assess the research as objectively as possible.

It appears that, in too many cases, neither side of the bargain is being kept. This weekend, several hundred researchers and medical journalists are supposed to be gathering in Barcelona for a conference organised by the World Association of Medical Editors on the effects of peer review on medical biomedical research. Some of the findings to be presented are damning of researchers, of reviewers and of journals themselves. Although, to their credit, authors of medical journals are seizing every opportunity to tighten the reigns of control over biomedical research, many of the problems defy any quick fix.

The issue topping the editorial agenda is the honesty of research that is sponsored by drug companies to test the safety of their wares. These companies have found that scientists at private contract-research firms can perform drug trials far more cheaply than the academic researchers who have traditionally been drafted in for this job. Such firms drew 60% of the research funds spent by drug companies last year; academic scientists were left scrabbling for the rest.

Unfortunately, the companies are saving more than money by using such hired guns; they are also saving face. Through the use of restrictive research agreements, drug companies ensure that only favourable results are published. In some cases, they not only retain the right to suppress publication of results, they also deny the authors of papers about such studies access to full sets of study data.

In the face of this, such authors cannot give honest accounts of the research, nor can editors and reviewers make honest appraisals of the results. This week, the authors of 13 top medical journals issued a statement condemning such abuses of corporate sponsorship. The members of the International Committee of Medical Journal Editors (ICMJE) will now revise their publication guidelines to require that researchers disclose their sponsors' involvement to editors when they submit articles. Some editors will require researchers to certify that they take responsibility for how the trial they are reporting was conducted. And authors may be asked to declare that they had access to all trial data, and had the power whether or not to publish the trials' results.

In a sense, these rules are formal demands for things that ought to be taken for granted. But, as a paper due to be presented at Barcelona demonstrates, it is not even clear that authors pay enough attention to the existing ICMJE guidelines, let alone that they will be willing to abide by extra ones.

The existing guidelines state that people on a people on a paper's author list should have contributed, at the very least, to the conception of a study, the analysis of the data and the writing of the manuscript. Yet when Susan Van Rooyen and Sandra Goldbeck-Wood of the British medical Journal and Fiona Godlea of Biomed Central, a database of scientific extracts, analysed 129 research articles submitted to the BMJ by 588 authors, they found that this was not always the case. Depending on how strivtly they interpreted the ICMJE guidelines, between 24% and 71% of authors qualified which is hardly a stellar performance.

Richard Horton, the editor of The Lancet took an even looser definition of authorship. He studied a small sample of research papers to find out if the views expressed in those articles were accurate representations of contributor's opinions, a basic enough requirement, one might think. He examined in detail, five papers published in The lancet last year, and found that contributors disagreed about almost everything including a paper's key findings, weaknesses, implications and future prospects.

That may not surprise anybody who has ever worked on a team project. Dr Horton, however, also sniffed out a more disturbing trend. The contributor who stood to gain the most credit from publication would often erase from the final report the weaknesses acknowledged by co contributors. A research paper rarely represents the opinions of all the scientists whose work it reports, Dr Horton concludes.

Ideally the 'suppressed opinion, censored criticism and serious bias' among contributors found by Dr Horton would be corrected by the unfettered opinions and criticism offered by unbiased reviewers. But authors may obstruct reviewers judgments by leaving out important information about their results, such as details about other similar trials. According to research conducted by The Cochrane Collaboration, an international group that studies clinical trials, reviewers do not seem to be catching these omissions. Indeed, they may be getting worse at their task.

In May 1997 a group of conference researchers, led by Ian Chalmers, investigated whether reports published in five top medical journals adequately outlined research on other relevant trials, by studying one month's worth of reports. Almost none of the 26 reports passed the test. This year, the team repeated the procedure, using the 33 trials published by these five journals in May. None of the trials put its findings in the context of earlier trials. Four reports falsely stated that they were the first published trial on a particular topic. Six more kept mum about related trials. Dr Chalmers and his colleagues found that the situation had deteriorated in the last four years.

This is strong stuff, but it may get worse. Phil Andersen and his colleagues performed a 'meta analysis' of editorial peer review- that is, they used published studies as their data and applied statistical methods to the outcomes of those studies, as though each study were a single experimental result. They looked at the effects of peer review upon various criteria, including methodological soundness, completeness and accuracy.

Although journal editors go to great pains to ensure that authors do not know the identity of their reviewers, or vice versa, the researchers found that this laborious and expensive process had little impact upon the reviewers' appraisals of quality of research. When they surveyed the sum of research on peer review, they found only scattered empirical evidence supporting the use of editorial peer review as a mechanism to ensure quality of biomedical research Until better studies are conducted, peer review may need to undergo, well, peer review.