Introduction
More than a decade ago, Douglas Altman noticed that it seemed “widely acceptable for a medical researcher to be ignorant of statistics” (1). Recently, in a discussion panel of the World Association of Medical Editors (WAME), Ana Marušić from the Croatian Medical Journal asked the editors their opinion on “how much knowledge of statistics a journal editor should have and how she/he would approach the problem in a biomedical journal” (2). Both Altman and Marušić touched on the sensitive question of statistics in biomedical science and drew attention to a possible problem: if both author and journal editor lack statistical knowledge, who is then responsible for the validity of statistical methods in published papers?
Statistics is defined as a scientific discipline, a branch of mathematics related to philosophy, logic, and informatics. However, in the field of biomedicine and biomedical sciences, it is mostly viewed as a simple, stylish or highly sophisticated set of methods for gathering, analysis, interpretation, explanation, clarification, and presentation of data (3); or a set of inquiry methods that enable us to think statistically; or a “body of methods for learning from experience” (4). Doubtlessly, statistics is important; therefore, researchers should have some knowledge of statistical methodology and use it properly. As for the statistical review of research papers, I agree with Paul Payson from the Canadian Psychiatric Association (2) who said that it is “crucial to have someone with advanced statistical knowledge either on the Editorial Board or acting as a consultant, who could determine if some of lesser known or more recent statistical tests have been used properly.”
Statistical reviewer and statistical editor
In general, biomedical journal editors do not have enough knowledge, training, or skills to evaluate statistical methods and computational analyses in all manuscripts submitted for consideration, especially in studies using complicated research methodology, investigating complex subject, including compound samples and groups, or applying sophisticated variants of statistical tests, or performing unusual data comparisons and presentation (2). They need help in a form of a professional evaluation of both research design and statistical methodology. This kind of help is provided by a competent scientist – a statistical reviewer or statistical referee. A statistical reviewer with a permanent position on the journal’s board is typically called a statistical editor (5,6).
It would be a desirable practice for journals if statistical editors read and commented on all manuscripts considered for publication. In some journals, such as Croatian Medical Journal, statistical review of all manuscripts accepted for publication by the editor-in-chief is part of a regular peer review process (7). In other journals, such as The Lancet, only manuscripts considered for publication are statistically reviewed after receiving positive peer reviews (8).
The main goal of a statistical review, which often combines the review of both statistical and epidemiological methodology, is twofold: to ensure that the study was properly conducted and results appropriately presented and to reveal possible errors and omissions. Let us first look at possible errors.
Errors in methodology
According to Altman (1), huge amounts of money are spent on poorly done research: inadequately designed studies, studies with unrepresentative samples or small sample sizes, studies using incorrect statistical methods or providing inappropriate interpretation of data. In their study, Lukić and Marušić (6) found that most frequent problems in manuscripts were errors in data presentation and interpretation, data analysis, and study design. Some errors could be considered relatively unimportant, but errors in study design usually have far-reaching implications on the study results, whose primary and only purpose is to show us “the truth”, the reality.
Fortunately, the most common statistical errors in published biomedical research papers are relatively simple, easy to find, and can be identified by a scientist with basic statistical knowledge (9,10). They range from using standard error of the mean instead of standard deviation in descriptive statistic, to reporting only P values without any other statistical data, to using linear regression analysis without previous confirmation of linear relationship, to no adjustment for multiple comparisons, to using figures and tables only to store data rather than assist the readers (9). In their study, published in this issue of Biochemia Medica journal, Simundic and Nikolac observed that a substantial proportion of manuscripts submitted and considered for publication in Biochemia Medica, involved at least some statistical error (11).
Pure listing of data with the sole purpose of presenting what was measured is sometimes called data torture (1) and is also considered inadequate.
Altman has grouped typical statistical errors into four categories (5):
1. errors in study design (e.g., no randomization in controlled trials; inappropriate control group),
2. errors in data analysis (e.g., unpaired test for paired data),
3. errors in data presentation (e.g., standard error instead of standard deviation to describe data; pie charts to present distribution of continuous variables), and
4. errors in data interpretation (e.g., “caused if associated” type of reasoning; interpretation of poorly done study as a well done one).
Recent analyses of scientific papers have indicated that the quality of statistical analysis in published research is often below satisfactory level. For example, Altman found that only 30–60% of papers published in selected biomedical journals from 1966 to 1996 used acceptable statistical methodology (5). Other authors reported even lower percentage, i.e., <15% (12).
Even this occasional report confirms that methodological errors – major and minor, made advertently or accidentally – are more common than not and that journals should consistently require statistical review of manuscripts considered for publication.
Statistical review
In general, there are three levels of manuscript review that should be taken into account by a journal editor before reaching a clear decision on whether or not to publish a manuscript:
- professional peer review,
- ethics review, and
- statistical review (including review of both statistical and epidemiological methodology).
Peer review is provided by scientists, professionals, or specialists with expertise in the subject to be reviewed. Although far from being perfect, peer review is still considered the best possible method for evaluation of the scientific quality of a study (13).
Ethics review is usually provided by independent ethics committees, also called institutional review boards or ethical review boards. Ethical issues fall in the domain of ethics, morality, and law. Statisticians are sometimes members of ethics committees that review biomedical research proposals. However, statistics and ethics deal with different problems, topics, and questions, which is the reason why statistical and ethical aspects of a study are reviewed separately (14).
Statistical review is provided by statisticians in a form of a written report containing clear and straightforward suggestions and comments for both journal editors and authors (5). A statistical reviewer reads a paper throughout, from the title and abstract, to the body text, to tables, figures, and references and makes notes on anything that requires clarification or explanation, or wherever a question may be raised in the text or data. For reports of randomized controlled trials (RCT), most biomedical journal editors also ask to see the protocol of a study (e.g., in The Lancet and Lancet specialty journals). The reviewer first reads the protocol of the study, and then reviews the report and evaluates if all objectives from the protocol were achieved correctly and reported as expected (the RCT report must fully match the protocol).
Statistical reviews typically contain methodological and statistical questions that should be addressed by the author. If both the study and the manuscript are considered statically acceptable, the statistical reviewer may suggest acceptance of the manuscript. If statistical errors mostly pertain to the presentation of methodology and data, a statistical reviewer may provide specific suggestions for the author on how to improve the manuscript. Such a practice contributes to a quicker preparation of the manuscript for publication. Nevertheless, the final acceptance of the paper is contingent on the corrections made by the author according to the statistical reviewer’s suggestions.
If errors were made in data analysis, data interpretation, and discussion of the results, the statistical reviewer usually requires more extensive changes to be made thorough the paper and asks the author to completely reanalyze the data and make a new concept of the report. However, if errors were made in the study design, it is rarely possible to accept the report, as such errors cannot be corrected without repeating the whole study. In such a case, the statistical reviewer mostly suggests to the editor not to accept the paper.
To provide a systematic and comprehensive statistical review, even a senior and skilful biomedical statistician needs guidelines, a checklist of instructions that would remind her or him on all points that should be evaluated during “statistical reading” of the paper. An example of a checklist, which I have been using recently, is presented in Table 1. The tips for reviewers are listed in five clusters. General comments refer to the IMRaD paper as a whole, whereas study design refers particularly to Introduction and Methods sections, methodology and data analysis the Methods section, data presentation to the Results section, and data interpretation to the Discussion section.
Reviewer’s suggestions and comments are usually divided in major and minor points. Major points usually refer to key errors and ignorance of basic statistical rules. Minor points list minor errors, mistakes, bugs, insufficiently explained results, or poor reporting.
Suggestions for authors
There are two major suggestions for authors: one is to gain appropriate knowledge of statistical methodology before making the final study protocol, and the other is to use the same checklists while preparing the manuscript as those used by statistical reviewers (Table 1).
Table 1. Checklist for editing and reviewing statistical and epidemiological methodology in biomedical research papers, according to the suggestions published in biomedical literature (5,7,15-25), including Biochemia Medica (26-29).
Many great books on biostatistical methodology have been published. I would like to recommend two, “Basic and clinical biostatistics” by Beth Dawson and Robert Trapp (30) and “Essential statistics for the pharmaceutical sciences” by Philip Rowe (31). Both textbooks consist of typical biostatistical lectures and both describe typical errors. In the latter textbook, “pirate boxes” – (code 78 in windings character set) – are used to alert the reader to the risk of misuse of methodology. One pirate box indicates minor possible hazard, two indicate moderate, and three severe hazard. For example, breaking one set of data into several subgroups, analyzing each subgroup of data, and then publishing only the subgroup results that were significant is considered a blatant fault and is marked by three pirate boxes (31). Such visual marking of possible errors is a good practice and makes learning more practical.
Some statistical textbooks, such as “Methodological Errors in Medical Research” by Born Andersen (32), provide examples of incorrect use of statistical methods found in published scientific papers. Other, such as Huff’s “How to lie with statistics” (33) do not even attempt to fight against flagrant misuse of statistics and bring only systematized recommendations in an obviously sarcastic tone.
Instead of conclusion
In conclusion, instead of using my own words, I would like to quote from the “Ethical Guidelines for Statistical Practice” of the American Statistical Association Committee on Professional Ethics: “The professional performance of statistical analyses is essential to many aspects of society. The use of statistics in medical diagnoses and biomedical research may affect whether individuals live or die, whether their health is protected or jeopardized, and whether medical science advances or gets sidetracked. Life, death, and health, as well as efficiency, may be at stake in statistical analyses…” (22). This extract indeed summarizes the reasons why the statistical reviewer does the work. No further explanation seems necessary. The rest is up to authors – to follow the rules, the simple rules of perfect statistical methodology.
Acknowledgment
I thank dr Aleksandra Mišak for her valuable help with copyediting the final text.
