In the era of evidence-based medicine, randomized controlled trials (RCTs) are the highest ranked means of research and are considered as “the most powerful tool in modern clinical research.”1 Even if recent research methods, such as meta-analyses and umbrella meta-analyses, provide more accurate data, the quality of RCTs remains central, as they represent the structural element of the aforementioned research methodologies. The process of randomization in clinical trials guarantees that the significant findings in the group comparisons regarding the matter under examination can be accredited to the intervention and not to other confounding factors,2 thus minimizing bias. However, available information in biomedical journals may be concealing a wide range of biases, such as publication, selection, and funding biases.3,4,5 It is crucial for readers and other researchers to be aware of the quality standards of RCTs so as to properly assess their strong and weak points.6,7
It is clear that evidence-based healthcare relies, among others, on the quality assessment of RCTs. The reporting of RCTs, as a type of medical literature, has in general got a certain form and adopted a basic structure that is more or less used by the majority of authors. However, several strategic methodological characteristics may be overlooked in published reports and, therefore, the assessment of reporting quality may be used as a surrogate for trial quality.8 A trial with inadequate reporting quality may be misclassified, while an unclear and inaccurate description may be attempting to conceal erroneous methods.7,9,10
Time and time again, evidence has stated that the reporting quality of RCTs is not optimal.11,12 Addressing these concerns, at the end of the 20th century, an international group of trial methodology experts, including statisticians, epidemiologists, science journal editors, and clinical trialists, created the Consolidated Standards of Reporting Trials (CONSORT) Statement, which was last updated in March 2010.This statement is an evidence-based set of recommendations, including a checklist and a flow diagram, for the reporting of RCTs; its aim is to promote the thorough and transparent reporting of trials while simultaneously assisting in their critical evaluation and understanding.11 Recommendations concern the title, introduction, methods, results, and other information that should be reported and cover virtually every aspect of conducting an RCT according to Good Clinical Practice. Separate items cover the methodology of the trials (randomization methods, blinding, etc.), the statistical analysis, the outcomes, as well as other information like the funding source and the online access of the study protocol. CONSORT stresses the importance of integrality, preciseness, and transparency of reporting, which are, all in all, indicative of the actual trial plan and conduct.13
A number of publications have examined the reporting quality of RCTs in several subspecialties of medicine.14,15,16,17,18,19,20,21 In a recent study by our team, concerning multiple sclerosis, the mean CONSORT compliance was found to be considerably low, since only 30% of the studies reported more than 75% of the CONSORT items.21 Similar studies in polycystic ovary syndrome, urology, and other surgical specialties also found the CONSORT compliance to be suboptimal,16,20 reflecting the intersectionality of the issue in the medical field.
Restless legs syndrome (RLS) is a relatively common neurological disease,22,23 which can be primary or secondary.24, 25, 26 The unpleasant sensation of RLS and the subsequent sleep disturbance have a significant socioeconomic impact on patients.27, 28 Following the ongoing series of reporting quality analyses in neurological diseases, in the present study, we analyzed the reporting quality of RCTs involving patients with RLS (primary or secondary), using the items of the revised CONSORT 2010 statement checklist.13 The period covered by this report extends from 1998 to 2017.
We searched the PubMed database for RCTs on patients with RLS published between the January 1, 1998 and December 31, 2017. As a search criterion, the phrase “Restless Legs Syndrome” or “RLS” or “Willis–Ekbom disease” was used. We used the “Randomized Controlled Trial” type of article, “English” language, and “Humans” for species as filters.
Trials were eligible for inclusion if the participants had been randomly assigned to at least two medicinal treatment arms (including iron and vitamin supplementation) and included patients with RLS regardless of the etiology (primary or secondary). Reports of trials without RLS-relevant symptoms as end points, dose comparison studies, those with non-medicinal regimens, and small pilot studies, alongside any article with information resulting from a previously conducted trial (such as post hoc or sub-group analysis and various sub-studies), were excluded. Crossover studies were also excluded due to their specific design.29
The CONSORT 2010 checklist consists of 25 items, which, alongside their sub-items, reach a total number of 37 items. This checklist is recommended to be used by authors during the reporting of RCTs to promote thorough work and transparency in their presentation. Alongside the explanation and elaboration document, CONSORT provides instructions for every section of an RCT article (title, introduction, methods, results, and funding) and covers every aspect of a well-conducted clinical trial (Supplementary File 2). We have included an additional item (No 13), whether an article included a participant flow diagram or not, in order to pertain to the authors’ (strong) recommendation of using one. The CONSORT explanation and elaboration document (available at the CONSORT website) was used to guide our evaluation process. Out of the total 54 eligible trials, 28 were published before 2010 and 26 were published after 2010, with 2010 being the year when the CONSORT Statement was revised. We used the revised CONSORT version for all the extracted articles. The full CONSORT checklist can be accessed online at http://www.consort-statement.org.
During the evaluation process, each author reviewed the selected articles one by one and assigned a positive (“yes”) or a negative (“no”) response to each CONSORT item, according to whether it was reported or not. Additionally, the following procedures were followed: (1) all of the checklist items were searched for into the published trials in terms of whether they were reported, and not if they were actually performed during the trial. In cases where a methodology followed by the trials’ authors was insinuated in the results or other sources, but there was no lucid reference in the article (or the supplementary files), the CONSORT item was marked as “non-reported.” For example, if a trial stated on its title that it is a “double-blinded” trial but there was no clear reference in the methodology section as for how it was blinded, the respective items were checked as “non-reported.” (2) In cases where it was not obvious whether the criterion of reporting was met, after consensus among all authors, the item was assigned a negative mark (non-reported), since they may conceal important information. (3) In cases where a procedure of the trial was not mentioned in the main manuscript of the trial but there was a reference to it in a supplementary file which addressed this information, the procedure was considered as adequately reported. This rule was not applied to item 8a, where the CONSORT Explanation and Elaboration Document specifically requires that “information on the process of randomization is included in the body of the main article and not as a separate supplementary file; where it can be missed by the reader.”30 Evaluation was made by two independent authors (D.R. and E.D.), and discrepancies were resolved by consensus among all authors. The articles were divided into three groups depending on the time of publication: from 1998 to 2004, from 2005 to 2009, and from 2010 to 2017. We also categorized the reported items into five groups as follows: (1) Title/Abstract and Introduction, (2) Methods, (3) Results, (4) Discussion, and (5) Other information.
The basic quality-of-reporting metric was the “CONSORT compliance,” meaning the percentage of the 38 CONSORT items that each article addressed. We calculated the greater than 75% compliance with the CONSORT statement items, as compliance with the CONSORT items to the extent of greater than 75% was considered an adequate cutoff in a number of other studies.14,31 This metric was considered as a measure of each article’s reporting quality. It must be noted at this point that items 3b (changes to methods), 6b (changes to trial outcomes), 7b (interim analyses and stopping guidelines), and 14b (why the trial had to be ended or terminated) are items that were not applicable in most of the trials. Item 17b (effect sizes of binary outcomes), though an important item in RCTs in general, may not be applicable to RLS trials where the most commonly used outcomes are categorical (e.g., IRLS SG score and polysomnographic data); hence, it is also justifiably under-reported. By removing the five aforementioned items, a total of 86.6% compliance was considered to be the maximum that an article could reach.
The comparison of the >75% compliance among the different time period groups was made using the Pearson chi-square statistic for a 2×3 table. We calculated the percentage of items per group (title and abstract, methods, results, discussion, and other) that was reported in at least 75% of the articles for the total and individual time periods. This metric is an indicator of which CONSORT items were adequately reported or under-reported by the articles in comparison to the compliance, which is a measure of each article’s reporting quality. We further calculated the median CONSORT compliance of the articles published in journals with a current impact factor greater than 7, between 2 and 7, and lower than 2, and additionally performed a Kruskal–Wallis non-parametric test to compare the three groups. Journal impact factor was accessed through the Thomson Reuters Journal Citation Reports website,32,33 and all statistical analyses were made using the IBM SPSS v.21 package.
The evaluation process was carried out in four steps, as can be seen in the search flow chart (Figure 1). Initial search for entries meeting the set criteria returned 307 related articles, whose titles were scrutinized. One hundred and eighty articles were excluded due to irrelevance, use of non-medicinal intervention (behavioral treatment, exercise, herbal), or not referring to randomized trials. The abstracts of the remaining 127 articles were reviewed and an additional 52 articles were excluded for the same reasons. The remaining 75 articles were retrieved in full text, 21 of which were found ineligible for reasons explained before, and finally 54 articles were included in the study. A list of these 54 RCTs, which included a total of 10,427 randomized patients, can be found in the Supplementary file.
Of the total 54 eligible trials, 9 were published between 1998 and 2004, 19 between 2005 and 2009, and 26 between 2010 and 2017, when the revised CONSORT version had been made available. The drugs in comparison against placebo or another active component were gabapentin (8 studies), cabergolide (2 studies), bupropion (1 study), iron (7 studies), oxycodone (1 study), pergolide (2 studies), pramipexole (12 studies), pregabaline (3 studies), ropinirole (9 studies), rotigotine (6 studies), sumanirole (1 study), vitamins (1 study), and xp13512 (1 study).
Twenty-one different scientific journals hosted the included articles. Nine of them are currently CONSORT-endorsing, corresponding to 76% (41) of the articles. The mean CONSORT compliance of articles published in these journals reached 58.47%. The remaining 13 articles, published in 12 non-endorsing journals, had a mean CONSORT compliance of 50%. The median CONSORT compliance scores of articles published in low (IF<2), medium (IF 2-7), and high-ranked (IF >7) journals were 52.63, 56.57, and 59.21%, respectively. These percentages were not found to be significantly different (Kruskal-Wallis (K-W) p = 0.272).The eligible RCTs were relatively small, with an average count of randomized patients of 196. Fifteen of them (27.7 %) randomized fewer than 100 patients.
Table 1 shows the percentage of articles that reported each individual item divided into publishing period groups. The numbers and percentages of CONSORT items reported by >75% of the articles by time period and by group are shown in Table 2.
|Data Item||Combined 1998–2016 (n = 54)||1998–2004 (n = 9)||2005–2009 (n = 19)||2010–2016 (n = 26)|
|Period/Checklist Items||1998–2016 (54 RCTs) N(%)||1998–2004 (9 RCTs) N(%)||2005–2009 (19 RCTs) N(%)||2010–2016 (26 RCTs) N(%)|
|Overall (38)||14 (36.8)||11 (28.9)||13 (34.2)||17 (44.7)|
|Title/abstract – intro (4)||3 (75)||3 (75)||3 (75)||4 (100)|
|Methods (17)||3 (17.6)||3 (17.6)||4 (23.5)||4 (23.5)|
|Results (11)||5 (45.5)||3 (27.3)||4 (36.4)||6 (54.5)|
|Discussion (3)||2 (66.6)||2 (66.6)||1 (33.3)||2 (66.6)|
|Other information (3)||1 (33.3)||0 (0)||1 (33.3)||1 (33.3)|
During the whole examined period of 1998–2016, 14 items (36.8%) were found reported in >75% of the articles, whereas the respective numbers for the 5-year periods were 11 (28.9%) in 1998–2004, 13 (34.2%) in 2005–2009, and 17 (44.7%) in 2010–2016, showing a very small, insignificant increase in the reporting of CONSORT items (p = 0.344).
The mean CONSORT compliance score was 56.5% (23.68–84.21%). The RCTs that covered more than 75% of the CONOSRT items (by time period) were as follows: 5 (9.25%); 1998–2004: 0; 2005–2009: 3(15.79%); 2010–2016: 2 (7.69%).
The conclusion that this study seems to reach is that the reporting of RLS-related RCTs is far from optimal. Furthermore, it provides a general view of RLS RCTs and its methodological components.
The number and size of RLS RCTs is smaller than that of other neurological diseases. We included 54 RCT articles referring to 10,427 randomized patients. In comparison to a recent similar study carried out by our group on the subject of multiple sclerosis RCTs, we found 103 studies regarding 42,031 patients21 for about the same time period (2000–2015). This is also made evident by the mean number of patients randomized in these 53 studies, which was less than 200 (193).
Only 14 of the 38 checklist items (36.8%) were addressed in 75% or more of the studies published between 1998 and 2016. There was a slight increase in checklist item reporting between the three time periods. Some of the underreported items were –in most cases – non-applicable (3b: important changes to methods; 6b: changes to trial outcomes; 7b: interim analyses and stopping guidelines). Other items underreported, though, contain important methodological information regarding randomization (9: allocation concealment method – 30%; 10: implementation – 26%). The allocation concealment method (concealment of the allocated intervention at the time of enrollment) and implementation (who generated the allocation sequence, who enrolled participants, and who assigned participants to trial groups) is an important piece of methodological information since trials in which the allocation sequence had been inadequately or unclearly reported yielded larger estimates of treatment effects than trials in which authors adequately reported allocation concealment.13,32,34 Item 24 (where the full trial protocol can be accessed, if available) was the most underreported item (0%). The overall CONSORT compliance was generally found to be poor as well, with the mean compliance calculated to be 56.6%, which is considerably low and, sadly, not showing signs of improvement over time. When considering the five usually not applicable items in RLS RCTs, an adjusted overall CONSORT compliance would be 71%, which is similar with the one found in our study regarding multiple sclerosis (average CONSORT compliance 68.2%). This finding, though, should be regarded with some degree of skepticism since not all studies included non-applicable items.
There are 585 journals currently endorsing CONSORT. In our study, 9 out of 21 journals that published RLS RCTs were CONSORT endorsers, and those articles’ compliance was larger – though insignificantly – than their non-endorsing counterparts (58.27% vs. 50.4%). At the same time, high impact factor journals had a better reporting, but it still did not reach the significance threshold.
The results of this study must be interpreted with caution and some points need to be addressed. First, we used the 2010 revised CONSORT checklist for all RCTs regardless of their publication date. Furthermore, the allocation of a negative or positive response on the used checklist has not always been clear and straightforward, making it susceptible to subjectivity. In addition, most of the CONSORT items have a title alongside a large paragraph of explanation and information that should be used while reporting an RCT. The CONSORT checklist was made to help researchers report studies in a more thorough and detailed manner rather than being used as an evaluation tool. Consequently, the evaluation of reporting is a more critical and delicate procedure.
Taking into consideration the aforementioned limitations and the data presented here, it is our feeling that RLS-related RCTs are in general badly reported, regardless of the time period, the quality of the publishing journal, and its CONSORT endorsement. To the extent that reporting reflects study design and conducting, along with the fact that RLS RCTs are relatively small and few in number, while also regarding many different medicinal products, this study raises important concerns. RLS is a disease whose management is troublesome and tricky, and the related research seems to have many methodological pitfalls; hence, designing, performing, and reporting RCTs all need to be as detailed and accurate as possible.
1 Citation: Rikos D, Dardiotis E, Aloizou A-M, Siokas V, Zintzaras E, Hadjigeorgiou GM. Reporting Quality of Randomized Controlled Trials in Restless Legs Syndrome Based on the CONSORT Statement. Tremor Other Hyperkinet Mov. 2019; 9. 10.7916/d8-0f2v-aq62
2 Editor: Elan D. Louis, Yale University, USA
3 Funding: None.
4 Financial Disclosures: None.
5 Conflicts of Interest: The authors report no conflicts of interest.
6 Ethics Statement: This study was performed in accordance with the ethical standards detailed in the Declaration of Helsinki. The authors’ institutional ethics committee has approved this study and all patient provided written informed consent.
Nystrom, L , Rutqvist, LE , Wall, S , Lindgren, A , Lindqvist, M , Ryden, S , et al. Breast cancer screening with mammography: overview of Swedish randomised trials. Lancet 1993;341(8851):973–978. doi:https://doi.org/10.1016/0140-6736(93)91067-V8096941
Stolberg, HO, , Norman, G , Trop, I, Randomized controlled trials. AJR 2004;183(6):1539–1544. doi: https://doi.org/10.2214/ajr.183.6.0183153915547188
Pfeifer, MP and , Snodgrass, GL, The continued use of retracted, invalid scientific literature. JAMA 1990;263(10):1420–1423. doi:https://doi.org/10.1001/jama.1990.034401001400202406475
Olkin, I Meta-analysis: reconciling the results of independent studies. Stat Med 1995;14(5–7):457–472. doi:https://doi.org/10.1002/sim.47801405077792440
Every-Palmer, S and , Howick, J, How evidence-based medicine is failing due to biased trials and selective publication. J Eval Clin Pract 2014;20(6):908–914. doi: https://doi.org/10.1111/jep.1214724819404
Moher, D , Pham, B , Jones, A , Cook, DJ , Jadad, AR , Moher, M , et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998;352(9128):609–613. doi: https://doi.org/10.1016/s0140-6736(98)01085-x9746022
Schulz, KF, , Chalmers, I and , Hayes, RJ, , Altman, DG, Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995;273(5):408–412. doi:https://doi.org/10.1001/jama.1995.035202900600307823387
Huwiler-Muntener, K, , Juni, P and , Junker, C, , Egger, M, Quality of reporting of randomized trials as a measure of methodologic quality. JAMA 2002;287(21):2801–2804. doi:https://doi.org/10.1001/jama.287.21.280112038917
Juni, P, , Altman, DG , Egger, M, Systematic reviews in health care: assessing the quality of controlled clinical trials. BMJ 2001;323(7303):42–46. doi: https://doi.org/10.1136/bmj.323.7303.4211440947
Liberati, A, , Himel, HN , Chalmers, TC, A quality assessment of randomized control trials of primary treatment of breast cancer. J Clin Oncol 1986;4(6):942–951. doi: https://doi.org/10.1200/JCO.19188.8.131.5223711962
Turner, L , Shamseer, L , Altman, DG , Weeks, L , Peters, J , Kober, T , et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst Rev 2012;11:Mr000030. doi: https://doi.org/10.1002/14651858.MR000030.pub223152285
Chan, AW and , Altman, DG, Epidemiology and reporting of randomised trials published in PubMed journals. Lancet 2005;365(9465):1159–1162. doi: https://doi.org/10.1016/s0140-6736(05)71879-115794971
Schulz, KF, , Altman, DG , Moher, D, CONSORT 2010 Statement: updated guidelines for reporting parallel group randomised trials. BMC Med 2010;8:18. doi: https://doi.org/10.1136/bmj.c33220334633
Ziogas, DC and , Zintzaras, E, Analysis of the quality of reporting of randomized controlled trials in acute and chronic myeloid leukemia, and myelodysplastic syndromes as governed by the CONSORT statement. Ann Epidemiol 2009;19(7):494–500. doi: https://doi.org/10.1016/j.annepidem.2009.03.01819523596
Lai, TY, , Wong, VW, , Lam, RF and , Cheng, AC, , Lam, DS, , Leung, GM, Quality of reporting of key methodological items of randomized controlled trials in clinical ophthalmic journals. Ophthalmic Epidemiol 2007;14(6):390–398. doi: https://doi.org/10.1080/0928658070134439918161613
Agha, R, , Cooper, D , Muir, G, The reporting quality of randomised controlled trials in surgery: a systematic review. Int J Surg 2007;5(6):413–422. doi: https://doi.org/10.1016/j.ijsu.2007.06.00218029237
Hill, CL, , Buchbinder, R , Osborne, R, Quality of reporting of randomized clinical trials in abstracts of the 2005 annual meeting of the American College of Rheumatology. J Rheumatol 2007;34(12):2476–2480.17985398
Mills, E , Loke, YK , Wu, P , Montori, VM , Perri, D , Moher, D , et al. Determining the reporting quality of RCTs in clinical pharmacology. Br J Clin Pharmacol 2004;58(1):61–65. doi: https://doi.org/10.1111/j.1365-2125.2004.2092.x15206994
Partsinevelou, A and , Zintzaras, E, Quality of reporting of randomized controlled trials in polycystic ovary syndrome. Trials 2009;10:106. doi: https://doi.org/10.1186/1745-6215-10-10619930576
Rikos, D, , Dardiotis, E, , Tsivgoulis, G , Zintzaras, E, , Hadjigeorgiou, GM, Reporting quality of randomized-controlled trials in multiple sclerosis from 2000 to 2015, based on CONSORT statement. Mult Scler Relat Disord 2016;9:135–139. doi: https://doi.org/10.1016/j.msard.2016.07.013
Allen, RP, , Bharmal, M , Calloway, M, Prevalence and disease burden of primary restless legs syndrome: results of a general population survey in the United States. Mov Disord 2011;26(1):114–120. doi: https://doi.org/10.1002/mds.2343021322022
Allen, RP , Picchietti, DL , Garcia-Borreguero, D , Ondo, WG , Walters, AS , Winkelman, JW , et al. Restless legs syndrome/Willis-Ekbom disease diagnostic criteria: updated International Restless Legs Syndrome Study Group (IRLSSG) consensus criteria--history, rationale, description, and significance. Sleep Med 2014;15(8):860–873. doi: https://doi.org/10.1016/j.sleep.2014.03.02525023924
Allen, RP, , Stillman, P , Myers, AJ, Physician-diagnosed restless legs syndrome in a large sample of primary medical care patients in western Europe: prevalence and characteristics. Sleep Med 2010;11(1):31–37. doi: https://doi.org/10.1016/j.sleep.2009.03.00719464949
Stefanidis, I , Vainas, A , Dardiotis, E , Giannaki, CD , Gourli, P , Papadopoulou, D , et al. Restless legs syndrome in hemodialysis patients: an epidemiologic survey in Greece. Sleep Med 2013;14(12):1381–1386. doi: https://doi.org/10.1016/j.sleep.2013.05.02224210601
Stefanidis, I , Vainas, A , Giannaki, CD , Dardiotis, E , Spanoulis, A , Sounidaki, M , et al. Restless legs syndrome does not affect 3-year mortality in hemodialysis patients. Sleep Med 2015;16(9):1131–1138. doi: https://doi.org/10.1016/j.sleep.2015.04.02326298790
Hadjigeorgiou, GM , Stefanidis, I , Dardiotis, E , Aggellakis, K , Sakkas, GK , Xiromerisiou, G , et al. Low RLS prevalence and awareness in central Greece: an epidemiological survey. Eur J Neurol 2007;14(11):1275–1280. doi: https://doi.org/10.1111/j.1468-1331.2007.01966.x17956448
Schormair, B , Plag, J , Kaffe, M , Gross, N , Czamara, D , Samtleben, W , et al. MEIS1 and BTBD9: genetic association with restless leg syndrome in end stage renal disease. J Med Genet 2011;48(7):462–466. doi: https://doi.org/10.1136/jmg.2010.08785821572129
Nolan, SJ, , Hambleton, I , Dwan, K, The use and reporting of the cross-over study design in clinical trials and systematic reviews: a systematic assessment. PLoS One 2016;11(7):e0159014. doi: https://doi.org/10.1371/journal.pone.015901427409076
Begg, C , Cho, M , Eastwood, S , Horton, R , Moher, D , Olkin, I , et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996;276(8):637–639. doi:https://doi.org/10.1001/jama.1996.035400800590308773637
Kober, T, , Trelle, S , Engert, A, Reporting of randomized controlled trials in Hodgkin lymphoma in biomedical journals. J Natl Canc Inst 2006;98(9):620–625. doi: https://doi.org/10.1093/jnci/djj160
Pildal, J, , Hrobjartsson, A, , Jorgensen, KJ and , Hilden, J, , Altman, DG, , Gotzsche, PC, Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007;36(4):847–857. doi: https://doi.org/10.1093/ije/dym08717517809
Wood, L , Egger, M , Gluud, LL , Schulz, KF , Juni, P , Altman, DG , et al. Empirical evidence of bias in treatment effect estimates in controlled trials with different interventions and outcomes: meta-epidemiological study. BMJ 2008;336(7644):601–605. doi: https://doi.org/10.1136/bmj.39465.451748.AD18316340