Quality assessment of systematic reviews and meta-analyses published in Saudi journals from 1997 to 2017 ======================================================================================================== * Zuhair S. Natto * Doaa S. AlGhamdi ## Abstract **Objectives:** To assess the quality of the meta-analyses (MAs) and systematic reviews (SRs) in Saudi journals indexed in PubMed using 2 scales: A MeaSurement tool to assess systematic reviews (AMSTAR) and the overview quality assessment questionnaire (OQAQ). **Methods:** This study focused on SRs/MAs published in 8 Saudi journals. We investigated, screened, and extracted the data, which included recording the main topic of each SRs/MAs and the date of publication. Furthermore, we assessed the quality of each included SRs/MAs using the AMSTAR and the OQAQ. The reviews concluded in January 2018. **Results:** The search uncovered 201 unique articles; of these, the researchers screened 110 full texts and included 103 in this review. Most of the included studies were published in Saudi Medical Journal (50 articles, 48.5%), followed by Saudi Journal of Gastroenterology (21 articles, 20.4%), and Annals of Saudi Medicine (16 articles, 15.5%). The main topics in these published articles were gastroenterology (20 articles, 19.5%), followed by oncology (14 articles, 13.7%), and pharmacology (9 articles, 8.7%). The AMSTAR and the OQAQ scales showed that most SRs/MAs were of medium quality. **Conclusion:** Quality of SRs and MAs published in Saudi journals was distributed in all categories (low, medium, and high) and it can be improved using critical evaluation by authors, journal editors, and readers. **PROSPERO REG. NO. CRD: 42018102210** The concept of evidence based medicine (EBM) was first introduced in 1992 and published in JAMA.1 This concept integrates clinical expertise with the best evidence available. One such method is conducting high-level systematic reviews (SRs) and meta-analyses (MAs). Clinicians can use them to access and implement the most up-to-date, precise, and reliable information available on a specific topic.2,3 These reviews provide a comprehensive coverage of the literature based on searches through Scopus, Web of Science and evidence-based bibliographic specialist databases.4,5 Saudi journals publish several MAs/SRs published every year. However, it may contain inappropriate or poorly-designed methodologies, biased information, or unreliable conclusions.6-8 There are several ways to check the quality of these SRs/MAs, such as Sack’s quality assessment (1988), overview quality assessment questionnaire (OQAQ) (1991),2 and A MeaSurement Tool to Assess Systematic Reviews scale (AMSTAR) (2007).3 Additionally, Critical Appraisal Skills Programme (CASP),9 the National Institute for Health and Care Excellence (NICE),10 and the Joanna Briggs Institute System for the Unified Management, Assessment and Review of Information (JBI-SUMARI) checklists or scales11 were developed by researchers to assess the quality of SRs/MAs published across a variety of disciplines, such as surgery,12,13 neuropathic pain, pulmonary disease,14 hand surgery,15 telerehabilitation, health literacy,16 nursing,17 cancer screening,18,19 and gastroenterology and hepatology.8,20-23 There are several SRs/MAs that have been published in Saudi journals. To assess the SRs/MAs in these journals, we evaluated the quality of articles appearing in 8 Saudi journals indexed in PubMed using 2 scales: OQAQ and AMSTAR. Thus, the aim of this review was to assess the quality of SRs/MAs published in Saudi journals using AMSTAR and OQAQ scales. The PICOS were as follows: P (population), articles published in Saudi journals indexed in PubMed using; I (interventions), OQAQ; C (comparisons), AMSTAR; O (Outcomes): quality scores; and S (study design), MA/SR. ## Methods We reviewed all SRs/MAs obtained through 3 databases: MEDLINE via PubMed, Scopus, and Web of Science. The screening focused on Saudi journals with universal access, which was limited to 8 journals indexed in PubMed and included the following: Annals of Saudi Medicine, Saudi Dental Journal, Saudi Journal of Gastroenterology, Saudi Journal of Anaesthesia, Saudi Journal of Biological Sciences, Saudi Journal of Kidney Diseases and Transplantation, Saudi Medical Journal, and Saudi Pharmaceutical Journal. These journals are all published in English therefore, all searches were carried out in English. We improved database searches using the screening bibliographies and the citations of included articles on Google Scholar. For each article, the investigators reviewed and recorded the abstract (main topic), bibliography, and date of publication. The reviews were concluded in January 2018 and the protocol was listed in PROSPERO (Reg. No. CRD42018102210). During the search, the investigators uncovered 201 unique articles; of these, 110 full texts were screened and 103 were included in this study and followed the PRISMA check list (**Figure 1** and **Table 1**). We evaluated the quality of each selected MA/SR using 2 scales. The first was the AMSTAR scale,3 which has 11 items divided as high (range 9-11), moderate (range 5-8), or low (range 0-4).24,25 The second scale was OQAQ. It is 10 items classified as high (7-9), moderate (4-6), or low (≤3).18,26 *P*<0.05 was statistically significant. Assessment of consistency was performed using McNemar-Bowker test. Any disagreement in the results achieved by the 2 investigators was resolved by discussion. ![Figure 1](http://smj.org.sa/https://smj.org.sa/content/smj/40/5/426/F1.medium.gif) [Figure 1](http://smj.org.sa/content/40/5/426/F1) Figure 1 PRISMA flow diagram used in this review. View this table: [Table 1](http://smj.org.sa/content/40/5/426/T1) Table 1 Frequency of systematic reviews and meta-analyses published in each selected Saudi journal from 1997 to 2017 (N=103). ## Results Majority of the included studies were published in Saudi Medical Journal (50 articles, 48.5%), followed by Saudi Journal of Gastroenterology, (21 articles, 20.4%), and Annals of Saudi Medicine (16 articles, 15.5%). **Table 1** presents an overview for the frequency of SRs/MAs published in each selected Saudi journal. The number of publications was much higher after 2010 when compared to previous years’ publications, as shown in **Table 2**. The main topics in these published articles included gastroenterology (20 articles, 19.5%), oncology (14 articles, 13.7%), and pharmacology (9 articles, 8.7%). **Table 3** summarized the complete distribution of articles by topic. **Figure 2** provides proportion of SRs/MAs satisfying the standards of each of the AMSTAR items. Eighty-four of the 103 reviews conducted with the AMSTAR scale implemented a comprehensive literature search (Q3), presence of statement of conflict of interest (Q11, 61 reviews), and a priori design (Q1, 55 reviews). The source of publication (gray literature or peer review) (Q4) was the least frequently included criteria among the reviews (10 reviews), followed by the list of included or excluded studies (Q5, 18 reviews) and suitable quality of the included studies used in the final conclusions (Q8, 36 reviews). The AMSTAR score for each article included are shown in **Table 3**. View this table: [Table 2](http://smj.org.sa/content/40/5/426/T2) Table 2 Frequency and percentage of systematic reviews and meta-analyses sorted by year of publication (N=103). View this table: [Table 3](http://smj.org.sa/content/40/5/426/T3) Table 3 Frequency and percentage of systematic reviews and meta-analyses sorted by main topic (N=10). ![Figure 2](http://smj.org.sa/https://smj.org.sa/content/smj/40/5/426/F2.medium.gif) [Figure 2](http://smj.org.sa/content/40/5/426/F2) Figure 2 Proportion of systematic reviews and meta-analyses satisfying the standards of each of the A MeaSurement Tool to Assess Systematic Reviews items. **Figure 3** provides proportion of SRs/MAs complying to the standards of each of the OQAQ items. Of the 103 reviews conducted with the OQAQ scale, 95 reviews incorporated main conclusions related to the primary questions (Q9), followed by the search strategy (Q1, 90 reviews), and the characteristics of the included studies (Q3, 90 reviews). The validity assessment of the included studies (Q5, 11 reviews), comprehensive search for evidence (Q2, 29 reviews), and selection bias assessment (Q4, 43 reviews) were the least frequently included criteria among the reviews. The OQAQ score for each article included are shown in**Table 4**. ![Figure 3](http://smj.org.sa/https://smj.org.sa/content/smj/40/5/426/F3.medium.gif) [Figure 3](http://smj.org.sa/content/40/5/426/F3) Figure 3 Proportion of systematic reviews and meta-analyses satisfying the criteria of each of the overview quality assessment questionnaire questions. View this table: [Table 4](http://smj.org.sa/content/40/5/426/T4) Table 4 Frequency and percentage of systematic reviews and meta-analyses published in Saudi journals categorized by methodological quality. **Figure 4** illustrates the numbers of SRs/MAs published in Saudi journals. Both scales had the same frequency of medium quality of 44 (4.9 ± 2.5) versus 43 (5.1 ± 2.0) studies. Overall, they had the same mean within a moderate category. The OQAQ scale provided more articles with a high quality compared with AMSTAR (34 versus 10 studies), while AMSTAR was stricter in its criteria and resulted in a higher number of studies with low quality when compared to the QQAQ scale (49 versus 26 studies). McNemar-Bowker test was statistical significant of consistency (*p*<0.001) (**Table 4**). ![Figure 4](http://smj.org.sa/https://smj.org.sa/content/smj/40/5/426/F4.medium.gif) [Figure 4](http://smj.org.sa/content/40/5/426/F4) Figure 4 Numbers of systematic reviews and meta-analyses published in Saudi journals categorized by methodological quality using A MeaSurement Tool to Assess Systematic Reviews (AMSTAR)and overview quality assessment questionnaires (OQAQ). ## Discussion This review revealed that SRs/MAs published in Saudi journals had a range of quality based on scoring of the articles using the AMSTAR and OQAQ scales. The tools indicated that some studies were of high quality, while others scored in the moderate and low ranges. Most of the weaknesses in these reviews were source of publication (gray literature or peer review), list of included or excluded studies, appropriate quality of the included studies used in final conclusions, the validity assessment of the included studies, comprehensive search for evidence, and selection bias assessment. Most SRs/MAs had comparable issues depending on the field of study. In quality assessments of SRs/MAs in hand surgery, the lowest scoring items were Q11 (conflict of interest included), Q1 (a priori design provided) and Q10 (likelihood of publication bias assessed).15 While in health literacy and cancer screening, the main issues were protocol registration, sources of literature searches, and a list of excluded articles.16,18 Moreover, SRs/MAs implemented a comprehensive literature search, included a statement of conflict of interest, and utilized a priori design. They also offered main conclusions and useful information related to the primary questions, the search strategy, and the characteristics of the included studies. These criteria were comparable to other studies, such as hand surgery systematic reviews, which scored the highest on Q3 (comprehensive literature search), Q7 (scientific quality assessed), and Q9 (methods used to combine the findings appropriate).15 Evidence based medicine is very popular in all specialties. Gastroenterology was the most common area investigated.8,20-23 In gastroenterology, EBM is part of the training and curriculum for many programs worldwide.8 In fact, a review published on the quality of gastroenterology articles in Saudi journals found that 80.7% were level IV based on Oxford’s level of evidence scale.23 The most common study designs were cross sectional (33.9%), followed by case reports (27.9%), and case series (18.8%).8 A total of 127 meta-analyses were included in this study.23 The overall compliance with the PRISMA statement was 20.8 ± 4.2 and the AMSTAR scale was 7.6 ± 2.4. The following items were inadequately reported, such as: protocol or registration, explaining the methods; results of additional analyses, duplicating study selections and data extraction, list of included or excluded studies. Articles published in latest years revealed a significantly better methodological quality than those issued in the preceding years.23 Based on these results, editors should encourage authors to prepare high quality of SRs/MAs and they should check the quality of each manuscript before any decision. Moreover, a growing number of MAs/SRs have been issued in all areas. However, quality is always a concern. In an SR that investigated 127 MAs in main principal gastroenterology and hepatology journals, compliance with the AMSTAR checklist was moderate (7.6 ± 2.4) while the median for hand surgery was approximately 7. A MeaSurement Tool to Assess Systematic Reviews and OQAQ scales are a valid and a reliable tool to evaluate the quality of SRs/MAs. Following the PRISMA (preferred reporting items for SRs/MAs) statement, developed in 2009, is another way to improve the quality.27 There are several reasons that may contribute to the occurrence of low quality articles in Saudi journals. One is the low or no impact factor in Saudi journals that make the articles unappealing to well-known investigators.8 Other factors that may affect the quality of the articles are the background of the editors, process of publication, possibility of publishing supplemental materials, number of editions per year, mandatory Arabic summary, and presence of a statistician or epidemiologist on the board. There are several limitations to this review. It would be more accurate to analyze the quality of SRs in the journals indexed by MEDLINE, which is currently one of the few evidence-based biomedical databases. Some high-quality Saudi medical journals were not processed for this article accidently (Neurosciences) or because it was indexed in 2018 (Journal of infection and public health). However, few systemic reviews were published in these journals, which the overall effect might be minimal. In conclusion, the quality of SRs/MAs published in Saudi journals was distributed in all categories (low, medium, and high) using OQAQ and AMSTAR scales. This study shows that it is possible to improve the methodological quality of MAs in Saudi journals. Using several published and validated tools, and ensuring that authors, journal editors, and readers conduct evaluations that are more critical would aid in the improvement. Specifically, greater attention could be paid to the reporting methodology of SRs/MAs to have this category as one of the top methods to obtain reliable evidence in any medical field. ## Footnotes * **Disclosure**. Authors have no conflict of interests, and the work was not supported or funded by any drug company. * Received December 2, 2018. * Accepted March 20, 2019. * Copyright: © Saudi Medical Journal This is an open-access article distributed under the terms of the Creative Commons Attribution-Noncommercial-Share Alike 3.0 Unported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. ## References 1. (1992) Evidence-Based Medicine. A new approach to teaching the practice of medicine. Jama 268:2420–2425. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1001/jama.1992.03490170092032&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=1404801&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=A1992JV69400030&link_type=ISI) 2. Oxman AD, Guyatt GH (1991) Validation of an index of the quality of review articles. J Clin Epidemiol 44:1271–1278. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1016/0895-4356(91)90160-B&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=1834807&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=A1991GP82500017&link_type=ISI) 3. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 62:1013–1020. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1016/j.jclinepi.2008.10.009&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=19230606&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=000270250500004&link_type=ISI) 4. Gasparyan AY, Ayvazyan L, Blackmore H, Kitas GD (2011) Writing a narrative biomedical review:considerations for authors, peer reviewers, and editors. Rheumatol Int 31:1409–1417. [PubMed](http://smj.org.sa/lookup/external-ref?access_num=21800117&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) 5. Misra DP, Agarwal V (2018) Systematic Reviews:Challenges for their justification, related comprehensive searches, and implications. J Korean Med Sci 33:e92. [PubMed](http://smj.org.sa/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) 6. Makhdom AM, Alqahtani SM, Alsheikh KA, Samargandi OA, Saran N (2013) Level of evidence of clinical orthopedic surgery research in Saudi Arabia. Saudi Med J 34:395–400. 7. Rohrig B, du Prel JB, Wachtlin D, Blettner M (2009) Types of study in medical research:part 3 of a series on evaluation of scientific publications. Dtsch Arztebl Int 106:262–268. [PubMed](http://smj.org.sa/lookup/external-ref?access_num=19547627&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) 8. Almaghrabi MM, Alamoudi AS, Radi SA, Merdad AA, Makhdoum AM, Batwa FA (2015) Quality of gastroenterology research published in Saudi Arabian scientific journals. Saudi J Gastroenterol 21:90–94. 9. CASP Qualitative Checklist, Available from: [www.casp-uk.net/#!casp-tools-checklists/c18f8](http://www.casp-uk.net/#!casp-tools-checklists/c18f8). [cited 2018]. 10. National Institute for Health and Clinical Excellence. The guidelines manual (National Institute for Health and Clinical Excellence, London (UK)) Available from: [https://www.nice.org.uk/process/pmg6/chapter/introduction](https://www.nice.org.uk/process/pmg6/chapter/introduction). [Updated 2012 November;Cited 2012]. 11. Munn Z (2016) Software to support the systematic review process:the Joanna Briggs Institute System for the Unified Management, Assessment and Review of Information (JBI-SUMARI). JBI Database Systematic Rev Implement Rep 14:1. 12. Adie S, Ma D, Harris IA, Naylor JM, Craig JC (2015) Quality of conduct and reporting of meta-analyses of surgical interventions. Ann Surg 261:685–694. [PubMed](http://smj.org.sa/lookup/external-ref?access_num=http://www.n&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) 13. Tan WK, Wigley J, Shantikumar S (2014) The reporting quality of systematic reviews and meta-analyses in vascular surgery needs improvement:a systematic review. Int J Surg 12:1262–1265. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1016/j.ijsu.2014.10.015&link_type=DOI) 14. Ho RS, Wu X, Yuan J, Liu S, Lai X, Wong SY, et al. (2015) Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease:a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respir Med 25:14102. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1038/npjpcrm.2014.102&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=25569783&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) 15. Wasiak J, Shen AY, Ware R, O'Donohoe TJ, Faggion CM Jr. (2017) Methodological quality and reporting of systematic reviews in hand and wrist pathology. J Hand Surg Eur Vol 42:852–856. 16. Rogante M, Kairy D, Giacomozzi C, Grigioni M (2015) A quality assessment of systematic reviews on telerehabilitation:what does the evidence tell us? Ann Ist Super Sanita 51:11–18. 17. Shi C, Zhu L, Wang X, Qin C, Xu Q, Tian J (2014) Epidemiology, methodological and reporting characteristics of systematic reviews of nursing interventions published in China. Int J Nurs Pract 20:689–700. 18. Lundh A, Knijnenburg SL, Jorgensen AW, van Dalen EC, Kremer LC (2009) Quality of systematic reviews in pediatric oncology--a systematic review. Cancer Treat Res 35:645–652. 19. Tao H, Zhang Y, Li Q, Chen J (2017) Methodological quality evaluation of systematic reviews or meta-analyses on ERCC1 in non-small cell lung cancer:a systematic review. J Cancer Res Clin Oncol 143:2245–2256. 20. Buscaglia J, Nagula S, Yuan J, Bucobo JC, Kumar A, Forsmark CE, et al. (2011) The practice of evidence-based medicine (EBM) in gastroenterology:discrepancies between EBM familiarity and EBM competency. Therap Adv Gastroenterol 4:283–94. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1177/1756283X11412240&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=21922027&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) 21. Dong J, Teng G, Wei T, Gao W, Wang H (2016) Methodological quality assessment of meta-analyses and systematic reviews of probiotics in inflammatory bowel disease and pouchitis. PloS One 11:e0168785. 22. Heavener T, Richards J, Bird H, Vassar M (2018) Heterogeneity assessment in gastroenterology systematic reviews:an analysis of current practices. Int J Evid Based Healthc 16:101–106. 23. Liu P, Qiu Y, Qian Y, Chen X, Wang Y, Cui J, et al. (2017) Quality of meta-analyses in major leading gastroenterology and hepatology journals:A systematic review. J Gastroenterol Hepatol 32:39–44. 24. Suebnukarn S, Ngamboonsirisingh S, Rattanabanlang A (2010) A systematic evaluation of the quality of meta-analyses in endodontics. J Endod 36:602–608. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1016/j.joen.2009.12.019&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=20307731&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=000276577800006&link_type=ISI) 25. Ernst E, Canter PH (2006) A systematic review of systematic reviews of spinal manipulation. J R Soc Med 99:192–196. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1258/jrsm.99.4.192&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=16574972&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=000236671000017&link_type=ISI) 26. Delaney A, Bagshaw SM, Ferland A, Manns B, Laupland KB, Doig CJ (2005) A systematic evaluation of the quality of meta-analyses in the critical care literature. Crit Care 9:R575–R582. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1186/cc3803&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=16277721&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=000232728800036&link_type=ISI) 27. Moher D, Liberati A, Tetzlaff J, Altman DG (2009) Preferred reporting items for systematic reviews and meta-analyses:the PRISMA statement. J Clin Epidemiol 62:1006–1012. [CrossRef](http://smj.org.sa/lookup/external-ref?access_num=10.1016/j.jclinepi.2009.06.005&link_type=DOI) [PubMed](http://smj.org.sa/lookup/external-ref?access_num=19631508&link_type=MED&atom=%2Fsmj%2F40%2F5%2F426.atom) [Web of Science](http://smj.org.sa/lookup/external-ref?access_num=000270250500003&link_type=ISI)