Evidence Based Library and Information Practice Evidence Based Library and Information Practice 2011, 6.1 78 Evidence Based Library and Information Practice Evidence Summary Wikipedia and the National Cancer Institute Website Appear to Offer Similar Osteosarcoma Information for Patients A Review of: Leithner, A., Werner, M., Glehr, M., Friesenbichler, J., Keithner, K., & Windhager R. (2010). Wikipedia and osteosarcoma: A trustworthy patients' information? Journal of the Medical Informatics Association, 17(4), 373-374. Reviewed by: Kate Kelly Royal College of Surgeons in Ireland Lower Mercer Street, Dublin 2. Ireland Email: katekelly@rcsi.ie Received: 29 Nov. 2010 Accepted: 2 Feb. 2011 2011 Kelly. This is an Open Access article distributed under the terms of the Creative Commons-Attribution- Noncommercial-Share Alike License 2.5 Canada (http://creativecommons.org/licenses/by-nc-sa/2.5/ca/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly attributed, not used for commercial purposes, and, if transformed, the resulting work is redistributed under the same or similar license to this one. Abstract Objective – To compare the completeness and accuracy of information about osteosarcoma in Wikipedia to information found on the patient and health professional versions of the U.S. National Cancer Institute (NCI) website. Design – Comparative study, test against 20 item questionnaire and expert opinion. Setting – n/a Subjects – n/a Methods – The authors developed a 20-item questionnaire to test the completeness and accuracy of information on osteosarcoma in Wikipedia and on the "patient version and the health professional version of the National Cancer Institute's website as 'official' reference websites" (p. 373). Three independent observers, two surgeons specializing in musculoskeletal tumour surgery and a medical student, tested the English language version of Wikipedia and the NCI “websites” on April 3, 2009. Answers to the 20 questions found on the websites were scored from zero to three and were discussed with a member of the "German board for guidelines in musculoskeletal surgery" (p. 373) and verified against international guidelines published by the World Health Organization. Data was analyzed using SPSS and group comparisons were performed using Mann-Whitney U test with p-values of less than 0.05 significance. mailto:katekelly@rcsi.ie� Evidence Based Library and Information Practice 2011, 6.1 79 Main Results – The quality of information about osteosarcoma found in the English language version of Wikipedia was good but inferior to the patient information from NCI. Out of a total of 60 points Wikipedia scored 33, NCI patient information 40 and NCI professional information 50. There was no significant difference between the NCI patient information and Wikipedia but a significant difference (p=0.039) between Wikipedia and NCI professional information. Conclusion – Non-peer reviewed websites providing health information, such as Wikipedia, should include links to sites such as NCI and other more definitive sources such as professional and international organizations. Frequent checks should be used to ensure external links are of the highest quality. Commentary This is a very short, interesting article, which suffers from lack of detail. The authors state they compared the "patient version and the health professional version of the National Cancer Institute's (NCI) website" (p. 373) with the Wikipedia osteosarcoma page but what this means is unclear. The NCI does not have separate websites for patients and health professionals. The Physicians Data Query (PDQ) Cancer Information Summaries, produced by the NCI, are available in separate versions for patients and professionals and can be accessed via several routes on the NCI website. Presumably it is the two versions of the PDQ summaries that the authors compared to Wikipedia but this is not stated. The reference to three websites throughout the article is confusing. The lack of detail on scoring criteria is also confusing. The one table in the article is the osteosarcoma questionnaire presenting the "points for each answer for each of the three different websites" (Table 1, p. 374). However, there is no information about the scoring criteria, for example, it is unclear why Wikipedia scores 0 points for “do you find web links to study centres (EURAMOS?)” (Table 1, p. 374) while NCI Patient and NCI Professional each score 2 points. While the version of Wikipedia current at the time of the test, last edited on March 19, 2009, did not have a link to EURAMOS, it did link to the NCI and the Mayo Clinic. Are these not "study sites"? EURAMOS is the European and American Osteosarcoma Study Group and the only reference to it this reviewer could find on the NCI website was in reports detailing the activities of all cooperative study groups, i.e., there was no direct link from either the patient or professional PDQ summary on osteosarcoma so it is unclear why it scored two points. Likewise there is no exploration of the reported result that "NCI professional" scored only 50 out of 60 points (it is the gold standard site after all) nor that Wikipedia outscored "NCI professional" on three items and matched it on eight. Detail on the scoring criteria would have been a helpful inclusion. The value of the little statistical data reported is debatable given the acknowledged small sample size and a similar lack of detail on what exactly was analyzed. A statement that the three reviewers preferred Wikipedia when asked about ease of use and explaining the page's oversight appears in the middle of the article and is not explained. Questions about ease of use were not included in the reported questionnaire used to compare websites and no other reference is made to the reviewers' opinions being part of the study. One of the conclusions states that sites such as Wikipedia should "include links to more definitive sources" (p. 374) which seems a little odd when the version of Wikipedia current at the time of the test did link to definitive sources such as those noted above plus the American Association of Family Physicians and a University of Minnesota research study. To be fair, the authors themselves draw attention to some of the most obvious limitations of the study such as small sample size in terms of items tested; potential bias in reviewers (the three "independent observers" are also three of the authors) and the "creation Evidence Based Library and Information Practice 2011, 6.1 80 of the questions without external review board" (p. 374). On that basis the validity and reliability of the questionnaire is questionable. The conclusion that "our study shows that the quality of osteosarcoma-related information found in the English Wikipedia is good but inferior to the patient information provided by the NCI" (p. 374) is not supported based on the evidence presented in the article. The genuine concern with accuracy of information in non- peer reviewed websites is clear but this particular study has the feel of something put together quickly rather than rigorously. It also uses a study design of clinician evaluation of consumer health sites. As the intended audience of these sites are patients, caregivers and other healthcare professionals the study may have benefited from including these potential users of information as reviewers. It is the authors desire that this study generate discussion among professionals and maybe also lead to a larger international study on various bone and soft tissue described in Wikipedia and this study serves that purpose well. They recommend that frequent checks be made on external links on Wikipedia to ensure they are of the highest quality. Maybe the authors have done what they suggest other professionals do and checked the Wikipedia osteosarcoma page - the editorial history shows it has been frequently edited and the external sites flagged several times for compliance with Wikipedia policy on linking to external sites. It is a shame that the methodology is not replicable as those with subject knowledge may find it interesting to adapt and repeat in other domains. As it stands, this study has no direct applicability to library practice but those interested in a test methodology should maybe consult the 2008 study on the accuracy of drug information in Wikipedia upon which this study was based (Clauson, Polen, Boulos & Dzenowagis, 2008). References Clauson, K. A., Polen, H. H., Boulos, M. N., & Dzenowagis, J. H. (2008). Scope, completeness and accuracy of drug information in Wikipedia. Annals of Pharmacotherapy, 42(12), 1814-1821. / Evidence Based Library and Information Practice Evidence Summary