130 Doctorates by thesis and publication in clinical medicine: An analysis of examiners’ reports Abstract National and institutional policies to escalate the production of doctorates have raised concerns about the quality of PhDs in South Africa. This study evaluates examiner reports of doctorates by thesis and publication in clinical medicine to ascertain the criteria that examiners used to define a successful doctoral thesis. A qualitative hermeneutic approach through document analysis was taken, involving 54 national and international examiner reports of 18 doctoral theses, half of which were by publication. The examiners rated the majority (81%) of these theses to be of a high quality. Our results show a scatter of positive and negative aspects distributed across all 18 theses, with corresponding commendations and criticisms. Notably, almost equal emphasis was placed on conceptual and communication issues. Ratings per thesis were not always concordant, with five theses each given widely divergent ratings by three respective examiners. The study also found differences in ratings between examiners, based on whether they were medical or non- medical, international or national, and with or without PhDs themselves. While no single criterion was identifiable as being diagnostic of doctoral quality, the identified strengths and shortcomings of these theses serve as a useful platform for supervisors and students aspiring to cross the doctoral threshold. Key words: doctoral study, clinical medicine, examination, thesis, health professions education 1. Introduction A PhD represents the pinnacle of academic achievement; the title confers an elite status within academia and an expectation of a lifetime commitment to research and teaching. Yet, it seems like the examiners are the custodians of “the PhD rite of passage” (Kumar and Stracke, 2018). Even though it is regarded as an entry requirement into academia, the nature and process of earning it remain something of a mystery (Holbrook et al., 2004a). Despite the significance placed on the PhD examination process, it is “enshrouded in mystery” and “secrecy”, with supervisors and candidates being apprehensive and failing to understand its “intricacies and subtleties”, leading to a call for detailed and explicit consensus on a more transparent process (Johnston, 1997). Given the value placed by both institutions and candidates on the PhD, it is a matter of concern that clear guidelines on the expectations of PhD Dr Suvira Ramlall Department of Psychiatry, Nelson R. Mandela School of Medicine, College of Health Sciences, University of KwaZulu-Natal Dr VS Singaram Clinical and Professional Practice, Nelson R. Mandela School of Medicine, College of Health Sciences, University of KwaZulu-Natal Prof TE Sommerville Department of Anaesthesia and Critical Care, Nelson R. Mandela School of Medicine, College of Health Sciences, University of KwaZulu-Natal DOI: http://dx.doi. org/10.18820/2519593X/pie. v37i1.10 ISSN 0258-2236 e-ISSN 2519-593X Perspectives in Education 2019 37(1): 130-147 Date Published: February 2020 Published by the UFS http://journals.ufs.ac.za/index.php/pie © Creative Commons With Attribution (CC-BY) http://dx.doi.org/10.18820/2519593X/pie.v37i1.10 http://dx.doi.org/10.18820/2519593X/pie.v37i1.10 http://dx.doi.org/10.18820/2519593X/pie.v37i1.10 http://journals.ufs.ac.za/index.php/pie https://creativecommons.org/licenses/by/2.0/za/ https://creativecommons.org/licenses/by/2.0/za/ 131 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine candidates and requirements of examiners are seldom explicitly articulated or uniformly defined so that the examination can be approached in an informed and consistent way across disciplines and universities worldwide. The completion of a doctoral degree by medical specialists poses unique requirements in the South African setting. The trajectory of clinicians’ development after the MBChB is often a period of specialisation towards a (master’s level) Fellowship of one of the Colleges of Medicine of South Africa (CMSA). There has been no provision for a developmental path such as integrated MD-PhD programmes seen elsewhere in the world. As noted by the Academy of Science of South Africa (ASSAf), 2010, the country has a very minor employment market for the clinician-PhD. The UKZN requirement of a PhD for all academics thus posed additional challenges for clinician-lecturers in the School of Clinical Medicine. Furthermore, the majority of clinicians are in honorary appointments, as they are employed by the Department of Health, which “rationalised all health care facilities to its control and prioritised health care and service delivery to the detriment of research activity” (Grossman and Cleaton-Jones, 2011: 111). National and institutional policies to escalate the production of doctoral degrees raised concerns about the quantity versus quality of PhDs and the significance of doctoral research (Jansen, 2011; Grossman and Cleaton-Jones, 2011). ASSAf (2010:17) highlighted the need for more research to develop a “comprehensive understanding of the dynamics of doctoral education”. Literature on the quality of PhDs from Africa is sparse; Mulinge and Arasa, 2013, qualitatively analysed 100 of the former’s examination reports of both master’s and doctoral dissertations, drawn from ten universities across sub-Saharan Africa, and found that 63% of the theses were thought to be of a low quality. However, only six of these were PhDs, none of which were from Medicine. This study explores the examiner reports of PhDs by thesis and publication of a cohort of South African clinical medicine staff, to ascertain what national and international examiners regarded as the sine qua non of a successful doctorate. 2. Literature The examination of a PhD is a highly complex process with the execution and judgment occurring at an intellectual level that is little understood and somewhat mysterious(Bourke and Holbrook, 2013). However, the lack of an explicit summary measure of thesis quality and differences in examiner evaluations cannot be linked to differences in university procedures (Holbrook and Bourke, 2004). While the traditional qualities defining a successful thesis are originality, scholarship and advancement of knowledge, Ballard (1996) expresses concern that examiners are transforming these into less lofty expectations of “imagination”, “competence” and “mastery”. Bitzer, Trafford and Leshem (2013) highlight that the doctorate represents knowledge, skill and attitude of a level that involves intellectualising, conceptualising and contributing to existing knowledge, and not just reporting facts. There is general agreement on the content and standard of a thesis: technical proficiency, originality (meaning ‘creativity’ or ‘significance’) and conceptual development are regarded as important (Mullins and Kiley, 2002). Criteria sought by examiners are described as “coherence”, “rigorous argument”, “meaty” and “thoroughness”, leading to the conclusion that the real role of examiners is to judge whether a student has “mastered appropriate indeterminate skills and displayed the right indeterminate quality” (Delamont et al., 2000). 132 Perspectives in Education 2019: 37(1) Leshem and Trafford (2007) confirm the value examiners attach to the significance, role and use of conceptualisation in a doctoral thesis, with evidence that the absence of a conceptual framework is unlikely to secure a pass; this however varies across disciplines. Poorly defined objectives, mystification of the process, confusion about what constitutes appropriate research, and lack of benchmarks may account for inconsistency between different examiners’ recommendations and comments on the same thesis, as well as between the recommendations and comments of an individual examiner(Holbrook et al., 2004a). Only a third of examiners surveyed by Mullins and Kiley (2002) considered institution-specific criteria when assessing theses; even while using the universities’ assessment forms, they considered themselves the arbiters of an acceptable thesis. While assessment at PhD level may not be a rigorously scientific process, it is not a lottery either, and while quality may be difficult to define, it may be more easily recognised (Becher et al., 1994). Examiners should be appointed on the basis of their subject expertise, examining and supervisory experience, and their possession of a PhD (Powell and McCauley, 2003). Experienced examiners are distinguished by their formative approach to the thesis, both anticipating and wanting it to pass. Conversely, inexperienced examiners rely more on institutional criteria, and approach the thesis summatively; uncertain about the boundaries between a good and a poor thesis, they often refer to their own postgraduate experience to guide their supervision and examination. Despite these differences, both experienced and inexperienced examiners look for consistency, coherence, and the investigation of an interesting problem. They are impressed – and unimpressed – by the same things (Kiley and Mullins, 2004). Furthermore, candidates’ commitment and effort do have an impact on examiners’ decision-making (Holbrook et al., 2004a), and examiners should have “enough humanity” to ensure that the examination process is a developmental experience for candidates (Joyner, 2003). The roles played by a PhD examiner have been scrutinised, as these impact on the content and purpose of the examination report. Holbrook argues that, while students want to know what examiners are looking for, there is a significant difference between what examiners look for and what they ‘privilege’ in their reports. Examiners use the report to “judge, instruct, amend, mentor and confer or confirm membership of the academy”. The “assessor-arbiter, mentor-colleague and supervisor-instructor” roles played by examiners within the assessment process need to be objectively defined for the training of both supervisors and examiners (Holbrook et al., 2004b). There is also a call for the learning role of assessment to be fulfilled by examiners; examiner guidelines should include the need for providing developmental feedback and negative assessments must be accompanied by advice. Kumar & Stracke assert that a thesis is ‘work-in-progress’, and apart from the gatekeeper (summative) role, examiners have a role as a teacher (formative) and would thus be promoting learning, which is the purpose of higher education. In their experience, even theses that required no amendments were accompanied by feedback which created learning opportunities for candidates and supervisors(Kumar and Stracke, 2018). The examination process does have an element of subjectivity, and while examiners consciously or subconsciously serve as gatekeepers to their community, they may also be partial to candidates who share their ideologies (Johnston, 1997). The massification of doctoral studies, coupled with the climate of student entitlement and consumer rights, highlight the need for national, if not international, examining standards for PhDs (Morley et al., 2003). At the 2010 Strategic Leaders Global Summit, the delegates identified the assessment of quality 133 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine in international collaborations as “integral to (post-)graduate research training in the 21st century”, and tasked graduate education leaders with the responsibility, among others, for “defining, measuring, benchmarking, and improving” the skills of students (Council of Graduate Schools, 2010). Bitzer et al. (2013:781) propose that candidates’ and examiners’ contributions to new knowledge could be enhanced if there were international quality measures for theses and ‘doctorateness’. Many institutions, including the University of KwaZulu-Natal (UKZN), have adopted the doctorate-by-publication model, but the full interpretation and impact of this approach is yet to be fully endorsed. The model is not universally accepted, with supervisors being sceptical and examiners questioning its validity (Dowling et al., 2012). While ‘new-route PhD’, ‘professional doctorate’ and ‘practice-based doctorates’ add to the traditional PhD and PhD by publication (Robins and Kanowski, 2008), several institutional and pedagogical reasons favour the publication route. Managerial and policy imperatives for shorter completion times, lower drop- out rates and higher productivity drive the pressure to publish, and incentives and rewards increase individual and institutional research productivity. Therefore, practices that increase publication numbers and thus influence institutional rankings and income, and also carry monetary and reputational benefits for supervisors, may be favoured (Frick, 2016, Lee, 2010). Students and supervisors must be adequately prepared for the publication format, which may call for a different doctoral supervisory pedagogy and infrastructure (Lee, 2010), and may require more supervisory capacity (Frick, 2016). Further, co-authorship, especially with supervisors, may leave examiners questioning the contribution of the student. Also, if papers are reviewed simultaneously with the thesis being examined, the task of double revisions may present itself and may make reconciliation of conflicting examiner/reviewer perspectives a challenge (Robins and Kanowski, 2008). A variety of views on this topic emerged from the survey conducted by Mullins and Kiley (2002). Half the examiners surveyed were favourably influenced by acceptance of thesis material by a reputable journal. A small number had reservations about early publication, on the basis of journals’ standards, or suspected that the work might not represent the work of the candidate but that of the supervisor. A small number rejected outright the influence of publications on their assessment, believing that examiners should be confident in their own opinions and not be influenced by peer reviewers. The international literature describes the desirable qualities of a doctoral thesis as reflecting the thought processes behind selecting an unresolved problem, delineating relevant prior work, crystallising the aspects to be studied and the applicable methodology to do so, using findings to build further knowledge and new understanding, and presenting all this in a reasoned, coherent, readable way. Given the subjective nature of discerning the necessary quality of the author’s thought processes through a manuscript alone, the supposed poor quality of theses in Africa, and the pressures and distractions of clinical work, we set out to explore the nature of ‘quality’ in locally generated theses through the eyes of national and international examiners. 3. Methodology and Context This study adopted a hermeneutic content analysis framework (Vieira and de Queiroz, 2017) to analyse the ratings and comments contained in examiner reports to establish the characteristics of successful and problematic theses respectively. 134 Perspectives in Education 2019: 37(1) Anonymised reports of a convenience sample of PhD candidates from the School of Clinical Medicine examined during 2013-2015 were obtained. Only the eighteen candidates whose examiners’ qualifications and affiliations could be obtained from the college central administrative office, which is the custodian of examiner information, were included in the study. The comments of the examiners were construed as representing formative feedback to candidates towards improving the quality of the PhD and were thematically analysed independently by each author (Mihas, 2019). Discussions were held to achieve consensus on emerging themes or main areas of feedback received from the examiners. On this basis, eight themes dominated this set of examiner reports. The overall ratings awarded by examiners according to the four categories stipulated by the university were regarded as summative feedback and were analysed quantitatively. It was expected that descriptive feedback would correlate with the final assessment of the PhD. The following instructions are sent by the university to the examiners on the requirements of a PhD. Clearance was obtained from the UKZN ethics committee and gatekeeper permission was obtained from the Registrar. 3.1 Guidance for examiners Examination of PhD theses at UKZN requires independent reports from three external examiners, at least one of whom should be international. UKZN requires each examiner to assign a summative assessment based on four categories: 1. The dissertation should be accepted without any corrections or revisions. 2. The dissertation should be accepted, provided certain corrections and revisions are carried out, to be approved by the supervisor, Dean and Head of School, and Dean of Research. 3. The dissertation requires substantive revision/extension by the candidate and should not be accepted in its present form. It should be returned to the candidate for revision/ extension and then be resubmitted for examination. 4. The dissertation should be rejected outright, according to a detailed report (University of KwaZulu Natal, Undated-a). Accompanying the thesis to be examined, examiners receive from the University a document to explain the process, expected standard, and reporting requirements for doctoral theses. This includes: Dissertations/theses must show that candidates are acquainted with the methods of research and their application in that they: understand the nature and purpose of their investigations, are sufficiently acquainted with the relevant literature, command the necessary techniques and are capable of assessing the significance of their findings. Evidence of independent/critical thought, thoroughness, consistency, logical development/ structure and strength of conclusion must be present. Additionally, dissertations/theses must be satisfactory in literary style and presentation. It is expected that a doctoral thesis will make an original contribution to the field. Systematic review type literature review – definitive in the area. Data collection usually not in one setting, often national sample. Must make a new contribution to knowledge in the field (University of KwaZulu Natal, Undated-b). 135 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine No specific guidelines were provided to examiners on the thesis by publication. These guidelines appear to be typical of those that were available from other local universities. 4. Findings Of the eighteen PhD submissions (all on clinical topics, with or without laboratory or other special investigations and/or audit), nine were traditional theses and nine were by publication. There were 52 examiners who submitted reports and two who returned only the rating form. Table 1 lists the examiners’ ratings of each thesis, the corresponding wordcounts of the examiners’ reports, and the qualifications and locality of each examiner. 4.1 Examiners’ profiles, ratings and reports Table 1: Analysis of examiners’ reports S t EXAMINER 1 EXAMINER 2 EXAMINER 3 C at . W or ds Q ua l. Lo c. C at . W or ds Q ua l. Lo c. C at . W or ds Q ua l. Lo c. 1 P 1 555 BSc + PhD RSA 1 481 MA + PhD RSA 1 784 MA + PhD USA 2 T 2 1042 Sci + DSc RSA 2 891 Med + PhD Ghana 2 882 Vet + PhD Canada 3 T 2 547 Nurs + PhD RSA 2 3920 Sci + PhD Malaya 2 5853 Med + PhD RSA 4 T 2 578 Med + DPhil RSA 1 0 Med + Spec RSA 1 1027 Med + DPhil UK 5 P 2 3200 Med + PhD RSA 1 119 Med + PhD Holland 1 278 Med + PhD USA 6 P 1 319 Med + PhD RSA 1 1812 Med + PhD Denmark 2 1365 Med + PhD Malaya 7 T 2 889 Sci + PhD RSA 1 427 Med + PhD USA 1 53 Med + Spec USA 8 T 2 563 Med + Spec RSA 2 783 Med + MD RSA 1 141 Med + Spec USA 9 T 1 32 Med + MD RSA 2 355 Med + MD UK 2 1060 Med + MD UK 10 T 2 1145 Med + DPhil RSA 2 792 Med + DPhil UK 1 510 Med + PhD Norway 136 Perspectives in Education 2019: 37(1) S t EXAMINER 1 EXAMINER 2 EXAMINER 3 C at . W or ds Q ua l. Lo c. C at . W or ds Q ua l. Lo c. C at . W or ds Q ua l. Lo c. 11 T 3 8176 Med + PhD RSA 2 511 Med + PhD Ethiopia 2 362 Med + MD UK 12 P 2 847 Sci + PhD RSA 2 1943 Med + MD UK 3 2696 Med + PhD UK 13 P 3 1809 Med + DSc RSA 3 1431 Med + Spec USA 2 695 Med + PhD Australia 14 P 3 1757 Med + PhD RSA 2 290 Med + DPhil RSA 1 636 Med + PhD Australia 15 P 3 396 Med + Spec RSA 1 84 PhD USA 2 646 Med + PhD Australia 16 T 3 1596 Med + PhD RSA 1 140 Med + MD RSA 1 130 Med + Spec USA 17 P 1 224 Med + PhD RSA 3 893 Med + MD RSA 1 454 Med + Spec USA 18 P 3 895 Med + MD RSA 1 506 Med + Spec USA 3 901 Med + Spec UK Cat = Category of rating Loc = Examiner’s country of residence Nurs = Nursing degree Med = Basic 0medical degree Qual = Qualifications of examiner Sci = Science degree Spec = Specialist St = Student (T = by thesis, P = by publication) Words = Wordcount of report. Vet = Veterinary degree 4.1.1 Examiner profiles As illustrated in Table 1, the majority (44) of examiners had a medical background, eight were scientists, one a nurse and one a veterinarian. Ten had only medical specialist qualifications but no doctorate. Of the 54 examiners, 26 were South African (RSA), two from elsewhere in Africa, eight British (UK), three European, eleven North American (USA), three Australian and one from Malaya. 4.1.2 Examiner ratings and reports The examiners awarded varied ratings to individual candidates. Their reports showed diverse formats, contents and lengths, ranging from broad summaries to detailed page-by-page and line-by-line feedback on content, referencing, and typographical or grammatical errors. Report lengths varied from 0 to 16.5 pages or zero to over 8000 words with a mean of 1082 words. The twenty-one reports related to the theses with the highest rating (category 1) varied in length from 58 to 1812 words (mean 413 words), with two examiners not submitting written 137 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine reports. The twenty-three theses rated 2 were accompanied by reports ranging from 290 to 5853 words (mean 1268 words). The reports of the ten examiners who awarded ratings of 3 were 396 to 8176 words in length (mean 2055 words). Hence, in general, the longer reports correlated with poorer ratings and more detailed feedback on grammar, formatting and referencing, except for one 1812-word report which comprised 424 words of comments and a 1388-word summary of the thesis (Candidate 6 Examiner 2). The majority of theses (81%, n=44) were rated as high quality (categories 1 and 2) by the examiners. The remainder required major corrections and resubmission (category 3). No thesis was rejected outright (category 4). Examiners’ ratings agreed fully for three theses (17%), and for ten theses (56%) two of the three examiners agreed, with the third giving an adjacent rating (1 vs.2 or 2 vs. 3). However, in three cases the discrepancy was greater (1 vs. 3), and two theses received three different ratings (1, 2 and 3). Medically qualified examiners with and without PhDs tended to rate theses less favourably and submitted shorter reports compared to those with a science background. South African examiners tended to give less favourable ratings and to write longer reports when compared to their international colleagues. Doctorates by publication tended to be given a poorer rating and required longer reports than those by thesis. 4.1.3 Examiner discordance For candidates 14 and 15 (see Table 1), each examiner rated the thesis differently, with scores of 1, 2 and 3. All the examiners commended the work and applauded the publications. Those who accepted the thesis (rating 1 or 2) focused their comments on the worthiness of the published papers and the contribution of the thesis to resolving current problems in resource- constrained environments. The dissenting examiners (rating 3) were unhappy with the depth, critique and discussion of the theories used, as well as the introduction and discussion. They focused on the weakness of the overall structure, the gaps in individual chapters, and grammatical errors. One of the examiners strongly highlighted the lack of guidelines for PhDs via publication as a factor contributing to criticisms of the structure of the thesis. As illustrated in Table 1, for candidates 16, 17 and 18, examiners were split between ratings of 1 and 3. Those awarding a 1 varied in their enthusiasm between noting that a thesis was in line with their own work, provided an important example, or was ‘acceptable’; each thought the topic important. One dissenting examiner congratulated the student and supervisor, but noted methodological and analytical weaknesses; in the other case, the dissenter felt that the publications included were disjointed, and that the student’s statements were inconsistent opinions unsupported by the literature or the study’s findings. In the third case, one examiner could not follow the thread that supposedly linked the included papers; another saw the links but felt that the matrix initially presented as a conceptual framework in which to consider the topic was thereafter ignored; the examiner suggested that either the publications must be related to the framework or the thesis must be rewritten as standard health service research. 4.2 Examiner feedback Each report contained one or more sections of, firstly, general comments (sometimes only a summary of the thesis, or, in one case, a list of references consulted by the examiner) without 138 Perspectives in Education 2019: 37(1) indication of approval or disapproval; secondly detailed critique (positive or negative) of the student’s thought process, or; thirdly, lists of technical errors requiring correction. Theses rated as Category 1 tended to attract non-specific approval, little critique and virtually no mention of corrections; those rated 2 garnered an equal amount of general comments, but far more critique and more lists of errors; 3-rated theses had fewer general comments and technical corrections than in category 2, the vast majority of their reports consisting of critique of the planning, execution and/or presentation of the doctoral study. We regard examiners’ descriptive feedback as formative, despite the fact that examiners did not always explain how a particular infelicity could be remedied. Some examiners provided explicit guidance: ‘…the candidate should… provide an explanation…be more explicit… state the significance and broader implications…’; ‘…the author is encouraged to provide potential reasoning…’. Some examiners referred candidates to lists of errors of interpretation or presentation: ‘Please attend to the following areas of concern’. Some examiners were less specific: ‘…requires substantive revision…’; ‘referred back to the candidate for attention to the significant shortcomings’. Eight key areas were identified from thematic analyses of the examiners’ critiques. The frequency of occurrence of each theme in the reports is included in parenthesis next to the theme. Direct quotes describing commendations or condemnations are used to illustrate the exact feedback received from examiners. 4.2.1 Discussion (34) The handling of the discussion seemed to distinguish between category 1 theses and others. The former attracted comments ranging from “extensive exploration and comprehensive synthesis drawing valid conclusions” to the more mundane addressing of study objectives and careful description of limitations. “brilliant synthesis” “exceptional, superb, well-written” “comprehensive and thorough” Theses of categories 2 and 3, despite in some cases having plausible, lucid and logical explanations, were generally found to display disorganised thinking, were hard to follow, lacked reflection or critical thought, failed to deal with the implications of the study’s findings, or simply presented the candidates’ opinions unsupported by the research findings. “assertions not supported by evidence … need to be more carefully argued” “statements of opinion not substantiated by actual clinical findings” 4.2.2 Methodology (29) Familiarity with research methodology and appropriate, adequately motivated choice of methods for the intended aims of the study were applauded. “critical insight displayed” “eloquently motivated design” 139 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine “appropriate choice of methods...correlates with aims” Lack of understanding of the techniques used or failure to justify their application, inadequate sample sizes or controls, or use of inappropriate measures, were censured. “choice of technique not justified” “method lacks rigour” 4.2.3 Literature Review (24) Familiarity with and critical appraisal of the literature in the field in order to contextualise the study was uniformly commended. “very sound and in-depth literature review” “adequately acquainted with the literature in the field” Inadequate coverage for the envisioned study, or for the level of a PhD, lack of a critical consideration of the literature, and candidates’ opinions or speculations unsupported by the quoted literature earned adverse comments. “critical review absent” “study not put into context due to lack of coverage of literature” 4.2.4 Publications (22) Some examiners commented positively on the thesis-by-publication format, as it increased the exposure of the work internationally. “pushed the boundaries of thesis by publication” “publications and manuscripts show international value” Others felt that thesis by publication posed several challenges. The inclusion of papers with multiple authors and the order of authorship led to speculation about candidates’ involvement in the work. “candidate is second author in publication” “candidate’s contribution to the publications … was unclear” Here, examiners suggested that the “candidate’s contribution to papers must be documented”. Further, the quality of submitted unpublished manuscripts was questioned, as a submission does not necessarily equate to a published paper that has undergone rigorous peer review. “submitted papers not synonymous with accepted for publication” While some examiners critiqued published papers as they did the rest of the thesis, others explicitly or implicitly suggested that the journals’ peer reviewers had already critiqued the papers, obviating the need for the examiner to do so. Those who did critique published papers sometimes disagreed with the conclusions, but felt unable to provide meaningful input since 140 Perspectives in Education 2019: 37(1) the papers were already in print. A suggestion was made that perhaps published papers need to be interleaved with introductory and commentary sections to link the papers to each other and to the research path, and to allow for post-hoc self-critique. “not happy with amount of detail in published papers” “disagree with conclusion of published paper” 4.2.5 Coherence and synergy (21) As suggested in the foregoing, even category 1 theses were not immune to the failings described. However, examiners were impressed by candidates’ demonstration of their acquaintance with their chosen topic, centred on a clearly delineated problem tackled in a well-conceived study, and by their robust findings. They also praised the arrangement of material in a sequence that supported a logical argument that demonstrated sophisticated and critical insight, and was presented fluently so as to engage the reader. “congratulations on a lucid manner of presentation” “detailed, thorough … aim is clear … good description of limitations … all objectives addressed” Lack of coherence was evidenced by inconsistency or a need for integration between different parts of the thesis, absence of a stated research problem, failure to support the argument with cogent references, or lack of a discernible thread linking the parts of the thesis. “lack of diligence in detail … has not demonstrated familiarity with theories quoted … not demonstrated critical thought … no evidence of reflection … introduction, methods and conclusion lacking” 4.2.6. Results (21) Clear presentation and systematic analysis of study results distinguished category 1 theses from the rest. “thinking is sophisticated and data critically evaluated” “unique and powerful dataset” Poorly described, inconsistent or sometimes missing data, inappropriate statistical analysis, or lack of interpretation of statistical outcomes characterised poorly rated theses. “results not consistent or systematically presented” “far too many tables and data over-analysed” 4.2.7 Presentation (14) Communicative aspects of format, grammar, writing style, and typographical errors tended to earn less censure in category 1 theses. The appeal of well-written and lucid texts that were a pleasure to read appeared to outweigh the irritation of errors and lack of clarity about the terms or concepts being used. “writing and diction – overall superior … fluent and versatile command of concepts ... exhibits all-too-rare combination of scholarly precision and linguistic plain spokenness” 141 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine Less admirable theses were characterised by more frequent errors in typing, referencing, and presentation of data, and in sloppy and imprecise language. “thesis not proof read” “inconsistent style, format” “grammar, punctuation, spacing inconsistent throughout” 4.2.8 Originality/Novelty (14) Despite the fact that a doctoral thesis must generate new knowledge, and that none of the reports rejected a thesis on the basis of a lack of originality, it was almost exclusively the category 1 theses in which this aspect was specifically remarked upon, often as being significant, substantial or valuable. “well conceptualised substantial, original and valuable contribution” “original, relevant, meaningful” Queries in this regard in lesser-ranked theses related to the inclusion of data gathered and publications authored a long time prior to the thesis submission dates. “10 years to complete … early part of thesis written long before the second part” “lack of publications for 12 year interval” “due to long time frame, relevance of data is questionable” Table 2. Themes and frequency of occurrence in examiner reports Holbrook et al Categories Corresponding Themes & Frequency Argument: Reasoning & coherence of argument Sufficiency & completeness of argument Depth & sophistication of thinking Discussion-14 Coherence and Synergy-21 Originality & Novelty-14 TOTAL: 69 Project: Theory Literature Data Analysis Literature-24 Results-21 TOTAL: 45 Fundamentals: Presentation Methods Methodology-29 Presentation-14 TOTAL: 43 Publications-22 142 Perspectives in Education 2019: 37(1) Table 2 categorises the themes of comments (favourable and negative) and their frequency in examiner reports; the themes used in this study are compared with the 3 categories proposed by Holbrook et al., 2014. While the eight themes identified in our analyses could be correlated with their three main categories, examiners’ comments on publications featured with similar frequency as for coherence and synergy, and the literature review. 5. DISCUSSION AND CONCLUSIONS The aim of this study was to explore the defining characteristics of ‘doctorateness’ by analysing examiners’ summative and descriptive/formative reports on a cohort of PhD theses from the School of Clinical Medicine at UKZN. Variations in the structure, length and focus of reports and a lack of consistency between examiners’ comments and ratings of the same thesis reflected a lack of standardisation of the doctoral examination process and a lack of clarity on what defines doctorateness, respectively. The lack of uniformity in the structure of reports was evident in both the length and content of examiner reports. Our experience is similar to the findings of the textual analysis of PhD examiner reports across disciplines in one Australian university which revealed no standard structure or style. Very few reports were organised by specific themes, topics or methods and assessments were presented in a variety of ways; there was also no consistency of reporting on themes with 69% making some reference to ‘significance’ and 60% to ‘approach’ and ‘methodology’ (Holbrook et al., 2004b). Generally, our findings on the length of examiner reports are consonant with the reported literature: weaker theses usually elicited lengthier reports with examiners commenting on both the academic and communication aspects. However, there were no sharp dividing lines between theses adjudged high or poor quality; reflections of excellence and of deficit were found throughout the three categories. It was not possible to construct a checklist based on institutional guidelines or literature that would unequivocally distinguish between theses that would pass or fail. The wide variation in length of reports correlates with the Broad Fields of Study of 1103 examiner reports which ranged from one line to 25 pages, with an average of 2-3 pages. No study was identified that examined PhDs from Medicine exclusively, but no significant difference in length of reports between ten studied discipline categories, which included ‘health’, has been reported (Holbrook and Bourke, 2004). Another study of 804 Australian theses revealed that satisfied examiners wrote shorter reports than dissatisfied examiners (Holbrook et al., 2008). Holbrook et al. (2004) note that examiners’ perceptions of the value of a thesis were quantitative rather than qualitative; commendations were more often attached to theses thought to be good, and criticisms more commonly attached to poor theses. We found, as they did, that the tone and intent of examiners’ reports varied between categories. Category 1 theses tended to attract briefer summative assessments with corresponding comments on their authors’ academic abilities; Category 2 and 3 theses tended to incur lengthier reports, generally of a formative nature, focused on improving the thesis to the point of acceptability. Our initial expectation that adverse reports are longer because of the need to justify their unfavourable conclusions was not borne out: examiners generally went to great lengths to suggest what candidates could do to amend their theses and supplement deficiencies. Direct comments on the quality of the candidates’ academic abilities were rare in these two categories. Kumar and Stracke(2018) found that longer reports contained more developmental feedback and 143 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine shorter reports contained more assessment and the authors called for the practice of negative assessment without advice to be discarded. In the cases of individual theses that were categorised differently, it appears that examiners awarding a rating of 1 focused on the importance of the topic, whereas dissenting examiners were more aware of the need for rigorous scholarship, cohesion and quality of communication in the presentation of a doctoral thesis. An examiner’s familiarity with the field of research appeared to provide an implicit understanding of the author’s intentions, and agreement with his or her conclusions may have swayed the examiner to overlook the logic of the work as a whole. We also compared the comments on theses categorised by examiners as 1 (Accept as is) with those categorised as 3 (Re-submit). While defects in presentation were widespread, they attracted more attention in poorer theses, as if quality of thought and quality of communication were directly related. The perceived value of the literature review varied even within categories, but tended to be less relevant and less inclusive in the lower quality categories. Critical mastery of the literature is suggested as the ‘litmus test’ for an ‘unambiguous academic entry point’ representative of the ‘essence of scholarship’. A good literature review favours a positive response from examiners (Holbrook et al., 2007). Candidates’ choices of methodology were questioned in all categories, but were generally considered better thought through and critiqued in category 1 theses, and less obviously appropriate or correctly applied to the research problem in the lower quality theses. Presentation and analysis of results were most likely to be complete and systematic in category 1 theses; statistical descriptions were less adequate or less apt, and prone to misinterpretation, in poorer quality theses. Students’ discussion and critique of study findings appeared to be comprehensive and thoughtful in category 1 theses, but opinions unsupported by study findings or viewed uncritically characterised poorer quality theses. General comments on theses showed, similar to examiners’ comments on specific sections, a spread of excellent and adverse features across all three categories. The coherence, lucidity of expression, clarity of sequencing and solid base for conclusions of the thesis as a whole were appreciated by examiners of category 1 theses. Discrepancies between the abstract and the body of the thesis, lack of integration within chapters, lack of application of theories adduced, unclear research questions, or a number of published papers without obvious links were noted in poorer quality theses. Surprisingly, although one would expect originality as a hallmark, only category 1 theses elicited specific comments on their creativity and freshness. Conversely, lower quality theses tended to generate queries as to the student’s contribution to previously published papers, implying some doubts as to the authenticity of the appearance of the paper in the thesis. The relevance of previous publications was also queried in terms of their subject matter or time lapse. This analysis of examiner reports illustrates a lack of uniformity between examiners in approaching the examination of a PhD, and fails to characterise the essence of a PhD for students and supervisors. There were no clearly defined shortcomings that consistently defined a poor thesis, as examiners variably focused on the conceptual/intellectual or communicative aspects of theses. It was evident that, while examiners placed equal value on both aspects, scant explicit mention was made of the role of synergy and coherence in the examined theses. Comment on the candidates’ demonstration of doctorateness and successful negotiation of thresholds (Trafford and Leshem, 2009) was lacking, begging the question of the difference between a good clinical study and a doctoral thesis. An analyses of more than 800 theses 144 Perspectives in Education 2019: 37(1) found a lack of significant differences between satisfied and dissatisfied examiners of the same thesis; even though examiners made substantially different recommendations, the content of their reports differed little (Holbrook et al., 2008). These findings highlight the challenges of seeking to characterise the defining features of a PhD by evaluating examiner reports. In their analyses of the focus and substance of formative comment by PhD examiners, Holbrook et al (2014), proposed a hierarchical sequence from the bottom up of three categories (and nine sub-categories) ranging from the ‘fundamentals’ to the ‘project’ and, at the top, the ‘argument’ which they propose as the ‘crux’ of the thesis. The themes identified in our study (excluding ‘publications’), when aligned with their three categories, indicate that examiner comments pertaining to the ‘argument’ were most frequent (Table 2). While this provides some evidence for the category prioritised by examiners, our analyses did not extend to the level that we could establish whether strengths or deficiencies in this category correlated with the final rating of the thesis. Although PhD by publication has been adopted nationally and internationally, our study found that doctorates by publication tended to be rated more poorly, and comments from examiners highlighted the need for explicit guidelines for students and supervisors regarding formatting of theses, authorship, publication timeframes and the need for a critical narrative linking the publications. The publication model lends itself particularly well to interdisciplinary research in clinical medicine in addition to the advantages to institution, student and supervisor, as highlighted by the literature. However, to ensure that pedagogical imperatives accompany scientific breakthroughs in keeping with doctorateness, heed needs to be paid to the communication and conceptual shortcomings highlighted by examiners. It is insufficient for a series of papers arising from clinical practice to be bound together. Guidelines for a critical narrative to complement the stream of published or publishable papers that would enhance the quality of PhDs by publication, ensuring that doctoral standards and institutions both benefit, are recommended (Badley, 2009, Mason, 2018). A traditional PhD and a PhD by publication are equally rigorous academically, but the question of academic level needs to be addressed if the latter model is to gain wider acceptance (Davies and Rolfe, 2009). The question of academic level leads one back to defining ‘doctorateness’. Should examiners focus on reviewing the published work – to which changes or revisions cannot be made – or should they rather focus on the critical self-appraisal that accompanies the portfolio of published work? Davies and Rolfe (2009) further suggest that critical self-assessment is the most important and difficult skill, and also the best index of doctoral level work; the publication route, far from being an easier route, is not for the fainthearted. Writing is central to academic identity formation, and therefore a PhD by publication facilitates crossing the threshold and plays an important formative role in developing the student towards becoming a member of the academic community (Mason, 2018). It is the reception and not just the production and publication of one’s scholarship that shapes this identity, which is an ongoing process or journey facilitated by the publication model (Dowling et al., 2012). As regards our first research question, we found examiners’ ratings of theses to be generally concordant, although in only three cases did they agree exactly, and in five of the 18 theses (28%) there was a wider discrepancy, reasons for which we have suggested. In their study of 804 theses, Holbrook et al (2008) found that 4% had one or more discrepant reports among examiners leading them to suggest that, even though there is no mandated curriculum for 145 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine the PhD, this indicates “innate robustness of the ‘invisible’ doctoral curriculum and evidence of consistently applied standards”(p46). The variety of opinions on a single thesis argues for reliance on multiple examiners, plus an independent internal university committee to reconcile conflicting reports. An interesting observation in this study is the comparison of ratings of examiners based on qualification and geographical location. Unlike a previous study (Prieto et al., 2016) we found that the geographical location and qualification of the examiners influenced their assessment of a thesis. This study found that international examiners rated theses higher than national examiners, whilst medically qualified examiners rated theses lower than examiners with a science background. These findings also contradict anecdotal assumptions that local examiners with similar qualifications will unfairly favour candidates. Perhaps this is the reason why local examiners rate theses more stringently. In qualitative terms, we established that this sample of doctoral theses, produced in a medical setting, conforms to standards and findings elsewhere in the world. This is heartening, despite it being a part-time endeavour, fashioned in between a full load of teaching and clinical service. The eight themes that emerged from our examiner’s reports spoke to the preparation (literature review and methodology), prosecution (results), intellectual preparation of findings (discussion, coherence and novelty) and dissemination (presentation and publication) as contributing to a satisfactory PhD. However, while there appears to be consensus with the literature on the ‘ingredients’, the ‘recipe’ for success remains elusive. In conclusion, our analyses of PhD examiner ratings and comments did not conclusively identify one or more distinctive ingredient or the hallmark that would assure success in crossing the doctoral threshold. Our findings support the view that scholarship in a PhD is subject to interpretation; because examiners identify and diagnose problems based on their expectations they should therefore clarify and teach what is required (Kumar and Stracke, 2018). We live and work in an age when doctoral programmes are following increasingly diverse paths, and while this makes standardization both less needed and more difficult, we also echo the plea of some of our examiners for guidelines on the basic conceptual and philosophical necessities of a doctoral thesis in any format. A framework for thesis quality that includes a set of stable indicators that distinguish between theses of ‘threshold quality’ and above is required (Holbrook et al., 2008). The processes of doctoral research and examination would be enhanced by the provision of explicit guidelines on the format and content of both PhDs (by thesis and publication) and examiner reports. This would also serve to guide students and supervisors to ensure that the essence of doctorateness is maintained while enhancing the doctoral experience and quality. 6. REFERENCES BADLEY, G. 2009. Publish and be doctor-rated: The PhD by published work. Quality Assurance in Education, 17, 331-342. https://doi.org/10.1108/09684880910992313 BALLARD, B. Contexts of Judgement: an analysis of some assumptions identified in examiners’ reports on 62 successful PhD theses. Conference on Quality in Postgraduate Research-is it happening, 1996. BECHER, T., HENKEL, M. & KOGAN, M. 1994. Graduate Education in Britain, London, Jessica Kingsley. https://doi.org/10.1108/09684880910992313 146 Perspectives in Education 2019: 37(1) BOURKE, S. & HOLBROOK, A. P. 2013. Examining PhD and research masters theses. Assessment & Evaluation in Higher Education, 38, 407-416. https://doi.org/10.1080/026029 38.2011.638738 BRAUN, V. & CLARKE, V. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology, 3, 77-101. https://doi.org/10.1191/1478088706qp063oa DAVIES, R. E. & ROLFE, G. 2009. PhD by publication: A prospective as well as retrospective award? Some subversive thoughts. Nurse Education Today, 29, 590-594. https://doi. org/10.1016/j.nedt.2009.01.006 DELAMONT, S., ATKINSON, P. & PARRY.O. 2000. The nature of the quest. The doctoral experience: success and failure in graduate school. London: Falmer. DOWLING, R., GORMAN-MURRAY, A., POWER, E. & LUZIA, K. 2012. Critical reflections on doctoral research and supervision in human geography: The ‘PhD by publication’. Journal of Geography in Higher Education, 36, 293-305. https://doi.org/10.1080/03098265.2011.638368 FRICK, L. 2016. PhD by publication: An institutional analysis. In: MAGDA FOURIE- MALHERBE, RUTH ALBERTYN, CLAIRE AITCHISON & BITZER., E. (eds.) The future of postgraduate supervision in the knowledge society. First ed. Stellenbosch: SunMedia. GROSSMAN, E. S. & CLEATON-JONES, P. E. 2011. Pipelines or pipe dreams? . Perspectives in Education, 29, 111-125. HOLBROOK, A. & BOURKE, S. 2004. An Investigation of PhD Examination Outcome in Australia Using a Mixed Method Approach. Australian Journal of Educational & Developmental Psychology, 4, 153-169. HOLBROOK, A., BOURKE, S., FAIRBAIRN, H. & LOVAT, T. 2007. Examiner comment on the literature review in Ph. D. theses. Studies in Higher Education, 32, 337-356. https://doi. org/10.1080/03075070701346899 HOLBROOK, A., BOURKE, S., LOVAT, T. & DALLY, K. 2004a. Investigating PhD thesis examination reports. International Journal of Educational Research, 41, 98-120. https://doi. org/10.1016/j.ijer.2005.04.008 HOLBROOK, A., BOURKE, S., LOVAT, T. & DALLY, K. 2004b. Qualities and Characteristics in the Written Reports of Doctoral Thesis Examiners. Australian Journal of Educational & Developmental Psychology, 4, 126-145. HOLBROOK, A., BOURKE, S., LOVAT, T. & FAIRBAIRN, H. 2008. Consistency and inconsistency in PhD thesis examination. Australian Journal of Education, 52, 36-48. https:// doi.org/10.1177/000494410805200103 JOHNSTON, S. 1997. Examining the examiners: an analysis of examiners’ reports on doctoral theses. Studies in higher education, 22, 333-347. https://doi.org/10.1080/030750797123313 80936 JOYNER, R. W. 2003. The selection of external examiners for research degrees. Quality assurance in education, 11, 123-127. https://doi.org/10.1108/09684880310471551 KILEY, M. & MULLINS, G. 2004. Examining the examiners: How inexperienced examiners approach the assessment of research theses. International Journal of Educational Research, 41, 121-135. https://doi.org/10.1016/j.ijer.2005.04.009 https://doi.org/10.1080/02602938.2011.638738 https://doi.org/10.1080/02602938.2011.638738 https://doi.org/10.1191/1478088706qp063oa https://doi.org/10.1016/j.nedt.2009.01.006 https://doi.org/10.1016/j.nedt.2009.01.006 https://doi.org/10.1080/03098265.2011.638368 https://doi.org/10.1080/03075070701346899 https://doi.org/10.1080/03075070701346899 https://doi.org/10.1016/j.ijer.2005.04.008 https://doi.org/10.1016/j.ijer.2005.04.008 https://doi.org/10.1177/000494410805200103 https://doi.org/10.1177/000494410805200103 https://doi.org/10.1080/03075079712331380936 https://doi.org/10.1080/03075079712331380936 https://doi.org/10.1108/09684880310471551 https://doi.org/10.1016/j.ijer.2005.04.009 147 Ramlall, Singaram & Sommerville Doctorates by thesis and publication in clinical medicine KUMAR, V. & STRACKE, E. 2018. Reframing doctoral examination as teaching. Innovations in Education and Teaching International, 55, 219-227. https://doi.org/10.1080/14703297.201 7.1285715 LEE, A. 2010. When the article is the dissertation: Pedagogies for a PhD by publication. In: AITCHISON, C., KAMLER, B. & LEE, A. (eds.) Publishing pedagogies for the doctorate and beyond. Oxon: Routledge. LESHEM, S. & TRAFFORD, V. 2007. Overlooking the conceptual framework. Innovations in education and Teaching International, 44, 93-105. https://doi.org/10.1080/14703290601081407 MASON, S. 2018. Publications in the doctoral thesis: challenges for doctoral candidates, supervisors, examiners and administrators. Higher Education Research & Development, 37, 1231-1244. https://doi.org/10.1080/07294360.2018.1462307 MIHAS, P. 2019. Qualitative data analysis In: NOBLIT, G. (ed.) The Oxford Encyclopedia of Qualitative Research Methods in Education Oxford University Press. https://doi.org/10.1093/ acrefore/9780190264093.013.1195 MORLEY, L., LEONARD, D. & DAVID, M. 2003. Quality and equality in British PhD assessment. Quality assurance in education, 11, 64-72. https://doi.org/10.1108/09684880310471489 MULLINS, G. & KILEY, M. 2002. ‘It’s a PhD, not a Nobel Prize’: how experienced examiners assess research theses. Studies in higher education, 27, 369-386. https://doi. org/10.1080/0307507022000011507 POWELL, S. & MCCAULEY, C. 2003. The process of examining research degrees: some issues of quality. Quality assurance in education, 11, 73-83. https://doi. org/10.1108/09684880310471498 PRIETO, E., HOLBROOK, A. & BOURKE, S. 2016. An analysis of PhD examiners’ reports in engineering. European Journal of Engineering Education, 41, 192-203. https://doi.org/10.108 0/03043797.2015.1056096 ROBINS, L. M. & KANOWSKI, P. J. 2008. PhD by publication: A student’s perspective. Journal of Research Practice, 4, 3. TRAFFORD, V. & LESHEM, S. 2009. Doctorateness as a threshold concept. Innovations in education and teaching international, 46, 305-316. https://doi.org/10.1080/14703290903069027 UNIVERSITY OF KWAZULU NATAL Undated-a. Examiners Questionnaire: College of Health Sciences In: UNIVERSITY OF KWAZULU NATAL (ed.). Howard College, Durban: UKZN. UNIVERSITY OF KWAZULU NATAL Undated-b. Information for the guidance of examiners’ of higher degrees In: UNIVERSITY OF KWAZULU NATAL (ed.). Howard College, Durban: UKZN. VIEIRA, K. A. L. & DE QUEIROZ, G. M. 2017. Hermeneutic Content Analysis: a method of textual analysis. International Journal of Business Marketing and Management 2, 8-15. https://doi.org/10.1080/14703297.2017.1285715 https://doi.org/10.1080/14703297.2017.1285715 https://doi.org/10.1080/14703290601081407 https://doi.org/10.1080/07294360.2018.1462307 https://doi.org/10.1093/acrefore/9780190264093.013.1195 https://doi.org/10.1093/acrefore/9780190264093.013.1195 https://doi.org/10.1108/09684880310471489 https://doi.org/10.1080/0307507022000011507 https://doi.org/10.1080/0307507022000011507 https://doi.org/10.1108/09684880310471498 https://doi.org/10.1108/09684880310471498 https://doi.org/10.1080/03043797.2015.1056096 https://doi.org/10.1080/03043797.2015.1056096 https://doi.org/10.1080/14703290903069027 __Fieldmark__198_1107328704 __Fieldmark__216_1107328704 __Fieldmark__258_1107328704 __Fieldmark__340_1107328704 __Fieldmark__349_1107328704 __Fieldmark__357_1107328704 __Fieldmark__397_1107328704 __Fieldmark__426_1107328704 __Fieldmark__435_1107328704 __Fieldmark__462_1107328704 __Fieldmark__482_1107328704 __Fieldmark__492_1107328704 __Fieldmark__526_1107328704 __Fieldmark__543_1107328704 __Fieldmark__557_1107328704 __Fieldmark__575_1107328704 __Fieldmark__606_1107328704 __Fieldmark__637_1107328704 __Fieldmark__648_1107328704 __Fieldmark__726_1107328704 __Fieldmark__749_1107328704 __Fieldmark__782_1107328704 __Fieldmark__791_1107328704 __Fieldmark__811_1107328704 __Fieldmark__823_1107328704 __Fieldmark__859_1107328704 __Fieldmark__884_1107328704 __Fieldmark__939_1107328704 __Fieldmark__963_1107328704 __Fieldmark__2627_1107328704 __Fieldmark__2663_1107328704 __Fieldmark__2672_1107328704 __Fieldmark__2794_1107328704 __Fieldmark__2905_1107328704 __Fieldmark__2937_1107328704 __Fieldmark__2999_1107328704 __Fieldmark__3011_1107328704 __Fieldmark__3082_1107328704 __Fieldmark__3125_1107328704 __Fieldmark__3156_1107328704 _GoBack