June 2021, Vol. 13, No. 2 AJHPE 129 Research In 2014, the World Health Assembly (WHA) Resolution A67.19 called on countries to strengthen and ensure equitable access to palliative care (PC). [1] This resolution also stated that PC education should form an ‘integral component’ of ongoing education, and that disciplines working routinely with patients with life-threatening illnesses should receive intermediate- level PC training.[1] The resolution was supported by African countries who signed the Kampala Declaration, and by the South African (SA) Department of Health (the SA National Policy Framework and Strategy on Palliative Care).[2,3] However, to date, no dedicated funding has been set aside to strengthen palliative care in SA. The need for integration of PC has never been more strongly felt than in the current African cancer care situation. The burden of cancer cases is rapidly increasing in low- and middle-income countries. Although situations differ from country to country, many patients generally present late, and struggle to access oncology care timeously.[4] In Africa, long distances to oncology centres and long waiting lists to receive oncology services compound the problem of access to PC. The barriers to the integration of PC into the health system and oncology care are well described, and some of these are the lack of available PC training and the misconception that PC is only needed for end-of-life care.[5] Academic cancer centres in SA provide training for radiation and/or clinical oncologists. These specialist trainees not only care for patients who require curative treatments, but frequently also serve the many patients whose care is of palliative intent. In response to the need for and the importance of PC in oncology care, especially in Africa, the University of Cape Town (UCT) developed a PC curriculum at the intermediate level. Background. Following a World Health Assembly call in 2014 to strengthen palliative care, the South African (SA) Department of Health approved this strategy as part of the SA National Policy Framework and Strategy on Palliative Care. In 2016, the University of Cape Town, together with the College of Radiation Oncology of SA, identified the need to integrate palliative care (PC) into the oncology curriculum. In collaboration with the Cancer Association of SA, a research project was developed to introduce a 12-module curriculum at five teaching hospitals. The aim of this research was to evaluate the impact of a 1-year PC course within the training programme for specialist oncologists in SA. Objective. To determine the reaction of oncology registrars and their supervisors to the course to determine changes in knowledge and skills, and to determine the application in oncology practice. Methods. This study was a mixed-method prospective evaluation of an educational intervention. The educational programme used a blended learning method to train and support registrars (n=32) and facilitators (n=5) across five universities from August 2017 to September 2018. Evaluation feedback was electronically collected to determine the registrars’ reactions to the course materials. Pre and post multiple-choice questions (MCQs) were used to review their knowledge. Focus group discussions (FGDs) were used to explore reactions, change in knowledge and skills and how registrars integrated PC into their daily work. Results. There was an overwhelmingly positive reaction to the PC course by the oncology registrars and their supervisors. The training was found to be feasible, and the topics addressed appropriate. Concerns previously raised by the College of Radiation Oncology of SA regarding the feasibility and appropriateness of the course and material were found to be unsubstantiated. The poor MCQ results can be ascribed to poor sequencing of the execution of the question. However, the MCQs in modules 7 and 8 (symptom management) demonstrated the most significant change in knowledge and skills (symptom management). The FGDs demonstrated a perceived change in knowledge and skills, especially for communication skills and pain and symptom management. The FGDs also indicated that the registrars’ approach to PC changed in that they were able to integrate the principles of PC into practice, and now saw PC as an essential component of oncology. Lastly, registrars and their supervisors felt that the course addressed topics that formed part of their daily clinical work. Conclusion. This research supports the view that PC training is an essential component of oncology training in the SA setting. PC forms part of the daily practice of oncologists, and a structured curriculum prepares clinicians to be able to integrate evidence-based PC into the practice of oncology if they receive appropriate training. Supervisors of the oncology training programme and registrars are confident that the training of 12 modules across 1 year is feasible and appropriate. Afr J Health Professions Educ 2021;13(2):129-134. https://doi.org/10.7196/AJHPE.2021.v13i2.1268 Evaluating palliative care training in the oncology registrar programme in South Africa R Krause,1 MB ChB, M Fam Med, MPhil (Pall Med); J Parkes,2 MB ChB, DA (SA), FC Rad Onc (SA); D Anderson,2 MB ChB, FC Rad Onc (SA); N Hartman,3 PhD; L Gwyther,1 MB ChB, BSc (Pall Med), PhD 1 Department of Public Health and Family Medicine, Faculty of Health Sciences, University of Cape Town, South Africa 2 Department of Radiation Oncology, Faculty of Health Sciences, University of Cape Town, South Africa 3 Education Development Unit, Faculty of Health Sciences, University of Cape Town, South Africa Corresponding author: R Krause (rene.krause@uct.ac.za) This open-access article is distributed under Creative Commons licence CC-BY-NC 4.0. https://doi.org/10.7196/AJHPE.2021.v13i2.1268 mailto:rene.krause@uct.ac.za 130 June 2021, Vol. 13, No. 2 AJHPE Research The intention was to strengthen the oncology curriculum and align it with international standards and the SA National Framework and Strategy on Palliative Care.[3,6] The curriculum was developed after a process of consultation, drawing from a survey among trainees, exploring the knowledge and attitudes towards PC. A focus group discussion (FGD) among experts in cancer care, including both academic and private oncology and PC experts, was also conducted, drawing on international recommendations on PC training.[7,8] The conclusions drawn from this process included that PC should indeed form part of all oncology services, that PC is a critical part of oncology training, that PC training should be delivered in a structured curriculum with well-defined outcomes, that PC training should be provided early in the training, that PC should form part of the summative academic assessment and that communication skills should be a critical component of the course. These findings were presented to the College of Radiation Oncologists of SA at the examiners’ meeting held in October 2016. A general agreement was reached that a PC course should be delivered to new registrars in oncology over a 1-year programme, to be conducted from August 2017 to September 2018, that should include 12 modules. Concerns were raised around the feasibility of such a course in an already full curriculum, and therefore not all universities participated in this pilot project. The course used a mixed mode of teaching, with online content and monthly facilitation sessions. The main modules included principles of PC (5%), communication skills (40%), pain and symptom control (30%), ethical and legal aspects around end-of- life care (5%), the implementation of advance directives (5%), bereavement and interdisciplinary teamwork (5%) and burnout and compassion fatigue (5%) (Table 1). It was also agreed that such a course should demonstrate evidence of improvement in teaching and learning outcomes in order to justify implementation, and that this would be best shown as part of a research project. Problem statement Before the introduction of this course, there was no structured PC curriculum within the SA oncology specialist training programme. Additions to an already full curriculum should not overburden trainees, should be appropriate for the SA setting and should demonstrate improvement in teaching and learning outcomes of oncology trainees. The rationale for this study was to evaluate a 1-year palliative care training module within the oncology curriculum. Methodology All oncology training centres across SA were invited to participate in the course, but owing to logistical constraints, only five universities participated (UCT, Stellenbosch University, Walter Sisulu University, the University of KwaZulu-Natal (UKZN) (Pietermaritzburg) and the University of the Free State). The main reason cited for not participating was resource constraints. Facilitators were trained at a 2-day workshop at the beginning of the course. During the workshop, the training team familiarised the facilitators with the Vula e-learning website and training materials, and upskilled them in communication skills and in providing student feedback. The modules were developed via a collaborative approach between oncology and PC. The final modules addressed concepts around self-care, and were developed with the help of psychologists. Five pre- and five post-training multiple-choice questions (MCQs) were developed for each assessment. Communication skills were assessed using a modified Calgary-Cambridge method.[9] The communication skill assessment formed part of the formative assessment, and provided an opportunity for registrars to receive feedback on their communication skills. The facilitators also met monthly online to discuss the new learning material and to reflect on the previous month’s lectures. Each site had a monthly meeting in which the material was discussed, and a task was completed to apply new theory learned. The participating universities invited all specialist trainees to participate in the research project, and assured them that their training would not be affected if they decided not to participate. Thirty-two oncology trainees from the five universities participated. These trainees included SA and international registrars. Although the course was geared towards adult PC, a paediatric oncology trainee also participated. Continuous evaluation of the course was made essential, and was used to demonstrate whether there was evidence of improvement in the identified proficiencies. The course was evaluated using the theory of the adapted version of the Kirkpatrick triangle.[10] The best evidence medical education (BEME) collaboration adapted a version of the Kirkpatrick triangle called ‘Kirkpatrick’s hierarchy’ system to higher education for evaluation.[10,11] This modified version was adopted to measure ‘soft outcomes’ of the course together with short-term and tangible outcomes. A mixed mode of evaluation was used to measure both quantitative and qualitative data to determine a comprehensive review of teaching activity.[10] This enabled the researcher to capture the softer nuances of the impact of the course. One of the objectives of the course was to evaluate the oncology trainees’ reaction to teaching and learning by completing anonymous online evaluations. These evaluated the structure of the course, the relevance of the discussion groups, the course material and the applicability of the material in daily clinical practice. FGDs were conducted to determine trainees’ perceived change in attitudes and perceptions towards PC. The second objective was to determine the trainees’ change in knowledge and skills by asking them to complete pre- and post-training MCQs after each module. The third objective was to determine the application of knowledge and skills of PC in oncology practice by interviewing the supervisors of trainees. Change in organisational practice and benefit to patients were not included in this study. The quantitative data gathered were statistically analysed, and the qualitative data were analysed by the research team using thematic analysis. The researchers familiarised themselves with the transcripts collected. Through inductive and iterative processes, themes were identified, and Table 1. Modules and content Module Topic 1 Principles of palliative medicine in oncology 2 Communication skills: Basic principles 3 Communication skills: Breaking bad news 4 Communications skills: Conversations around serious illness and care planning 5 Communication skills: Managing conflict 6 Pain management 7 Gastrointestinal symptoms 8 Dyspnoea and delirium 9 Constitutional and treatment-related symptoms 10 End-of-life care 11 The role of the oncologist providing comprehensive care 12 Self-care June 2021, Vol. 13, No. 2 AJHPE 131 Research were adjusted and/or confirmed. Confirmation bias was limited by using a multidisciplinary research team and an anonymous evaluation platform, and by including feedback from senior supervisors. Ethics approval was obtained from the UCT Human Research Ethics Committee (ref. no. HREC 851/2016). Ethics approval was also obtained from the ethics departments of all the universities who participated in this study (UKZN ref. no RECIP 299/17; other universities letters provided). Twelve online modules were developed with MCQs. Results Feedback from the five sites indicated that trainees participated enthusiastically, and that the material resonated with their daily experiences. The Vula e-learning site provided researchers with the number of times the site was visited by the registrars (Fig. 1) up to 6 months after the completion of the course, and data on resources most used (Table 2). Videos were the most frequently used resource, and the site was most often visited when communication skills were addressed. The videos that were most watched were those on pain management (174 views) and breaking bad news (125 views). The written resource the registrars used most was ‘Guide to the treatment of cancer pain’.[12] There was excellent attendance at the monthly meetings. Although not all registrars completed the required online preparation, there was active participation by all at the meetings themselves. Unfortunately, owing to workload restrictions, only a 1-hour meeting was allocated per module. This should ideally be longer, as each module generated enthusiastic discussions that had to be cut short. The modules that generated the most discussion were communication, self-care, end-of-life care and comprehensive care. This may be due to these subjects not having been part of any formal oncology training in the past. These modules are also very emotive, and the meetings allowed the registrars to express and debate ideas that would be difficult to discuss in a forum or chat room. The registrars often incorporated real-life cases into these meetings, which helped add a valuable practical side to the course. The structure of the modules (Fig. 2), the relevance of the discussion groups (Fig. 3), the appropriateness of the course material (Fig. 4) and of the applicability of the material in daily clinical practice (Fig. 5) were anonymously evaluated. Modules 7 and 8 received 100% positive feedback. This evaluation also assisted in reviewing modules for further courses (module 4 was not evaluated owing to technical difficulties, so is excluded from the figures). The changes between the pre- and post-test results of the MCQs were analysed by running repeated sample t-tests on each module. The differences between the pre- and post-test results are generally not statistically significant, except for those from module 1 and module 7, which demonstrated negative (module 1) and positive (module 7) outcomes in terms of a change of knowledge. In modules 5 and 8 the sample size was small,[5] tending to significance (p=0.099 and 0.089, respectively). This may require further exploration in a larger group to determine whether the effect is real. The small number of participants completing the MCQs was the main reason that the statistical tests were not of any value. The problem was not the construction of the MCQ questionnaire, but rather the way that it was situated within the programme, and the fact that it was not made obligatory. Online assessment may not be the best form of assessment if it is not rigorously enforced. Focus groups In general, better feedback information was obtained from the FGDs than from the assessment documents. Data were analysed using systematic thematic analysis with inductive coding. The FGD notes were read and reread, and discussed with the research team. Themes emerged as shown in Table 3. Theme 1: Attitudes towards the course Positive feedback It was clear that registrars deal with PC on a daily basis. The course introduced topics that form part of their daily work. There was an overwhelmingly positive attitude towards the course, and all registrars felt that they wanted to continue with it. The course changed their approach to PC, and most stated that PC is part of oncology care. Many felt that the course highlighted the fact that PC is an essential component of oncology training not only for palliative patients, but for better management of all oncology patients: ‘It is actually part of the bread and butter of oncology in my experience and there is so much that we’ve learnt throughout the module with regards to palliative care.’ Student, UKZN (StUK) ref 3 ‘For palliative care is, even in the non- metastatic setting, in patients with curable intent or radical intent, there is still so much that you can learn from palliative care in terms of communication skills, breaking bad news and so I found it extremely important in the oncology setting.’ StUK ref 4 Negative feedback It was an adjustment for the registrars to become online students. There was also variability in terms of the different facilitators at each university, which affected registrars’ attitude towards the course. Registrars also felt that Table 2. Resources most frequently opened Resource Vula visits, n Pain module mp4 174 Breaking bad news mp4 125 Serious illness conversation mp4 91 Dyspnoea in end-stage cancer mp4 89 The oncologist providing comprehensive care 81 End-of-life care 74 2017 2018 2019 500 450 400 350 300 250 200 150 100 50 0 Si te v is it s, n Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Jan Feb Mar Study period 141 54 316 49 54 222 432 285 201 63 200 187 326 116 406 150 183 207 196 77 21 55 18 7 1 Fig. 1. Vula visits by registrars over course period, n. 132 June 2021, Vol. 13, No. 2 AJHPE Research many of the skills were not mentored to them, because supervisors had no training in PC. This lack of comprehensive training affected integration into the daily care of patients: ‘That one is just an IT challenge, where sometimes every time you needed to log in.’ Student, Stellenbosch University (StUS) ref 3 ‘I think, I know is probably idealistic thinking but I do think that not, maybe if you could have like a crash course version for even consultants … I also think it will be good for them to understand what it is that we are learning in the curriculum and it would help them support us when we try to apply these things in the day-to-day dealing with patients in clinic and all of these.’ Student, University of Cape Town (StUU) ref 3 Theme 2: Workload and structure of the course Feasibility The registrars felt that the workload was achievable and that the course was feasible in their own setting. The supervisors’ concern was that some registrars did not prepare for lectures beforehand. There were some concerns about how to better teach communication skills and mentoring, and about how to use these in current clinical practice: ‘I have to just generalise I would say that the curriculum is quite comprehensive and that the pace at which the curriculum was presented was a good, manageable for the students and for the facilitator and that the variety of faculty brought depth and richness to the curriculum.’ Supervisor, Stellenbosch University (SupUS) ref 1 ‘I know this is not the question but um, probably the communication part is something that is still struggling to, you know, kind of apply practically just because of the kind of feasibility of it in our setting.’ StUU ref 1 ‘I think a negative thing for me was that during the course, most of the time, most of the students did not, you know, did not prepare beforehand.’ Supervisor, University of the Free State ref 1 Structure of the course Registrars wanted short, precise and very practical material. They enjoyed the video material and group interactions. However, supervisors felt that the course needed to be more academic, with journal articles and assessments built in, in order to engage registrars in their capacity as postgraduates: ‘I think it will be nice to be, maybe interactive with the other groups in other parts of the country.’ StUS ref 1 ‘… felt that the videos were very, very nice.’ StUU ref 4 ‘I also feel that way and I do think that it sometimes takes the academics out of it. We mustn’t take the academics out of it.’ SupUU ref 7 100 90 80 70 60 50 40 30 20 10 0 Module % Positive Neutral Negative 1 2 3 5 6 7 8 9 10 11 12 Fig. 3. Participant (registrar) evaluation of discussion group relevance. 100 90 80 70 60 50 40 30 20 10 0 Module % Positive Neutral Negative 1 2 3 5 6 7 8 9 10 11 12 Fig. 2. Participant (registrar) evaluation of module structure. 100 90 80 70 60 50 40 30 20 10 0 Module % Positive Neutral Negative 1 2 3 5 6 7 8 9 10 11 12 Fig. 4. Participant (registrar) evaluation of appropriateness of course material. 100 90 80 70 60 50 40 30 20 10 0 Module % Positive Neutral Negative 1 2 3 5 6 7 8 9 10 11 12 Fig. 5. Participant (registrar) evaluation of ability of course material to assist registrars in daily practice. Table 3. Focus group discussions Theme 1: Attitude towards the course Positive Negative Theme 2: Workload and structure of the course Feasibility Structure Modules Theme 3: Application in the workplace Theme 4: Integration of palliative care principles into oncology June 2021, Vol. 13, No. 2 AJHPE 133 Research Modules The module that contributed most to registrars’ learning was communication skills. The registrars felt that this assisted them in learning and development beyond working with PC patients: ‘I also think that communication and breaking bad news is the main thing.’ StUF ref5 ‘…will improve my way of relating to patients. And I found it very valuable when I was doing my exams because it helped me really communicate very well during the oral exams. Communications were very key both to the question, in taking history and also to the examiners.’ StUU ref 6 Pain management was a vital component to the course, and this was supported by supervisors and by registrars: ‘I can say that I have seen changes and it will mostly be around pain management.’ SupUS ref1 ‘I think the module on pain control and it was more I think the practical advice that you as a palliative care physician could actually offer, that you can’t really get in textbooks which were very, very useful.’ StUU ref 1 Self-care was a new topic introduced into oncology as a formal component in the curriculum: ‘…think for me probably the one module that stood out the most for me was the self-care. I think at the end of the day as a clinician in order for you to be able to manage your patient well, you have to be physically, mentally, you have to be mentally, you know, in the right space to be able to treat with, to deal with your patient.’ Student Eastern Cape ref 1 Theme 3: Application in the workplace The registrars indicated that the course contributed to their daily practice and that it gave them the confidence to manage situations that they confronted regularly and in which they had not previously received training. The training also contributed to their approach to dealing with patients for curative intent, and they stated that they could use the skills in activities other than patient interactions: ‘It gave us a bit more confidence to approach a lot of the topics. So we were able to manage our patients a bit more confidently with good backing and theoretical knowledge of each subject matter.’ StUK ref 2 ‘I can say that I have seen changes and it will mostly be around pain management.’ SupUS ref 1 Theme 4: Integration of PC into oncology The course enabled registrars to integrate key concepts in PC: PC starts at diagnosis; holistic care; teamwork; pain and symptom control; and the involvement of the family. The course also enabled registrars to see the role of a PC team in the care of oncology patients: ‘I feel is very important because we deal with a lot of patients that require palliative care and palliative care being not just for the end-stage patients, at beginning, right from the beginning from the diagnoses.’ StUS ref 1 ‘We generally tend to focus more towards chemotherapy and radiation not forgetting the more important biopsychosocial, the psychosocial part of it. So it made us more aware of trying to address those problems providing us with insight into an approach to these problems and enable us to provide holistic management in all aspect of care.’ StUKref6 ‘I think also is good to also be aware of your own limitations. You know to be aware that there are palliative care facilities and the team of palliative care specialists. To know when your role, you know, I have done all I can and I need to call someone else in.’ StUF ref 1 Discussion The course was developed using minimal resources but with strong collaboration among groups. The collaborators were the departments of palliative medicine (UCT), radiation oncology (UCT), health professional education (UCT), radiation oncology (Stellenbosch University), a private palliative care physician and the Cancer Association of SA (CANSA). It used blended learning to enable facilitation in different oncology settings and to support facilitators across SA. The facilitators came from five universities, which enabled a community of trainers to develop. Blended learning is a new method of learning and assessment for many registrars and supervisors, and some registrars found this challenging. In future, more time needs to be spent initially familiarising registrars and supervisors with an online teaching platform. Although blended learning has its challenges, it enables support and development of more geographical sites in integrating PC into a curriculum using minimal resources. The first objective of this research was to determine the reaction of oncology registrars and their supervisors to a PC course. There was an overwhelmingly positive reaction towards the course, and it surpassed our expectations. Concerns from supervisors about issues such as the feasibility and appropriateness of the course and material were unsubstantiated. PC training in oncology is feasible, and the topics addressed were appropriate. This was supported by both anonymous evaluation forms and qualitative data from the FGDs. The modules on communication skills and pain and symptom management were found to be the most valuable components of the course for registrars. This was supported by evaluation forms, data from the Vula site and FGDs. Self-care was also found to be a valuable component of the course, and may be included in more programmes, according to comments received from the FGDs. The second objective was to determine changes brought about in knowledge and skills in PC. Some negative aspects of this study were that there were generally no statistically significant changes in pre- and post- test results, and that there was poor completion of the online assessment. Knowledge in practical aspects such as pain and symptom control improved, however. The FGDs demonstrated that the registrars’ knowledge of what PC is and how they could integrate the principles in their daily work improved. The third objective was to determine the application of knowledge and skills of PC in oncology practice. The registrars felt that the course made oncology ‘more fluid’, which enabled them to move between disease- specific care and a patient-centred approach, and between a curative and a palliative approach, more easily. The course addressed topics that formed part of their daily work. The skills and knowledge most applicable in the oncology setting are communication skills and pain and symptom management. These skills therefore need to be core components that must be included in all oncology curricula. However, these competencies do not stand alone, and are best managed if the principles of PC are integrated into all components of care. 134 June 2021, Vol. 13, No. 2 AJHPE Research Conclusion Internationally, PC is an essential component in oncology training, and should be incorporated into the SA specialist oncology training programme.[1] PC forms part of the daily practice of oncologists, and a structured curriculum enables clinicians to practise using an evidence-based approach. This provides registrars with confidence when managing a patient with PC needs. Supervisors of oncology training and registrars in the present study proved confident that the training of 12 modules across 1 year is feasible and appropriate. These skills are part of daily practice, and support the WHA resolution 67.19 that PC training is an integral component of oncologist training: ‘Intermediate training should be offered to all health care workers who routinely work with patients with life-threatening illnesses, including those working in oncology, infectious diseases, paediatrics, geriatrics, and internal medicine.’[1] The sustainability of this course can only be ensured by advocating for strong collaboration between academic oncology units and academic PC units. PC training must also become available to the oncologist who has not been previously exposed to it, through short courses. This may serve as a module to integrate PC into other disciplines. Declaration. This study formed part of RK’s PhD research. Acknowledgements. CANSA. Author contributions. RK: data collection; RK and LG: analysis; all authors: manuscript writing; all authors: scientific input. Funding. CANSA. Conflicts of interest. None. 1. World Health Assembly. Strengthening of palliative care as a component of integrated treatment within the continuum of care. Geneva: World Health Organization, 2014. 2. African Palliative Care Association. Kampala Declaration. Kampala: 5th International African Palliative Care Conference, 2016. 3. National Department of Health, South Africa. National Policy Framework and Strategy on Palliative Care 2017 - 2022. Pretoria: NDoH, 2017. 4. Bray F, Jemal A, Grey N, Ferlay J, Forman D. Global cancer transitions according to the Human Development Index (2008 - 2030): A population-based study. Lancet Oncol 2012;13(8):790-801. https://doi.org/10.1016/s1470- 2045(12)70211-5 5. Aldridge MD, Hasselaar JG, Garralda E, et al. Education, implementation, and policy barriers to greater integration of palliative care: A literature review. Palliat Med 2016;30(3):224-239. https://doi.org/10.1177/0269216315606645 6. Smith TJ, Temin S, Alesi ER, et al. American Society of Clinical Oncology provisional clinical opinion: The integration of palliative care into standard oncology care. J Clin Oncol 2012;30(8):880-887. https://doi. org/10.1200/jco.2011.38.5161 7. Gwyther L, Parkes J, Anderson D, Hartman N, Krause R. Building a palliative care curriculum for oncologists. Specialist Forum 2018;18(4):42-43. 8. Gamondi C, Larkin P, Payne S. [Core competencies in palliative care: An EAPC White Paper on palliative care education – part 1]. European Association of Palliative Care, 2013. Eur J Palliative Care 2013;20(2):86-91. 9. Kurtz SM, Silverman JD. The Calgary-Cambridge referenced observation guides: An aid to defining the curriculum and organizing the teaching in communication training programmes. Int J Med Educ 1996;30(2):83- 89. https://doi.org/10.1111/j.1365-2923.1996.tb00724.x 10. Yardley S, Dornan T. Kirkpatrick’s levels and education ‘evidence’. Med Educ 2012;46(1):97-106. https://doi. org/10.1111/j.1365-2923.2011.04076.x 11. Praslova L. Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educ Assess Evaluation Account 2010;22(3):215-225. https://doi. org/10.1007/s11092-010-9098-7 12. South African Cancer Pain Working Group. Guide to the Treatment of Cancer Pain in South Africa. 2015. https://painsa.org.za/wp-content/uploads/2016/02/A5-Guide-to-treatment-of-Cancer-Pain-2015.pdf (accessed 7 June 2021). Accepted 24 June 2020. https://doi.org/10.1016/s1470-2045(12)70211-5 https://doi.org/10.1016/s1470-2045(12)70211-5 https://doi.org/10.1177/0269216315606645 https://doi.org/10.1200/jco.2011.38.5161 https://doi.org/10.1200/jco.2011.38.5161 https://doi.org/10.1111/j.1365-2923.1996.tb00724.x https://doi.org/10.1111/j.1365-2923.2011.04076.x https://doi.org/10.1111/j.1365-2923.2011.04076.x https://doi.org/10.1007/s11092-010-9098-7 https://doi.org/10.1007/s11092-010-9098-7 https://painsa.org.za/wp-content/uploads/2016/02/A5-Guide-to-treatment-of-Cancer-Pain-2015.pdf