Paper title (Paper Title style) Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 1 The effectiveness of SMS Reminders and the impact of patient characteristics on missed appointments in a public dental outpatient clinic Emilia Bellucci Deakin Business School Deakin University emilia.bellucci@deakin.edu.au Lasitha Dharmesena Deakin Business School Deakin University Lemai Nguyen Deakin Business School Deakin University Hanny Calache School of Health and Social Development Deakin University Abstract This paper reports on the Failure To Attend (FTA) rate of appointments as well as patients following the implementation of SMS reminders in a public dental outpatient service. Given the ineffectiveness of the intervention and a highly representative patient’s profile, this paper identifies the demographic characteristics of patients who miss all of their appointments. Data on appointment attendance, patient demographics and dental service type was collected over a time period of 46 consecutive months. Using descriptive and inferential statistics (chi- square, two sample tests and Marascuilo procedure) we found the SMS intervention was ineffective in reducing the FTA rates. Further, patients associated with high rates of non- attendance exhibited one or more of the following characteristics: male; age 26 – 44; non- concession card holders; a person of Indigenous, local, Asian or African descent, and of refugee status, persons living in low socio-economic areas; and appointments in General Care and Student Clinics. Whilst the literature overwhelmingly attributes SMS reminders to improving the attendance rate of patients in outpatient clinics, our contradictory findings suggest a more targeted approach in settings whose patients exhibit strong characteristics associated with non-attendance. Keywords: Non-attendance; SMS reminder service; Public Outpatient Clinics; Dental Health Services; Patient demographic and appointment characteristics. 1 Introduction Patients failing to attend appointments is a very prevalent problem for outpatient clinics around the world. Non-attendance can result in the underutilization of resources and extended waiting lists (Geraghty et al. 2008), contribute to the loss of clinician time (Bos, Hoogstraten & Prahl-Andersen 2005) and delayed access to care for other patients (Stubbs et al. 2012). Patient failure to attend appointments also results in lower healthcare outcomes for patients, often due to missed opportunities to continuous healthcare (Cashman et al. 2004). The financial expense of missed appointments can be quite significant and can increase the cost of the service (Stubbs et al. 2012; Milne 2010; Perron et al. 2010). When a patient fails to attend an appointment, the outpatient service must still pay for the room, equipment, the salaries of clinicians and administrative staff for setting up the appointment. In addition, missed appointments contribute to already prolonged waiting periods for primary care Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 2 appointments and surgery. These costs are particularly concerning for outpatient clinics in the public health system as these measures are often used to calculate future funding. In Australia, public dental services are funded by the government. Access to dental services are usually restricted to adults in the public welfare system and to children. During 2007 – 2008, the Australian government spent approximately $6.1 billion (6.2% of the total health expenditure) on dental health services (Dental Health Services Victoria 2011). Given this substantial investment, it is imperative we attempt to limit opportunities for wasted resources such as those attributed to unattended appointments. In addition, research over the last 20 years has drawn significant links between oral health and serious health conditions including chronic diseases such as cardiovascular disease, diabetes, respiratory diseases, stroke, kidney disease, dementia and other serious aliments (Dental Health Services Victoria 2011). Therefore, ensuring the public maintain good oral health is not only important to help prolong life expectancies and to protect good quality of life, but it may have financial implications for increased expenditure across the whole health services network. Whilst Failure to Attend - FTA (also referred to as non-attendance rates, missed appointments, Did not attend - DNA or no-shows) vary significantly within and between outpatient specializations, the literature overwhelmingly finds the intervention of SMS reminders as major contributors to higher attendance rates. There are however, instances where SMS reminders have had no effect. In this article, we refute the universal application of SMS reminders, with the view that if the technology does not arrest the non-attendance rate, its’ adoption further impinges on limited financial resources. This study reports on the non-attendance rates of patients at a major dental outpatient clinic co-located on the same site as a public dental hospital. It is located within three kilometers of the CBD. The dental outpatient service provides very affordable dental services to persons on low incomes and to the socially marginalized - that is, to those who cannot afford private dental care. Further, the clinic is a key provider of dental services to the Victorian public. Given the highly representative demographic profile of its patients, this study also investigated a patient non-attendance score and whether a profile of characteristics common in patients who fail to attend appointments can be proposed. 2 Background 2.1 Literature review Interventions to tackle non-attendance have included postal, telephone and electronic reminders such as SMS. Outpatient clinics often use SMS to remind patients of their appointments, which are sent close to the date and time of the appointment. SMS text messages are found to be more effective or as effective as other forms of intervention (Milne 2010; Perron et al. 2010; Chen et al. 2008; Leong et al. 2006; Hasvold & Wootton 2011). SMS reminders are preferred due to lower costs and easy delivery of the intervention (Koshy, Car & Majeed 2008; McIver et al. 2015) in comparison telephone calls and mailed letters. SMS is a common mode of communication, particularly as the prevalence of mobile phone ownership is high in Australia, with penetration rates of 81% among the Australian public (Deliotte 2014). The literature overwhelmingly supports the adoption of SMS reminders to improve attendance rates at medical clinics around the world (Hasvold &Wootton 2011; Guy et al. 2012; Boksmati et al. 2016). A recently published meta-analysis and systematic review by Boksmati et al. (2016) shows an increased Odds Ratio compared to a previous meta-analysis of SMS reminder effectiveness in appointment adherence by Guy et al. (2012). Table 1 lists papers reporting on FTA rates by comparing patient cohorts sent SMS reminders (SMS) and those not sent a SMS reminder (Control) in medical outpatient settings. Given some studies suggest an analysis of patient characteristics could help to increase the effectiveness of appointment reminders (Cashman et al. 2004; Perron et al. 2010; Norton et al. 2014), the table also distinguishes any patient or appointment characteristics analysed and whether an association between such characteristics and the effectiveness of SMS reminders was reported. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 3 The association between non-attendance and Visit type (such as New or Follow up appointments) has been confirmed by five studies; whilst the other characteristics reflect mixed findings. In addition to the studies listed in the table, other studies analysing patient demographics of Gender and Age (irrespective of a reminder intervention) show an association between these demographic characteristics and general non-attendance (Cashman et al. 2004; Perron et al. 2010). Other characteristics mentioned in the non-attendance literature include associations with Socio-economic group (Cashman et al. 2004; Lalloo & McDonald 2010; McInnes et al. 2014) and Asylum seeker status (Perron et al. 2010). From Table 1, we can see the majority of articles report SMS reminders are helpful in reducing the non-attendance rate of patients, with a small number of studies reporting SMS reminders have no effect, suggesting the universal application of SMS as an effective measure to counter non-attendance should not be assumed. Equally, this table shows patient and appointment characteristics may help to predict the success of an intervention such as SMS reminders on patient non-attendance. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 4 Author, year Out-patient Clinic Country SMS successful? Non-attendance/attendance rates Characteristics associated with non-attendance within SMS groups Adankin et al. 2014 Post Natal Nigeria Yes Non-attendance rate = 42.85 (control); 21.3% (SMS). None tested Bos et al 2005 Orthodontic Netherlands No change Change did not reach statistical significance Age, Gender, Insurance Type were not associated with non-attendance Brannan et al. 2011 Ophthalmology UK Yes Non-attendance rate = 12% (control); 5.5 % (SMS) None tested Chen et al. 2008 Hospital Health promotion China Yes Attendance rate: 80.5% (control); 87.5% (SMS) None tested Da Costa et al 2010 General practice Brazil Yes Non-attendance rate = 25.57% (control); 19.42% (SMS) None tested Downer et al. 2005 Pediatric Hospital Australia Yes Non-attendance rate = 23.4 % (control); 14.2 % (SMS) None tested Downer et al. 2006 Pediatric Hospital Australia Yes Attendance rate: 80.5% (control); 90.2% (SMS). Non-attendance rate fell by 5.5% in New appointments and 11.6% in follow up appointments Visit type were associated with non- attendance Fairhurst & Sheikh, 2008 Primary care Scotland No change Change did not reach statistical significance None tested Farmer et al. 2013 Sexual health and HIV UK Yes and No Overall Non-attendance rate = 28% (control); 24% (SMS). Male clinic Non- attendance rate = 28% (control); 18 (SMS). No change in HIV clinic. Gender and Clinic Type were associated with non-attendance Foley, 2009 Pediatric Dental Scotland Yes and No Overall Non-attendance rate: 23.9% (control); 10.4% (SMS). Clinic type was associated with non- attendance Geraghty et al 2008 ENT Ireland Yes Non-attendance (mean) rate: 33.6% (control); 22% (SMS). None tested Koshy et al 2008 Ophthalmology UK Yes Non-attendance rates = 18.1% (control); 11.2% (SMS) None tested Leong et al 2006 Primary care Malaysia Yes Attendance rates: 48.1% (control); 59.6% (SMS) None tested Liew et al. 2009 Primary Care Malaysia Yes Non-attendance rate = 23.0 (control); 15.6 (SMS) None tested McInnes et al. 2014 Urban Veterans Primary Care US Yes 19% downward change in SMS group, compared to non-SMS group None tested Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 5 Author, year Out-patient Clinic Country SMS successful? Non-attendance/attendance rates Characteristics associated with non-attendance within SMS groups Milne et al. 2006 Hospital outpatient Scotland Yes and No Lower attendance rates for new appointments. No change in follow up appointments Visit Type was associated with non- attendance Milne, 2010 Hospital Scotland Yes New appointments Non-attendance rate = 6.8 (control); 3.7 (SMS). Return appointments Non-attendance rate = 5.2 (control); 4.0 (SMS). Visit Type was associated with non- attendance Narring et al. 2013 Hospital youth Switzerland No change Change did not reach statistical significance None tested Norton et al. 2014 HIV US No change Change did not reach statistical significance None tested Parikh et al. 2010 University US Yes Non-attendance rate = 23.1% (control); 17.3% (SMS) Age, Visit Type were associated with non- attendance Perry, 2011 Dental Scotland Yes Non-attendance rate = 31% (control); 14% (SMS) None tested Prasad & Arnand, 2012 Dental University India Yes Rate of attendance: 35.5% (control); 79.2% (SMS) None tested Sims et al. 2012 Mental Health England Yes Attendance rates were significantly higher for the 2009 and 2010 samples (SMS) than for the 2008 sample (control) Delivery time was not associated with non-attendance. Taylor et al. 2012 Physical Therapy Not Stated Yes Non-attendance rate = 16% (control); 11% (SMS) Age, clinic type, visit type and day was associated with non-attendance Youssef 2013 Hospital Saudi Arabia Yes Non-attendance rate = 39.8% (control); 26.3% (SMS) Age, Gender, nationality were not associated with non-attendance. First appointment was found associated with non-attendance. Table 1: Empirical studies assessing non-attendance /attendances rate in SMS reminder trials. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 6 2.2 Setting The dental outpatient clinic used in our study is located on the same site as a public dental hospital located in the Melbourne city centre. In the clinic, all patient appointment records are captured electronically using the Titanium patient management system of Spark Dental Technology (https://www.spark-dental.com/). The SMS Reminder system was programmed and integrated into the Titanium system by the hospital’s internal IT department. SMS reminders are sent in English one day prior to the appointment. The text reminds patients of the appointment time and date and contains instructions on how to cancel the appointment if required: “Reminder [Firstname]: Appointment at [ClinicName] on [AppointmentDate] at [AppointmentTime]. Call [ClinicPhone] ONLY if you CANNOT attend.” In Australia, dental care services are provided by the public and private sector, with the large majority of dentists working in the private sector. To be eligible for public dental care, certain eligibility criteria must be met. For adults to be eligible, they must be recipients of either a healthcare card or pensioner concession cards. These cards are offered to residents who are on low incomes and to refugees, asylum seekers and all Indigenous (Aboriginal and Torres Strait Islander) persons. The benefits of eligibility are affordable dental care, with most cardholders paying a small co-payment for services. The value of the co-payment varies according to whether patients attend a general service or a specialist service. In addition, if healthcare cardholders are being treated by dental students, then the treatment is free. The hospital also runs an emergency service for all persons. 3 Objective The demographic of patients attending the clinic is typical of persons who cannot afford private dental care. With a large proportion of the dental outpatient services’ patients falling into this group, it is important to investigate possible associations between patient demographics and non-attendance in addition to assessing the effectiveness of SMS reminders. In light of the literature review, the objective of the paper is two-fold: 1) To determine the effectiveness of an SMS reminder system to decrease the FTA rate of a public dental outpatient clinic. 2) To develop a patient profile with characteristics which are significantly associated with non-attendance. 4 Materials and Methods Prior to the commencement of the study human research ethics approvals for use of secondary de-identified data were obtained from the health service and affiliated university. Patient appointment data was obtained for a 46-month period from September 2009 to June 2013. During this time, not all appointments were sent SMS reminders, with text reminders sent according to clinic type, part of a trial rotation of dental rooms. Over the time period, all dental rooms included in the study were subjected to the trial. Records were automatically updated by a “1” in the “SMS sent” field if the SMS message was sent successfully to the registered mobile number. Data records included patient numbers, the date and time of the appointment, patient characteristics and the dental service type. Data was retrieved from the one site. Demographic factors collected include gender, country region, suburb region, age groups, Indigenous status, refugee (and asylum seeker) status, concession card holders and dental service type. The data attached to each appointment was extracted from the system and analysed. Adults (persons aged above 18) were included in the study and no identifying data such as names, addresses or phone numbers was collected. Patients were separated into age groups Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 7 representing distinct adult stages: young adult (18 – 25), adult (26 – 44), mature adult (45 – 65), seniors (65 +). Our groupings are consistent with Cashman et al (2004). Country regions (indicating a patient’s identifying cultural background) were organized by continent, the only exception was Australasia where local patients (predominately identifying as Australian or New Zealanders) were separated from the rest of Australasia, such as the Pacific Islands. Suburb regions were aggregated according to regions set by the Victorian Electoral office.1 Dental service types were grouped into five categories: general care, special needs, specialist care, surgery and student clinics. General dental care refers to check ups and other minor procedures such as teeth cleaning associated with primary care. Special needs appointments are for patients with disabilities. Student clinics refer to student clinicians treating patients. Specialist care clinics refer to procedures such as prosthodontics, periodontics and orthodontics, while surgery refers to theatre procedures. Table 2 provides a breakdown of demographic factors of patients in our sample. The number of appointments made during this time was 862,745 and of these, 139,924 appointments had a SMS reminder sent to the patient’s mobile phone. The number of patients sent SMS reminders was 34415. Demographic Factor Levels Missing Data Gender Male (40%), Female (59%) 1% Identifying cultural background Africa (3.39%), America (0.15%), Asia (12.39%), Australasia (0.33%), Australia & New Zealand (56.14%), Europe (8.26%), North America (0.32%), South America (0.46%) 18.55% Suburb region Eastern Metropolitan (9.72%), Eastern Rural Victoria (8.32%), Northern Metropolitan (28.59%), Northern Rural Victoria (5.73%), South Eastern Metropolitan (10.71%), Southern Metropolitan (7.78%), Western Metropolitan (21.91%), Western Rural Victoria (6.89%) 0.35% Age groups 18-25Yrs (25%), 26-44Yrs (40%), 45-65Yrs (24%), 65+Yrs (11%) - Indigenous Indigenous (2.1%), Non-Indigenous (83.5%) 14.4% Refugee (including Asylum Seeker status) Yes (0.2%), No (99.8%) - Concession card type Veterans Pensioner Concession Card (0.03%), Health Care Card (55.97%), No Card (3.93%), Pensioner Concession Card (40.08%) - Dental Service type General Care (18.13%), Special Needs (3.34%), Specialist Care (47.85%), Surgery (13.65%), Student Clinics (17.03%) 0.01% Table 2: Demographic factors of patients (Number of patients=34415) sent an SMS and frequency distributions 1 https://www.vec.vic.gov.au/ElectoralBoundaries/StateRegionProfiles.html Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 8 The collected data were analysed using the Minitab (Version 17) statistical software. All assumptions of statistical tests were examined and met before the analysis. The analysis was conducted at the significance level of 5%. The next section presents the findings in response to the two research objectives formulated above. 5 Results 5.1 Effectiveness of SMS Reminders to decrease FTA rates To answer the question of whether SMS reminders improved patient attendances, we used the appointment as the unit of analysis to determine: a) the non-attendance rate of the appointment before (πPre) and after the SMS implementation (πPost) b) following the implementation, the non-attendance rate of the appointment population sent an SMS reminder (πPost, SMS) against the population not sent an SMS reminder (πPost, SMS1). Table 3 reports a) non-attendance rate of the scheduled appointments before SMS implementation from Sep 2007 to Oct 2009 and after SMS implementation from Oct 2009 to Sep 2011. To determine whether there is a difference between the two attendance rates (πPre i.e. Failure To Attend – Yes % before SMS Implementation and πPost i.e. Failure To Attend – Yes % after SMS Implementation), we formulated the following hypotheses: H0: πPre ≥ πPost H1: πPre < πPost. The p-value was 0.016 smaller than 5%, leading us to conclude that the data provide statistically significant evidence to infer that the non-attendance rate before SMS implementation is lower than the one after SMS implementation. Prior SMS Reminder service Post SMS Reminder service Did not attend 28,848 (9.3%) 31,456 (9.5%) Total Appointments 309,147 331,533 Table 3: Summary of non-attendance rates – Yes % prior to and after introducing SMS reminder service Table 4 reports sample statistics in regards to b) the non-attendance rate of the appointment population sent an SMS reminder (π Post, SMS) against the population not sent an SMS reminder (π Post, SMS1). Appointments where SMS was sent (SMS) Appointments where SMS was not sent (SMS1) Totals Attended 125122 654297 779419 Did not Attend 14802 68524 83326 Total 139924 722821 862745 % non- attendance 10.58% 9.48% 9.66% Table 4. Proportion of non-attendance vs attended appointments (sent an SMS vs not sent an SMS Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 9 The following hypotheses were formulated: H0: πPost, SMS ≤ πPost, SMS1 H1: πPost, SMS > πPost, SMS1 We found statistical evidence (p value = 0.000) for the ineffectiveness of the SMS Reminder Service in reducing non-attendance. 95% Confidence Interval for difference (πPost, SMS - πPost, SMS1): 0.93%, 1.28%. Given this result, we explored explanations from an analysis of patients’ characteristics highlighted in the literature and available in the data set. This analysis provides impetus for the second research objective. 5.2 Patient profile associated with high non-attendance The second research question asked to explore the potential relationship between patient non- attendance and their demographics. The patient was used as the unit of analysis. Step 1: An analysis was conducted to explore the relationship between the number of missed appointments and the scheduled appointments per patient. To enable analysis on a patient level, we calculated a FTA score2 for each patient in the dataset as follows: Patient FTA score = (number of missed appointments /number of appointments scheduled)*100%. Table 5 shows Patient FTA scores grouped into non-overlapping intervals and the corresponding frequency distribution used to examine the relationship between missed appointments and demographic factors. Findings in Table 5 reveal amongst the patients who received an SMS reminder, 69% of patients attended all of their scheduled appointments (FTA score = 0 %). About 9.5% of patients failed to attend all of their appointments (FTA score = 100%). FTA Score Number of Patients % Patients 100% 3281 9.5336% 90-99% 3 0.0087% 80-89% 76 0.2208% 70-79% 213 0.6189% 60-69% 62 0.1802% 50-59% 1891 5.4947% 40-49% 1377 4.0012% 30-39% 1053 3.0597% 20-29% 1629 4.7334% 10-19% 1057 3.0713% 0% 23773 69.0774% 34415 100.0000% Table 5: Percentage distribution of FTA Score per patient During the 46-month study period, the number of appointments scheduled ranged from 1 to 72, with the modal number of appointments scheduled per patient being 1 (38.7%) and the arithmetic mean being 4. Covering 77% of the patients in the sample had no more than 4 2 A “FTA score” is a calculation assessing a patient’s level of non-attendance at multiple appointments. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 10 appointment, 90% had nine or less appointments, 94.4% had 14 or less appointments and about 97% had 21 or less appointments. ()* FTA score Table 6: Percentage Distribution of Patients across Missed Appointments and Number of Scheduled Appointments per patient Percentages in Table 6 were calculated based on actual data, for example in the cell marked $, 0.3% of patients do not attend 8-9 appointments out of ten appointments. As per Table 6, 38.68% of the patients made a single appointment, and of these, 77.4% attended and 22.6% missed the appointment. A number of observations can be made. Firstly, the percentage of patients whose number of missed appointments is none (0) i.e. (FTA = 0%), decreases as the number of scheduled appointments increases. From this observation we can conclude that patients are increasingly failing to attend all their scheduled appointments (FTA Score=0%) as they are given more appointments to attend. Secondly, the percentage of ‘never show up’ patients (FTA score =100%) declines considerably as the number of appointments per patient increases, suggesting some patients will try not to miss all their scheduled appointments when they are given more appointments to attend. Moreover, regardless of the total number of visits scheduled per patient, patient percentage for a single missed dental appointment is concentrated around 25%. For a given number of scheduled appointments per patient, patient percentages for multiple missed dental appointments scheduled (FTA score = 20%, 30%, … 100%) decrease significantly as the number of scheduled appointments increase. Thus, appointment keeping appears to be related to the number of dental appointments scheduled. Percent (%) 0 1 2 3 4-6 7-9 10-12 13-14 1 77.4% (FTA=0%) 22.6% (FTA=100%) 13313 38.68 2 73.2% (0%) 23.2% 3.6% (100%) 6738 19.58 3 70.1% (0%) 24.30% 4.90% 0.7% (100%) 4095 11.9 4 60.9% (0%) 27.5% 9.0% 2.4% 0.2% (100%) 2383 6.92 5 59.8% (0%) 26.4% 10.1% 2.9% 0.7% (≥90%) 1471 4.27 6 59.1% (0%) 25.4% 10.6% 3.8% 1% (≥75%) 1069 3.11 7 59.5% (0%) 23.8% 11.8% 2.9% 2% (≥57%) 790 2.3 8 57.7% (0%) 25.2% 10.4% 3.6% 3.1% (≥50%) 636 1.85 9 54.3% (0%) 26.0% 11.5% 5.5% 2.7% (≥44%) 477 1.39 10 53.2% (0%) 26.7% 9.9% 7.2% 2.7% 0.3% (≥80%)$ 374 1.09 11-20 51.3% (0%) 24.0% 12.7% 6.2% 5.1% 0.6% 0.1% 2047 5.95 21-30 38.1% (0%) 23.1% 15.7% 7.9% 11.9% 2.4% 0.7% 0.1% 706 2.05 31-40 30.6% (0%) 23.1% 15.7% 8.7% 16.5% 3.7% 1.2% 0.4% 242 0.7 41-50 16.4% (0%) 29.5% 21.3% 3.3% 18.0% 6.6% 1.6% 3.3% 61 0.18 51-60 33.3% (0%) 25.0% 8.3% 0.0% 25.0% 8.3% 0.0% 0.0% 12 0.03 72 100% 1 0 34415 100 Number of scheduled appointments Per Person Number of Missed Appointments Patient Total Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 11 Furthermore, we examined the relationships between number of ‘No show’ appointments and number of 'No show' appointments by those that never show up. Year % "No Show" Appts. % Patients "Never Show Up" (FTA = 100%) % No show Appts. by those that never show up3 Sep - Dec 2009 (3 Months) 10.32% 8.30% 43.02% Jan - Dec 2010 10.16% 8.60% 29.32% Jan - Dec 2011 10.61% 9.08% 31.35% Jan - Dec 2012 10.56% 9.85% 36.60% Jan - Jun 2013 (6 Months) 11.55% 10.85% 48.86% Table 7: Comparison between ‘No show appointments’ % and “Never show up patients’% Table 7 demonstrates a consistent observation that at least 30% of missed appointments are by patients who never attend their appointments. This finding suggests further analysis into the demographic profile of patients who are at high risk of not making all scheduled appointments, Patient FTA score = 100%. Step 2: A Chi-square test (Table 8) was conducted to detect associations between the Patient FTA score and demographic factors i.e. to test whether different patient groups in terms of a demographic factor have the same proportion of FTA score. Demographics factors include gender, identifying cultural background, suburb region, age groups, Indigenous, refugee or asylum seeker status, concession card type, and dental service type. The two hypotheses were formulated as: H0: There is no association between FTA score levels and the demographic factor (e.g. Gender), i.e. they are independent. H1: There is an association between FTA score levels and the demographic factor (e.g. Gender), i.e. they are dependent. Variable p-value Gender 0.011 Identifying cultural background 0.000 Suburb region 0.000 Age in years 0.000 Indigenous 0.000 Refugee or Asylum seeker status 0.000 Concession card type 0.000 Dental Service type 0.000 Table 8 Association between FTA score and demographic factors: Results of Pearson Chi- Square Test 3 (% No show Appointments by those that never show up =Number of 'No show' appointments by those that never show up / Number of "No show" Appts.*100%). Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 12 Results evident in Table 8 revealed that H0 was rejected for all demographic factors, including p-value for all variables 0.00 as well as for Gender (p-value of 0.011). Therefore, all associations were statistically significant at 5% level of significance (p-value < 0.05). Rejecting the null hypothesis of equality allows us to conclude that not all patient groups (grouped by demographic characteristics) are equal with respect to the proportion of Patient FTA scores. In addition, it does not tell us which patient group or patient groups cause the rejection. Thus, a two sample t-test was conducted to test the difference between two patient groups and when there are several patient groups under investigation; the Marascuilo procedure was employed to simultaneously test the differences of all pairs of proportions. Step 3: Patients with an individual FTA score 100% were analysed to isolate profiles of high risk patients. Marascuilo procedure or two sample t- test were employed to test the differences of all pairs of patient proportions across different levels of patient demographic factors. Variable Proportion Sample size Difference in proportions p-Value†/ Critical Value‡ Gender Female (F) Male (M) 8.69% 9.78% 20,463 13,615 p(F)-p(M)* 0.011† Identifying Cultural Background Europe (Eur) Asia Australia, New Zealand (Aus&NZ) North America Africa Australasia South America America 4.96% 8.25% 8.81% 9.09% 10.54% 10.62% 10.83% 11.54% 2,843 4,265 19,322 110 1,167 113 157 52 p(Eur)- p(Africa)* p(Eur)-p(Asia)* p(Eur)- p(Aus&NZ)* 0.037‡ 0.022‡ 0.017‡ Suburb Region Eastern Metropolitan (EM) Northern Rural Victoria (NRV) Eastern Rural Victoria (ERV) South Eastern Metropolitan (ESM) Southern Metropolitan (SM) Western Rural Victoria (WRV) Western Metropolitan (WM) Northern Metropolitan (NM) 7.02% 8.26% 8.74% 8.95% 9.56% 10.00% 10.33% 10.34% 3,346 1,973 2,862 3,687 2,679 2,370 7,539 9,839 p(EM)-p(NM)* p(EM)-p(WM)* p(EM)-p(WRV)* 0.020‡ 0.021‡ 0.028‡ Age in years 1: 18-25 2: 26-44 3: 45-65 4: 65+ 10.07% 12.61% 6.56% 3.62% 9,767 12,884 8,370 3,394 p(2)-p(1) p(2)-p(3) p(2)-p(4) p(1)-p(3) p(1)-p(4) p(3)-p(4) 0.0118‡ 0.0114‡ 0.0124‡ 0.0111‡ 0.0121‡ 0.0117‡ Indigenous Yes (Y) No (N) 15.96% 9.24% 664 28,737 p(Y) – p(N)** 0.000† Refugee (and Asylum Seeker) Yes No 27.54% 9.50% 69 343,46 p(Y) – p(N)** 0.000† Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 13 Variable Proportion Sample size Difference in proportions p-Value†/ Critical Value‡ Concession card type Pensioner Concession Card (PCC) Health Care Card (HC) No Card (NC) 7.83% 10.05% 19.60% 13,792 19,271 1,352 p(NC)-p(HC)** p(NC)-p(PCC)** p(HC)- p(PCC)** 0.000† 0.000† 0.000† Dental Service Type 1.General Care 2.Student Clinics 3.Surgery 4.Specialist Care 5.Special Needs 13.58% 13.05% 8.33% 7.49% 3.92% 6,238 5,861 4,696 16,467 1,149 p1-pi †† i = 2,3,4 p2-pj †† j = 3,4,5 p3-p5 †† p4-p5 †† 0.022‡,0.015‡ ,0.018‡ 0.019‡, 0.22‡,0.022‡ 0.015‡ 0.018‡ * Test for difference =0 (vs < 0) ** Test for difference =0 (vs > 0) †p-Value ‡ Critical value †† List of differences in patients’ proportions that are statistically significant Table 9 Characteristics of patients who failed to keep their appointments (FTA score = 100%) As per Table 9, female patients were significantly more likely to attend their appointments than male patients. Older patients were less likely than younger patients to miss appointments. Interestingly, patients who were in 26-44 age group were more likely than other patients to miss all of their appointments. The group (18-24 years) was the second most likely to miss their appointments. These differences across age groups were found to be statistically significant. Patients from socially disadvantaged groups, such as identifying as Indigenous (Aboriginal, Torres Strait Islander), refugees and asylum seekers were significantly more likely to miss appointments. Patients who had pensioner concession card were more likely than others to keep their appointments. Patients who must pay the full fee were more likely to miss appointments than concession and healthcare cardholders. Proportions of FTA score =100% varied markedly by cultural background. Europeans were more likely to keep appointments and the percentage differences between them and Australians and New Zealanders, Asia, Africa were statistically significant. Patients living in low income and disadvantaged regions such as Western Rural Victoria, Western Metropolitan and Northern Metropolitan, were most likely to miss appointments whereas those from Eastern Metropolitan (and the most affluent) were less likely. Furthermore, the differences in proportions between Eastern Metropolitan and three regions including Western Rural Victoria, Western Metropolitan, Northern Metropolitan were statistically significant. Patients with appointments in General Care and Student Clinics were more likely to miss appointments, while special needs patients were more likely to keep appointments. In addition, the differences in FTA score =100% proportions between patients’ scheduled for General Care and each service type of Student Clinics, Surgery, and Specialist Care; between Student Clinics and each of Surgery, Specialist Care and Special Needs; and between Surgery and Special Needs Specialist Care and Special Needs were statistically significant. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 14 6 Discussion 6.1 Discussion of findings in relation to previous research This section discusses the findings in relation to previous work in the literature. Our reported non-attendance rate of 9.48% (no SMS sent) is on par and compares well against rates reported in medical specializations and dental literature. Non-attendance rates range from 12% in ophthalmology clinics (Brannan et al. 2011), 33.6% in an ENT outpatient clinic (Geraghty et al. 2008) and 23.4% in four specialized outpatient clinics (Downer, Meara & Da Costa 2005), while in dental outpatient clinics, non-attendance rates range from 11% for tooth scaling reminders (Cheng et al. 2013), 31% and 12% in pediatric dental (Parikh et al. 2010; Cheng et al. 2013) respectively and 21.3% in general dental clinics (Lalloo & McDonald 2010). Our statistical analysis found insufficient evidence to infer that the SMS reminder intervention improved the non-attendance rate for the outpatient clinic. This result contravenes the majority of literature which overwhelmingly reports SMS reminders’ effectiveness in lowering non-attendance rates. Similar to our finding, a small number of studies reported no change in the non-attendance rate from a SMS intervention (Table 1), suggesting SMS reminders may not be appropriate for all patient cohorts, particularly for low socio-economic groups who may experience additional barriers to accessing health care, particularly dental care (Norton et al. 2014). Our study found the percentage of patients not attending appointments appears to increase significantly as the number of scheduled appointments increases, and reaffirms the finding that multiple SMS reminders did not help patients who repeatedly fail to attend their general practice appointments (Fairhurst &Sheikh 2008). This finding suggests that technology by itself does not solve the patient non-attendance problem. Our finding that missed appointments were significantly associated with persons aged 25 – 44 may be due to work and other priorities of adults in this age group, such as a taking care of children (Lalloo & McDonald 2010). We also found older patients are more likely than younger patients to keep their appointments, as similarly reported in the literature (Cashman et al. 2004; Perron et al. 2010; Parikh et al. 2010). SMS reminders were found ineffective in a primarily youth (aged 12 – 24) clinic in a similar study (Narring et al. 2013), where it was reported most patients did not initiate appointments as they were often referred by a healthcare professional or their parents. In our study, the second age group most likely to miss appointments was the younger 18 – 24 age group. Further research is warranted into understanding how young adults make appointments and took responsibility for appointments made on their behalf. Previous studies reported mixed findings regarding gender. Males are more likely to miss appointments (Perron et al. 2010; Hamilton, Round & Sharp 2002), while in another study women are most likely to miss appointments (Neal et al. 2001). Yet other studies concluded there is no significant relationship between gender and non-attendance (Bos, Hoogstraten & Prahl-Andersen 2005; Cashman et al. 2004; Parikh et al. 2010). Whilst our study found men are more likely to miss appointments, this association was the least in statistical significance compared to other patient characteristics. Males may tend to prioritize work schedules before health related appointments, particularly if the appointment is not absolutely necessary. More studies are required to investigate whether a strong association exists. We also found patients with appointments in General Care or in Student Clinics contribute to the non-attendance rate. Clinical general care includes appointments for non-complicated dental care and routine check-ups. Patients may tend to miss appointments if symptoms are self-resolving (Parikh et al. 2010) and may fail to attend routine check-ups if they feel well, particularly if they are busy with other priorities such as paid work or caring for children. Our study suggests patients tend to miss appointments with students clinicians. Similar findings were reported where patients were likely to miss appointments if they were cared for by a junior doctor rather than by a senior doctor (Perron et al. 2010). The dental outpatient Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 15 service in our study provides training facilities and supervising clinicians for student dentists from the University of Melbourne. Appointments with student clinicians can be lengthy as students must have each stage of the procedure checked before they can move on, with supervisors in charge of 5 – 8 students at a time. Patients are aware of the increased time required for appointments in Student Clinics when appointments are made. Even though appointments with student clinicians are free, patients may not comprehend the high quality of supervision provided or are concerned with the longer time required to complete a dental procedure. Patients identifying with a cultural background of Australian, New Zealand, Asian or African, are of Indigenous descent or have refugee or asylum seeker status were associated with higher rates of consistently missing appointments (that is, a higher proportion of individual patient FTA Score =100%) compared to others. Riggs et al. (2012) found disadvantaged communities such as refugees and migrants may lack oral health knowledge usually accepted in a developed country, preferring to use anecdotal care conventional in their culture. Similarly, some migrant groups do not value dental health, for example the Chinese view of dental disease as ‘normal’ (Kwan & Holmes 1999). Aboriginals have a concept of time which differs from a system based on setting specific appointments, which may explain aboriginal non-attendance (Lalloo & McDonald 2010). In addition, Perron et al. (2010) found asylum seeker status is related to high non-attendance rates and in this case cites poor socio-economic status as an indicator of non-attendance status (Perron et al. 2010). Persons living in low socio-economic regions in Melbourne and Victoria, in particularly Western Victoria, Western Metropolitan and Northern Metropolitan tend to be associated with non-attendance. These areas of Melbourne are identified as low socio-economic by the latest ABS Census data (ABS, 2011). This finding concurs with similar conclusions relating low socio- economic status to non-attendance (Cashman et al. 2004, Norton et al. 2014; Neal et al. 2001). The finding that non-concession card holders (patients who are ineligible for discounted dental care) are the most likely to miss appointments is supported by Parikh et al. (2010) which found patients who must pay for their treatment tend not to attend. An explanation may be related to the clinic in our study treating emergency patients, many of whom would not normally attend the clinic as an eligible patient. Patients treated in emergency are often booked into a follow up appointment at the outpatient clinic. These patients, for example, may prefer to attend a private dentist who is located closer to home, and may do so without cancelling the follow up appointment. Our results suggest patients living in low socio-economic regions of Victoria contribute to the FTA rate. Seemingly contravening these results are our findings suggesting non-concession card holders (predominately patients in high socio-economic groups) also contribute to the non-attendance rate. The proportion of non-concession holders however is very small, and we suggest a most likely explanation for this high FTA rate is due to emergency services connected to the clinic, as explained above. As evidenced in Table 2, the clinic predominately services patients holding health card and pensioner concession cards. 6.2 Practice implications Firstly, SMS reminders should not be implemented in outpatient clinics simply because the intervention is considered low cost. We have shown patient and appointment characteristics are strongly associated with non-attendance, and suggest clinics review the characteristics of their patient cohort to inform a targeted approach to reducing non-attendances. Other implications of this work includes revising assumptions into how the SMS reminder service supports the patient appointment system and its interactions with patients. The SMS system in our study requests a patient contacts the service only if they are not attending the appointment. If no response is received, the outpatient service assumes the patient will be attending. In doing so, the clinic also assumes the patient saw the message and has acted upon it. In contrast, if the emphasis for responding to an SMS is on confirming attendance, the service will also capture patients who are cancelling their appointment. Another suggested Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 16 revision to the SMS reminder service includes provision for patients to automatically confirm attendance through touch screen technology. For example, patients may confirm or cancel their appointment by pressing a button labelled “Yes” or “No” respectively, whilst at the same time management can ensure the SMS has been received and acted upon. These software changes may assist patients to better communicate their intentions directly to the clinic and hence contribute to the re-scheduling of appointments. Targeted interventions should be considered for people with the characteristics we have found contribute to the non-attendance rate. Reminders targeted to such persons shortly before the appointment (including telephone calls) may assist in rescheduling the appointments that patients cannot attend. Other recommendations include double booking appointments for multiple non-attending patients and implementing a fee-based system to act as a deterrent. Policy makers may consider increasing dental health promotion for the socially marginalized (including Indigenous communities, refugees, and asylum seekers) and for communities commonly associated with low income. For people who do not attend appointments for service types such as General Care and with student clinicians, these groups may benefit from health promotion programs on the importance of preventive and continuous care. Likewise patients may not be aware of the adequate level of care provided by student clinicians who are supervised by senior dentists. Targeted interventions for patients who do not hold a concession card should be considered. We have found paying for an appointment as a predictor of non-attendance, and hence recommend targeted reminders or even a fee based system for missed appointments by non-concession cardholders. 6.3 Limitations Limitations of the study include restricted generalizability as it reports on a single outpatient clinic, even though it is common for studies in this area to draw conclusions based on populations from the one site4. Outpatient clinics often see numerous patients who present with different conditions and are attended to by various health professions, hence resembling a diversity which is representative of multiple site data collection. Given the consistency of our data sampling method and findings with previous studies reporting on low-income and disadvantaged patients, it is reasonable to assume our findings may be replicated in clinics serving similar patient groups. The study was also limited in the range of criteria predicting non-attendance. The analyses in this study assess some of the factors present in the literature that have been shown to be associated with patient appointment keeping. Others such as the time and day of the appointment were not included. Finally, we could only make inferences from the data to explain why patients missed appointments. Other factors such as occupation, highest education level, marital status, nature of the job (part time, casual, fulltime etc.), number of dependents, owner of the mobile, mode of transport, travel time to clinic etc will be considered in our future work. 7 Conclusion Based on a large data set of 862,745 appointment records over the period of 46 months, our findings in relation to the ineffectiveness of SMS and the confirmed associations between patient characteristics and non-attendance strongly suggest patient characteristics are important factors to consider in the adoption of SMS reminders. This study is of particular importance to public health care settings, and can be used to inform interventions that will have a high likelihood of reducing missed appointments. Future work will involve a follow up study to investigate our interpretations of demographic and patient characteristic predictors, such as differences in patient location, country of origin, age and gender. The new concept of a Patient FTA score will be used to conduct different types 4 For example, of the 25 papers listed in Table 1, 18 of these used data retrieved from a single site. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 17 of analytics to further investigate non-attendance. In addition, a qualitative research study is planned to understand the reasons for patient non-attendance. Interviews will be conducted with patients who fit the above discussed profile and miss appointments, to understand why they miss appointments and how we can best assist them in taking care of their health. Such a qualitative study will contribute to the Information Systems literature by examining contextual and human factors affecting the effectiveness of technology when introduced to healthcare service provision. Such factors can be explored and analyzed using socio-technical theories such as Activity Theory (Vygotsky 1978; Engeström 2000) or Actor Network Theory (Callon 1986; Latour 2005). Implications for practice from these studies include strategies to increase attendance at outpatient clinics. Future research into the relationships between patient demographics and non-attendance is vital to providing underprivileged communities with realistic opportunities for improved healthcare. Author Contributions Emilia Bellucci and Lemai Nguyen collaborated on the protocol for the research, reviewing the statistical results and developed the full manuscript. Lasitha Dharmasena contributed to the statistical analysis of the data and Hanny Calache provided access to the data and clinical expertise required in interpreting our results. Acknowledgements Many thanks to the anonymous reviewers and this journal’s editors for their comments on earlier drafts of the manuscript. We also thank the participating dental outpatient clinic’s administrative staff for their contributions to this project. Statement of Conflict of Interest There was no conflict of interest relevant to this research. Summary Table What is already known of the topic: • The literature overwhelmingly attributes SMS reminders to improving the attendance rate of patients in outpatient clinics • There are limited studies reporting SMS reminders have no effect on the non- attendance rate, suggesting universal and successful application of SMS reminders cannot be assumed. • Patient and appointment demographics may help predict patient non-attendance. Key Results: • The FTA rate before the implementation of the SMS reminder system is lower than the FTA rate following implementation. • SMS Reminders were ineffective among the population sent an SMS. • Patients who miss all their appointments exhibited one or more of the following characteristics: male; age 26 – 44; non-concession card holders; a person of Indigenous, local, Asian or African descent, and of refugee status, persons living in low socio-economic areas; and have appointments in General Care and Student Clinics. Contributions: • SMS reminder systems have a limited effect on certain demographics and appointment characteristics, including those synonymous with the public health system. • A demographic and appointment profile of patients who never attend appointments is proposed. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 18 • Outpatient clinics should not assume a cost effective intervention such as SMS reminders will be successful in their context, and suggests clinics focus on eliciting patient profiles to better understand patient non-attendance. REFERENCES ABS. (2011) Census of Population and Housing: Socio-Economic Indexes for Areas (SEIFA). Australian Bureau of Statistics. Adanikin, A.I., Awoleke, J.O & Adeyiolu, A. (2014) “Role of reminder by text message in enhancing postnatal clinic attendance”. International Journal of Gynecology & Obstetrics 126(2):179-180. Boksmati, N., Butler-Henderson, K., Anderson, K. & Sahama, T. (2016) “The Effectiveness of SMS Reminders on Appointment Attendance: a Meta-Analysis”. Journal of Medical Systems 40:90. Bos, A., Hoogstraten, J. & Prahl-Andersen, B. (2005) “Failed appointments in an orthodontic clinic”, American Journal of Orthodontics and Dentofacial Orthopedics 127(3), 355-357. Brannan, S.O., Dewar, C., Taggerty, L. & Clark, S. (2011). The effect of short messaging service text on nonattendance in a general ophthalmology clinic. Scottish Medical Journal 56(3):148-150. Callon, M. (1986). The sociology of an actor-network: the case of the electrical vehicle. In M. Callon, J. Law, & A. Rip (Eds.), Mapping of the dynamics of science and technology. Basingstoke: The Macmillan Press Lt. Cashman, S.B., Savageau, J. A., Lemay, C. A. & Ferguson, W. (2004) “Patient Health Status and Appointment Keeping in an Urban Community Health Center” Journal of Health Care for the Poor and Underserved 15(3): 474-488. Chen, Z-w., Fang, L-z., Chen, L-y., Dai, H-l. (2008) “Comparison of an SMS text messaging and phone reminder to improve attendance at a health promotion center: A randomized controlled trial”. Journal of Zhejiang Unversity - Science B 9(1):34-38. Cheng C-CJ, Li C-Y, Hu Y-J, Shen, H-C. & Huang, S-M. (2013) “Effects of tooth scaling reminders for dental outpatients”. Journal of Telemedicine & Telecare 19(4):184-89. da Costa, T.M., Salomão, P.L., Martha, A.S. Pisa, I.T. & Sigulem, D. (2010) “The impact of short message service text messages sent as appointment reminders to patients' cell phones at outpatient clinics in São Paulo, Brazil”. International Journal of Medical Informatics 79(1):65-70. Deloitte. (2014) Media consumer survey 2014: Australian media and digital preferences – 3rd edition. Dental Health Services Victoria. (2011) Links between oral health and general health - the case for action , Available at: https://www.dhsv.org.au/__data/assets/pdf_file/0013/2515/links-between-oral- health-and-general-health-the-case-for-action.pdf [Accessed 10th September 2016]. Downer, S.R., Meara, J.G. & Da Costa, A.C. (2005) “Use of SMS text messaging to improve outpatient attendance”. Medical Journal of Australia 183(7):366-68. Downer, S.R., Meara, J.G., Da Costa, A.C. & Sethuraman, K. (2006) “SMS text messaging improves outpatient attendance”. Australian Health Review 30(3), 389-396. Engeström, Y. (2000). "Activity Theory and the Social Construction of Knowledge: A Story of Four Umpires". Organization, 7(2), 301-310. doi: 10.1177/135050840072006 Fairhurst, K., Sheikh, A. (2008) “Texting appointment reminders to repeated non-attenders in primary care: Randomised controlled study”. Quality and Safety in Health Care 17(5):373-376. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 19 Farmer, T., Brook, G., McSorley, J., Murphy, S. & Mohamed, A. (2013) “Using short message service text reminders to reduce 'did not attend' rates in sexual health and HIV appointment clinics”. International Journal of STD & AIDS 0(0), 1-5. Foley, J. & O'Neill, M. (2009) “Use of mobile telephone short message service (SMS) as a reminder: the effect on patient attendance”. European archives of paediatric dentistry: official journal of the European Academy of Paediatric Dentistry 10(1):15-18. Geraghty, M., Glynn, F., Amin, M. & Kinsella, J. (2008) “Patient mobile telephone 'text' reminder: A novel way to reduce non-attendance at the ENT out-patient clinic” Journal of Laryngology and Otology, 122(3): 296-298. Guy, R., Hocking, J., Wand, H., Stott, S., Ali, H. & Kaldor, J. (2012) “How Effective Are Short Message Service Reminders at Increasing Clinic Attendance? A Meta-Analysis and Systematic Review”. Health Services Research 47(2):614-632. Hamilton, W., Round, A. & Sharp, D. (2002) “Patient, hospital, and general practitioner characteristics associated with nonattendance: a cohort study”. British Journal of General Practice 52: 317-19. Hasvold, P.E. & Wootton, R. (2011) “Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review”. Journal of Telemedicine and Telecare 17(7): 358-364. Koshy, E., Car, J., & Majeed, A. (2008) “Effectiveness of mobile-phone short message service (SMS) reminders for ophthalmology outpatient appointments: Observational study”. BMC Ophthalmology 8:9. Kwan, S.Y.L. & Holmes, M.A.M. (1999) “An exploration of oral health beliefs and attitudes of Chinese in West Yorkshire: a qualitative investigation”. Health Education Research 14(4):453-60. Lalloo, R. & McDonald, J. M. (2010) “Appointment attendance at a remote rural dental training facility in Australia”. BMC Oral Health 13(1):1-8. Latour, B. (2005). Reassembling the social: An introduction to actor-network theory. Oxford, UK: Oxford University Press. Leong, K.C., Chen, W.S., Leong, K.W., Mastura, I., Mimi, O., Sheikh, M.A., Zailinawati, A.H., Ng, C.J., Phua, K.L. & Teng, C.L. (2006) “The use of text messaging to improve attendance in primary care: a randomised controlled trial”. Family Practice 23: 699– 705. Liew, S.M., Ng, C.J., Tong, S.T., Lee, V.K.M., Teng, C.L. & Leong K.C. (2009) “Text messaging reminders to reduce non-attendance in chronic disease follow-up:a clinical trial”. British Journal of General Practice, 59: 916–920. McInnes, D.K.K, Petrakis, B.A., Gifford, A.L, Rao, S.R., Houston, T.K., Asch, S.M. & O’Toole, T.P. (2014) “Retaining Homeless Veterans in Outpatient Care: A Pilot Study of Mobile Phone Text Message Appointment Reminders”. American Journal of Public Health 104(S4):S588-S94. McIver, R., Dyda, A., McNulty, A.M., Knight, V., Wand, H. C. and Guy, R. J. (2015) “Text message reminders do not improve hepatitis B vaccination rates in an Australian sexual health setting”, Journal of the American Medical Informatics Association 0:1-5 Milne, R.G. (2010) “Reducing non-attendance at specialist clinics: an evaluation of the effectiveness and cost of patient-focussed booking and SMS reminders at a Scottish health board” International Journal of Consumer Studies 34(5): 570-580. Milne, R.G., Horne, M., & Torsney, B. (2006) “SMS reminders in the UK National Health Service: An evaluation of its impact on "No-Shows" at hospital out-patient clinics”. Health Care Management Review 31(2):130-136. Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 20 Narring, F.O., Perron, N.J., Dao, M.D., Righini, N.C, Humair, J-P., Broers, B., Gaspoz, J-M. & Haller, D.M. (2013) “Text-messaging to reduce missed appointment in a youth clinic: a randomised controlled trial”. Journal of Epidemiology & Community Health 67(10):888-891. Neal, R.D., Lawlor, D.A., Allgar, V., Colledge, M., Ali, S., Hassey, A., Portz, C. & Wilson, A. (2001) “Missed appointments in general practice: retrospective data analysis from four practices”. British Journal of General Practice 51:830-832. Nelson, T.M., Berg, J.H., Bell, J.F., Leggot, P.J. & Seminario, A.L. (2011) “Assessing the effectiveness of text messages as appointment reminders in a pediatric dental setting”. Journal of the American Dental Association 142(4):397-405. Norton BL, Person AK, Castillo C, Pastrana, C., Subramanian, M. & Stout, J. E. (2014) “Barriers to Using Text Message Appointment Reminders in an HIV Clinic”. Telemedicine and E- health 20(1):86:89 Parikh, A, Gupta, K, Wilson, A.C., Fields, K., Cosgrove, N.M., & Kostis, J.B. (2010) “The Effectiveness of Outpatient Appointment Reminder Systems in Reducing No-Show Rates”. American Journal of Medicine 123(6):542-48. Perron, N.J., Dao, D. M., Kossovsky, M.P., Miserez, V., Chuard, C., Calmy, A. & Gaspoz, J-M. (2010) “Reduction of missed appointments at an urban primary care clinic: a randomised controlled study”. BMC Family Practice 11:79 - 87. Perry, J.G. (2011) “A preliminary investigation into the effect of the use of the Short Message Service (SMS) on patient attendance at an NHS Dental Access Centre in Scotland”. Primary dental care: Journal of the Faculty of General Dental Practitioners (UK) 18(4):145-149. Prasad, S., & Anand, R. (2012) “Use of mobile telephone short message service as a reminder: The effect on patient attendance”. International Dental Journal 62(1):21-26. Riggs, E., van Gemert, C., Gussy, M., Waters, E., & Kilpatrick, N. (2012) “Reflections on cultural diversity in oral health promotion and prevention”. Global Health Promotion 19(1): 60- 63. Stubbs, N.D., Geraci, S.A., Stephenson, P.L., Jones D.B. & Sanders, S. (2012) “Methods to Reduce Outpatient Non-attendance” American Journal of the Medical Sciences 344(3): 211-219. Taylor, N.F., Bottrell, J., Lawler, K. & Benjamin, D (2012) “Mobile telephone short message service reminders can reduce nonattendance in physical therapy outpatient clinics: a randomized controlled trial”. Arch Phys Med Rehabil 93(1):21-26. Vygotsky, L.S. (1978). Mind in Society: The Development of Higher Psychological Processes. Cambridge: Harvard University. Sims, H., Sanghara, H., Hayes, D., Wandiembe, S., Finch, M., Jakobsen, H., Tsakanikos, E., Okocha, C.I. & Kravariti, E. (2012) “Text message reminders of appointments: a pilot intervention at four community mental health clinics in London”. Psychiatric Services 63(2):161-168. Youssef, A. (2014) “Use of short message service reminders to improve attendance at an internal medicine outpatient clinic in Saudi Arabia: a randomized controlled trial”. Eastern Mediterranean Health Journal 5:317. Copyright: © 2017 Bellucci, Dahrmesana, Nguyen & Calache. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non-commercial use, distribution, and reproduction in any medium, provided the original author and AJIS are credited. http://creativecommons.org/licenses/by-nc/3.0/au/ http://creativecommons.org/licenses/by-nc/3.0/au/ Australasian Journal of Information Systems Bellucci, Dahrmesana, Nguyen & Calache 2017, Vol 21, Research on Health Information Systems Effectiveness of SMS reminders 21