Paper title (Paper Title style) Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 1 A Participatory Model for Multi-Document Health Information Summarisation Dinithi Nallaperuma La Trobe University dinithi.mjayaratne@gmail.com Daswin De Silva La Trobe University Abstract Increasing availability and access to health information has been a paradigm shift in healthcare provision as it empowers both patients and practitioners alike. Besides awareness, significant time savings and process efficiencies can be achieved through effective summarisation of healthcare information. Relevance and accuracy are key concerns when generating summaries for such documents. Despite advances in automated summarisation approaches, the role of participation has not been explored. In this paper, we propose a new model for multi- document health information summarisation that takes into account the role of participation. The updated IS user participation theory was extended to explicate these roles. The proposed model integrates both extractive and abstractive summarisation processes with continuous participatory inputs to each phase. The model was implemented as a client-server application and evaluated by both domain experts and health information consumers. Results from the evaluation phase indicates the model is successful in generating relevant and accurate summaries for diverse audiences. Keywords: Summarization; Multi-document; User Participation; Health Informatics; MEAD, Ontology, Templates 1 Introduction Retrieval of relevant and reliable health information is becoming a formidable challenge with increasing volumes of resources available for perusal via the Internet. This is significantly experienced in the healthcare domain as it is crucial for practitioners as well as researchers to have quick access to up-to-date information (Afantenos et al., 2005). Besides providers, healthcare consumers are also inundated with large volumes of information. Their limited knowledge of relevance due to lack of contextual and discourse awareness entails a further degree of complexity. (Moen, et al., 2016) (Pivovarov and Elhadad, 2015). For instance, health forums consist of many topics that would frequently repeat the same question with minor changes followed by a multitude of responses that reflect differences of opinion. In such cases, the automated generation of a summary of multiple documents relevant to a selected topic can contribute towards addressing information needs of consumers as well as healthcare providers and researchers. The information retrieval needs of health information consumers have been appropriately addressed by a substantial body of research literature on smart health information portals. The delivery of user-sensitive, relevant, timely and accurate health information to various user groups was the focus of these endeavours (Burstein et al., 2006). Notable research outcomes include, resource description quality criteria modelling (McKemmish et al., 2009), automated quality assessment (Xie and Burstein 2011) and decision support systems perspective on portals (Burstein et al 2005). The need for a multi-document information summarisation approach for health information consumer has been indicated in these research endeavours. The main goal of a summary is to extract the important content in document(s) and present them in a short space (Afanteno et al., 2005). Aligned with Radev et al., (2002) definition of a summary, Afanteno et al. (2005) demonstrated that a document contains information bursts and an automated summary should distinguish these bursts of information from the rest of the Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 2 document. Thereby, identifying relevant information bursts is the key challenge in document summarisation. We posit that end-user participation is an important consideration to identifying such relevant information bursts in health document summarisation and has been consistently overlooked by all existing approaches. Use of participatory design methods where ‘end users’ take part in service development processes is currently widespread in the health sector. With the complexity embedded in health information, specifically in health forums, system developers have to identify and manage social conflicts and technical issues (Pilemalm & Timpka, 2008). Thursky and Mahemoff (2007) and Weng, et al. (2007) have shown promising results by using user participation in different sites such as hospital and clinical. It is mainly in response to the diversity of the problems encountered in the design processes that user participatory design principles have been used in health information system development. Kappelman and McLean (1992) defined participation as the observable behaviour of system users such as involvement in development and implementation activities of information systems. Markus and Mao (2004) extended the traditional concept of system success into development success and implementation success, alongside explicit articulation of participants in each stage. Zowghi et al. (2015) conducted a systematic review of the relationship between user involvement and system success to reveal a positive correlation between the two, alongside factors for effective management of user involvement. Eichhorn & Tukel (2015) present a further review of user involvement, taking into account multiple factors such as user roles and activities, selection of users, type of communications used, and timing and level of their involvement. Both these reviews emphasise on the importance of user participation to effectuate relevance in design, development and implementation success. In this paper, we introduce a participatory model for automated summarisation of multi- document health information. The paper delineates the design, development and evaluation of the proposed model. The paper is organised as follows. Section 2 reviews literature on document summarisation and user participation in information system design. Section 3 presents overall design of the model and further discusses the participation model in order to validate the proposed participatory model for multi-document summarisation. Section 4 discusses implementation; the three tier architecture of the summarisation model and the multi-document summarisation process. Section 5 reports on both qualitative and quantitative evaluation. Finally, concluding remarks are presented with a brief discussion on limitations and future work. 2 Document summarisation Jones (2007) conducted a comprehensive survey on document summarisation using several frameworks that evaluate key factors affecting summaries. He also evaluated statistical and symbolic methods that do not require model instantiation such as in-domain ontologies. Saggion and Poibeau (2012) have reported that extractive summarisation on multi-documents is challenging as it would lead to redundancies as well as inconsistent and/or hard to read summaries. Lloret and Palomar (2012) proposed sentiment based summaries where high degree of subjectivity is applicable and update summaries where only the most recent information on the topic would be considered for summarisation. Aliguliyev (2009) reviewed salient notions and developments in summarisation such as partitioning-based clustering methods in generic document summarisation. He has shown that summarisation depends on optimising functions and on similarity measures which would be more useful in improving performance compared to traditional summarisation approaches. The primary methods of summarisation are extractive and abstractive (Wan, 2008) (Mani and Maybury, 1999). These two methods are discussed in detail in the following subsections. 2.1 Extractive summarisation Most existing systems for summarisation use sentence extraction from the source. The advantage of extraction is that it does not require automated natural language generation. Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 3 Therefore, domain independence summaries are easy to implement as extraction does not require the vocabulary of the particular domain. However, poor linguistic coherence of the extracted summaries is the main disadvantage of the extraction based systems. A collection of sentences from one or more documents will be input to the extraction based summarisation process and the output would comprise of a subset of these sentences (Radev et al., 2004) (Salton et al., 1997). Length of the required summary is taken into account when generating the subset. As shown by Radev et al. (2004), ranking approaches based on similarity and centrality are used to rank sentences for inclusion in the summary. Wan et al. (2007), have proposed a novel approach based on manifold ranking of sentences for query based multi- document summarisation. In their approach, firstly, the manifold ranking score for each sentence will be computed where biased information richness of the sentence is used. This is followed by greedy algorithm to penalize the sentences with the highest overall scores, which will indicate the sentence is more informative and is highly biased to the given query. MEAD is widely used open source summarisation method, it is an implementation of the centroid- based method for either single or multi-document summarizing (Erkan and Radev, 2004) (Radev et al., 2004). 2.2 Abstractive summarisation In direct contrast to extractive summarisation, abstractive summarisation emphasises on semantic structure of information content based on sentence compression and sentence reformulation (Radev et al. 2002). SUMMONS (McKeown and Radev, 1995) attempts abstractive summarisation using a set of heuristic rules that perform operations such as contradiction, change of perspective, refinement, etc. Some of these operations require resolving conflicts such as information within different opinions. Although SUMMONS generates accurate summaries when the domain is quite simple, a generalized system for more complex domains is problematic. McKeown et al. (1999) and Barzilay et al. (1999) have reported an improvement where the input is raw text such as the text that is retrieved by a standard search engine in response to a query. The concept has been adapted by Belkebir and Guessoum (2016) in order to derive a more general representation where different concepts appear in the same context. 2.3 Multi-document summarisation Although summarising multiple documents could lead to overlap, the value generated by bursts of supplementing information is significant. The primary issues in multi-document summarisation are to identify and cope with redundancy across documents, recognize novelty and ensure that the final summary is coherent and complete. Existing research on multi- document summarisation attempts to address these issues. Making use of similarity measures between sets of sentences, different approaches such as clustering were used on implement extractive techniques. One approach identified common themes through clustering and then select one sentence to represent each cluster (McKeown et al., 1999; Radev et al., 2000); another generates a composite sentence from each cluster (Barzilay et al., 1999), while some approaches work dynamically by including each candidate passage only if it is considered novel with respect to the previous included passages, via maximal marginal relevance (Carbonell and Goldstein, 1998). Multi-document summarisation has also been extended to multilingual environments (Evans, 2005). 2.4 Adaptation of abstractive and extractive techniques for participatory health information summarisation Documents containing health information are exceptional due to volume and heterogeneity (Radev et al., 2004). Given the number and diversity of health information sources, methods must be found that will enable users to quickly understand and choose the relevant document. In our adaptation for health information summarisation, extraction is used to identify the information bursts using the key sentences from multiple health documents based on the centroid method. This is followed by abstractive summarisation based on templates that allow Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 4 generalization of sentences. A health care ontology is used to find the generalized content to match the template from sentences captured through extraction. As accuracy is vital in healthcare, healthcare domain experts are required to impart their knowledge to determine accuracy. Simultaneously, health information consumers specify and rank their information expectations in support of increased accuracy. Therefore, user participation makes a significant contribution to the proposed multi-document health information summarisation model. Past research endeavours on participative behaviour were limited to system development. However, Markus and Mao (2004) present participation as personally relevant and important which results in user participants being committed to the inputs given to the system development as well as adoption and usage. Moreover, when system users’ need-based attitudes were coupled with participative behaviours, the relationship between system success and user participation appeared to be stronger compared to when the system researchers operationalized the participation construct in purely behavioural terms. 3 Design of a participatory model for multi-document health information summarisation It is pertinent to extend Markus and Mao (2004) new IS theory of participation for the proposed model as it successfully addresses logical gaps and gaps created by the practice of system design and development. It was also supportive to determine participants and their contributions towards multi-document health information summarisation. Figure 1: Adaptation of IS user participation theory for multi-document health information summarisation (Markus and Mao, 2004) As illustrated in Figure 1, the two key groups of participants can be identified as stakeholders (medical practitioners) and change agents (health information consumers). The participation activity for medical practitioners is the use of domain knowledge for template design whereas health information consumers contribute ratings and comments to convey their position on relevance of summarisation outcomes. As reported by Markus and Mao (2004), these activities would lead to the development and implementation success of a summarisation system that takes into account participatory activities. Medical practitioners initiate their contribution of domain knowledge from template generation for the abstractive summarisation process. Given the nature of health information, it is evident that different user groups seek different levels of information from a summary, for instance, a health information consumer requires a general understanding regarding the topic; a medical student requires an in-depth understanding of the topic and a medical practitioner requires a quick summary. Therefore with the support of ten experts from the health domain, templates were generated for different types of information requirements (Figure 2). These Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 5 templates can be further extended to include other stakeholders such as carers, family and support providers. Figure 2: Templates generated for three distinct groups of end-users The following high level diagram provides an overview of complete model and the role and purpose of participation. Figure 3: High-level schematic of the proposed model 4 Development of the summarisation model This section provides a detailed account of each module illustrated in Figure 3. The extraction module, abstraction module and NLP-based integration of the outcomes from these two modules are explicated below. 4.1 Extraction module For the extraction module, MEAD (Radev et al., 2004) an open-source framework for multi document extractive summarisation was extended to suit the requirements of the healthcare domain. Hence, the adapted MEAD will be referred to as MEAD*. The MEAD framework decomposes sentence extraction into three steps. In the first step numerical features (position- based, centroid-based, largest common subsequence and keywords) are calculated for each Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 6 sentence. These features are assumed to correlate with how informative the sentence would be for summarising the given set of documents. For instance, one feature is based on the position of the sentence in the document, as sentences at the beginning (or at the end) of a document tend to be more general and informative. In MEAD*, the sentence feature was set to centroid- based which measures how similar the sentence is to the set of documents to be summarised. This is done by using a corpus where the set of documents will be represented as a centroid vector. As a result, each word is associated with the product of its term frequency (TF) in the documents times its inverse document frequency (IDF) in the entire corpus. The centroid feature for a sentence can be then computed as the cosine overlap between the word occurrence vector for the sentence and the centroid vector of the set of documents. Following feature calculation, a new classification phase was introduced to MEAD*. All the sentence features computed in the previous step are combined into a single numerical score for each sentence. Finally, MEAD* performs re-ranking based on centroids where the numerical score for each sentence is adjusted relative to other sentences. This allows the system to avoid redundancy in the final set of sentences by lowering the scores of sentences that are similar to already selected sentences. 4.2 Abstraction module The abstraction module aims to capture and summarise the semantics of content rather than statistical contribution of the text. Abstraction consists of two major components: a content planner that selects information to be include in the summary through combination of input templates, and a sentence generator that selects appropriate words to express the information in a grammatically correct format. 4.2.1 Content planning with user participation Using a well-established domain ontology, SNOMED CT, content would be extracted to suit the templates above (Figure 2). Health information consumers will seek a generic template out of the sentences extracted whereas medical students will seek a more detailed template. Once the connections are made using content planning component and the template is populated. Keywords are retrieved from the extracted sentences and matched for meaning using the SNOMED CT ontology. For instance, the sentence 'Pre-eclampsia is a Disease' is identified as Pre-eclampsia as a child node of Disease in the ontology. Table 1 presents a sample of sentences extracted for different templates. It can be observed that same content has been extracted for the criteria disease, reason, and relevance. Even though, diagnosis of the health information consumer is stated as uncertain whereas for the medical student’s template diagnosis has a value. This is due to the sentences extracted for health information consumer where diagnosis has retrieved a low importance in the document. Also, more details such as sections of a sentence was extracted for medical student and practitioner as it is critical not to lose information from the extracted sentences. Moreover, in practitioner’s template (Table 1) infection has three infections identified through the ontology. Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 7 Table 1: Sentence extraction for different templates 4.2.2 NLP-based sentence generation Sentence generation is a key component of the proposed model. The sentence generator gathers all the combined information and uses connective phrases to synthesize a summary, devised by adapting existing language generation tool; Natural Language Tool Kit (Bird, Klein, & Loper, 2009). Content were derived from the generated templates and diverse levels of healthcare terms were used to generate feature based sentences for different information consumers. Sentence generation is carried out using feature structures which contains various kinds of information about grammatical entities. In the case of a verb, it is often useful to know what the semantic role played by the arguments, for example - {'Disease': OBJ, 'Prevention': SUB, 'prevent': V}. Further, grammar is concerned with how words and sequences of words combine to form sentences. A distinct and complementary approach, dependency grammar, was used instead of how words relate to other words. Therefore, {'Disease': OBJ, ‘Reason’: SUB, cause: V} (Table 1) feature structure was used to generate the sentence with the appropriate tense such as “High blood pressure and proteinuria cause Pre-eclampsia” 5 Results Experiments were conducted using a test dataset of health documents from different domains. The two modules; extraction, abstraction (content planner and sentence generator). 5.1 Extraction module Module was tested with a test case which comprises three documents in order to check whether the relevant yet different level of information is extracted for different users. In the interest of space, results from three documents used to test the extraction module are shown below. Out of three documents, two were in the domain of Obstetrics and Gynecology whereas the other was in the domain of Cardiac Defibrillators which is outside the domain of concern. Difference in sentence extraction is highlighted. Health information consumer Disease: Pre-eclampsia Reason: high blood pressure and proteinuria Whom: pregnancy Diagnosis: uncertain Medical student Disease: Pre-eclampsia Whom: pregnancy Diagnosis: no preventative measures or screening tools for early detection of the condition Reason: one or more infectious agents present in the placenta, with a predominant organism triggering the condition Impact: responsible for about 50,000 maternal deaths and 28% of all pregnancy complications worldwide annually Medication: defective placentation Medical practitioner Disease: Pre-eclampsia Reason: high blood pressure and proteinuria Infection: respiratory tract, female reproductive tract, periodontal cavity Prevention: no preventative measures or screening tools for early detection of the condition Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 8 User Sentences Extracted M ed ic a l D o ct o r [1] Microbiome of the placenta in pre-eclampsia supports the role of bacteria in the multifactorial cause of pre-eclampsia [2] Pre-eclampsia, a condition of high blood pressure and proteinuria in pregnancy, is a leading cause of maternal and infant morbidity and mortality. [3] It is thought that the presence of bacteria in the placenta could trigger increased release of anti-angiogenic factors, like soluble fms - like tyrosine kinase 1 (sFlt - 1) and decreased release of pro-angiogenic factors, like placental growth factors (PIGF) and vascular endothelial growth factor (VEGF) from the placenta. [4] A meta-analysis for the association of infectious agents with pre-eclampsia reports a twofold increase in the risk of pre-eclampsia in the presence of infection (odd ratio [OR] 2.1, 95% confidence interval [CI] 1.6 2.7). [5] On the other hand, pre-eclampsia may be triggered by the infectious burden of multiple organisms rather than the presence of a specific infectious agent that may not be adequate enough to cause a symptomatic infection but may activate the inflammatory cells and increase of anti-angiogenic factors. M ed ic a l In fo rm a ti o n C o n su m er [1] Microbiome of the placenta in pre-eclampsia supports the role of bacteria in the multifactorial cause of pre-eclampsia. [2] Pre-eclampsia, a condition of high blood pressure and proteinuria in pregnancy, is a leading cause of maternal and infant morbidity and mortality. [3] In the USA, pre-eclampsia is responsible for around 61% of pregnancy complications and accounts for 15.9%maternal deaths. [4] It is thought that the presence of bacteria in the placenta could trigger increased release of anti-angiogenic factors, like soluble fms - like tyrosine kinase 1 (sFlt - 1) and decreased release of pro-angiogenic factors, like placental growth factors (PIGF) and vascular endothelial growth factor (VEGF) from the placenta. [5] A meta-analysis for the association of infectious agents with pre-eclampsia reports a twofold increase in the risk of pre-eclampsia in the presence of infection (odd ratio [OR] 2.1, 95% confidence interval [CI] 1.6 2.7). [6] On the other hand, pre-eclampsia may be triggered by the infectious burden of multiple organisms rather than the presence of a specific infectious agent that may not be adequate enough to cause a symptomatic infection but may activate the inflammatory cells and increase of anti-angiogenic factors. [7] This could lead to a series of events, such as abnormal trophoblast function leading to endothelial dysfunction and impairment of oxygen nutrients to the placenta, which raises the maternal blood pressure causing pre-eclampsia. [8] The alternate, therefore, would be to examine the placenta of women with pre-eclampsia harvested at the time of delivery using culture independent highly sensitive methods that would enable one to detect traces of DNA of infectious agents left behind by infection at the time of placentation. [9] Researchers have discovered how one genetic and one sexual risk factor can combine to increase the risk of pre-eclampsia. M ed ic a l S tu d en t [1] Microbiome of the placenta in pre-eclampsia supports the role of bacteria in the multifactorial cause of pre-eclampsia [2] Pre-eclampsia, a condition of high blood pressure and proteinuria in pregnancy, is a leading cause of maternal and infant morbidity and mortality. [3] In Sri Lanka, pregnancy induced hypertension account for 7% of maternal and 24% of neonatal deaths due to premature delivery. [4] In the USA, pre-eclampsia is responsible for around 61% of pregnancy complications and accounts for 15.9%maternal deaths. [5] It is thought that the presence of bacteria in the placenta could trigger increased release of anti-angiogenic factors, like soluble fms - like tyrosine kinase 1 (sFlt - 1) and decreased release of pro-angiogenic factors, like placental growth factors (PIGF) and vascular endothelial growth factor (VEGF) from the placenta. [6] A meta-analysis for the association of infectious agents with pre-eclampsia reports a twofold increase in the risk of pre-eclampsia in the presence of infection (odd ratio [OR] 2.1, 95% confidence interval [CI] 1.6 2.7). [7] The first study that examined amniotic fluid of women with pre-eclampsia using both conventional microbial culture techniques and broad rage and group specific polymerase chain Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 9 reaction (PCR) assays targeting the 16S ribosomal RNA (rRNA) gene reported the presence of Lactobacillus iners, Ureaplasma, Sneathia, Leptotrichia and Streptococcus species. [8] On the other hand, pre-eclampsia may be triggered by the infectious burden of multiple organisms rather than the presence of a specific infectious agent that may not be adequate enough to cause a symptomatic infection but may activate the inflammatory cells and increase of anti-angiogenic factors. [9] This could lead to a series of events, such as abnormal trophoblast function leading to endothelial dysfunction and impairment of oxygen nutrients to the placenta, which raises the maternal blood pressure causing pre-eclampsia. [10] The current study was based on the hypothesis that pre-eclampsia is caused by one or more infectious agent present in the placenta, with a predominant organism triggering the condition. [11] The alternate, therefore, would be to examine the placenta of women with pre-eclampsia harvested at the time of delivery using culture independent highly sensitive methods that would enable one to detect traces of DNA of infectious agents left behind by infection at the time of placentation. [12] Researchers have discovered how one genetic and one sexual risk factor can combine to increase the risk of pre-eclampsia. [13] Elizabeth Triche and colleagues at Brown University say their findings suggest there could be new ways for couples to plan pregnancy with improved awareness of the disorder, as well as improved management of the risk. Table 2: Sentences extracted from multiple documents Therefore, it is evident that for different users’ different level and number of sentences are extracted. Yet, all the sentences would be in the domain on Obstetrics and Gynecology. The content submitted from outside the domain was detached. 5.2 Abstraction module The sentences extracted from the same documents for different users were used to test the abstraction module. First the content planner generated the following templates using the SNOMED CT ontology. 5.2.1 Content Planner Table 1: demonstrates the templates generated for different users. It can be observed that same content has been extracted for the disease, reason, whom etc. Even though, diagnosis of the medical information consumer is stated as uncertain whereas for the medical student’s template diagnosis has a value. This is due to the content extracted and the diagnosis has retrieved a low importance for the medical information consumer. Also, more details such as sections of a sentence was extracted for medical student and medical doctor as it is critical not to lose information from the extracted sentences. Moreover, in medical doctor’s template infection has three infections identified through the Ontology and is concatenated using comma. 5.2.2 Sentence Generator Using the templates generated for different users, following sentence were generated; Medical Information consumers will get a generic and more standard output, Medical Students will get a more detailed summary, whereas Medical Doctors will get a more compressed summary in order to give the medical doctor a basic idea on the content. Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 10 User Summary Generated Medical Information Consumer “High blood pressure and proteinuria causes Pre-eclampsia. Pregnancy gets Pre-eclampsia. Uncertain symptoms Pre-eclampsia. Detective placentation cures Pre-eclampsia.” Medical Student “One or more infectious agents present in placenta, with a predominant organism triggering the condition causes Pre- eclampsia. Pregnancy gets Pre-eclampsia. No preventative measure or screening tools for early detection of the condition is a symptom of Pre-eclampsia. Defective placentation cures Pre-eclampsia.” Medical Doctor “Respiratory tract, female reproductive tract, periodontal cavity infects Pre-eclampsia. No preventive measures or screening tools for early detection of the condition prevents Pre-eclampsia. “ Table 3: Summary generated for different user groups 6 Evaluation Online articles on obstetrics which are used by community (e.g. babycenter.com) was used for testing and evaluation of the developed multi-document summarisation system. The system was evaluated by 10 domain experts, obstetricians, and 50 healthcare information consumers in the obstetrics healthcare domain. The health information consumers were the users of online health forums such as http://www.babycenter.com/ and were willing to voluntarily participate in the study. 6.1 Extraction module Precision, recall, F-score and compression rate measures were used to evaluate the extraction module using the statistical values. Figure 4: Evaluation of extraction module As shown in Figure 4, accuracy of extraction was high with an F-score close to 1 across all templates (0.73 for health information consumers, 0.69 for medical students and 0.86 for practitioners). Figure 4 also presents compression rate of extractive summaries. Compression rate (Knight & Marcu, 2000) depends on the details that are extracted from the content as well as the comprehensive rate of sentence generation. The evaluations show high compression rate values closer to 0 which denotes higher level of summarisation. Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 11 6.2 Abstraction module Evaluation of the abstraction module required metrics of relevance of levels of sentences. In order to determine these metrics a qualitative study was required. The domain experts volunteered to compose manual summaries of the same documents used for this evaluation. The results are shown in Table 4. Manual analysis of the system generated summaries and expert generated summaries shows that same content and idea has been reflected in both the summaries. It is evident that the sentence generated could be more comprehensive although system generated summaries consist of many sentences than expert generated summaries. Example 1 Example 2 Health information consumer System summarised: High blood pressure and proteinuria causes Pre-eclampsia. Pregnancy gets Pre-eclampsia. Uncertain symptoms Pre-eclampsia. System summarised: Contraceptive actions causes progesterone. Women gets progesterone. Bio-markers symptoms progesterone. Expert summarised: 'Pre-eclampsia is caused by high blood pressure and proteinuria in pregnancy which is diagnosed by uncertain and is medicated by defective placentation' Expert summarised: ‘Progesterone issues are caused by contraceptives taken by women and could be diagnosed with bio-markers.’ Medical student System summarised: One or more infectious agents present in the placenta, with a predominant organism triggering the condition causes Pre-eclampsia. Pregnancy gets Pre-eclampsia. No preventative measures or screening tools for early detection of the condition is a symptom Pre-eclampsia. Defective placentation cures Pre-eclampsia. System summarised: Menstrual abnormalities during the later reproductive years causes progesterone. Women gets progesterone. Bio-markers is a symptom of progesterone. Pre-term delivery is an impact of progesterone. Use of levonorgestrel instrauterine cures progesterone. Expert summarised: 'Pre-eclampsia is caused by high blood pressure and proteinuria in pregnancy. Pre-eclampsia is diagnosed by no preventative measures or screening tools for early detection of the condition. Pre-eclampsia is due to one or more infectious agents present in the placenta, with a predominant organism triggering the condition. Pre-eclampsia is medicated by defective placentation.' Expert summarised: ‘Progesterone issues could be due to menstrual abnormalities during the later reproductive years of women which could be diagnosed with bio- markers. Pre-term delivery could occur due to progesterone issues and should use levonorgestrel instrauterine to avoid that.’ Healthcare practitioner System summarised: Respiratory tract, female reproductive tract, periodontal cavity infects Pre- eclampsia. No preventative measures or screening tools for early detection of the condition prevents Pre-eclampsia. System summarised: Contraceptive actions causes progesterone. Effects on all uterine cell types infects progesterone. Limit the damage from being born before organ maturity prevents progesterone. Expert summarised: 'Pre-eclampsia is infected by respiratory tract, female reproductive tract, periodontal cavity and is prevented by no preventative measures or screening tools for early detection of the condition' Expert summarised: ‘Progesterone issues are caused by contraceptives which would affect all uterine cell types and medication should be taken to avoid pre-term delivery.’ Table 4: Qualitative evaluation of system generated and expert generated abstraction summaries Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 12 7 Conclusion In this paper we have presented a novel participatory model for multi-document health information summarisation. Although many summarisation techniques have been proposed in research literature, none consider the crucial participatory role of both end-users and domain experts. The updated IS participation theory was extended to identify how user participation can be incorporated to a summarisation approach. The proposed model initiates with participatory template generation, followed by an abstractive and extractive summarisation approach. The generated summaries are integrated using NLP-based semantics integration and then presented based on the original template structure. The proposed model was developed using a client-server architecture and evaluated by 10 domain experts and 50 health information consumers in obstetrics. Both quantitative and qualitative evaluation outcomes are indicative of an effective participatory model for summarisation. Proposed model proves that user participation is essential in document summarisation as a summary depends on user preference. User participation is beneficial in health domain due to the diversity embedded in health information. Health forums demonstrate the need for user participation where different users look for different levels of information. The proposed model extends user participation in IS theory with different user group participation where medical practitioners participate with their domain knowledge and health information consumers participate by providing ratings and comments. The use of the model allows users to customize and generate relevant and accurate summaries for diverse expectations. As future work, we intend to extend the summarisation process to uses domain ontologies and semantic reasoning to generate a hybrid version of summaries with aggregate reasoning outcomes. Moreover, the summarisation process will be expanded to capture social media inputs as part of the summaries as well as conduct evaluations into other healthcare domains. References Afantenos, S., Karkaletsis, V., & Stamatopoulo, P., (2005). “Summarisation from medical documents: a survey”, Artificial Intelligence in Medicine, 33(2), p. 157-177. Aliguliyev, R.M., (2009). “A new sentence similarity measure and sentence based extractive technique for automatic text summarisation”, Expert Systems with Applications, (36), p. 7764–7772. Barzilay, R., McKeown, K. and Elhadad, M., (1999). “Information fusion in the context of multi-document summarisation.”, In Proceedings of ACL '99. Belkebir, R. and Guessoum, A., (2016). Concept Generalization and Fusion for Abstractive Sentence Generation. Expert Systems with Applications. Bird, S., Klein, E., & Loper, E. (2009). Natural Language Processing with Python (1 edition). Beijing ; Cambridge Mass.: O’Reilly Media. Burstein, F., Fisher, J. L., McKemmish, S. M., Manaszewicz, R., Malhotra, P. (2005). “User Centred Quality Health Information Provision: Benefits and Challenges,” in: Proceedings of the Thirty-Eighth Annual Hawaii International Conference on System Sciences Burstein, F., McKemmish, S.M., Fisher, J.L., Manaszewicz, R., Malhotra, P. (2006). “A Role for Information Portals as Intelligent Decision Support Systems: Breast Cancer Knowledge Online Experience”, in: Intelligent Decision-making Support Systems: Foundations, Applications and Challenges, Springer, London UK, pp. 359-383. Burstein, F., McKemmish, S.M., Fisher, J.L., Manaszewicz, R., Malhotra, P. (2006). “A Role for Information Portals as Intelligent Decision Support Systems: Breast Cancer Knowledge Online Experience”, in: Intelligent Decision-making Support Systems: Foundations, Applications and Challenges, Springer, London UK, pp. 359-383. Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 13 Collins, M., (1999). “Head-Driven Statistical Models for Natural Language Parsing”, University of Pennsylvania. De Silva, D., Burstein, F. and Fisher, J., (2012). Supporting personalised content management in smart health information portals. In Proceedings of the 23rd Australasian Conference on Information Systems 2012 (pp. 1-11). Eichhorn, B. R., & Tukel, O. I. (2015). A Review of User Involvement in Information System Projects. Int. J. Inf. Technol. Proj. Manag., 6(1), 26–53. Erkan, G. and Radev, D.R., (2004). “Lexrank: Graph-based lexical centrality as salience in text summarisation”, Journal of Artificial Intelligence Research, (22), pp. 457-479. Eysenbach, G., (2000). “Consumer health informatics”, BMJ, (320), pp. 1713-1716. Eysenbach, G. and Köhler, C., (2002). “How do consumers search for and appraise health information on the World Wide Web? Qualitative study using focus groups, usability tests, and in¬depth interviews”, BMJ, (324), pp. 573-577. Jadad, A.R. and Gagliardi, A., (1998). “Rating health information on the Internet: navigating to knowledge or to Babel”, JAMA, (279:8), pp. 611-614. Jones, K.S., (2007). “Automatic summarizing: The state of the art”, Information Processing and Management, (43), pp 1-41. Kappelman, Leon. A., and McLean, Ephraim. R., (1992). “Promoting Information System Success: The Respective Roles of User Participation and User Involvement,” Journal of Information Technology Management, (3:1), p.1-12 Knight, K., & Marcu, D., (2000). “Statistics-Based Summarisation - Step One: Sentence Compression”, In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, p. 703–710. Lafferty, J.D. and McCallum, A., (2001). “Conditional random fields: Probabilistic models for segmenting and labeling sequence data”, In Proceeding of the 18th international conference on machine learning, San Francisco, p. 282-289. Lloret, E. and Palomar, M., (2012). Text summarisation in progress: a literature review. Artificial Intelligence Review, 37(1), pp.1-41. Mani, I. and Maybury, M.T., (1999). “Advances in automated text summarisation”, Cambridge: MIT Press. Markus, M. L., and Mao, J.-Y. (2004). "Participation in Development and Implementation- Updating an Old, Tired Concept for Today's IS Contexts," Journal of the Association for Information Systems (5:11), p. 14. McKemmish, S.M., Manaszewicz, R., Burstein, F., Fisher, J.L. (2009). “Consumer Empowerment through Metadata-Based Information Quality Reporting: The Breast Cancer Knowledge Online Portal,” Journal of The American Society For Information Science And Technology (60:9), John Wiley & Sons, pp 1792-1807. McKeown, K., Klavans, J., Hatzivassiloglou, V., Barzilay, R. and Eskin, E., (1999). “Towards multidocument summarisation by reformulation: Progress and Prospects”, In AAAI/IAAI, p.453-460. McKeown, K.R., and Radev, D.R., (1995). “Generating summaries of multiple news articles”, In Proceedings of SIGIR '95, Seattle, Washington, p.74-82. Moen, H., Peltonen, L.-M., Heimonen, J., Airola, A., Pahikkala, T., Salakoski, T., & Salanterä, S., (2016). “Comparison of automatic summarisation methods for clinical free text notes”, Artificial Intelligence in Medicine, (67), p. 25-37. Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 14 Pilemalm, S., & Timpka, T. (2008). Third generation participatory design in health informatics—Making user participation applicable to large-scale information system projects. Journal of Biomedical Informatics, 41(2), 327–339. Pivovarov , R., & Elhadad, O., (2015). "Automated methods for the summarisation of electronic health records", JAMAI, p. 938-947. Radev, D., Hovy, E., and McKeown, K., (2002). “Introduction to the special issue on text summarisation”, Comput Linguist, (28:4). Radev, D.R., and Fan, W., (2000). “Automatic summarisation of search engine hit lists”, Proceedings of the ACL-2000 workshop on recent advances in natural language processing and information retrieval, Hong Kong, p.99-109. Radev, D.R., Jing, H., Sty, M., and Tam, D., (2004). “Centroid-based summarisation of multiple documents”, International Journal on Information Processing and Management, (40:6), p. 919-938. Saggion, H. and Poibeau, T., (2013). Automatic text summarisation: Past, present and future (pp. 3-21). Springer Berlin Heidelberg. Sarkar, A., Molla, D., and Paris, C., (2013). “An Approach for Query Focused Text Summarisation for Evidence based medicine”, Lecture Notes in Computer Science, 7885:10), p. 295-304. Thursky, K. A., & Mahemoff, M. (2007). User-centered design techniques for a computerised antibiotic decision support system in an intensive care unit. International Journal of Medical Informatics, 76(10), 760–768. Wan, X., (2008). “Using only cross-document relationships for both generic and topic-focused multi-document summarisations”, Information Retrieval, (11), p. 25-49. Wan, X., Yang, J., and Xiao, J., (2007). “Manifold-ranking based topic-focused multidocument summarisation”, In Proceedings of the 20th international joint conference on artificial intelligence, Hyderabad, p.2903-2908. Weng, C., McDonald, D. W., Sparks, D., McCoy, J., & Gennari, J. H. (2007). Participatory design of a collaborative clinical trial protocol writing system. International Journal of Medical Informatics, 76 Suppl 1, S245-251. Xiaoyan, C., and Wenjie, L., (2011). “A spectral analysis approach to document summarisation: Clustering and ranking sentences simultaneously”, Information Sciences, (181), p. 3816 - 3827. Xie, J. and Burstein, F., (2011). Using Machine Learning to Support Resource Quality Assessment: An Adaptive Attribute-based Approach for Health Information Portals, Proceedings of the 16th International Conference on Database Systems for Advanced Applications International Workshops, pp. 526-537 Copyright: © 2017 Nallaperuma & De Silva. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 3.0 Australia License, which permits non-commercial use, distribution, and reproduction in any medium, provided the original author and AJIS are credited. http://creativecommons.org/licenses/by-nc/3.0/au/ Australasian Journal of Information Systems Nallaperuma & De Silva 2017, Vol 21, Research on Health Information Systems Participatory Model for Multi-Document Summarisation 15