HUNGARIAN JOURNAL OF INDUSTRY AND CHEMISTRY Vol. 47(1) pp. 79–83 (2019) hjic.mk.uni-pannon.hu DOI: 10.33927/hjic-2019-12 RULE-BASE FORMULATION FOR CLIPS-BASED WORK ERGONOMIC AS- SESSMENT BENEDEK SZAKONYI *1 , TAMÁS LŐRINCZ1 , ÁGNES LIPOVITS2 , AND ISTVÁN VASSÁNYI1 1Department of Electrical Engineering and Information Systems, University of Pannonia, Egyetem u. 10, Veszprém, 8200, HUNGARY 1Department of Mathematics, University of Pannonia, Egyetem u. 10, Veszprém, 8200, HUNGARY Modern societies are dominated by computer-based work. As a result, people tend to be seated for most of their working life. Prolonged sedentariness is known to significantly increase the risk of developing unwanted conditions. This paper presents the development of a rule-based expert system module using CLIPS that provides ergonomic assessment. The system was validated by evaluating pre-recorded user logs from real-life office environments. The tests showed that the system is able to perform the required basic assessment functionality, thus, the implementation of more complex features to advance its development is viable. Keywords: expert system, CLIPS, work ergonomics 1. Introduction Civilization-related diseases have become more widespread over time, although small improvements to our lifestyle could effectively reduce both their severity and number of sufferers. As adults spend a significant proportion of time working, which for the vast majority involves sitting in front of a computer [1], under what conditions this time is spent is very much relevant. Thanks to tools of Information Technology and er- gonomics experts, it is possible to provide automated so- lutions, in terms of monitoring and assessing user be- haviour, to help more employees avoid the adoption of undesirable and harmful postures whilst at work. Re- search has already shown that providing such real-time feedback to users can promote this, thus, reducing the im- pact of related negative effects [2, 3]. But in these cases, wearable sensors were used to track user motion, which can be considered problematic to be applied in real life applications. A possible solution to this might be Mi- crosoft’s Kinect sensor [4] that, being a good value-for- money motion capture device, has already been used in similar cases [5, 6]. In spite of this, just as in numer- ous other papers that concern work-related assessment [7–10], considerably more focus is placed on manual labour where “blue-collar workers” are subject to phys- ically demanding conditions (e.g. production lines, agri- cultural or construction work, etc.). Even though these fields are of equal importance, such conditions cannot be applied to office environments, where “white-” and *Correspondence: benedek.szakonyi@virt.uni-pannon.hu “pink-collar workers” spend most of their time, as here the majority of problems does not originate from incor- rect movement (e.g. bad techniques when lifting heavy items) but rather from the (utter) lack of movement itself and prolonged sedentariness. The aim of this research was to develop an expert sys- tem module for ergonomic assessment (based on a previ- ously created Lifestyle Coach Framework [11] as shown in Fig. 1 with an emphasis placed on the formulated work ergonomics-based rule set, and to investigate if such a system is capable of properly evaluating user behaviour in office environments. 2. Experimental 2.1 The Framework used The Lifestyle Coach Framework used in the develop- ment process is a rule-based expert system framework using CLIPS (C Language Integrated Production System) [12]. The main reason for using a rule-based solution (in- stead of neural networks, fuzzy logic, etc.) is the fact that most healthcare experts can express their knowledge using simple IF-THEN-like statements, which is exactly how rule-based systems work. The CLIPS runtime is responsible for providing the basic functionality of evaluating the current state of the system and deducing the most suitable response and re- actions. The statements that describe the current state and different events are called facts, while rules convey the relevant information concerning what actions to take in mailto:benedek.szakonyi@virt.uni-pannon.hu 80 SZAKONYI, LŐRINCZ, LIPOVITS, AND VASSÁNYI Figure 1: The components of the framework used. the event (or absence) of such facts, in the form of “sim- ple” IF-THEN expressions. The majority of these compo- nents are neither predefined nor hardcoded in the frame- work itself (except for those that are automatically gener- ated) but are to be defined in light of the specific topic and task at hand in the form of a rule set. Hence the frame- work can be applied in different fields as long as the prob- lem is interpretable as the evaluation of a log of succes- sive events. These events are transmitted to the system via the Protobuf Interface (that uses Google Protobuf [13]), which also serves as a tool to receive the corresponding responses. The messages of which can be categorized into two types, namely control and event. The former is used to manage the expert system itself, e.g. to signal that a unit of time has passed and re-evaluation of the state of the system should be commenced. The latter serves as a way of inserting new facts (incoming data) and handling consequences (outgoing reactions). How such messages are interpreted and forwarded is the responsibility of the Event Dispatcher. Apart from bridging the gap between the interface and CLIPS runtime, it is also connected to a database that is used to initialize (i.e. it stores the rule- base used), record system behaviour and act as a backup should anything fail. 2.2 Work Ergonomics To create the rule base required for ergonomic evaluation, methods used to estimate the risks of developing Muscu- loskeletal Disorders (MSDs) were analysed, with the help of ergonomics experts. The most popular and widely ac- cepted tools used for such tasks are RULA (Rapid Upper Limb Assessment) [14], REBA (Rapid Entire Body As- sessment) [15], OWAS (Ovako Working Posture Analy- sis System) [16] and HARM (Hand Arm Risk-assessment Method) [17]. RULA applies a scoring method for measuring the physical load that workers are subjected to when adopting specific postures by taking into consideration six main body regions: neck, trunk, legs, upper arms, forearms and wrists. REBA works in a similar fashion by using scores but takes into consideration the whole body when such postures are adopted. OWAS uses a 4-digit score for each posture where the digits describe the back, arms, legs and load. HARM accounts for the head, neck, arms and wrists as well as the forces that are applied and the duration over which each posture is adopted. However, these methods cannot be used directly for the matter at hand as originally none of them were intended to be used in regular office environments: RULA was developed to assess work in the textile industry, REBA in healthcare, OWAS in the steel industry, and HARM in hand-intensive professions (e.g. barbers, product assembly/disassembly, woodwork, etc.). Based on these tools, a basic method of postural as- sessment was derived as presented in Table 1. It is im- portant to note that when subject to these constraints, the values used were chosen as initial limits as a proof of concept in light of the fact that they may need to be up- dated. For each body part, the relevant axes of movement were selected and for each axis, ranges were defined. A range can be characterised into three types: appropriate, incorrect and harmful. Appropriate ranges define the de- sired position the user should take, while harmful ones represent postures that should be avoided or even prohib- ited. Incorrect ones fall somewhere between the other two categories, when, as far as is feasible, a posture should be prevented since it is considered unhealthy, however, when adopted for short intervals of time it is still acceptable. For the ranges of the inappropriate postures, a frequency is given, that describes that in one work hour, how much time spent in them (in total) is still considered as tolera- ble. Apart from evaluating specific body parts based on their position, the time the user spends sitting was also selected to be taken into account, as a general aspect. 2.3 Rule-base Formulation The formulation of the constraints introduced in Table 1 that are applied to a CLIPS-based rule set could have been determined using numerous different approaches. The one that was selected, builds on the fact that the im- plementation of these aspects has the same general struc- ture: 1. there are numerical measurement values to be eval- uated over each time interval 2. there is a constraint that the measured values are compared to when evaluated 3. there is a frequency that, if exceeded, should trigger a warning to be sent. Therefore, instead of creating multiple rules for each aspect, one main rule can be defined and applied during the evaluation which can make use of an “aspect tem- plate” to check if the related conditions have been ad- hered to or not. To achieve this, as rules in CLIPS are gov- erned by facts, the aspects are to be in the form of facts. In Fig. 2 the template used for defining aspects is shown. The aspect_type serves as a name/identifier for the aspect, while measurement_type defines which measurements Hungarian Journal of Industry and Chemistry Rule-Base Formulation for Clips-Based Work Ergonomic Assessment 81 Table 1: The advised constraints to be used in the formu- lated rule base. For each body part, separate ranges were used for the different planes of movement. The acceptable duration within each range was defined (as % of 1 working hour). Body part Axis Range Frequency Head/neck Sagittal < 0 < 20% 0 < < 10 10 < < 30 < 40% 30 < < 20% Horizontal < ±10 ±10 < < ±30 < 40% ±30 < < 20% Frontal < ±5 ±5 < < ±10 < 40% ±10 < < 20% Trunk Sagittal < 0 < 40% 0 < < 15 15 < < 25 < 40% 25 < < 20% Horizontal < ±5 ±5 < < ±10 < 40% ±10 < < 20% Frontal < ±5 ±5 < < ±10 < 40% ±10 < < 20% Upper arm Sagittal < −10 < 20% −10 < < 0 < 40% 0 < < 10 10 < < 45 < 40% 45 < < 20% Frontal < 10 10 < < 20 < 40% 20 < < 20% Forearm Sagittal < 60 < 20% 60 < < 80 < 40% 80 < < 100 100 < < 110 < 40% 110 < < 20% should be inspected. In spite of some axes, where ranges of constraints are “evenly distributed” or “symmetrical” (e.g. head frontal, trunk sagittal), the relation between the measured data and the constraint must be provided in the constraint_type template, as for other cases, the ranges are “uneven”. This type can be either smaller than or equal to, or greater than or equal to, followed by the value to use in the value_constraint. The horizon_minute slot is used to determine the duration of measurement that is to be examined (e.g. the last 60 minutes). As cases may oc- cur when, despite being warned, users continue to follow their unhealthy lifestyle, a feature that permits repeated warnings of increasing severity (or even more extensive interventions such as turning off the computer screen) could be useful. In the logic developed, checking whether the user has taken the advice given or not is considered to takes less time than the original evaluation interval (this will have an impact on the main evaluating rule, that is de- tailed later on). With regard to this feature, apart from the first_occurrence_constraint that corresponds to the fre- quency in Table 1 and the first_reaction_event_code that defines what actions to take if the limit has been reached, (deftemplateaspect_to_monitor (slot aspect_type (type SYMBOL)(default ?NONE)) (slot measurement_type (type SYMBOL)(default ?NONE)) (slot constraint_type (type SYMBOL)(default ?NONE)) (slot value_constraint (type NUMBER)(default ?NONE)) (slot horizon_minute (type NUMBER)(default ?NONE)) (slot first_occurrence_constraint (type NUMBER)(default ?NONE)) (slot first_reaction_event_code (type NUMBER)(default ?NONE)) (slot repeated_occurrence_constraint (type NUMBER)) (slot repeated_reaction_event_code (type NUMBER)) (slot repeated_feedback_horizon (type NUMBER)) ) Figure 2: The CLIPS fact template used for describing the structure of ergonomic aspects. optional repeated_ slots can be used for this purpose in the template. As the evaluating CLIPS rule is lengthy, only a short explanation of the logic behind it is provided here. Over each evaluation interval, the number of measurements that exceed the given constraints during the correspond- ing time horizon is counted, and if this value is above the acceptable limit, a request to send feedback is made and the related measurements are labelled as evaluated (ac- cording to the current aspect). If only a few “bad” mea- surements are identified, the evaluation is considered to be finished, unless repeated feedback is received with re- gard to the current aspect. In that case, it must be de- termined if a “first feedback” was given recently, and if it was, the number of measurements that exceed the limit must be compared to the occurrence constraint of the sec- ond feedback. If this is exceeded, a second feedback is sent. 2.4 Experiments for Rule-base Validation As the aim of the research was to provide feedback for users concerning their postures by using a rule base con- sisting of ergonomic rules, a method for recording and identifying their postures was needed. To accomplish this, a Microsoft Kinect v2 sensor was used. As it is an easy-to-use and relatively small device, it was possible to insert it into the real working setups of the willing par- ticipants, with only slight modifications in their environ- ments. The general setup that was created for all users was the following: each user was seated at a desk with a personal computer that consisted of 1 or 3 displays, a key- board, a mouse and a landline telephone. The sensor was placed on a tripod behind and slightly above the “main” screen (i.e. the one in front of the user). The Kinect was tilted forward in order to ensure as much of the partic- ipant as possible was in view (from the top of the head down to the waist/hips). Users were monitored according to two different “be- haviour modes”, one being general “everyday” attitude where they completed their daily computer-based tasks as usual. As this inherently meant that users might have had to leave their desks, long measurement sessions were recorded (up to 8 hours in duration) where logging was 47(1) pp. 79–83 (2019) 82 SZAKONYI, LŐRINCZ, LIPOVITS, AND VASSÁNYI suspended once the absence of the participant was de- tected until their return, when it was resumed (such “gaps”, of course, were then taken into account as part of the evaluation). In the other “mode”, where consid- erably shorter measurement intervals were used in order to ensure sessions that definitely consisted of unsuitable behaviour, the users were asked to adopt some evidently inappropriate postures. As the framework used offers a customisable time unit for defining when an interval stops (i.e. it can last for an hour, a minute, a second, etc.), it was possible to load and evaluate the logs gathered using this method quickly. 3. Results and Discussion Based on the evaluation method suggested by the er- gonomist experts, a total of 11 aspects have been formu- lated (by using 34 facts), 9 for the positions of the body parts (one for each axis using the constraints shown in Table 1) and 2 regarding sedentariness, where the accept- able duration of continuous sitting was chosen to not ex- ceed 60 minutes (the first aspect was related to sending a warning once this limit had been reached, the second was used to alert when a participant had been sitting continu- ously for 3 hours). Subsequently the recorded user logs were evaluated, for the majority of aspects used (10 out of 11) the ex- pected functionality was achieved. However, in the case of the facts responsible for assessing head positions, con- siderably more warnings were sent by the system than was acceptable. As for manually created input (simulated user postures), this behaviour had not emerged again, ex- amination of the monitoring software developed have be- gun. Upon inspection of the logs created, it was found that this error was a result of improper calibration: the inaccuracy of the sensor itself was higher than antici- pated and the distortions that resulted meant that even when the user evidently adopted a suitable posture, the logged value exceeded the threshold defined as accept- able. Based on this information, the related constraints were adjusted accordingly and this undesirable behaviour of the system was successfully removed. This source of inconvenience, however, highlights the possible sensitiv- ity and dependency of the system, i.e. the reliability of the measurement data received. Still, its significance might be diminished by utilizing methods developed for im- proving the accuracy and error tolerance of Kinect, such as the ones proposed in [18–20]. In general, it can be said that the system developed and the initial rule set created are capable of providing the required functionality. However, there is still room for improvement as the constraints involved could be fine-tuned and additional or more precise aspects imple- mented. Moreover, a more complex system capable of adapting to user behaviour could also be created, the main goal of which would be to assist users, without being too rigorous or repetitive, by changing how frequently and in what manner its responses are displayed. This could help to maintain the motivation of workers to break bad habits concerning improper postures whilst seated. Even though the number of participants in the ex- periments conducted was sufficient to validate the ini- tial version of the expert system, further investigations with considerably larger user groups are desirable as they would provide a more thorough validation and may yield additional insight into what other aspects the sys- tem could provide assistance to users. Additionally, while the recording sessions, on average of 5-6 hours in dura- tion, implemented so far have been useful, by extending these to a few days or even a couple of weeks, more com- plex behavioural patterns could be identified and anal- ysed. Furthermore, by providing real-time assessment over much longer periods, the effects caused by this inter- vention could be investigated and subsequently the meth- ods used to assist users improved. Nevertheless, from the results that have already been logged, it can be clearly seen that the global tendency of users to be seated for longer periods of time than is advisable was also exhibited by the cases investigated. Most participants reported that over such prolonged ses- sions whilst seated, a need to change position is common, however, usually a better alternative cannot be found. A solution to this problem could be to provide workers with electric height adjustable desks, which may facili- tate the ability to switch easily between sitting and stand- ing working postures and customise the height of desks to match the anthropometric features of the individual. Moreover, such furniture could enable the expert system itself to “take action”, when needed, to raise the desk when users fail to heed previous warnings. It should be noted that, when asked, most of the par- ticipants considered such a monitoring and assessment system to be useful, however, some, while welcoming the idea of an adjustable desk, expressed concerns about personal security and were reluctant to be “continuously monitored”. This reveals that if such a system enters the market, apart from providing the necessary security mea- sures, assuring users that their personal data is safe would also be necessary. 4. Conclusion In this research a proof-of-concept CLIPS-based rule- base for an expert system was created to facilitate er- gonomic evaluation of users in office environments. It has been shown that such a system is capable of thor- oughly evaluating user behaviour, thus, implementing in- terventions in order to decrease the negative effects of prolonged sedentariness. Acknowledgement The research was supported by the ÚNKP-18-3 New Na- tional Excellence Program. The authors thank the assis- tance of the Department of Ergonomics and Psychol- ogy of the Budapest University of Technology and Eco- nomics. The authors also acknowledge the support of the Hungarian Journal of Industry and Chemistry Rule-Base Formulation for Clips-Based Work Ergonomic Assessment 83 Széchenyi 2020 programme under the EFOP-3.6.1-16- 2016-00015 project. REFERENCES [1] Smith, M. J.; Conway, F. T.; Karsh, B. T.: Oc- cupational stress in human computer interaction, Ind. Health, 1999 37(2), 157–173 DOI: 10.2486/ind- health.37.157 [2] Vignais, N.; Miezal, M.; Bleser, G.; Mura, K.; Gorecky, D.; Marin, F.: Innovative system for real- time ergonomic feedback in industrial manufactur- ing, Appl. Ergonomics, 2013 44(4), 566–574 DOI: 10.1016/j.apergo.2012.11.008 [3] Battini, D.; Persona, A.; Sgarbossa, F.: Innovative real-time system to integrate ergonomic evaluations into warehouse design and management, Computers Ind. Eng., 2014 77, 1–10 DOI: 10.1016/j.cie.2014.08.018 [4] Zhang, Z.: Microsoft Kinect sensor and its ef- fect, IEEE Multimedia, 2012 19(2), 4–10 DOI: 10.1109/MMUL.2012.24 [5] Plantard, P.; Shum, H. P. H.; Le Pierres, A. S.; Multon, F.: Validation of an ergonomic assessment method using Kinect data in real workplace con- ditions, Appl. Ergonomics, 2017 65, 562–569 DOI: 10.1016/j.apergo.2016.10.015 [6] Cippitelli, E.; Gasparrini, S.; Spinsante, S.; Gambi, E.: Kinect as a tool for gait analysis: Validation of a real-time joint extraction algorithm working in side view, Sensors, 2015 15(1), 1417–1434 DOI: 10.3390/s150101417 [7] Rivero, L. C.; Rodríguez, R. G.; Pérez, M. D. R.; Mar, C.; Juárez, Z.: Fuzzy logic and RULA method for assessing the risk of working, Pro- cedia Manufacturing, 2015 3, 4816–4822 DOI: 10.1016/j.promfg.2015.07.591 [8] Kong, Y. K.; Lee, S. Y.; Lee, K. S.; Kim, D. M.: Comparisons of ergonomic evaluation tools (ALLA, RULA, REBA and OWAS) for farm work, Int. J. Occup. Safety Ergon., 2018 24(2), 218–223 DOI: 10.1080/10803548.2017.1306960 [9] Golabchi, A.; Han, S.; Fayek, A. R.: A fuzzy logic approach to posture-based ergonomic analysis for field observation and assessment of construction manual operations, Can. J. Civil Eng., 2016 43(4), 294–303 DOI: 10.1139/cjce-2015-0143 [10] Berlin, C.; Adams, C.: Production ergonomics: De- signing work systems to support optimal human performance, (Ubiquity Press, London, England) 2017 pp.139-160 DOI: 10.5334/bbe [11] Szakonyi, B.; Lőrincz, T.; Lipovits, Á.; Vassányi, I.: An expert system framework for lifestyle coun- selling. In: eTELEMED 2018: The tenth interna- tional conference on eHealth, telemedicine, and so- cial medicine, 2018 pp. 92-96 ISBN: 978-1-61208-618-7 [12] Riley, G.: CLIPS: a tool for building expert sys- tems, http:\www.clipsrules.net/?q=AboutCLIPS [ac- cessed September 20, 2018] [13] Google Inc.: Protocol Buffers, http:\developers. google.com/protocol-buffers [accessed September 20, 2018] [14] Lueder, R.: A proposed RULA for computer users, In: Proceedings of the ergonomics summer work- shop, UC Berkeley Center for Occupational & En- vironmental Health Continuing Education Program, San Francisco, August 8-9, 1996 24, 91–99 [15] Hignett, S.; McAtamney, L.: Rapid Entire Body As- sessment (REBA), Appl. Ergonomics, 2000 31(2), 201–205 DOI: 10.1016/S0003-6870(99)00039-3 [16] Karhu, O.; Kansi, P.; Kuorinka, I.: Correcting work- ing postures in industry: A practical method for analysis, Appl. Ergonomics, 1977 8(4), 199–201 DOI: 10.1016/0003-6870(77)90164-8 [17] Douwes, M.; Boocock, M.; Coenen, P.; van den Heuvel, S.; Bosch, T.: Predictive validity of the Hand Arm Risk assessment Method (HARM), Int. J. Ind. Ergonomics, 2014 44(2), 328–334 DOI: 10.1016/j.ergon.2013.09.003 [18] Plantard, P.; Hubert, H. P.; Multon, F.: Filtered pose graph for efficient kinect pose reconstruction, Mul- timedia Tools Appl., 2017 76(3), 4291–4312 DOI: 10.1007/s11042-016-3546-4 [19] Shum, H. P. H.; Ho, E. S. L.; Jiang, Y.; Tak- agi, S.: Real-time posture reconstruction for Mi- crosoft Kinect, IEEE Trans. Cybernetics, 2013 43(5), 1357–1369 DOI: 10.1109/TCYB.2013.2275945 [20] Zhou, L.; Liu, Z.; Leung, H.; Shum, H.P.H.: Pos- ture reconstruction using Kinect with a probabilistic model, In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology (VRST), 2014, pp. 117-125 DOI: 10.1145/2671015.2671021 47(1) pp. 79–83 (2019) https://doi.org/10.2486/indhealth.37.157 https://doi.org/10.2486/indhealth.37.157 https://doi.org/10.1016/j.apergo.2012.11.008 https://doi.org/10.1016/j.apergo.2012.11.008 https://doi.org/10.1016/j.cie.2014.08.018 https://doi.org/10.1109/MMUL.2012.24 https://doi.org/10.1109/MMUL.2012.24 https://doi.org/10.1016/j.apergo.2016.10.015 https://doi.org/10.1016/j.apergo.2016.10.015 https://doi.org/10.3390/s150101417 https://doi.org/10.3390/s150101417 https://doi.org/10.1016/j.promfg.2015.07.591 https://doi.org/10.1016/j.promfg.2015.07.591 https://doi.org/10.1080/10803548.2017.1306960 https://doi.org/10.1139/cjce-2015-0143 https://doi.org/10.5334/bbe http:\www.clipsrules.net/?q=AboutCLIPS http:\developers.google.com/protocol-buffers http:\developers.google.com/protocol-buffers https://doi.org/10.1016/S0003-6870(99)00039-3 https://doi.org/10.1016/0003-6870(77)90164-8 https://doi.org/10.1016/j.ergon.2013.09.003 https://doi.org/10.1016/j.ergon.2013.09.003 https://doi.org/10.1007/s11042-016-3546-4 https://doi.org/10.1007/s11042-016-3546-4 https://doi.org/10.1109/TCYB.2013.2275945 https://doi.org/10.1145/2671015.2671021 Introduction Experimental The Framework used Work Ergonomics Rule-base Formulation Experiments for Rule-base Validation Results and Discussion Conclusion