INTERNATIONAL JOURNAL OF COMPUTERS COMMUNICATIONS & CONTROL Online ISSN 1841-9844, ISSN-L 1841-9836, Volume: 16, Issue: 2, Month: April, Year: 2021 Article Number: 4166, https://doi.org/10.15837/ijccc.2021.2.4166 CCC Publications Human-In-The-Loop: Role in Cyber Physical Agricultural Systems M. Sreeram, S.Y. Nof Maitreya Sreeram School of Industrial Engineering Purdue University, USA 315 N Grant Street, West Lafayette, IN 47907 *Corresponding author: msreeram@purdue.edu Shimon Y. Nof School of Industrial Engineering Purdue University, USA 315 N Grant Street, West Lafayette, IN 47907 Abstract With increasing automation, the ‘human’ element in industrial systems is gradually being re- duced, often for the sake of standardization. Complete automation, however, might not be optimal in complex, uncertain environments due to the dynamic and unstructured nature of interactions. Leveraging human perception and cognition can prove fruitful in making automated systems robust and sustainable. “Human-in-the-loop” (HITL) systems are systems which incorporate meaningful human interactions into the workflow. Agricultural Robotic Systems (ARS), developed for the timely detection and prevention of diseases in agricultural crops, are an example of cyber-physical systems where HITL augmentation can provide improved detection capabilities and system perfor- mance. Humans can apply their domain knowledge and diagnostic skills to fill in the knowledge gaps present in agricultural robotics and make them more resilient to variability. Owing to the multi-agent nature of ARS, HUB-CI, a collaborative platform for the optimization of interactions between agents is emulated to direct workflow logic. The challenge remains in designing and inte- grating human roles and tasks in the automated loop. This article explains the development of a HITL simulation for ARS, by first realistically modeling human agents, and exploring two different modes by which they can be integrated into the loop: Sequential, and Shared Integration. System performance metrics such as costs, number of tasks, and classification accuracy are measured and compared for different collaboration protocols. The results show the statistically significant advan- tages of HUB-CI protocols over the traditional protocols for each integration, while also discussing the competitive factors of both integration modes. Strengthening human modeling and expanding the range of human activities within the loop can help improve the practicality and accuracy of the simulation in replicating a HITL-ARS. Keywords: Agricultural Robotics; Collaborative Intelligence; Human-in-the-Loop; Multi-Agent Simulation https://doi.org/10.15837/ijccc.2021.2.4166 2 1 Introduction Stress detection in agricultural plants has been a source of improving research over the past few years. Developing a resilient response network to agricultural stresses with appropriate collaborative control protocols is of immense importance to the food supply chain in terms of reliability and food security. Dynamic conditions give rise to a host of abnormalities, each of which must be dealt with in a timely manner through detection, monitoring, predictive and preventive maintenance [18]. Current research in precision agriculture (PA) includes the development of Cyber-Physical Systems (CPS) with Collaborative Intelligence (CI) capable of performing the aforementioned tasks [19]. There is a need to harmonize the interactions between agents in such systems, in order to locate and ensure optimal collaborative performance. HUB-CI (HUB with Collaborative Intelligence) is one such platform de- veloped to enable collaboration of information, knowledge and decisions [12]. HUB-CI encapsulates tenets from the Collaborative Control Theory [28], which emerged from the realization that with in- creasing complexity and inter-dependency of automated systems, they will collapse unless designed and optimized for effective cyber-supported collaboration. With the advancement in sensors and robotic technologies, the ideas of applying these technologies to agricultural settings have been discussed with increasing attention in controlled environments such as greenhouses [7]. The development of CPS integrating robotics, sensors and human entities for PA provide potential test beds for “resilient” agricultural systems. Monitoring of environmental condi- tions such as temperature and humidity can be accomplished through effective sensor deployments, but often these systems are affected by external noise due to the large amounts of data produced by the sensors. Lack of quantifiable data also prevents the systems from working dynamically, and this is one of the primary motivations behind Human-in-the-loop (HITL) systems. Robotic Agents (RAs) perform well in environments which are largely constant, structured and predicable [1, 11]. Developing autonomous systems for agricultural environments, however, is difficult because of the unpredictable nature of the terrain, disruptions [27] and variable environmental parameters. System simplification and performance can be improved by the introduction of a human agent (HA) into the system [3]. Inspired by Sreeram [33], this research aims to simulate and study the integration of human operators as intelligent agents within the agricultural systems, specifically for accurate detection of stress in agricultural crops. The objectives are summarized as follows: • Model and simulate human operators as intelligent agents capable of integration within the information and decision flows; Explore different integration architectures of HITL. • With emphasis on Collaborative Intelligence, develop robust multi-agent workflows for agricul- tural CPS to enable early detection of stresses and infections; This article is organized as follows. In Section 2, literature review and related work on Cyber-Physical Systems for precision agriculture, Collaborative Intelligence, HUB-CI, and HITL Systems are de- scribed. Section 3 contains the methodology involved for the design of the HITL simulation, human agent modeling and different levels of interaction with the system. Section 4 discusses the observa- tions, and evaluation of workflows. Concluding remarks and future research are proposed in Section 5. 2 Background 2.1 Cyber-Physical Systems in Precision Agriculture Cyber-Physical Systems (CPS) are traditionally defined as integrations of computation with phys- ical processes, where embedded computing devices and networks integrated with feedback loops mon- itor, control and support physical process [22]. An important paradigm of CPS is the parallelism to Industry 4.0, which considers technologies such as sensor and actuator networks, human-robot inter- actions, big data analytics and decentralization. Internet of Things (IoT) and Internet of Services (IoS) have vastly improved the effectiveness of CPS in real-time decision support and administration, virtualization of physical components and data collection. The current Industrial Revolution driven https://doi.org/10.15837/ijccc.2021.2.4166 3 by CPS and IoT is expected to have a major impact on the future of work and on agriculture as well, as there is a natural relation between industry and agriculture [8]. Currently, there is a need for effective early disease detection techniques to control plant diseases for food security and sustainability of Agro-ecosystems [17, 35]. Precision Agriculture is an application of an Agricultural CPS consisting of three layers: physical, network, and decision layer [29]. It comprises a set of technologies that combines sensors, information systems, enhanced machinery, and informed management to optimize production by accounting for variability and uncertainties within agricultural systems [23]. The network layer, defined by the use of Wireless Sensor Networks (WSN), and have been used for the monitoring of environmental factors such as temperature, humidity, soil content successfully in agricultural settings [2] leveraging different IoT technologies for real-time flow of information. The decision layer, founded on human knowledge, is aided by the rise of non-invasive remote sensing techniques such as fluorescence and hyperspectral imaging, spectroscopy [6], which can prove to be sensitive and consistent in detecting defects in plants. Robots in agriculture, which redefine the physical layer (as opposed to human operators) are devel- oped to reduce the reliance on farmers for tedious and repetitive tasks such as fertilization, irrigation, and disease detection. With the rise of sensor reliability, development of agricultural robots fitted with remote sensing and vision systems for accurate detection of crop diseases has grown [15]. Robots must be integrated with intelligent control and communication systems and made capable of working in unstructured environments and terrains. 2.2 Human-in-the-loop Systems While the complexity of performable tasks by robots has increased, there still exist many tasks where robots show a considerable gap in performance when compared to humans. Complex operations such as object detection under uncertainty [9], task classification in human-robot work cells [10] require human cognitive decision-making and properly designed collaboration between humans and robots. Multi-agent systems designed with optimized human-robot interactions and collaborative intelligence are termed as “human-in-the-loop (HITL)”. Previous literature on this topic includes human-robot interactions for target recognition [5], human decision modeling [34], robots querying humans for help [20]. Current approaches work asynchronously with HAs who are expected to provide input in terms of decisions and data interpretations. Leveraging human cognition and perception can improve the versatility of complex tasks, help in reducing errors and conflicts with dynamic environments, and drive productivity [4]. Bechar & Edan [3] also reported the advantages of HITL Systems for melon detection tasks, on average showing higher detection rates and lower detection times as compared to fully autonomous and manual systems. Figure 1: Partial Autonomy in HITL Systems For traditional agricultural systems, HAs are usually trained inspectors tasked with sampling, inference and physical movement between locations. Appropriate task taxonomies should also be studied to understand the range of tasks which can be performed by the HAs in the human-robot collaboration context. Bechar et al.,[4] develop different levels of Human-Robot Collaboration (HRC) based on the four degrees of autonomy [32], with each level corresponding to a unique HRC workflow to elucidate the different possible integrations of humans in the loop. Each integration entails a different https://doi.org/10.15837/ijccc.2021.2.4166 4 workload for the HA, and appropriate workload levels must be maintained for sustainable benefit, since human performance is not perfect and subject to fatigue [36]. Limited by time, physical and mental constraints, increasing workload often result in reduced performance, increased chance of error [21, 36]. This is an important measure in high-risk systems such as machine operation, maintenance, emergency protocols – it needs to be ensured that the HAs interface for the optimal amount of time, as long as their benefits can overbear their disadvantages. 2.3 Collaborative Intelligence and HUB-CI Effective collaboration implies the timely sharing of information, resources and responsibilities between distributed agents to plan, implement and accomplish individual and common goals of the combined system [37]. For CPS, often it is necessary to ensure that the collaboration caters across different agent and interaction types, and the suitability of collaboration is judged the “Collaborative Intelligence” of an agent. Collaborative Intelligence (CI) can be measured by agents’ ability to interpret new information, share resources and information with other peers to resolve new local and global problems in a dynamic environment [37]. Figure 2: Overview of HUB-CI as a CPS control platform for collaborative interactions among IoT/IoS, physical and virtual agents. There have been several collaborative tools designed for knowledge and information sharing [13]. HUB-CI is a tool designed for cyber-supported collaboration between physical and virtual agents to optimize and harmonize agent actions across domains. It is a platform of e-Work systems equipped with algorithms and cyber collaborative protocols specifically developed to provide versatility in collaboration-support [16], telerobotics [38] and network optimization. The main objective of HUB-CI is to enable agents to identify the following as a function of time and current conditions: (1) what information to contribute; (2) which tools to use; (3) with whom to collaborate. The key innovation of HUB-CI is that it can enable and facilitate physical collaboration between several groups of human participants, along with relevant cyber-physical agents [38] while previous HUBs were limited to vir- tual interactions between agents [24]. In this article, HUB-CI logic [26] is used to develop a simulation platform for the communication and interaction of different agents comprising a cyber physical ARS. HUB-CI is the central control platform responsible for the exchange and distribution of information https://doi.org/10.15837/ijccc.2021.2.4166 5 and decisions between these agents, in an optimized and harmonized manner. By managing informa- tion from three central algorithms (Sampling/Routing, Adaptive Search, and Human-in-the-Loop), HUB-CI provides concise, timely and efficient collaboration. 3 Methodology 3.1 Task Description A cyber physical agricultural robotic system (ARS), described in [14] is a multi-agent system com- prising of humans (experts), an autonomous robotic cart, and a sensor system. The robotic cart is mounted with remote sensing equipment and vision systems to effectively maneuver to specific loca- tions within a greenhouse and inspect plants for the early and timely detection of stress or infections, while the human experts provide real-time decision/inference support. Decision support is required for two reasons: 1) robot motion occurs in an unstructured and unpredictable environment, and 2) the reliability of the sensor data and automated predictive models can vary, and any potential for error and conflict must be mitigated in real time. This challenge is one of the reasons humans are required in the loop: human cognition and versatility can be leveraged to improve dynamic response and fill the diagnostic gap. In order to test the inclusion of humans in the loop, a simulation-based approach is employed for the provided task description. By simulating the ARS, the scope of human control can be varied with different configurations, which can be tested and evaluated. The objective is to detect the early onset of diseases in greenhouse plants using the best combination of given agents. 3.2 Sampling/Routing Current monitoring systems in greenhouses involve a manual process where a trained inspector scouts the plants on foot within the plot, usually sampling a few locations at each plot. The sampling locations are determined arbitrarily, following a fixed heuristic (policy). Constrained by large plots, limited human resources, time constraints and high costs, sampling rates even for trained operators are low. Usage of robotic agents can improve sampling rates by shifting the bottleneck from the human inspectors, but due to added costs it is essential to employ sampling strategies which improve detection rates while reducing redundant sampling. To estimate optimal sampling plan for all spots, a sampling model is employed which applies pre-determined infection parameters, such as initial chance of infection diagnosis, and chance of plant-to-plant propagation, to estimate high-risk locations and prioritize them in the sampling queue. A routing heuristic is applied to generate the shortest path between the set of recommended sampling nodes in a given grid. 3.3 Adaptive Search Figure 3: Adaptive Search Algorithm (Source: Dusadeerungsikul and Nof [14]). https://doi.org/10.15837/ijccc.2021.2.4166 6 The need for preventive and predictive maintenance of agricultural crops in a timely manner was established by [18]. The Adaptive Search (AS) algorithm, as reported in [14] is a sensing algorithm used to indicate the extent or severity of a stress/disease in greenhouse plants. Stresses in plants usually propagate in predictable directions, which are often influenced by external factors, such as sunlight and airflow. Leveraging this knowledge, the AS algorithm has been developed to extend the sampling space to include predictable locations with relatively higher risk of stress. Depending on factors such as the type of disease-stress detected, type of plant sampled, and environmental conditions, the search region for AS varies. For example, if a plant is found to have a stress/disease based on the sampling data, AS is initiated to add the locations of plants located vicinity of the plant, depending on the type and severity of the stress/disease. For the purpose of this research, only the plants in the immediate vicinity of the stressed/diseased plant are considered. 3.4 HITL Integrations The major challenges in integrating HAs include 1) understanding the range of applications of HITL control, or the different taxonomies based on the controls that HAs employ, 2) modeling human behavior of various types, and 3) identifying the optimal modeling schemes for each type [25, 31] also address the question of workflow control in such systems: which agent should control the system at a given time period; how do they delegate tasks and avoid decision conflicts? This research attempts to evaluate different workflow control structures, based on individual and shared control. HAs are modeled as dynamic agents, able to provide decision inferences on different plant locations. To model these agents, the following attributes were chosen: • Task performance: It is assumed that HAs can perform the tasks at a relatively better rate than the corresponding RAs, but this relative advantage is achieved at a higher cost. RAs are developed to reduce physical load and provide automated and generic control, but they lack versatility and cognition that HAs possess. • Fatigue Models: HAs are prone to relatively higher error rates and inferior task performance (longer time taken to service task) under increased task load based on pre-defined thresholds. • Limited availability: HAs are assumed to be available for task assignment only for fixed time periods (HITL Level) Task control is defined as the agent currently controlling the system: for e.g., if the robot performs and validates a scan on a certain plant, task control is said to be with the RA; if the HA provides the validation, task control is maintained by the HA. Task control is dynamic and switches between agents based on their availability. For a specific task, it can either be solely maintained by a single agent (Robot/Human scans individually) or shared by multiple agents (e.g. Robot can request Human for help). Based on these differences, two workflow modes are simulated and compared: Sequential, and Shared Integration. 3.4.1 Sequential Integration In this mode of integration, task control is maintained solely by either of the agents, but not shared. Task control switches sequentially between RA and HA. The human agents are available for a fixed time period (HITL Level) and within this period are completely assigned to provide scanning inferences for all tasks. This mode of integration requires available HAs to necessarily perform tasks, and thus it is necessary to monitor fatigue. It is also useful to find a theoretical optimum time period, which is denoted as Optimal HITL Level (OHL) for which the HAs can contribute maximum benefit to the system. Figure 4 (L) shows the sequential integration workflow. 3.4.2 Shared Integration As opposed to Sequential Integration, task control in the case of Shared Integration is collabora- tively distributed, with the ability of agents to collaborate with other agents when required. Agent https://doi.org/10.15837/ijccc.2021.2.4166 7 Figure 4: Sequential (L) vs Shared (R) Integration Workflows Table 1: Performance Variables α = Distance Normalization Constant β = Time Normalization Constant γ = Penalty Constant Cmove = Motion Cost Cpenalty = Penalty Cost Cscan = Scan Cost Ctot = Total Cost D = Total number of infected plants detected ej = Error instance of at jth task i = Index of previous task completed j = Index of task being completed MDR = Missed Detection Ratio RSR = Redundant Sampling Ratio S = Total number of sampled plants Tinf ected = Total number of infected plants tj = Scan time of jth task xi,yi = Coordinates of ith task availability does not necessarily correspond to task assignment, especially in the case of human agents (in sequential integration, an available HA must take up task control). For instance, if a result of the robot scan does not provide definite inference, it can request help from a human agent, if now available; the human agent can provide their final inference based on trained expertise. Figure 4 (R) shows the workflow for shared integration, indicating the difference between the two workflow modes. 3.5 Performance Metrics Two types of performance metrics are calculated, based on costs and routing parameters. The variables for difference calculations are shown in Table 1. 3.5.1 Cost Calculations The overall costs incurred in the system due to the agents and their interactions are captured in Equation 1. Cost can be incurred from three operations: motion, scanning and error penalties. Motion costs are incurred by the RAs which traverse between different plant locations. Scanning and error cost https://doi.org/10.15837/ijccc.2021.2.4166 8 parameters vary depending on the agent performing the scan. The probability of error for the human agent depends on the fatigue level which increases with the number of tasks performed by the human agent. Fatigue also effects task performance, defined by the time required for the agent to complete the chosen tasks (scanning/decision inference); with increased fatigue, task performance decreases (task time increases). The probability of error for the robot agent remains constant throughout the simulation since fatigue for robotic agents is not considered. Ctot j = Cmove(i,j) + Cscan(j) + Cpenalty(ej) (1) Cmove(i,j) = α(|xj −xi|+|yj −yi|) (2) Cscan(j) = βtj (3) Cpenalty(ej) = γCscan(j) (4) 3.5.2 Sampling/Routing Metrics To measure the effectiveness of the sampling, routing and Adaptive Search techniques, the follow- ing metrics are developed and observed during the experiments for each complete run of the simulation. Total Sampled/Task Performance (S): Number of tasks performed by the HITL ARS for a single run of the simulation. Missed Detection Ratio (MDR): Percentage of missed infections which the system might have missed or cannot capture within the given run. MDR = Tinfected −D Tinfected (5) Redundant Sampling Ratio (RSR): Percentage of sampled locations which were unnecessarily sampled. RSR = S −D D (6) Classification Observations: For each run, the sampling model estimates whether a plant has an infection (positive outcome) and compares this outcome to the outcome of the scanning task, which depends on the agent with task control: • True Positives: Sampling and scanning indicate positive outcome • False Positive: Sampling indicates positive, scanning indicates negative outcome • False Negative: Sampling indicates negative, scanning indicates positive outcome • True Negative: Sampling and scanning indicate negative outcome 3.6 Experiment Design Two simulation experiments are designed to analyze and evaluate respective metrics across dif- ferent HITL Levels. Various simulation tools such as V-REP, Gazebo, AgROS and Webots [30] are extensively used to test agricultural robotics. These tools, however, are specifically dedicated for the integration of agricultural robotics into real-world environments, and none of them are specifically designed to evaluate the integration of HAs into such robotic systems. The simulation reported in this research is developed using an object-oriented programming (OOP) approach in Python, and it follows a discrete-time, event-based architecture. This approach allows us to dynamically model humans in the simulation by providing flexibility in defining attributes such as fatigue and chance of error. The simulator is designed for the task description provided in Section 3.1. Within the simulation, an event is defined as any of the motion or scanning tasks occurring as part of the task sequence. Each event is performed for a discrete time-period, A single run in the simulation consists of 100 time steps, and the number of tasks (events) completed in the given time frame for a fixed HITL Level (varied across runs) are monitored. For e.g., if the HITL Level is 30%, HAs are available and can perform https://doi.org/10.15837/ijccc.2021.2.4166 9 tasks for 30 timesteps of the overall 100 timesteps. When the HAs are available (depending on the HITL Level), they perform tasks within the system with active fatigue models to monitor the fatigue and error levels after each task. The following simulated collaboration protocols are considered: 1. Random (baseline) protocol: The set of locations to be visited are generated at random for a chosen HITL Level. (a) Without Adaptive Search (b) With Adaptive Search 2. HUB-CI protocol: The set of locations to be visited are based on a probability (risk)-based sampling model with Adaptive Search (simulating the cyber collaborative protocol of HUB-CI) for a chosen HITL Level. Based on the above description, two simulation experiments are performed to evaluate sequential and shared integration for the different collaboration protocols across HITL Levels. 4 Results Table 2: Comparing HUB-CI performance with Random Protocol for each Integration Mode Integration Cost Task Performance RSR MDRw/o AS AS w/o AS AS w/o AS AS w/o AS AS Sequential +10% +0.8% +44% +15% -30% -26% -50% -41% Shared -5% -5% +44% +18% -32% -27% -44% -37% 4.1 Experiment 1: Sequential Integration Figure 5: True and false classification observations for Experiment 1: (a) True Positives, (b) False Positives, (c) False Negatives, (d) True Negatives for HITL Level Sequential Integration relies on completely utilizing the available agents to complete the required tasks. True and false classification observation graphs for each HITL Level are shown in Figure 5. The parabolic nature of the graphs suggest the existence of point of inflection in each graph, coinciding https://doi.org/10.15837/ijccc.2021.2.4166 10 approximately at the same HITL Level. This is the first evidence of an optimal HITL Level (OHL) for this integration, since at these points of inflection the classification accuracy of the system seems to be optimal. To elaborate, at this OHL (approximately 45% HITL Level), the true positives and negatives both are maximum, while both false positives and negatives are at a minimum. In Figure 6, the task performance (a) and costs (b) for different simulation schemes over different HITL Levels are plotted. From Figure 6(a), the difference is task performance is significant, and the HUB- CI protocol outperforms the Random protocol (with and without AS). Comparing task performance across HITL Levels, HUB-CI performs on average 4.67 (44%) tasks more than Random (without AS), and 2.08 (15%) more tasks than Random (with AS) (Table 2). To highlight improvements due to HITL, we compare the HUB-CI protocol performance for two fixed HITL Levels: 45% (M = 15.44, S = 0.07) and 0% (M = 14.06, S = 0.06); the results (t(196.65) = 143.56, p < 2.2E-16) provide strong evidence to showcase the benefits of HITL augmentation. Figure 6: (a)Task(L) and (b)Cost Performance(R) for different simulated collaboration protocols Figure 6(b) shows the costs of each collaboration protocol for different HITL Levels. A one-way ANOVA test (F(2) = 9.026, p = 0.0001) determined that the difference between costs are statistically significant. HUB-CI incurs, on average, 10% added costs when compared to Random (without AS) and 0.8% when compared to Random (with AS). The cost trend seems to increase exponentially with increasing HITL Level, but since motion and scanning costs are directly related to the number of tasks performed (concave downward trend as shown in Figure 6(a)), this increase can be attributed to penalty costs occurring due to higher instances of errors, and also increased costs from lowered task servicing due to fatigue. An interesting point to note is that at the OHL (45% HITL Level), the costs for each protocol remain similar, but the task performance at the same level shows large differences, thus indicating the advantages of the HUB-CI protocol. Table 3 shows the performance metrics at the OHL to elaborate further on the previous statement. Table 3: Performance Metrics for HITL Level = 45. Protocol # Tasks Cost RSR MDR Mean SD Mean SD Mean SD Mean SD HUB-CI 15.44 0.07 897.82 6.75 43.28 0.32 33.10 0.52 Random(AS) 13.27 0.04 911.17 4.75 59.63 0.53 60.04 0.58 Random(no AS) 10.458 0.03 851.65 4.86 64.66 0.55 72.99 0.38 Difference* 16.35%* 1.45%* 33.06%* 44.87%* *Note: Difference between HUB-CI and Random (with AS); statistically significant (p < 0.0001) Redundant Sampling Ratio (RSR) and Missed Detection Ratio (MDR) for different HITL Levels are plotted in Figure 7. The concave nature of both plots provides more evidence to support the https://doi.org/10.15837/ijccc.2021.2.4166 11 existence of the OHL, with the local minima of both plots coinciding approximately and the formerly stated OHL (45%). At this level, the benefits of HITL are maximized and beyond this level negative effects due to human error are prevalent. In Figure 8, the MDR and RSR for different collaboration protocols is over different HITL Levels is plotted. It is clearly evident that the HUB-CI protocol is more efficient in terms of sampling and routing for infections. Figure 7: RSR and MDR Comparison for HUB-CI Figure 8: RSR and MDR Comparison across HITL Levels 4.2 Experiment 2: Shared Integration In Experiment 2, the collaborating agents share task control and agent utilization prioritizes ne- cessity over availability. Tasks are shared only when the RA cannot provide definite results, and when the HA is available. The true and false classification in Figure 9, unlike Sequential Integration do not indicate the existence of an Optimal HITL Level for this integration. Shared task control ensures that HAs only get tasks when necessary, and this reduces the impact of errors on task classification. For e.g., in Figure 9(a), true positives increase with increasing HITL Level: increased HITL Level does not necessarily mean that the HAs perform more tasks, but that they are more available to provide decision inference on tasks which could not be classified initially (by the RAs). Figure 10 plots the task and cost performance for each collaboration protocol, and it is evident that HUB-CI protocol is relatively the most productive: 4.13 (44%) tasks more than Random (without AS), and 2.1 (18%) more tasks than Random (with AS), as seen in Table 2. The decrease in mean tasks per https://doi.org/10.15837/ijccc.2021.2.4166 12 Figure 9: True and false classification observations for Experiment 2: (a) True Positives, (b) False Positives, (c) False Negatives, (d) True Negatives for HITL Level run can be due to the increased availability of HAs in the loop, consequently increasing cumulative scan time. The cost performance provides an important observation: as opposed to sequential inte- gration, the costs for HUB-CI protocol (M = 738.28, S = 5.27) decrease slightly with increasing HITL Level, and the differences between protocols are statistically significant (F(2) = 57.62, p < 2.2E-16). HUB-CI incurs, on average, 5% reduced costs when compared to Random (without AS) and 5% when compared to Random (with AS). Figure 10: (a)Task(L) and (b)Cost Performance(R) for different simulated collaboration protocols The lack of impact due to fatigue is visible in the RSR and MDR metrics from Figure 11, which decrease monotonically with increasing HITL Level. The best case MDR for shared integration, however, (in Figure 11, at HITL Level = 100% - 40.01%) underperforms when compared to Sequential Integration, which had a lower best case MDR (at OHL - 32.02%). Best case RSR for shared integration (at HITL Level = 100 – 41.08%) performs marginally better than sequential integration (at OHL – 42.70%). Figure 12 cements HUB-CI simulated protocol as a better sampling, routing, and search https://doi.org/10.15837/ijccc.2021.2.4166 13 protocol. Figure 11: RSR and MDR for HUB-CI Figure 12: RSR and MDR Comparison across HITL Levels 4.3 Cost Comparisons An agent-based cost comparison was also performed to understand the nature of costs in each integration. From Figure 13, it is evident that with increasing HITL Level, the cost in Sequential Integration is preferred for lower HITL Levels, while Shared Integration ensures that the cost per task does not increase exponentially across HITL Levels. This is also backed up by Figure 14, where an agent-based cost-breakdown is provided. The stacked bar charts show that for Experiment 1, the con- tribution of HAs to the total cost dominates (due to increased penalty costs, lower task performance) over high HITL Levels, overshadowing the RA costs involved. This is not the case for Experiment 2, even though HA cost does increase, it remains nominal compared to the RA costs, suggesting that this integration is more cost efficient and does ensure that HAs are tasked appropriately to lessen the degree of penalty and performance costs. https://doi.org/10.15837/ijccc.2021.2.4166 14 Figure 13: Comparing cost/task for shared and sequential integrations across HITL Levels Figure 14: Agent-wise cost breakdown across HITL Levels https://doi.org/10.15837/ijccc.2021.2.4166 15 5 Conclusion This research examines Human-in-the-loop systems within the scope of Agricultural Robotic Sys- tems. There is a need for humans in the loop given the unstructured nature of the environment and their ability to leverage specific knowledge bases for decision-making and dynamic events. The objective is twofold: 1. To create a precise HA model for accurate integration into the loop, including active fatigue and error models, and 2. To explore different possible integration modes of humans within the cyber-physical loop, with the target of early detection of stress leading to infections in greenhouse plants To manage communication and interaction between diverse agents (robots, humans) and algo- rithms (Sampling, Adaptive Search, and HITL), a simulation based on HUB-CI is developed. The simulation monitors KPI’s of the system in relation to sampling and routing efficiency, task classifica- tion and productivity, and overall costs. By comparing the system performance for different simulated protocols, the devised experiments provide insight into the augmentation provided by the three pri- mary algorithms, Sampling/Routing, Adaptive Search, and HITL under two HITL workflow modes: Sequential, and Shared HITL Integration modes. Both Integration modes establish the relative advan- tage of the HUB-CI collaboration protocol in providing significantly better performance and results, as follows: • In Sequential Integration, the priority of task management is agent utilization, and task con- trol is maintained individually based on availability. Based on the established workflow, the performance metrics calculated indicate the existence of an Optimal HITL Level (OHL), where the productivity gains from HAs are at a maximum. The main drawback of this integration is the cost management and the effect on the agents, since by compulsorily pushing tasks to HAs, increased costs due to fatigue and error penalties lead to an exponential increase in system costs. As shown in the cost comparison, the cost contribution due to HAs overshadows the cost from the RAs for higher HITL Levels, thus indicating the cascading effects of fatigue and error • For Shared Integration, HAs are inserted into the workflow (subject to availability) only when collaboration is absolutely essential to provide an inference. Task control is now shared, leading to lower task performance, since both agents now contribute to contentious tasks. The experi- ment results do not provide evidence of an Optimal HITL Level (OHL) for this integration mode, since HAs are not driven to complete utilization (and thus maximum productivity benefits). The positives of this integration mode, however, are evident in the cost performance, which remains largely stable over the different HITL Levels. Compared to Sequential Integration, it can be inferred that this cost stability of the Shared Integration mode is the result of no productivity losses and penalty losses due to fatigue and induced errors By establishing the critical factors of both HITL integration modes, the advantages of the HUB- CI protocol for ARS workflow collaboration are elaborated. The experiment results also provide a measure of the difference in productivity, which is enhanced by applying the HUB-CI collaboration protocol. Future research can be targeted towards hybrid interactions consisting of elements from both integration modes. For example, a hybrid workflow where a priority-based time-out protocol can be used to determine the amount of time the HA has to dynamically interface in the system, based on task characteristics. It is also important to improve HA models to make them more realistic towards decision making, sampling plans, and expanding the range of performable tasks in the loop. Funding This work is supported by the Production, Robotics, and Integration Software for Manufacturing & Management (PRISM) Center at Purdue University. The research on ARS is supported by BARD project Grant IS-4886-16R, “Development of a Robotic Inspection System for Early Identification https://doi.org/10.15837/ijccc.2021.2.4166 16 and Locating of Biotic and Abiotic Stresses in Greenhouse Crops”; and NSF project Grant 1839971, “FW-HTF: Collaborative Research: Pre-Skilling Workers, Understanding Labor Force Implications and Designing Future Factory Human-Robot Workflows Using a Physical Simulation Platform”. Author contributions The authors contributed equally to this work. Conflict of interest The authors declare no conflict of interest. References [1] Ajoudani, Arash, Andrea Maria Zanchettin, Serena Ivaldi, Alin Albu-Schäffer, Kazuhiro Kosuge, and Oussama Khatib. 2018. “Progress and Prospects of the Human–Robot Collaboration.” Au- tonomous Robots 42 (5): 957–75, 2018. [2] Aqeel-Ur-Rehman, Abu Zafar Abbasi, Noman Islam, and Zubair Ahmed Shaikh. 2014. “A Re- view of Wireless Sensors and Networks’ Applications in Agriculture.” Computer Standards and Interfaces 36 (2): 263–70, 2014. [3] Bechar, Avital, and Yael Edan. 2003. “Human-Robot Collaboration for Improved Target Recog- nition of Agricultural Robots.” Industrial Robot30 (5): 432–36. [4] Bechar, Avital, Yael Edan, and Joachim Meyer. 2006. “Optimal Collaboration in Human-Robot Target Recognition Systems.” Conference Proceedings - IEEE International Conference on Sys- tems, Man and Cybernetics 5: 4243–48, 2006. [5] Bechar, Avital, Joachim Meyer, and Yael Edan. 2007. “An Objective Function to Evaluate Perfor- mance of Human-Robot Systems for Target Recognition Tasks.” Conference Proceedings - IEEE International Conference on Systems, Man and Cybernetics 84105: 967–72, 2007. [6] Behmann, Jan, Jörg Steinrücken, and Lutz Plümer. 2014. “Detection of Early Plant Stress Re- sponses in Hyperspectral Images.” ISPRS Journal of Photogrammetry and Remote Sensing 93: 98–111, 2014. [7] Belforte, G.; Deboli, Gay, R. P.; Piccarolo,P.; Ricauda Aimonino, D. (2006). “Robot Design and Testing for Greenhouse Applications.” Biosystems Engineering 95 (3): 309–21, 2006. [8] Blanchet, Max, Thomas Rinn, Georg Von Thaden, and G De Thieulloy. 2014. “Industry 4.0: The New Industrial Revolution-How Europe Will Succeed.” Hg. v. Roland Berger Strategy Consultants GmbH. München. Abgerufen Am 11.05. 2014, Unter Http://Www. Rolandberger. Com/Media/Pdf/Roland_Berger_TAB_Industry_4_0_2014 0403. Pdf. [9] Branson, Steve, Catherine Wah, Florian Schroff, Boris Babenko, Peter Welinder, Pietro Perona, and Serge Belongie. 2010. “Visual Recognition with Humans in the Loop.” In Proceedings of the 11th European Conference on Computer Vision: Part IV, 438–51. ECCV’10. Berlin, Heidelberg: Springer-Verlag. http://dl.acm.org/citation.cfm?id=1888089.1888123. [10] Bruno, Giulia, and Dario Antonelli. 2018. “Dynamic Task Classification and Assignment for the Management of Human-Robot Collaborative Teams in Workcells.” The International Journal of Advanced Manufacturing Technology 98 (9): 2415–27. https://doi.org/10.1007/s00170-018-2400- 4. [11] Cai, Hong, and Yasamin Mostofi. 2019. “Human-Robot Collaborative Site Inspec- tion under Resource Constraints.” IEEE Transactions on Robotics 35 (1): 200–215. https://doi.org/10.1109/TRO.2018.2875389. https://doi.org/10.15837/ijccc.2021.2.4166 17 [12] Devadasan, Prabhu, Hao Zhong, and Shimon Y. Nof. 2013. “Collaborative Intelligence in Knowledge Based Service Planning.” Expert Systems with Applications 40 (17): 6778–87. https://doi.org/10.1016/j.eswa.2013.06.048. [13] Durugbo, Christopher. 2016. “Collaborative Networks: A Systematic Review and Multi- Level Framework.” International Journal of Production Research 54 (12): 3749–76. https://doi.org/10.1080/00207543.2015.1122249. [14] Dusadeerungsikul, Puwadol Oak, and Shimon Y. Nof. 2019. “A Collaborative Control Protocol for Agricultural Robot Routing with Online Adaptation.” Computers and Industrial Engineering 135 (August 2018): 456–66. https://doi.org/10.1016/j.cie.2019.06.037. [15] Dusadeerungsikul, Puwadol Oak, Shimon Y Nof, Avital Bechar, and Yang Tao. 2019. “Collab- orative Control Protocol for Agricultural Cyber-Physical System.” Procedia Manufacturing 39: 235–42. https://doi.org/https://doi.org/10.1016/j.promfg.2020.01.330. [16] Dusadeerungsikul, Puwadol Oak, Maitreya Sreeram, Xiang He, Ashwin Nair, Karthik Ra- mani, Alexander J. Quinn, and Shimon Y. Nof. 2019. “Collaboration Requirement Plan- ning Protocol for Hub-Ci in Factories of the Future.” Procedia Manufacturing 39: 218–25. https://doi.org/10.1016/j.promfg.2020.01.327. [17] Fang, Yi, and Ramaraja P Ramasamy. 2015. “Current and Prospective Methods for Plant Disease Detection.” Biosensors 5 (3): 537–61. https://doi.org/10.3390/bios5030537. [18] Gongal, A., S. Amatya, M. Karkee, Q. Zhang, and K. Lewis. 2015. “Sensors and Systems for Fruit Detection and Localization: A Review.” Computers and Electronics in Agriculture 116: 8–19. https://doi.org/10.1016/j.compag.2015.05.021. [19] Guo, Ping, Puwadol Dusadeeringsikul, and Shimon Y. Nof. 2018. “Agricultural Cyber Physical System Collaboration for Greenhouse Stress Management.” Computers and Electronics in Agri- culture 150 (June 2017): 439–54. https://doi.org/10.1016/j.compag.2018.05.022. [20] Holzinger, Andreas, Markus Plass, Michael Kickmeier-rust Katharina, Holzinger Gloria, and Cerasela Cris. 2019. “Interactive Machine Learning: Experimental Evidence for the Human in the Algorithmic Loop A Case Study on Ant Colony Optimization,” 2401–14. [21] Jaber, M. Y., and W. P. Neumann. 2010. “Modelling Worker Fatigue and Recovery in Dual-Resource Constrained Systems.” Computers and Industrial Engineering 59 (1): 75–84. https://doi.org/10.1016/j.cie.2010.03.001. [22] Lee, Edward A. 2010. “CPS Foundations.” Proceedings - Design Automation Conference, 737–42. https://doi.org/10.1145/1837274.1837462. [23] Lentz, E C, Compendium Project, B L Gardner, G C Rausser, M T Ruel, Food Security Assess- ment, Food Security Statistics, et al. 2010. “References and Notes 1.,” no. February. [24] Mclennan, Michael, and Rick Kennell. 2010. “HUBzero: A Platform for Dissemination and Col- laboration in Computational Science and Engineering.” Computing in Science and Engineering 12 (2): 48–52. https://doi.org/10.1109/MCSE.2010.41. [25] Munir, S., Stankovic, J. a, Liang, C.-J. M., Lin, S. 2013. “Cyber Physical System Challenges for Human-in-the-Loop Control.” The 8th International Workshop on Feedback Computing. [26] Nair, A S, A Bechar, Y Tao, and S Y Nof. 2019. “The HUB-CI Model for Telerobotics in Green- house Monitoring.” Procedia Manufacturing 39: 414–21. [27] Nguyen, Win P V, and Shimon Y Nof. 2019. “Collaborative Response to Disruption Propagation (CRDP) in Cyber-Physical Systems and Complex Networks.” Decision Support Systems 117: 1–13. https://doi.org/10.15837/ijccc.2021.2.4166 18 [28] Nof, S. Y. 2007. “Collaborative Control Theory for E-Work, e-Production, and e-Service.” Annual Reviews in Control 31 (2): 281–92. https://doi.org/10.1016/j.arcontrol.2007.08.002. [29] Rad, Ciprian-Radu, Olimpiu Hancu, Ioana-Alexandra Takacs, and Gheorghe Olteanu. 2015. “Smart Monitoring of Potato Crop: A Cyber-Physical System Architecture Model in the Field of Precision Agriculture.” Agriculture and Agricultural Science Procedia 6: 73–79. [30] Ramin Shamshiri, Redmond, Ibrahim A. Hameed, Lenka Pitonakova, Cornelia Weltzien, Siva K. Balasundram, Ian J. Yule, Tony E. Grift, and Girish Chowdhary. 2018. “Simulation Software and Virtual Environments for Acceleration of Agricultural Robotics: Features Highlights and Performance Comparison.” International Journal of Agricultural and Biological Engineering 11 (4): 12–20. https://doi.org/10.25165/j.ijabe.20181104.4032. [31] Schirner, Gunar, Deniz Erdogmus, Kaushik Chowdhury, and Taskin Padir. 2013. “The Future of Human-in-the-Loop Cyber-Physical Systems.” Computer 46 (1): 36–45. https://doi.org/10.1109/MC.2013.31. [32] Sheridan, Thomas B, and William L Verplank. 1978. “Human and Computer Control of Undersea Teleoperators.” [33] Sreeram, Maitreya. 2020. “Human-in-the-Loop of Cyber Physical Agricultural Robotic Systems.” Purdue University Graduate School. https://doi.org/10.25394/PGS.13303490.v1. [34] Stewart, A, M Cao, A Nedic, D Tomlin, and N Leonard. 2012. “Towards Hu- man–Robot Teams: Model-Based Analysis of Human Decision Making in Two-Alternative Choice Tasks With Social Feedback.” Proceedings of the IEEE 100 (3): 751–75. https://doi.org/10.1109/JPROC.2011.2173815. [35] Wang, Dongyi, Robert Vinson, Maxwell Holmes, Gary Seibel, Avital Bechar, Shimon Nof, and Yang Tao. 2019. “Early Detection of Tomato Spotted Wilt Virus by Hyperspectral Imaging and Outlier Removal Auxiliary Classifier Generative Adversarial Nets (OR-AC-GAN).” Scientific Re- ports 9 (1): 4377. https://doi.org/10.1038/s41598-019-40066-y. [36] Winkelhaus, S., F. Sgarbossa, M. Calzavara, and E. H. Grosse. 2018a. “The Effects of Hu- man Fatigue on Learning in Order Picking: An Explorative Experimental Investigation.” IFAC- PapersOnLine 51 (11): 832–37. https://doi.org/10.1016/j.ifacol.2018.08.442. [37] Zhong, H.; Levalle,R.R.; Mohsen Moghaddam, Nof, S.Y. 2015. “Collaborative Intelligence - Defini- tion and Measured Impacts on Internetworked e-Work.” Management and Production Engineering Review 6 (1): 67–78, 2015. [38] Zhong, H.;, Wachs,J.P.; Nof, S.Y. 2014. “Telerobot-Enabled HUB-CI Model for Collaborative Lifecycle Management of Design and Prototyping.” Computers in Industry 65 (4): 550–62, 2014. https://doi.org/10.15837/ijccc.2021.2.4166 19 Copyright ©2021 by the authors. Licensee Agora University, Oradea, Romania. This is an open access article distributed under the terms and conditions of the Creative Commons Attribution-NonCommercial 4.0 International License. Journal’s webpage: http://univagora.ro/jour/index.php/ijccc/ This journal is a member of, and subscribes to the principles of, the Committee on Publication Ethics (COPE). https://publicationethics.org/members/international-journal-computers-communications-and-control Cite this paper as: Sreeram, M.; Nof, S.Y. (2021). Human-In-The-Loop: Role in Cyber Physical Agricultural Systems, International Journal of Computers Communications & Control, 16(1), 4166, 2021. https://doi.org/10.15837/ijccc.2021.2.4166 Introduction Background Cyber-Physical Systems in Precision Agriculture Human-in-the-loop Systems Collaborative Intelligence and HUB-CI Methodology Task Description Sampling/Routing Adaptive Search HITL Integrations Sequential Integration Shared Integration Performance Metrics Cost Calculations Sampling/Routing Metrics Experiment Design Results Experiment 1: Sequential Integration Experiment 2: Shared Integration Cost Comparisons Conclusion