Mathmet Measurement Uncertainty Training activity – Overview of courses, software, and classroom examples ACTA IMEKO ISSN: 2221-870X June 2023, Volume 12, Number 2, 1 - 6 ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 1 Mathmet Measurement Uncertainty Training activity – Overview of courses, software, and classroom examples Francesca R. Pennecchi1, Peter M. Harris2 1 Istituto Nazionale di Ricerca Metrologica (INRIM), Strada delle Cacce 91, 10135 Torino, Italy 2 National Physical Laboratory (NPL), Hampton Road, Teddington TW11 0LW, UK Section: Technical note Keywords: Measurement Uncertainty (MU); training; Mathmet; overview; MU courses; classroom examples and software Citation: Francesca R. Pennecchi, Peter M. Harris, Mathmet Measurement Uncertainty Training activity – Overview of courses, software, and classroom examples, Acta IMEKO, vol. 12, no. 2, article 12, June 2023, identifier: IMEKO-ACTA-12 (2023)-02-12 Section Editor: Eric Benoit, Université Savoie Mont Blanc, France Received July 1, 2022; In final form March 14, 2023; Published June 2023 Copyright: This is an open-access article distributed under the terms of the Creative Commons Attribution 3.0 License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Funding: NPL’s work was supported by the UK Government’s Department for Business, Energy, and Industrial Strategy (BEIS) as part of its National Measurement System (NMS) programme. Corresponding author: Francesca Pennecchi, e-mail: f.pennecchi@inrim.it 1. INTRODUCTION In October 2021, a two-year Mathmet [1] activity was launched [2] with the aim of developing new training material and establishing an active community for those involved in teaching measurement uncertainty. This “Measurement Uncertainty Training activity” [3] is conducted by a consortium of Mathmet and non-Mathmet members who committed themselves, on a voluntary basis, to develop new training material on measurement uncertainty (MU) and to strengthen collaborations among experts and interested people at metrology institutes, universities, industry and within accreditation and legal metrology communities. Concerning the development of new material for MU training, this will include an overview of existing courses, software and examples, which can guide trainees across the tools and materials already available at different levels and in different fields of application. It is also planned to prepare some short videos explaining the need for, and common difficulties in, evaluating MU. All material will be made publicly available on the dedicated webpage [3] of the Mathmet website and will be actively disseminated to a large set of practitioners in metrology, academia, and industry. In the present abstract, we will focus on the survey of existing courses and software for MU evaluation, together with the review of selected examples, suitable for MU training, which will be revisited in the form of proper classroom examples. Further new training material that will be developed from scratch by the MU Training Activity is presented separately at this joint symposium [4] and will not be detailed here. 2. OVERVIEW OF EXISTING COURSES AND SOFTWARE Starting from the plethora of courses on MU usually offered by the partners of the consortium, the first step was to undertake a review of such courses, in order to inform the wider audience about their availability and characteristics. In this respect, Mathmet will serve as a reference point to make connections among trainers and trainees at a European level and beyond. The ABSTRACT A collaborative activity on “Measurement Uncertainty (MU) Training” under the auspices of the European Metrology Network for Mathematics and Statistics (Mathmet) is underway. This abstract reports on the progress to undertake surveys of existing training courses on MU, software for MU evaluation, and classroom examples to support the understanding of methods for MU evaluation. An appreciable number of training courses, software and examples have been identified and are currently under review. These tools and materials will be analysed and categorised according to their main features and characteristics. Special attention will be given to their adherence to the JCGM guidelines (i.e., JCGM 100:2008, JCGM 101:2008 and JCGM 102:2011). It is hoped that the knowledge assembled in this activity will help practitioners to make good choices about appropriate material to support their training needs, as well as help developers of training material to ensure good coverage of their training products and target them at user needs. mailto:f.pennecchi@ ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 2 courses will be categorized according to their main features to enable the audience to easily identify which course would fit the best with their need. Analogously, based on the availability of a variety of software (SW) performing MU evaluation, a critical overview of such tools is currently underway to analyse their characteristics, such as the status, the kind of methods they implement and the main operating conditions. 2.1. Existing courses on MU So far, information on 41 training courses taught by 14 partners and 1 stakeholder of the activity consortium has been collected, for a total amount of more than 880 hours of lessons per year. For each course, a specifically developed template was completed by a reference contact, who was free to provide the following details: - General information (title of the course, its integration into a training framework or project, specific field of application, organizing body, website advertising the course, duration, frequency, language(s), location, material provided to attendees, attendance fee, final examination, kind of certification, etc.) - Audience (target audience, specific constraints and prerequisites, average number of attendees) - Teacher/s or technical contact/s - Technical contents - Classroom examples From a preliminary analysis of the features undertaken it is worth noting that most of the courses are specifically dedicated to MU evaluation but a good number have a broader scope (covering, for example, metrology in general or for a specific SI quantity). These courses were included in the review as they make a strong effort in teaching MU evaluation. Figure 1 gives an idea of their specific fields of application. The majority of courses (78 %) is given on a recurring basis but some are e- Learning courses and are available on demand. 15 % are offered in more than one language (see the languages distribution in Figure 2). 50 % require some sort of final examination and 50 % have an enrolment fee. 15 % of courses are aimed at legal metrology, 37 % at NMIs, 46 % at calibration and testing laboratories and 24 % at academia (with overlapping categories). Concerning the “Technical contents”, the contact person completing the template was required to describe the main topics of the course and the extent to which they comply with the prescriptions of the JCGM WG1 suite of documents [5], with a special focus on the teaching of the Law of Propagation of Uncertainty (LPU) and the Monte Carlo Method (MCM) for propagation of distributions: - Review of mathematical tools (linear algebra, partial derivatives, linear regression, …) - Review of probability concepts (random variables, distributions, …) - Basic metrological concepts (measurand, measurement model, error, accuracy, precision, repeatability, reproducibility, …) - Input standard uncertainties and covariances (GUM Type A and Type B) - LPU (GUM 1st or also higher-order Taylor series expansion, expanded uncertainty) - LPU (JCGM 102 multivariate models) - MCM for propagation of distributions (JCGM 101 univariate models) - MCM for propagation of distributions (JCGM 102 multivariate models) - Validating LPU against MCM - Reporting the measurement result As a result, 34 % of courses provide a review of mathematical concepts, 85 % of probabilistic topics and 95 % of metrological topics. Almost all discuss how to model and evaluate input standard uncertainties and covariances, as well as the application of LPU to univariate models. Interestingly, though, only 20 % address LPU for multivariate models (JCGM 102). Concerning the teaching of MCM, 44 % of courses treat MCM for univariate models (JCGM 101) but only 15 % for multivariate models (JCGM 102): see Figure 3 and Figure 4. Moreover, the training on MCM is not homogenous across the audience: it is almost never taught in courses for the legal metrology community, a third of the time to calibration and testing laboratory personnel, half of the time to NMI employees and most of the time in courses for academia. As a general comment, it seems there is a gap in the treatment of multivariate models, both from the side of LPU and even worse for what concerns application of MCM. This implies that little attention is given to the training on calculation of covariances among measurands depending on some common input quantities and hence being correlated. This seems in Figure 1. Fields of application of the courses. Figure 2. Languages used in the courses (the total number of occurrences of languages used in the courses, also considering the various combinations of languages, was 49). ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 3 contrast with the fact that the main target audience (46 %) are calibration and testing laboratories and calibration procedures often involve multivariate models. An encouraging result is that 75 % of the courses dealing with LPU for multivariate models address the problem by teaching also the corresponding MCM. In the questionnaire, it was also possible to specify which references are used and if any software is applied or mentioned in the course. The references mainly reported are documents of the JCGM WG1 suite, ISO and OIML standards, EURAMET and ILAC Guides, as well as documents by EA, UKAS, DIN, EURACHEM/CITAC, etc. 68 % of the courses rely on, or at least mention, use of some SW or programming language, like Excel, Matlab, R, LabVIEW, Origin, the NIST uncertainty machine and the GUM Workbench. Among the technical topics treated on top of standard ones (i.e., LPU and MCM), the following are also mentioned: Bayesian inference, conformity assessment, linear regression, and quality control. Comments concerning the “Classroom examples” are left until Section 3. 2.2. Software for MU evaluation A further survey was initiated by a subset of the activity partners, i.e., INRIM, NPL, LNE, IPQ, IMBiH, METAS and POLITO, with the aim of categorizing available software related to MU and summarizing the methods offered by such software to the end users. A list of software was agreed within the consortium encompassing 50 SW for MU evaluation, coming from several sources (mainly from a Wikipedia webpage [6]). 35 SW were already analysed by involved partners, by filling in an agreed list of characteristics/features. The software ranged from basic uncertainty calculators to quite complex, broad-scope software, and from user-friendly web applications to comprehensive collections of libraries and tools for uncertainty quantification. Some of those SW are currently under analysis by the partners, considering the following characteristics: - General information - Technical features - Adherence to JCGM 100:2008 - Adherence to JCGM 101:2008 - Adherence to JCGM 102:2011 For the “General information” and “Technical features” items, information is reported on license, version, programming language, whether the SW is computer-based or a web application, its language(s), documentation, and evidence of verification and validation. Concerning the SW so far analysed, 74 % of them are cross-platform, 85 % computer-based (15 % web application), 54 % provide some evidence of validation, and all are available in English version (some also in other languages). The distribution of the programming languages is shown in Figure 5. In the questionnaire, moreover, it has to be stated whether the SW is able to handle correlated input quantities, nonlinear models, more than one output quantity (most of the analysed SW have these features, i.e., 86 %, 89 % and 57 %, respectively), complex-valued quantities, implicit models, symbolic uncertainty evaluation, repeated input observations, and input imported from previous analyses (these features, instead, are generally less covered by the SW). Information will be also given on the output results and their format. The adherence of the implemented methods with the JCGM documents [5] is investigated in some detail. The aim is to assess the metrological relevance of each SW and its level of compliance with recognized guidelines. Concerning JCGM 100:2008, the SW is checked against its ability to implement the LPU (without or with correlation among input quantities and implementing the first or higher-order Taylor series approximation), to (analytically or numerically) calculate the sensitivity coefficients, to provide a summary of standard uncertainty components, and to calculate the effective degrees of freedom and the expanded uncertainty at a prescribed coverage probability. In this respect, the majority of the SW implement LPU based on the first-order Taylor approximation of the model (71 %) and provide sensitivity coefficients (55 %) and expanded uncertainties (54 %). The remaining capabilities are less frequently addressed. Concerning the JCGM 101:2008 and JCGM 102:2011 documents, the main features under investigation are the maximum numbers of Monte Carlo trials and of input quantities, Figure 3. Courses teaching MCM (JCGM 101). Figure 4. Courses teaching MCM (JCGM 102). Figure 5. Programming languages. ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 4 the gallery of available (univariate or multivariate) input probability density functions, the application of LPU to explicit or implicit (univariate or multivariate) measurement models, the ability to provide a coverage interval for the output quantity at a prescribed coverage probability (also considering probabilistically symmetric and shortest coverage intervals), to perform an adaptive Monte Carlo procedure, and to validate the GUM uncertainty framework against the Monte Carlo Method. As concerns the adherence to JCGM 101:2008, most of the analysed SW respect the document prescriptions on how to assign the input probability density functions (60 %) and calculate an estimate and the associated uncertainty from the simulated output distribution (60 %). The other features are less well covered. As concerns the adherence to JCGM 102:2011, it is evident a large gap exists in this respect: only 20 % of SW implement LPU for explicit multivariate models, and a meagre 6 % for implicit ones; 31 % of SW apply a Monte Carlo procedure to calculate an estimate and the associated covariance matrix from the simulated multivariate distribution of the measurand. In parallel with the above-described revision of the available SW for MU evaluation, it is worth mentioning that Mathmet is developing a Quality Management System (QMS) for software, data and guidelines as one of the main outputs of the Mathmet- related Joint Network Project. Relevant output is available in a dedicated publication [7] and uploaded on a Mathmet webpage dedicated to Quality Assurance Tools [8]. 3. OVERVIEW OF CLASSROOM EXAMPLES Concerning the “Classroom examples” offered in existing courses on MU, contact persons were asked to provide information about some of the main examples treated and their characteristics, comprising: - Title - Short description - Application area (calibration, testing, conformity assessment, etc.) - Metrology area (mass, length, etc.) - Approach to MU evaluation (JCGM 100, JCGM 101, etc.) - Level of difficulty (simple, medium, difficult) - Existing supporting material exists or to be developed So far, 69 examples have been collected from 36 training courses, of which 46 identify “calibration” as the main application area. The examples are spread over 15 different metrology areas (including “not specified”), with the top two listed as “dimensional” (18/69) and “temperature” (13/69) accounting for almost one-half of the examples. There is a focus on applying the LPU approach of JCGM 100:2008, either on its own (40/69) or in combination with other approaches (55/69) for comparison. Very few examples are classified as “difficult” (4/69) with most classified as “simple” (33/69), but in this regard the classification for the different levels of difficulty is likely to be quite subjective. It is planned to review other sources of examples used for teaching the principles of MU and for demonstrating different methods for MU evaluation. One such source is the compendium [9] of examples that was the main output of the EMUE project [10]. The compendium presents 41 examples from six broad application areas: - Industry and society - Quality of life - Energy - Environment - Conformity assessment - Calibration, measurement, and testing Figure 6, Figure 7 and Figure 8 present graphically a comparison of the examples taken from the two sources in terms of the categories of metrology area, approach (to MU evaluation) and level (of difficulty). The examples taken from the EMUE project are spread over 19 different metrology areas, with a more uniform distribution across them, and the top two listed are “chemistry” (8/41) and “flow metrology” (6/41). In terms of metrology area, the examples from the two sources appear complementary, perhaps reflecting in existing courses how examples from the “dimensional” and “temperature” areas are more accessible to a general audience and easier to teach, whereas the examples collected in the EMUE project reflect the wider interests of the partners involved in the project. There is again a focus on applying the LPU approach of JCGM 100:2008, either on its own or in combination with other approaches, but the examples from the EMUE project offer a wider range of approaches, including Bayesian, regression and “top-down” approaches to MU evaluation. Finally, a judgment about the level of difficulty of each example was made by one of the authors regarding how a “non-expert” faced with the example in a training course might perceive the example. The result was that all examples were classified as “medium” to “difficult” and so, in this regard, the examples from the sources again can be considered complementary. The data collected from these different sources serve as a basis for identification of interesting cases to be further developed in the form of classroom examples. In general, the analysis of the results, and the comparison for different sources, will support the identification of needs not covered by existing training courses, or deficiencies in those courses, and it will facilitate the exchange of knowledge between people teaching MU. 4. CONCLUSIONS A collaborative activity on “Measurement Uncertainty Training” under the auspices of the European Metrology Network for Mathematics and Statistics (Mathmet) is underway. This abstract reports on the progress to undertake surveys of existing training courses on MU, software for MU evaluation, and examples to support the understanding of methods for MU evaluation. It appears that an appreciable amount of material is available: 41 training courses, 69 examples, and 50 items of software. It is hoped that the knowledge assembled in this activity will help practitioners to make good choices about appropriate material to support their training needs, as well as help developers of training material to ensure good coverage of their training products and target them at user needs. Actual and future updated versions of the three surveys will be published on a dedicated webpage [11]. ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 5 Figure 6. Classification according to “metrology area” for the examples taken from existing training courses (top) and developed in the EMUE project (bottom). ACKNOWLEDGEMENT The authors of this abstract wish to thank the colleagues of all the partners of the activity consortium for participating in the survey of courses, software and classroom examples. The consortium comprises [3]: PTB (coordination), CEM, GUM, IMBIH, IMS SAS, INRIM, IPQ, LNE, METAS, NPL, SMD, ACCREDIA Ente Italiano di Accreditamento, Deutsche Akademie für Metrologie (DAM), National Standards Authority of Ireland (NSAI), Politecnico di Torino, University of Konstanz. Figure 7. As Figure 6 but using the classification of “approach (to MU evaluation)”. Figure 8. As Figure 6 but using the classification of “level (of difficulty)”. REFERENCES [1] European Metrology Network for Mathematics and Statistics home page. Online [Accessed 27 February 2023] https://www.euramet.org/european-metrology- networks/mathmet [2] EMN Mathmet starts initiative on measurement uncertainty training News story. Online [Accessed 27 February 2023] https://www.euramet.org/european-metrology- networks/mathmet/bugermenu/news [3] Mathmet Measurement Uncertainty Training Activity home page. Online [Accessed 27 February 2023] https://www.euramet.org/goto/g1-e34bee [4] K. Klauenberg, Introduction to the Mathmet activity MU Training, presented at the joint IMEKO TC1-TC7-TC13-TC18 & 0 5 10 15 20 Biology Chemistry Dimensional Electrical Force Frequency General Illuminance Mass Material properties Pressure Radiometry, Photometry Temperature Time Not Specified COURSE EXAMPLES Atmospheric measurement Chemistry Dimensional Electrical Flow metrology Gas metrology General Hardness Imaging Key Comparisons Mass Material combustion Precipitation Pressure Temperature Thermal comfort Thrust Torque Viscosity EMUE EXAMPLES https://www.euramet.org/european-metrology-networks/mathmet https://www.euramet.org/european-metrology-networks/mathmet https://www.euramet.org/european-metrology-networks/mathmet/bugermenu/news https://www.euramet.org/european-metrology-networks/mathmet/bugermenu/news https://www.euramet.org/goto/g1-e34bee ACTA IMEKO | www.imeko.org June 2023 | Volume 12 | Number 2 | 6 Mathmet Symposium, Porto, Portugal, 31 August-2 September 2022. [5] JCGM Publications: Guides in Metrology home page. Online [Accessed 27 February 2023] https://www.bipm.org/en/committees/jc/jcgm/publications [6] List of uncertainty propagation software webpage. Online [Accessed 27 February 2023] https://en.wikipedia.org/wiki/List_of_uncertainty_propagation _software [7] Keith Lines, Jean-Laurent Hippolyte, Indhu George and Peter Harris, A MATHMET Quality Management System for data, software, and guidelines, Acta IMEKO Vol. 11 No. 4 (2022). DOI: 10.21014/actaimeko.v11i4.1348 [8] Quality Assurance Tools webpage. Online [Accessed 27 February 2023] https://www.euramet.org/european-metrology- networks/mathmet/activities/quality-assurance-tools [9] Good practice in evaluating measurement uncertainty, Compendium of examples, Adriaan M.H. van der Veen and Maurice G. Cox (editors), 27 July 2021. Online [Accessed 27 February 2023] http://empir.npl.co.uk/emue/wp- content/uploads/sites/49/2021/07/Compendium_M36.pdf [10] Examples of Measurement Uncertainty Evaluation Project home page. Online [Accessed 27 February 2023] http://empir.npl.co.uk/emue/ [11] Mathmet Measurement Uncertainty Training Activity, For Trainees - Measurement Uncertainty Training webpage. Online [Accessed 27 February 2023] https://www.euramet.org/european-metrology- networks/mathmet/activities/measurement-uncertainty-training- activity/for-trainees-measurement-uncertainty-training https://www.bipm.org/en/committees/jc/jcgm/publications https://en.wikipedia.org/wiki/List_of_uncertainty_propagation_software https://en.wikipedia.org/wiki/List_of_uncertainty_propagation_software https://doi.org/10.21014/actaimeko.v11i4.1348 https://www.euramet.org/european-metrology-networks/mathmet/activities/quality-assurance-tools https://www.euramet.org/european-metrology-networks/mathmet/activities/quality-assurance-tools http://empir.npl.co.uk/emue/wp-content/uploads/sites/49/2021/07/Compendium_M36.pdf http://empir.npl.co.uk/emue/wp-content/uploads/sites/49/2021/07/Compendium_M36.pdf http://empir.npl.co.uk/emue/ https://www.euramet.org/european-metrology-networks/mathmet/activities/measurement-uncertainty-training-activity/for-trainees-measurement-uncertainty-training https://www.euramet.org/european-metrology-networks/mathmet/activities/measurement-uncertainty-training-activity/for-trainees-measurement-uncertainty-training https://www.euramet.org/european-metrology-networks/mathmet/activities/measurement-uncertainty-training-activity/for-trainees-measurement-uncertainty-training