Microsoft Word - 150105 35 Color Culture and Science Journal Vol. 15 (1) DOI: 10.23738/CCSJ.150105 Imaging colorimeters to evaluate Camera Monitor Systems image quality Cristian Bonanomi1and Kedar Sathaye2 1 Konica Minolta Sensing Europe B.V., Milan, Italy. Cristian.Bonanomi@seu.konicaminolta.eu 2 Konica Minolta Sensing Europe B.V., Nieuwegein, Netherlands. Kedar.Sathaye@seu.konicaminolta.eu Corresponding author: Cristian Bonanomi (Cristian.Bonanomi@seu.konicaminolta.eu) ABSTRACT Over the last few years, we have experienced a gradual increase in autonomous and driver assistance technology. Generally, we refer to these systems as ADAS (Advanced driver-assistance systems). A particular aspect of ADAS is Camera Monitor Systems (CMS), a system composed of a camera, a software that performs image processing operations, and a monitor for the driver. These systems help increase the overall safety aspect of the vehicle and increase the visibility of the drivers’ surroundings; therefore, the original equipment manufacturers (OEMs) must adhere to country specific regulations, necessary to test the robustness of the system. There are several test procedures for assessing CMSs, in this paper we will focus to the optical performance evaluation of the system. This includes lighting system, test patterns and an imaging colorimeter accompanied by a software which performs measurements according to the regulations mentioned in ISO16505:2019 (ISO, 2019). KEYWORDS Advanced Driver-Assistance Systems, Autonomous Driving, Camera Monitor Systems, 2D Colorimeters, Display Evaluation RECEIVED 09/03/22; REVISED 19/10/22; ACCEPTED 08/02/23 Imaging colorimeters to evaluate Camera Monitor Systems image quality 36 Color Culture and Science Journal Vol. 15 (1) DOI: 10.23738/CCSJ.150105 1. Introduction Over the last few years, we have experienced a gradual increase in autonomous and driver assistance technology and according to ABI research forecasts almost 8 million cars with autonomous or semi-autonomous level by year 2025 (ABI research, 2021). The Society for Automotive Engineers (SAE) defines six levels of driving automation when talking about vehicles (SAE, 2021): from no automation (level 0) to full driving automation (level 5). Beyond these levels, many efforts have been made to provide systems that facilitate enhanced driving situations. Generally, we refer to such systems as ADAS (Advanced driver-assistance systems) as electronic systems that assist users in driving and parking functions. A particular case of ADAS is the Camera Monitor Systems (CMS): a system composed of a camera, a software that performs image processing operations, and a monitor to illustrate the possible dangers as well as the blind spots around the car, mainly integrated in rear view mirror or side view mirror. Although the technical name used in the standards, is Camera Monitor Systems, the automotive market uses different names, and these systems can be referred to as virtual mirrors, digital mirrors, or electronic mirrors. Whatever name is used, these systems help increase the overall safety aspect of the vehicle and increase the visibility of the drivers’ surroundings; therefore, the original equipment manufacturers (OEMs) must adhere to country specific regulations, necessary to test the robustness of the system. In this paper we will discuss about the ISO 16505: 2019 “Road vehicles — Ergonomic and performance aspects of Camera Monitor Systems — Requirements and test procedures” (ISO, 2019), with particular attention to the optical performance evaluation of the system. To ensure image quality, several tests need to be performed to evaluate monitor characteristics e.g., directional uniformity, luminance, color rendering and sharpness, etc. The components (camera and display) can be measured separately, to easily discover where degradation has occurred, however sometimes it is necessary to test the complete system, a task that requires a high-resolution imaging colorimeter. In the following, we will focus on the materials and methods necessary to test the robustness of the system, which include: a lighting system to simulate different lighting conditions, i.e., direct sunlight or diffuse sky exposure, several test patterns to be used and imaging colorimeter accompanied by a software which performs measurements according to the standard regulations mentioned in ISO16505:2019 (ISO, 2019). The state-of- the-art imaging colorimeter and the dedicated software ensures that CMS under test is correctly validated. 2. Camera Monitor Systems A CMS is a possible technology to replace exterior mirrors, to display the side or rear view on a monitor inside a vehicle (see figure 1). Fig. 1. Example of a CMS system for a truck. Top row: CMS camera, bottom row: CMS display However, since exterior mirrors are fundamental for the safety, it is important to evaluate if a CMS can be a source of reduced safety or provide equal or more information to the driver. Generally, a CMS improves the aerodynamics of the vehicle decreasing wind resistance coefficient and noise. Furthermore, it reduces the blind zone area, improving the safety of driving. In 2015 an extensive work (Schmidt et al., 2015) was done to evaluate CMS as replacement for exterior mirrors in cars and trucks. The authors tested technical aspects as well as human-machine interaction scenarios. Although it has only been seven years since the report, many technical issues have been overcome, though some of the aspects underlined in the document are still of interest and Imaging colorimeters to evaluate Camera Monitor Systems image quality 37 Color Culture and Science Journal Vol. 15 (1) DOI: 10.23738/CCSJ.150105 concern. We recall some of them in the following, leaving to the reader the task of reading the complete report. Both external mirrors and CMSs have advantages and disadvantages. Some of these are related to technical aspects, such as optical quality; for example, resolution, color and contrast rendering, or time behavior properties that happen in critical situations. Additional aspects such as exposure adjustments when entering or exiting from a tunnel, or when a road surrounded by trees creates a succession of shadows and sunny areas are considered. These situations and aspects have been improved in the last years, thanks to technological advancements. Other aspects are related to intrinsic properties of the two systems: a mirror follows the reflection law, and movements of the head can add 3D information to the driver, while these movements do not affect the vision on a display. Furthermore, weather conditions can affect the two systems in different way: under the rain, the drops on the driver’s side window as well on the mirror itself can reduce the mirror visibility, while this condition seems to affect less the CMS, if the camera is in a well-covered position and since the display is inside the vehicle. Direct sunlight, snow or night driving are other non-standard conditions that must be considered. Finally, there are aspects related to the human-machine interaction: some experiments with human drivers have been carried out, resulting in a different perception of speed and distance when objects are viewed through a CMS. However, generally, people can adapt quite quickly to this new situation. All these aspects need consideration when using a CMS, therefore a procedure performing a range of tests on these systems has been developed in ISO16505:2019 (ISO, 2019). The standard includes several tests dealing with operating readiness, time behavior properties (evaluating frame rate, system latency), failure behavior, quality and ergonomic requirements, etc. In the following section we are going to focus only on image quality tests. 3. Testing Camera Monitor Systems Testing Camera Monitor Systems requires several items:  Test charts to evaluate different properties of the CMS. These charts can be found in specialized stores (i.e., (Imatest, 2021)).  Illumination of the charts, to simulate different lighting conditions.  The CMS camera installed outside of the car.  The computer that elaborates the data of the camera.  The CMS monitor used by the driver.  A digital camera or a 2D colorimeter to evaluate the result of CMS monitor.  A light source that illuminates the CMS display, to simulate i.e., direct sunlight. An image presenting the setup and the necessary elements for evaluating the image quality of the CMS is shown in figure 2 (the bottom row represents a schematic view of the same setup). However, other light sources can be added to this setup to evaluate other conditions, like the contrast rendering under different ambient illumination conditions (please refer to section 3.2). Fig. 2. Setup to evaluate a CMS system: 1) CMS camera, 2) lighting system for the target, 3) printed target, 4) CMS display, 5) 2D colorimeter for CMS evaluation. The evaluation of a CMS requires several tools and a dedicated facility. For this reason, such tests are usually conducted directly by OEM (Original Equipment Manufacturer) or by specialized independent third-party laboratories. These laboratories provide technical services offering verification, inspection, and certification of several products in accordance with international and national standards as well as audits for systems management certification (i.e., (TUV, 2021)). In the following, we are going to address the previous elements, with particular attention to the evaluation of the optical characteristics of the CMS system. These tests can be divided in five sets: 1) to verify the basic Imaging colorimeters to evaluate Camera Monitor Systems image quality 38 Color Culture and Science Journal Vol. 15 (1) DOI: 10.23738/CCSJ.150105 characteristics of the display like luminance, contrast and uniformity; 2) to evaluate potential issues related to the driving conditions: readability of the display, lens flare due to direct light, etc.; 3) to ensure the colors are correctly reproduced, i.e., for identifying the traffic lights; 4) to take in consideration or alert the driver about possible artefacts, and 5) to assess the resolution and sharpness of the system, in order to identify details. 3.1. Monitor Isotropy The Monitor isotropy test aims at evaluating the optical characteristics of the display according to different positions and viewing directions, using a uniform 70% gray scale image. Measurements of the directional uniformity are performed using a goniometer or a conoscope. A conoscope (figure 3, left) is a special lens that can be plugged to a 2D colorimeter to measure the angular distribution of luminance, contrast and color of a display. Lateral uniformity is measured on 9 positions on the display which coordinates are specified in the standard. 3.2. Luminance and contrast rendering The evaluation of luminance and contrast rendering is done on five different ambient illumination conditions that can affect the monitor readability. These conditions simulate: direct sunlight, diffuse skylight in day condition, night condition and sunset condition. Generally, a test chart composed by a white and black chessboard, is illuminated by two light sources, with a defined spectral power distribution, color temperature and illuminance value. The sunset condition is simulated using a direct light source reflected in a mirror towards the CMS camera, to evaluate artifact like blooming, smear and flare. 3.3. Color rendering This test is used to verify the CMS capability to reproduce eight specific colors (red, green, blue, yellow, cyan, magenta, black and white, placed on a 18% neutral gray background), in an accurate way. The test chart used for the color rendering should satisfy a range of conditions, including (see also figure 4c):  The color patches are placed on a circle, in order to keep the same distance from the center.  Opposite patches should have complementary colors.  The illumination of the chart should simulate CIE D65 and have a CCT of 6500 K +-1500K. To verify the accuracy of color rendering, a spectroradiometer or a colorimeter should be used to measure the chromaticity of the chart as well of the CMS monitor and convert them to the CIE 1976 uniform color space. The measurements done on the monitor are converted in chromatic hue angle to verify that they are in the correct range, while the measured data of the chart are not used for any calculation but to confirm the appropriateness of the used color chart and illumination. 3.4. Artefacts The possible artifacts and their drawbacks should be listed in the opertaor’s manual: smear, blooming and lens flare, can cause partial occlusion of the field of view that shall not cover more than a specifc percentage; point light sources, simulating low beam headlamps of another car, should be rendered as distinguishable lights; color noise and chromatic aberration should be avoided or minimized details. 3.5. Sharpness and depth of field Other important tests of CMS regard the sharpness and the related properties: resolution and depth of fields. Sharpness is measured evaluating the MTF50(1:1) (modulation transfer function) of a chart composed by five black squares slightly tilted. In order for a CMS to recognize object of interested behind the vehicle, also the depth of field needs to be measured. 3.6. Regulation No. 46 – Addendum 45 European regulation No. 46 (ECE R46, 2020) regulates the principles for the approval of motor vehicles for the installation of rear-view and side-view mirrors, in Europe. This standard is based on ISO 16505: 2019. The addendum 45 specifically, addresses considerations and procedures about the display-based systems as an alternative to conventional mirrors. This addendum adds a couple of tests for evaluate CMS:  Gray-scale rendering: this test aims at verifying that the CMS can display at least 8 tonal gray steps distinguishable from the darkest to the brighter.  Point light sources: this test is used to verify that the CMS can recognize and render as distinguishable two-point light sources (to simulate passing beam headlights). In particular, a set of two point light sources located at a distance of 250m from the CMS camera, having a luminous intensity of 1750 cd and separated each other by 1.3m, should be distinguishable. 4. Using a colorimeter for CMS evaluation Systems All the mentioned tests need a specific instrument able to inspect luminance, color and fine-details. A spot meter device (both a filter-based chroma meter or a spectro- radiometer) could be used, due to its capability of measuring the luminance and color in an accurate way. Imaging colorimeters to evaluate Camera Monitor Systems image quality 39 Color Culture and Science Journal Vol. 15 (1) DOI: 10.23738/CCSJ.150105 However, doing repeated measurements is time consuming, and spatial measurements, like sharpness or blooming test, cannot be performed. Fig. 3. Example of a 2D colorimeter, used for CMS evaluation. On the right, the colorimeter with a conoscopic lens mounted. On the other hand, a digital camera can provide high resolution images, but cannot measure luminance in accurate way or a traceable color rendering. To follow this standard, color accuracy is an extremely important factor, and digital cameras, which are typically used in consumer photography, are designed to please the viewers, enhancing color saturation, rather than reproduce color accurately. Furthermore, the color generation in digital cameras is achieved by a Bayer pattern, and the process of raw conversion can affect color. Spot meter Digital camera 2D colorimeter Directional uniformity D X Y (conoscope) Lateral uniformity D D Y Gray scale rendering D D Y Color rendering D X Y Point light source D Y Y Sharpness / D.o.F. X Y Y Table 1. Comparison between devices to perform specific tests. X: cannot be done, Y: can be done, D: can be done with difficulty. An imaging colorimeter, also known as 2D colorimeter, (figure 3) is the optimal solution to evaluate the image characteristic of the CMS, since it comprises the accuracy of a chroma meter and the flexibility of a digital camera. The color measurement is done through four filters that carefully simulate the CIE color matching functions. The fourth filter is used to simulate the small peak of the �̅� CMF in the blue side of the spectrum. These filters are placed on a rotating filter wheel, so that four different images are taken, to maximize resolution without spatial interpolation, as happen in typical digital cameras. This is a key point to consider, since for testing CMS, the 2D colorimeter needs much higher resolution than the camera and monitor that constitute the CMS. Another significant point requested by the standard is the necessity to evaluate the directional uniformity of the monitor. Some colorimeters allow the use of a special conoscopic lens (figure 3, right), which through Fourier optics can map an emitting spot so that each pixel of the sensor corresponds to a different emission angle. Radiant Vision Systems (RVS, 2021), provides hardware that fulfills the standard requirements and a comprehensive software suite to evaluate Camera Monitor Systems. Table 1 shows the ease to perform the required tests using different type of devices. In figure 4 three screenshots from the software are presented. Figure 4a) shows the interface, which allows the selection of an Analysis test (top image), and the parameters that can be set for the specific function (bottom image). In the example, the test “Contrast rendering” is selected, with the “Direct sunlight” as lighting condition, to reflect one of the standard requirements. Figure 4b) shows an image taken with the conoscope. This is a false color representation of the luminance in a specific point of the display, expressed in polar coordinates. For the display under test, the luminance decreases with the angular viewing. When observed perpendicularly (white area), the luminance is around 700 cd/m2. At the cursor point, with coordinates [Inclination 50°, Azimuth 135°], luminance is around 380 cd/m2 (light blue). For this specific display the image shows that the display is very dim when observed from below, a condition that of course does not occure in a car. Figure 4c) is an acquisition of the color rendering chart. The software helps the user to correctly register the patches, and after the execution of the analysis, returns a pass/fail result according to the standard requirements. Figure 4d) shows how two point light sources appear in the acquisition that simulates passing headlights. To perform this test, it is usually used a lighting device composed by two high intensity LEDs placed not too far from the CMS camera. Figure 4e) represents the typical checkboard pattern to evaluate the contrast of the display, under various daylight conditions. Figure 4f) shows the acquisition of the gray scale rendering pattern, to evaluate if the CMS is able to display at least eight distinguishable tonal gray levels. Imaging colorimeters to evaluate Camera Monitor Systems image quality 40 Color Culture and Science Journal Vol. 15 (1) DOI: 10.23738/CCSJ.150105 5. Conclusions In this paper we have seen how CMSs offer technological innovation yet create a set of new challenges for inspection that must be validated for the use in advanced levels of autonomous driving. Since replacing side and rear mirrors is a safety concern, a complete protocol to test these new technology platforms is necessary. Different regulations are in force in different countries (i.e in US: (FMVSS111, 2019) and in Canada (CVMSS111, 2017)). In Europe UN Regulation No. 46 (ECE R46, 2020) concerns the approval of devices for indirect vision and of motor vehicles with regards to the installation of these devices. This regulation incorporates test standard from ISO16505:2019 (ISO, 2019), which primary is to offer a guide to evaluate in an objective and critical way the quality and reliability of the CMSs. All the tests need instruments able to measure photometric and colorimetric properties as well as some spatial characteristics. In principle, different types of devices can serve the scope: spot meter devices, calibrated digital cameras or 2D colorimeters. However, only the latters can be considered as the optimal solution to evaluate the optical performance of the CMS, combining the accuracy of a chroma meter and the flexibility of a digital camera. Futhermore, the use of a special conoscopic lens can measure in a single step the luminance at different emission angle. All these aspects have been reviewed in the paper, with a special focus on the optical properties evaluation described in the European standard, and on the necessary tools that an OEM or a third part laboratory should use to carry on these tests. Fig. 4. Screenshots from CMS evaluation software: a) analysis selection, b) conoscope measurement for evaluating directional uniformity, c) patch registration for the color rendering test, d) simulation of two-points light source, e) checkboard pattern to evaluate contrast, e) gray-scale rendering pattern. Imaging colorimeters to evaluate Camera Monitor Systems image quality 41 Color Culture and Science Journal Vol. 15 (1) DOI: 10.23738/CCSJ.150105 6. Conflict of interest declaration The authors of this article are employees of Konica Minolta Sensing, part of Konica Minolta Inc., the company that acquired Radiant Vision System, whose colorimeter was described and tested in this article. 7. Statement on sources of funding The authors did not receive funding from any public, private, or non-profit funding agencies in support of this research. 8. Acknowledgment The authors would like to thank Weiyu Shao (Felix) and Yusaku Kawahara for providing some of the material used in this paper, and Matt Scholtz for the constant support and guidance. 9. Short biography of the authors Cristian Bonanomi – He received his degree and PhD in Computer Science, at Università degli Studi di Milano, where he worked as research fellow for six years, on colorimetry, human perception, digital imaging. He is currently employed in Konica Minolta Sensing, as Light and Display sales specialist. Kedar Sathaye - He received the PhD from Telecom Bretagne, where he focused on Optics and Photonics. He is now Product Manager of Light and Display in Konica Minolta Sensing. Previously he worked as display expert in Panasonic Automotive, and in Valeo as Optical Innovation Engineer. His fields of experience are in Optical Engineering, Spectroscopy, Lighting, Liquid Crystals, and Product Development. Licensing terms Articles published in the “Cultura e Scienza del Colore -Color Culture and Science" journal are open access articles, distributed under the terms and conditions of the Creative Commons Attribution License (CC BY). You are free to share (copy and redistribute the material in any medium or format) and adapt (remix, transform, and build upon the material for any purpose, even commercially, under the following terms: you must give appropriate credit to authors, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use, you may not apply legal terms or technological measures that legally restrict othersfrom doing anything the license permits. Copyright: The authors keep the rights to further publish their contents where they want and can archive pre-print and post-print (submitted version and accepted version) and the published version of the PDF of their article with no embargo period. References ABI research (2021) ABI Research Forecasts 8 Million Vehicles to Ship with SAE Level 3, 4 and 5 Autonomous Technology in 2025, Available at: https://www.abiresearch.com/press/abi-research-forecasts-8-million- vehicles-ship-sae-level-3-4-and-5-autonomous-technology-2025/ (Accessed: 20 December 2021). CMVSS111 (2017) Canada Motor Vehicle Safety Standard – CMVSS 111 – Mirrors and Rear Visibility Systems. ECE R46 (2020) Addendum 45, Uniform provisions concerning the approval of devices for indirect vision and of motor vehicles with regards to the installation of these devices. FMVSS111 (2019) Federla Motor Vehicle Safety Standard No. 111, Rear Visibility. Imatest (2021) Digital Image Quality Testing, “Camera Monitor Systems” (Accessed: 7 June 2021). https://www.imatest.com/solutions/camera- monitor-systems/ ISO (2019) ISO 16505:2019. Road vehicles — Ergonomic and performance aspects of Camera Monitor Systems — Requirements and test procedures (Accessed: 7 June 2021). https://www.iso.org/standard/72000.html RVS (2021) https://www.radiantvisionsystems.com/ (Accessed: 20 December 2021). SAE (2021), SAE Levels of Driving Automation™ Refined for Clarity and International Audience Available at: https://www.sae.org/blog/sae-j3016- update (Accessed: 20 December 2021). Schmidt, E. A., H. Hoffmann, R. Krautscheid, M. Bierbach, A. Frey, J. Gail and C. Lotz-Keens (2015) ‘Final report: camera-monitor systems as a replacement for exterior mirrors in cars and trucks’. Federal Highway Research Institute, 91p. TUV (2021) Camera Monitor Systems for the Automotive Industry https://www.tuv.com/japan/en/camera-monitor-systems-for-the- automotive-industry.html (Accessed: 7 June 2021).