Annual report 2021/2022 From idea to implementation: We make AI fly www.iosb.fraunhofer.de

We make AI fly Artificial Intelligence – there is probably no other field of technology that is currently associated with so many hopes for innovations, new business models, growth opportunities and disruption. For Germany alone, its economic potential for the coming years is estimated at several hundred billion euros. We are experi- encing huge changes to our living and working environments associated with AI, with more to come in the future. But the challenges of turning all of these visions into reality are also enormous. Artificial intelligence now produces eloquent-sounding texts, fakes deceptively real-looking videos and can even beat world champions at complex board games. Some practical digital tools that are now part of everyday life, such as speech recognition on smartphones, were only possible in the first place thanks to AI. However, progress is more arduous in many sectors. Where AI is just one compo- nent in a complex system, where something is at stake, where systems interact with the physical world in ways that could break things or even endanger people, the successful application of AI still faces major challenges, some of which are very fundamental. For us at Fraunhofer IOSB, researching AI applications is a part of our DNA, not only since the hype of the last few years, and we certainly do not limit ourselves to the commonly referenced subfield of deep neural learning. Whether it is log- ical reasoning, solution seeking, environmental perception, understanding and modeling, motion planning or making the best decisions in the face of uncer- tainty – our focus is diversified. We are also familiar with important application domains, as our research and development – which is not just limited to the field of AI – is focused on practical issues from sectors such as industrial production, mobile systems, human-machine interaction, testing and sorting tasks, public safety or utility infrastructure management. Making AI fly in these domains too, through appropriate methods and tools – that is our strength. This is our mission. It is a common thread across the many activities and focuses of our institute. It is therefore also a theme that runs through this Annual Report and turns up in many chapters. 1 According to market research studies published by PwC and eco/Arthur D. Little in 2018 and 2020, the potential effects on the German GDP amount to 430 and 480 billion euros, respectively (see https://www.pwc.de/de/digitale-transformation/business-analytics/kuenstliche-intelligenz-sorgt-fuer-wachstumsschub. html and http://go.eco.de/ki-studie2019).

and knowledge at your service. Therefore, if you work for an authority, company or other organization whose challenges we could help solve with our competences – please do not hesitate to contact us! Sincerely, Prof. Dr.-Ing. habil. Jürgen Beyerer Head of Fraunhofer IOSB semi-autonomous survey system for rivers and lakes, the utilization of light field technology and AutoInspect, a system for comprehensive, automated, multisensory inspection that also integrates manual inspection by human operators and combines all the results into a single digital quality twin. We are equally proud of our leading role in a large number of publicly funded major research and development projects such as the Competence Center for AI Systems Engineering CC-KING, the Competence Center Robotic Systems for Decon- tamination in Hostile Environments ROBDEKON, the AI Living Lab for Industrial Production, the European defense research project OCEAN2020 and the Bauhaus.MobilityLab. This report will also go into more detail on these and other projects. Of course, all these pre-competitive and publicly funded research activities ultimately have one goal – otherwise we would not be Fraunhofer: They are intended to help transform scientific knowledge into practical applications, in projects for and with our customers, and to pave the way for techno- logical innovations that generate tangible added value and competitive advantages. This is our mission at Fraunhofer, and with this goal in mind, we would also like to put our expertise

Highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 1. Making AI fly . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2. Other initiatives and infrastructures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 3. Locations and buildings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4. When research meets people – our events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 The institute in profile: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Expertise in applications and market-oriented solutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 How to cooperate with us . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Pooling expertise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Fraunhofer Strategic Research Field Artificial Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Fraunhofer Segment for Defense and Security VVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Business units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Artificial Intelligence and Autonomous Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Automation and Digitalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Defense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Energy, Environmental and Security Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Inspection and Optronic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Departments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Laser Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 Optronics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Signatorics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Visual Inspection Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Cognitive Energy Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Digital Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 Embedded Intelligent Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Information Management and Production Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 Interoperability and Assistance Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Machine Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Systems for Measurement, Control and Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Underwater Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Interactive Analysis and Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Object Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Scene Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Video Exploitation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Variable Image Acquisition and Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Facts and figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Board of trustees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Internal working groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Scientific consultants of Fraunhofer IOSB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Editorial notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

Highlights » The aim of AI Systems Engineering is to develop AI and ML for use in engineering, in a similar way to clas- sical engineering,« says Prof. Jürgen Beyerer, Head of institute 15 QuickChecks performed at CC-KING Websites www.ki-engineering.eu www.iosb.fraunhofer.de/ml4p 7

Highlights Development and demonstration centers for the production of the future: Karlsruher Forschungsfabrik (left) and SmartFactoryOWL. Federal Ministry of Economic Affairs and Climate Action. Led by Fraunhofer IOSB-INA in Lemgo, it aims to create an open data platform for the application of AI in Industrie 4.0, which relies on open standards such as the Asset Administration Shell (AAS). The AI Living Laboratory brings together companies, AI experts and researchers in a safe environment and creates added value for all parties. Some parties can bring in real- world use cases and challenges for which they need solu- tions. Others get the data sets they need to (further) develop ML-based solutions. Based on case studies from operational practice, the success story of AI in production can be fleshed out, retold and communicated to new target groups. As such, the AI Living Laboratory is not just a virtual network. Rather, it is closely interlinked with the Industrie 4.0 develop- ment and demonstration center SmartFactoryOWL and enables tangible collaborations in the real world. At the end of 2021, for example, a production line for sustainable coffee cups moved into the SmartFactoryOWL and is now becoming a test case for AI-optimized production of the future (see p. 75). ROBDEKON Competence Center Here too, the potentials of AI are systematically leveraged – except in this case we are talking about embodied AI, namely “robotic systems for decontamination in hazardous environ- ments”: In ROBDEKON, autonomous and semi-autonomous robot systems are being researched specifically for use in the dismantling of nuclear facilities, in the decontamination of plant components and in the remediation of landfills and contami- nated sites. This is so that, in the future, people can stay away from the danger zone while this work is being carried out. Fraunhofer IOSB contributes its expertise in sensor technol- ogy, environmental sensing, navigation over unstructured terrain and autonomous manipulation, for example. These are incorporated into the autonomy kits developed in Ilmenau and Karlsruhe, which can be used to convert near-series construc- tion machines into (partially) autonomous robots. For example, during the positive regular evaluation in 2021, there was a live demonstration of how a driverless 24-ton excavator autono- mously removes layers of soil and loads them onto the trailer of a tractor that also operates autonomously. Preparations are now underway for a continuation of the competence center’s initial four-year term. Launched in 2018, ROBDEKON was initially funded with 12 million euros by the German Federal Ministry of Education and Research as part of the “Research for civil security” program. The coordinator is Fraunhofer IOSB, where the ROBDEKON coordination office is also located. The Karlsruhe Institute of Technology (KIT), the German Research Center for Artificial Intelligence (DFKI) Bremen, the FZI Research Center for Information Technol- ogy and a number of industry and application partners are involved. Websites www.karlsruher-forschungsfabrik.de www.ki-reallabor.de www.robdekon.de 9

Highlights UVC disinfection From physics-based simulation to prototype imple- mentation and characterization: 3 questions for Thomas Westerhoff on a booming theme Mr. Westerhoff, the coronavirus pandemic has hit home the importance of hygiene. How does UVC radiation make an impact in this area? Radiation with UVC light, which is made up of particularly short waves, has been proven to be an effective means of inactivating germs. However, safe disinfection means ensuring that people are reliably kept away from the source of radiation. Further- more, the radiation dose must be sufficient for every surface area, or everywhere in the liquid or gas be disinfected. We have made it our mission to meet these requirements with cost-efficient solutions. In doing so, we are focusing on UVC LEDs as particularly flexible light sources. How can Fraunhofer IOSB contribute to this purpose? By doing what we do best – by combining technolog- ical and scientific expertise from disciplines ranging from physics and engineering to informatics to drive practical innovations. In specific terms, we simulate the propagation of the radiation and, if required, the flow conditions to calculate the incoming radiation dose. By combining this result with the spectrum of UV efficacy of individual microorganisms, we are able to very precisely derive the microbiological effect of the radiation. We are designing suitable light geometries and arrangements as well as developing LED control electronics and cooling concepts. Thanks to funding from the German state of Thuringia, we are also currently building a reference laboratory to thoroughly evaluate all possible UV devices from a technical perspective. A lot has been reported on the UVC disinfection of ambulances when they are not transporting patients (p. 68). Can you name other application examples? Whether you want to disinfect the cell phones of medical personnel, incubators for premature babies, water directly in the faucet, air in air purifiers or even in refrigeration units designed to keep food fresh on long journeys: The list of potential applications is long. As part of the Fraunhofer MobDi – Mobile disinfection project we have developed a module for mobile robots that can disinfect door handles or light switches fully automatically. By combining our tech- nological skills with the expertise of microbiologists and companies providing the specific applications in question, we are able to produce promising solutions. 11

New site: Rostock New site: Berlin Highlights Highlights New site: Oberkochen » With the new Oberkochen branch lab, we as Fraunhofer IOSB will cover the entire value chain for laser components in the >1.6 µm range,« says Prof. Dr. rer. nat. Marc Eichhorn Director Fraunhofer IOSB Ettlingen +3 new entries on the location map other things, a tunnel and a crystal growing laboratory are planned here, as well as an indoor laser test track more than 250 meters long, which connects the new and the old build- ings and can be used by a number of new laser laboratories. The building contractor is the Federal Republic of Germany, which owns the entire Ettlingen property. Karlsruhe Research Campus The plans for a new building on the Karlsruhe Institute grounds, essentially in the area of the current gateway and parking lot, have also taken on a more concrete form. The Fraunhofer Institute for Systems and Innovation Research ISI will move into the resulting “Karlsruhe Research Campus”, too. The ISI has clearly outgrown its current building in the Wald- stadt district of Karlsruhe. The second institute building on the current IOSB site will provide an additional 7500 square meters of usable space: 3900 square meters for Fraunhofer ISI, with the remainder being split into new office and experimental spaces for Fraunhofer IOSB and common areas. Regarding research topics, there is a common vision linked to the forthcoming immediate neighborhood: to responsibly shape the development of AI and autonomous systems. The opportunity to combine our technological with ISI’s social science expertise is a perfect fit for a topic with such a high impact potential. This will enable us to take a holistic view that integrates aspects such as the regulation and control of AI, but also participation and acceptance. The new building has now been approved and the next planning steps in collaboration with Fraunhofer ISI have begun. 13

Highlights While digital events such as Fraunhofer's “Solution Days” (left) played an important role in 2020 and 2021, more in-person events have become possible lately. One prominent example was the ceremonial opening of Karlsruher Forschungsfabrik. The photo shows how Prof. Beyerer (right) gave State Premier Winfried Kretschmann a tour of the IOSB hall. 4. When research meets people – our events Digital trade fairs and events The event celebrating our anniversary was to be the last large organized event for some time. Internal and external events, whether meetings, conferences or trade shows, were either canceled or, as was more often the case, moved to a digital format and reinvented as needed. The Fraunhofer-Gesellschaft set up its own digital trade show “Solution Days” within a few months, at which Fraunhofer IOSB was represented with a number of solutions as digital exhibits as well as presentations. Research projects were started and finished in online conferences, and the functioning of developed technologies was captured on video rather than demonstrated live. Hackathons got more attention than ever. In 2020, employees of Fraunhofer IOSB-INA participated in the #WirVSVirus major development competition organized by the German federal government, before founding the AICommunityOWL with other volunteers from partner organizations shortly after. This network of AI enthusiasts has already hosted two successful online hackathons around SmartFactoryOWL and the AI Living Lab, where participants were asked to solve challenges based on real industry data. The learning curves for all involved were steep. In many cases, the virtual formats worked surprisingly well – but in some cases, especially for making new contacts, they unfortunately proved not to be a wholly adequate substitute for the usual event formats. Return to in-person events This made the few in-person events that had once more become possible by the time this report went to press all the more valuable. Of particular note is the IAA Mobility, which was held as an in-person event during the breather from the pandemic in late summer 2021. There, Fraunhofer IOSB presented the OCTANE simulation platform and its VERTEX test vehicles for experiments related to automated driving functions. The Karlsruhe Research Factory was inaugurated with a cere- monial opening at the end of March 2022. Around 500 guests were curious to see the research projects and demonstrators on display in the three halls, including the State Premier, the Ministers of the Economy and Research of Baden-Württem- berg, as well as numerous representatives of manufacturing companies. The following day, the first AI Systems Engineering Day for production took place in a somewhat smaller group in the IOSB hall of the Research Factory. 15

The institute in profile Core competences and structure Optronics System Technologies Image Exploitation Understanding and control of the generation of light, its beam forming, propagation and conversion into electronic signals Laser Technology Optronics Signatorics Visual Inspection Systems The competence triangle Ability to analyze, understand, model, develop and control complex systems Preparation and real-time processing of images and videos as well as automatic and interactive information extraction Cognitive Energy Systems Digital Infrastructure Embedded Intelligent Systems Information Management and Production Control Interoperability and Assistance Systems Machine Intelligence Systems for Measurement, Control and Diagnosis Underwater Robotics System Technologies Interactive Analysis and Diagnosis Object Recognition Scene Analysis Video Exploitation Systems Variable Image Acquisition and Processing research group The competence triangle positions the departments of Fraunhofer IOSB in relation to the three core competences. Optronics Image Exploitation 17

The institute in profile Selected strategic collaborations, expert networks and platforms we are participating in CC-KING Competence Center KI-Engineering Lemgo Digital                    Open District Hub Für die Zukunft der Energiewende 19

The institute in profile Key figures The institute in numbers / Budget and Staff Business expenses Funding How much money did we spend? Where did the money come from (in 2021)? * Industrial sector 32% Basic funding 21% Others 4,5% Public sector 42,5% 70 60 50 40 30 20 10 '17 '18 '19 '20 '21 Investment Research expenditure Staff How many persons worked at Fraunhofer IOSB? ** '21 '20 '19 '18 '17 Scientists and engineers Other regular employees Research assistants, student assistants and interns 100 200 300 400 500 600 700 800 * Preliminary figures as of March 2022 ** Headcount reporting date: 31 December 2021 21

The institute in profile Ilmenau /Görlitz Lemgo Advanced System Technology branch Fraunhofer IOSB-AST Industrial Automation branch Fraunhofer IOSB-INA Prof. Dr.-Ing. Peter Bretschneider Prof. Dr.-Ing. habil. Thomas Rauschenbach Directors Ilmenau Prof. Dr.-Ing. Jürgen Jasperneite Director Lemgo EIS | Embedded Intelligent Systems DIS | Digital Infrastructure Administration Prof. Dr.-Ing. Andreas Wenzel Dr.-Ing. Sebastian Schriegel KES | Cognitive Energy Systems MIT | Machine Intelligence Prof. Dr.-Ing. Peter Bretschneider Dr. rer. nat. Oliver Niehörster UWR | Underwater Robotics Prof. Dr.-Ing. habil. Thomas Rauschenbach Rostock Berlin SOT | Smart Ocean Technologies (research group) Prof. Dr.-Ing. habil. Thomas Rauschenbach SIRIOS | Fraunhofer Center for the Security of Socio-Technical Systems Dr.-Ing. Markus Müller Beijing Representative office China Dipl.-Ing. Hong Mu Head of administration / Commercial and Technical Management Dipl.-Betriebsw. (FH) Nicole Keller-Rau IT security Dipl.-Inform. Birger Krägelin Staff department Head of Staff depart- ment and Strategy and Innovation Management Dr. rer. nat. Frank Lorenz Press and Communi- cations Dipl.-Phys. Ulrich Pontes 23

The institute in profile Fraunhofer structure contribute to achieving the: Impact Goals Affordable healthcare Energiewende accomplished Digitalized value chain Fully circular economy Security and resilient society and on: Transfer: Fraunhofer Alliances aimed at addressing lead markets Construction industry Mobility sector Aerospace economy Healthcare sector Digital economy Chemical industry Plant, mechanical and vehicle engineering Energy sector Agriculture and food industry focusing on: Pre-competitive research: Fraunhofer Strategic Research Fields Hydrogen Technologies Quantum Technologies Bioeconomy Next Generation Computing Digital Healthcare Artificial Intelligence Resource Efficiency and Climate Technologies and as: Fraunhofer Cluster of Excellence Research Fabs High-Performance Centers Fraunhofer Project Centers organized as: Fraunhofer Groups Innovation Research ICT Group Light & Surfaces Microelectronics Production Health Resource Technologies and Bioeconomy Energy Technologies and Climate Protection Materials and Components Fraunhofer Institutes and research units Units of the Fraunhofer Segment VVS 25

The institute in profile Fraunhofer Strategic Research Field Artificial Intelligence In the factory, office and everyday life, artificial intelligence systems are taking on ever more routine tasks. Technologies like machine learning are trans- forming entire industries and are playing a key role in the future transformation of our society and econo- my. Independent and comprehensive AI expertise is important so as to avoid becoming dependent on play- ers with possibly lower requirements in terms of data protection or security, for example, and to guarantee Germany’s and Europe’s competitiveness in these stra- tegically and economically crucial technologies. At present, the Strategic Research field AI is focus- ing on seven future-oriented topics: AI Systems Engineering – Systematic engineering of AI – Specification, planning, dimensioning of AI – Energy optimized AI Embodied AI – Robots and autonomous systems – Edge and IoT systems – Hardware/software co-design Hybrid AI – Combining data-driven learning with knowledge and AI technology you can trust models The Fraunhofer Strategic Research Field Artificial Intelligence (FSF AI) explores key technologies for AI-based services and products. With our comprehensive methodical and applica- tion competence, we are developing trustworthy, secure and sustainable AI technologies to help our customers solve their practical problems. Fundamental questions, such as the explainability of AI meth- ods, the systematic engineering of AI-based systems or the integration of expert knowledge into machine learning meth- ods, are focal points of our developments. The FSF AI aims to take AI technologies from concept to application; our solutions will be applied in many different fields – from autonomous driv- ing to intelligent production plants and medical engineering. – Small/relevant data – Development/optimization of algorithms AI and the Human – Speech and dialog systems – Intelligent agents, collaborative systems Certified AI – Explainability, traceability – AI safeguarding (safety/security), attack and defense – Standardization – Human control of AI AI and Hardware – Hardware/software co-design, neuromorphic computing – Quantum AI and corresponding algorithms Data and sustainability underpin each of the focus topics. The sustainable use of computing resources and data over their entire life cycle (data value chain) plays an equally important role as data quality and data sovereignty. Fraunhofer Strategic Research Field Artificial Intelligence Spokesmen Prof. Dr.-Ing. habil. Jürgen Beyerer, Fraunhofer IOSB Prof. Dr. Stefan Wrobel, Fraunhofer IAIS Deputy spokesman Prof. Dr. habil. Alexander Martin, Fraunhofer IIS 27

The institute in profile ASSTA 3.1-Tornado. Fraunhofer Institute for High Frequency Physics and Radar Techniques FHR Radar – A Key Technology Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE Command, Control and Reconnaissance Fraunhofer Institute for Applied Solid State Physics IAF Sensors for Safety, Security and Reconnaissance Fraunhofer Institute for Chemical Technology ICT Security, Safety and Energetic Materials Technology Fraunhofer Institute for Technological Trend Analysis INT Planning Support for State and Industry Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB From Networked Sensor Data to Decision Fraunhofer Institute for Experimental Software Engineering IESE Software and Systems Engineering Fraunhofer Institute for Integrated Circuits IIS Communication, Positioning Technologies and X-Ray for Safety and Security Applications Fraunhofer Institute for Structural Durability and Systems Reliability LBF Secure Processes for secure Structures Fraunhofer Segment for Defense and Security VVS Chairman Prof. Dr.-Ing. habil. Jürgen Beyerer, Fraunhofer IOSB Deputy chairman Prof. Dr. rer. nat Peter Martini, Fraunhofer FKIE vvs.fraunhofer.de Deputy chairman Univ.-Prof. Dr.-Ing. Dr. rer. pol. habil. Michael Lauster Fraunhofer IN Managing director Dipl.-Ing. Caroline Schweitzer, Fraunhofer IOSB caroline.schweitzer@iosb.fraunhofer.de 29

Business units Artificial Intelligence and Autonomous Systems ..............................32 Automation and Digitalization ........................................................36 Defense ...........................................................................................40 Energy, Environmental and Security Systems ...................................46 Inspection and Optronic Systems .....................................................50 31

Business units » Our XAI tools shed light on the black box« Explainable Artificial Intelligence (XAI): important tool for AI systems engineering and the predictable, systematized use of AI in demanding applications Mr. Frey, why is the concept of explain- ability such a big issue when it comes to AI methods? Although AI algorithms such as deep learn- ing methods often deliver impressively good results, it is usually hard to understand how and why these algorithms produce certain results. XAI aims to fill this gap. XAI stands for a group of methods that in a sense shed light on the black box and are designed to make it easier to interpret the decisions of AI models. This is not only important for the issue of acceptance – in other words, that humans trust and accept the decisions made by an AI. It is also an important tool for the develop- ment phase and the subsequent life cycle of AI components in the context of AI systems engineering, for example to track down the causes of errors. Do you have any concrete examples? We use a variety of methods, depending on whether explanations are required only for a specific case or in general, how much is known about the underlying AI model, etc. Semantic XAI, for example, is based on knowledge models and can generate textual reasoning along the lines of: “This object is very probably a car because I have detected two wheels, headlights, exterior mirrors and a license plate in the image.” Or, in the case of AI-assisted route planning for autonomous systems, XAI methods visualize the deci- sions that lead a robot system on the best path through unknown terrain in a way that humans can clearly understand. What specific range of services does the IOSB offer in this area? In the context of an internal research and development program, we have focused on building up targeted XAI expertise for more than two years. This expertise is embedded in our comprehensive and applied AI knowledge in sectors ranging from industrial production, automotive and medicine to energy. One of the results is an XAI toolbox that is available immediately for analyzing data, debugging and explaining the predictions of any black- box model. This means that we can support virtually any type of AI application project and contribute significantly to developing not just “pretty” proof-of-principle demonstrators, but robust, practical and accepted productive solutions. Dipl.-Ing. Christian Frey Business unit spokesperson Head of MRD department Phone +49 721 6091-332 christian.frey@ iosb.fraunhofer.de 33

Artificial Intelligence and Autonomous Systems The project is creating a cloud platform for various AI applica- tions in urban environments. Mock-up system with same kinematics as the project's battery- powered electric target systems captures data on airport apron. Bauhaus.MobilityLab in Erfurt − the living lab of the future Energy management and automation of battery- powered special-purpose vehicles The “Bauhaus.MobilityLab” project is using Erfurt’s Brühl district as a living lab to develop and test a range of AI-based applications: Traffic lights will be controlled according to traffic volume, deliveries will be made in a more customer-oriented manner, local energy generation will reduce electricity costs and intelligent tariff systems will determine the charge price for electric cars. The applications will be developed and provided on a cloud platform that collates data from different areas, such as transport, logistics and energy. Under the consortium leadership of Fraunhofer IOSB-AST, the “BML Ecosys” platform forms the basis for the overarching development and testing of AI-based mobility, logistics and energy services in a real-world urban environment. The lab platform will be run by an operating company with the involvement of all project partners. The concept of the Bau- haus.MobilityLab will act as a blueprint in terms of a “lab as a service” for the development of future urban living labs and even for other fields of application. German and international cities have already expressed their interest in the project. At the core of the ALEC project funded by the Federal Ministry for Economic Affairs and Climate Action is the development of battery-powered electric vehicles for the airport apron. The main focus is on equipment carriers with tool changing systems. One aim is to develop an energy management system for the airport and a charging management system for the vehicles that makes use of their flexibility combined with local photovoltaics to save CO². This is the basis for a concept for the future supply of energy to the airport. Another focus is automatic driving, which increases the reli- ability and efficiency of vehicle use. To this purpose, a specially adapted sensor concept is being developed to identify the enviroment of the vehicles on the airport apron, ensuring stable localization and the reliable detection of obstacles. The prototypes for energy and charging management have been successfully implemented and tested in simulations using realistic input data, allowing CO²-optimized battery charging to be demonstrated and future potential for development to be identified. The future energy concept for the airport has also been created. As well as the sensor and navigation concept for autonomous driving that has been developed, a specially built mock-up system has been used in the project to record sensor data at the airport and develop simulations based on it. These simulations allow to adjust the transport processes on the air- port apron to evaluate the localization, path planning and path adjustment algorithms developed for autonomous driving in different situations and with different vehicle kinematics. Dipl.-Wirtsch.-Inf. Oliver Warweg, group manager energy informatics, Phone +49 3677 461-111 www.bauhausmobilitylab.de Prof. Dr.-Ing. Andreas Wenzel, head of department of EIS, Phone +49 3677 461-144 www.iosb.fraunhofer.de/EIS 35

Business units » The AIoT trend is heralding new possibilities« Artificial Intelligence of Things: How combining networked sensors and actuators with artificial intelligence is raising new research questions and creating new busi- ness modelssystematized use of AI in demanding applications Dr. Usländer, what does Artificial Intelli- gence of Things – AIoT for short – refer to? Thomas Usländer: AIoT is all about merging Internet of Things technologies with methods and tools used in artificial intelligence. This interplay creates huge added value in every IoT domain, especially automation technol- ogy and networked industrial production. That’s because the “things” in the IoT, such as machine tools, can use sensors to supply exactly the data that machine learning (ML) methods need and that can be quick- ly evaluated and efficiently converted into useful information by these very methods. This opens up the scope for new applications and business models – and it raises exciting research questions. Could you give us some examples? A fascinating area of research is federated learning, in which an ML model is trained on multiple devices. Instead of exchanging all data, parts of the ML model are generated locally and then combined to form one large model – an approach that has particular advantages for data protection. An exciting concrete innovation topic is the application of ML methods in digital twins of production facilities and their assets, whereby the digital twins are continually synchronized with the states and behavior of the real “things”. How can companies exploit this potential? Many companies have already invested in digitalizing production processes and the accompanying IoT technologies: for example, to monitor their facilities remotely. The logical next step is to evaluate data using AI solutions that can be implemented in the edge or in the cloud. However, this does mean bringing together a range of areas and disciplines – specifically, production engineering, informat- ics/IT and data science/AI. As we know from many projects in practice, this is by no means trivial: The use of AI methods is hard to plan and subject to its own laws, which are often alien to engineers and can be difficult to rec- oncile with the operational culture of estab- lished production facilities. However, we have the expertise and the AI systems engineering methods and tools that are needed to provide companies with the best possible support during this transformation journey. Dr.-Ing. Thomas Usländer Business unit spokesperson Head of ILT department Phone +49 721 6091-480 thomas.uslaender@ iosb.fraunhofer.de 37

Automation and Digitalization Built from real-world components, our demonstrators enable cyberattacks and countermeasures to be tested realistically. CC-KING is funded by the Baden-Württemberg Ministry of Eco- nomic Affairs, Labor and Tourism. Cybersecurity training labs We operate two cybersecurity training labs, both of which have been among Fraunhofer Academy‘s “outstanding learn- ing sites” since February 2021. Our cybersecurity training lab for energy and water supply in Ilmenau and Görlitz sensitizes participants to the increased vulnerability of digitized energy and water infrastructures to cyberattacks. The threats are addressed in the 2021 white paper, “Current Cybersecurity Challenges in Energy Supply.” Realistic replication of power and water systems is guaranteed by real telecontrol, protection and control system components as well as real-time simulations. Our cybersecurity training lab for industrial manufacturing, based in Karlsruhe and Lemgo, combines learning modules with a tailored education environment. It features experimen- tal hard- and software systems that are built from real-world automation components. Like the cybersecurity training lab for energy and water supply, it lets users see the consequenc- es of cyberattacks and apply and test countermeasures. Since October 2021, we have been managing a “Cybersecurity Fundamentals” training program for VDMA (mechanical engineering industry association) members. The program is a cooperation between the International Society of Automa- tion (ISA) and Maschinenbau Institut GmbH (MBI), a service company of VDMA. Making AI manageable for engineers CC-KING, the Competence Center Karlsruhe for AI Systems Engineering, develops methods for engineers to apply artificial intelligence (AI) and machine learning (ML) in practice in a pre- dictable and secure way. In AI systems engineering, CC-KING is combining cutting-edge AI research with established engineer- ing disciplines to investigate methods and tools that directly benefit the economy, and small and medium-sized enterprises in particular. As a leading partner in the consortium, we are contributing our extensive IT expertise in industrial automation and control systems as well as in the fields of AI and ML. Our partners in the consortium are FZI Research Center for Information Technology and Karlsruhe Institute of Technology (KIT). Practical relevance is ensured through transfer measures such as training in the CC-KING training lab, low-threshold advisory services (quick/transfer checks) and close collaboration with businesses in the CC-KING innovation committee. In order to support businesses with the successful execution of AI projects in demanding areas such as mobility or industrial production, the first systematic Process Model for AI Systems Engineering was published as a white paper in late 2021. PAISE® describes and guides through the entire process from the design of AI-based technical systems to data acquisition, operation and maintenance, and supporting the development of AI-based sub-systems in an agile manner. Dr.-Ing. Christian Haas, group manager industrial cybersecurity, Phone +49 721 6091 605, www.iosb.fraunhofer.de/llcs-p, www.iosb-ast.fraunhofer.de/llcs-ew Dr. Thomas Usländer, CC-KING project manager, head of department ILT, Phone +49 721 6091-480 www.ki-engineering.eu 39

Business units » We are making deep learning more robust« The specific challenges of data-driven processes: How the right tools can help to supply sufficient training data and avoid misclassifications Dr. Arens, AI processes, for instance those used to classify images, are very efficient. In spite of this, they are only hesitantly making their way into practical applica- tions. Why is this? Michael Arens: Despite their renowned perfor- mance, deep learning processes also con- tinuously attract attention when they make serious mistakes: AI processes are often over- whelmed by specifically selected input data. This is why a key issue for us is robustness: How can we make sure that AI will deliver the desired outcome for any situation and that it will not, for example, think that a picture of a butterfly is actually a plane because of a cer- tain cloud texture in the background? Some- thing like this could happen as a result of a paradigm change borne by machine learning: Rather than on algorithms, results depend on training data. When the AI system is learning, millions of parameters are set based on the training data, resulting in a type of black box that will determine – in complex and barely comprehensible ways – which outputs are caused by specific inputs at a later date. So, can the key to robustness be found in the training data? Exactly. The training data must describe the problem comprehensively. Ideally, they cover the whole range of what might be input at a later date. There are, however, other aspects which play a defining part in our work. Such as: Can we find weak spots and use them to outwit the AI, for example to avoid detection? We are covering this in a project on camou- flage and camouflage assessment (see next page). Or, another important aspect, what will happen when the input data systematically shift because the sensor gets old, for example – will the process still be robust? To address this kind of question and systematically make the process more resilient, we are also investi- gating various tools that allow for a look into the black box. What approaches could ensure that there is sufficient training data? In our field of business, a general lack of train- ing data poses a challenge. This data doesn’t accumulate on a daily basis to the same degree that it does for most other fields of applica- tion. This is especially the case for multispectral data, which plays an important role in defense research. Data produced by simulations may help. So may artificial variations of real data. A third approach is to transform existing data. In such a way, infrared data can be generated from conventional images acquired in the vis- ible spectrum, though an appropriate AI pro- cess is required for this. We are exploring and developing tools for all of these approaches. Dr. rer. nat. Michael Arens Business unit spokesperson Head of OBJ department Phone +49 7243 992-147 michael.arens@ iosb.fraunhofer.de 41

Defense 4 New laboratory for SWIR high-energy lasers. 1 2 3 1–3: Temporal evolution of laser interaction with a steel plate. 4: Impact patterns after laser interaction with a target for different interaction times. Dr. Christelle Kieleck, head of department LAS, Phone +49 7243 662-310 www.iosb.fraunhofer.de/LAS 43 Coherently coupled high-energy SWIR fiber lasers Power scaling of fiber lasers in the short-wave infrared (SWIR) range by coherent coupling is a promising way to realize high-energy lasers (HEL) at new wavelengths. Taking into account the spectral dependency of molecular transmission, specific wavelengths need to be selected and novel fiber laser architectures need to be found to amplify them efficiently. The strategy we pursue can be outlined as follows: Based on numerical simulations of atmospheric propagation, suitable wavelengths are defined. In combination with our own laser dynamics simulations, different ultra-narrow-linewidth fiber lasers and amplifiers are realized for assessing the atmospheric effects of high-power laser propagation, and specific fiber lasers and their components are developed to provide kW-class single-frequency output power. Using our own algorithms and technologies, power scaling by coherent coupling will be demonstrated and investigated. The aim is to conceive SWIR HEL laser systems that combine high beam quality with laser power far beyond the kW range. Along that route, we have identified different wavelengths. Our research has proven power scalability of laser architectures and provided new insights into some limits. One example concerns narrow linewidths. In this case, stimulated Brillouin scattering becomes a major concern. Based on these results and our ongoing research, we will be able to develop demonstrators and specify the necessary archi- tecture for future laser systems with the desired characteristics.

Defense 3D-GV recording of a dynamic scene with high depth resolution. Quantum Ghost Imaging of an object with only one photon per pixel on average. Active laser sensorics for reconnaissance Military reconnaissance under adverse atmospheric conditions demands imaging techniques capable of returning high-reso- lution images through scattering and turbulent media. Active imaging is the method of choice for critical applications where high precision and reliability is required. Complete control over the illuminating light source enables extraction of a high level of information about the characteristics of distant targets, especially detailed range 3D information, as shown in the picture on the left. In addition to that, pulsed active imaging systems can gate the time of detection which suppresses noise and enhances visibility through fog, smoke and fire. operational use, we are investigating novel approaches to active imaging such as Compressive Sensing and Quantum Imaging. With Compressive Sensing it is possible to detect distant objects using fewer detections compared to established active imaging systems by having control over the spatial distribu- tion of the illumination pattern for each frame. This method requires only a single element detector for detection as the spatial information of the illuminated object can be recon- structed from the reflexion of these patterns. Gated viewing (GV) offers the perfect blend of 3D detection of non-cooperative targets in adverse conditions and detec- tion speed. It utilizes a pulsed laser that is synchronized with a camera to illuminate distant objects of interest. A suitable timed gate only selects photons that are reflected by the object and suppresses photons belonging to its fore- and background. Thereby, you obtain a much higher target/back- ground contrast than for a non-gated image. Furthermore, backscattered photons from particles in the atmosphere between sensor and object are not captured, thus providing advantages in conditions with poor visibility. In a project commissioned by the German government, we are working on enhancing the performance of systems in use and developing and benchmarking modern systems for active imaging. The systems typically operate in the short-wave infrared, ret- ina-safe wavelength range with good properties for atmo- spheric propagation. The pulse energies range from several tens of microjoule in scanning mode up to tens of millijoule in flash mode. As non-detectability as well as extended cov- erage of spectral regions is of outstanding importance for Quantum Imaging has the potential to obtain images with a minimal number of single photons (picture on the right) and those systems can stay below detectability for exter- nal observers. The broadband quantum light source emits photons randomly in space and time and can thereby ideally mimic background noise. Only the sender has the key to separate real background noise from real signal via additional information from entangled photon partners. Furthermore, quantum systems are inherently hardened against hostile jamming attacks. Dr. Dominik Walter, project manager and POC Quantum Technologies, Phone +49 7243 992-159 www.iosb.fraunhofer.de/active-sensorics 45

Business units » Climate and energy price crises offer an opportunity« Cross-sectoral coupling and cognitive systems are important building blocks for mastering the recent challenges of energy supply Prof. Bretschneider, as a result of the war in Ukraine, the energy transition is to be accelerated: The German government wants to generate 80 percent of elec- tricity from renewables by 2030. Is this a realistic goal? Peter Bretschneider: Given the current share of around 50 percent, that is more than ambi- tious. The problem may not be with the tech- nologies themselves, but also their supply avail- ability in the face of rapidly growing demand, potential supply chain problems, cost increases, increasing shortage of skilled workers and so on. In addition to this, there are competing goals: The so-called energy policy triangle includes not only developing renewables, but also affordability and security of supply. On the other hand, the current crisis offers a unique opportunity for disruptive change that could significantly reduce our dependence on energy imports in the medium to long term. How- ever, we must not consider electricity alone: Cross-sectoral energy supply – the coupling of the electricity, gas, heating/cooling and mobili- ty sectors – is becoming increasingly important. Significant synergies and cost savings are possi- ble, especially in buildings and neighborhoods, but this requires local energy storage and sophisticated energy management systems. The entire energy system is therefore changing. This is not new: Liberalization and the climate crisis are long-standing issues, but the current crisis is stepping up the pace and, among other things, it puts a new focus on energy efficien- cy, which can be tackled in the short term. What contribution can cognitive energy systems make? The change described above involves increasing complexity and requirements. This can only be mastered by means of digitalization. In order to operate robustly and as efficiently as possible, energy management systems including AI tech- nology are necessary – whether for load and network loss forecasts, energy use optimization including redispatch or intelligent charging. What are services that Fraunhofer IOSB can offer in this context? Technology companies, energy suppliers and platform operators can rely on our compre- hensive competence – from modeling, data preparation and automation of processes to ongoing optimization during operation, not forgetting cybersecurity, too. With our energy management suite EMS-EDM PROPHET®, we have been successful in the market for more than 20 years. New additions are the aforementioned AI solutions and cross-sector applications. With Bauhaus.MobilityLab, the growth hub SMOOD® and as part of the Open District Hub, we are involved in key flagship projects in Germany. Our customers benefit from this experience. Univ.-Prof. Dr.-Ing. Peter Bretschneider Business unit spokesperson Head of KES department Phone +49 3677 461-102 peter.bretschneider@ iosb-ast.fraunhofer.de 47

Energy, Environmental and Security Systems Example of an incident of interest to the police. Intelligent video surveillance in public places For many years now, Fraunhofer IOSB has worked on intelli- gent video surveillance technology in the field of preventative policing and investigative technology. The institute develops the latest methods of intelligent video surveillance and tests them in real applications. One such example of this is our ongoing pilot project in Mannheim, which runs until 2023. In this project, we are working with the State of Baden-Würt- temberg and Mannheim Police Headquarters (Polizeipräsidium Mannheim) on an intelligent video surveillance system. The aim is to develop an assistance system that alerts police officers − those who are monitoring the video images in the command and situation center − to situations that require their attention so that the officers do not have to constantly watch images from multiples cameras simultaneously. Instead, they only need to look at specific situations flagged by the system and evaluate these individually. The benefit of this is twofold: Firstly, it enables officers to intervene at an earlier stage. Secondly, the technology has the potential to improve data protection and the general public’s privacy. Once the system has proven itself in practice and relevant incidents can be reli- ably detected, the video images will be visible in full resolution during normal operation only after a situation of interest to the police is detected by the system. While such a solution offers huge potential, it also poses a significant developmental challenge on account of its complexity. While barely perceptible to the naked eye, hyperspectral infor- mation can clearly identify the ripeness level of avocados. Incoming quality control for food Affordable sensors in conjunction with AI-supported analysis are opening up a wide range of new application possibilities for evaluating food. Digitally recording as many characteristics as possible and their combinations using commercial data from goods logistics plays a key role in such applications. Our portfolio now provides the option of creating digital twins (e.g., from a crate or pallet of the same products) and includes evaluations of spectroscopic features in the near infrared (NIR) range. This spectral footprint provides additional information, such as the products’ internal composition. Past data form the basis for these evaluations and the new measurements can be used to continuously improve this database as required. Our technology thus enables food to be appropriately classified and evaluated − be it identifying the apple variety at the checkout, determining ripeness during goods receipt or shortly before delivery, or checking com- pliance with standards or the authenticity of a food’s prov- enance. The results are integrated into the establishment’s internal IT system via digital interfaces and are made available for future transactions. Optical detection of the quality of incoming goods using camera/sensor technology enables comparisons between individual products and product deliveries; this can be used to evaluate the possible influence of natural (e.g., seasonal) factors or different product origins. In addition, data on the relevant product/characteristic history can be quickly retrieved and easily referenced. Dr.-Ing. Jürgen Metzler, group manager video-based safety and assistance systems, Phone +49 721 6091-453 www.iosb.fraunhofer.de/intelligente-videoueberwachung Dipl.-Ing., Dipl.-Wirt.-Ing. Henning Schulte, business unit developer, Phone +49 721 6091 275 www.iosb.fraunhofer.de/IO 49

Business units » Our technology helps to close material cycles« Plastics recycling: how the IOSB’s expertise in bulk material sorting and digitaliza- tion is contributing to increasing sustainability and resource efficiency Prof. Längle, what is the link between the technologies found in the inspection and optronic systems business unit and climate protection? Thomas Längle: Our skills in bulk material sorting have already been used in recycling for many years. It doesn’t matter whether the material is used glass, construction waste or plastic – separating the different substances as effectively as possible is a key prerequisite when it comes to producing high-quality recy- clates. With regard to plastics in particular, we need to do much better in terms of recycling if we are going to achieve the general target of climate neutrality by 2050. According to current estimates, around 50 million tons of CO² are produced as part of the plastics value chain in Germany alone each year – the majority of which is down to the fact that a good half of all used plastic is simply sent for thermal recycling, i.e., burnt. Not only would other recycling methods be more sustain- able, they are also becoming more and more attractive from an economic point of view as a result of CO² pricing. That’s why Fraunhofer is championing the vision of a circular plastics economy. Exactly. The aim is to significantly increase the proportion of materials that are recycled. Where that isn’t possible, it’s important that plastic waste is at least turned into a second- ary source of raw materials through chemical recycling, replacing the need for fossil raw materials. This would keep the carbon within the economic cycle instead of releasing it into the atmosphere as CO². The Fraunhofer lighthouse project Waste4Future is develop- ing a comprehensive approach here and our expertise is playing a key role in several ways. How exactly? Firstly, we are developing a digital twin that will map the processes and material flows comprehensively and interoperably to enable the plastics cycle to be evaluated in a transpar- ent and digital manner. This will allow optimi- zation across the board and the best recycling approach to be defined for each different part of the material flow – taking technical, economic and ecological factors into account. We are also making further developments to sensors and sorting technology. Com- monly used sensors have weaknesses – for example, when distinguishing different black plastic materials or if parts have an internal structure. One area on which the project is focusing is therefore multi-sensor data fusion, which when combined with machine learning methods will enable materials to be identified reliably and in real time. Prof. Dr.-Ing. Thomas Längle Business unit spokesperson Head of SPR department Phone +49 721 6091-212 thomas.laengle@ iosb.fraunhofer.de 51

Inspection and Optronic Systems Deflectometry makes it possible to detect even minute dents or scratches in reflective surfaces. This drone and hyperspectral camera succeeded in detecting oil reliably in the test scenario. AutoInspect: Combining optical inspection results into one digital representation Hyperspectral pipeline monitoring from the air Digitally fused measurement data is playing an increasingly important role in automated test procedures and downstream process control. Information about various quantities describ- ing surfaces must be brought together at the correct location. The combination forms a weighted quality measure which allows for further analysis, e. g. with respect to user-defined quality metrics or with methods based on machine learning. In a self-financed project, Fraunhofer IOSB has developed a large-scale, modular demonstrator for the inspection of many classes of objects. It showcases a wide variety of inspection methods for surfaces, simplifies the workflow configuration and combines the results into what might be called a digital quality twin. Results from sensors such as laser scanners, ellipsometry, and deflectometry are registered on a texture map of the object under inspection. For maximum flexibili- ty, the objects can be moved around on automated guided vehicles (AGVs) in the inspection space past stationary sensors, as well as be inspected in detail by mobile sensors on a robotic arm, and/or by humans assisted by newly developed AR user interfaces. The demonstrator, with example sensor systems acquiring 3D shape, dents and coating irregularities, is set up at the Karls- ruhe Research Factory (see p. 8), where other processes can easily be incorporated, such as surface inspection by camera drones. This paves the way for a Fraunhofer inspection center for complex products, combining the rich expertise of several IOSB research groups into one unique solution. To prevent environmental damage from leakages, oil pipelines are generally subject to costly testing procedures that normally require an interruption of regular operations. If aerial inspec- tions are carried out at all, they are visual checks only. In order to identify even the smallest oil spills from the air early on, we are working with industry partners to develop an aircraft platform fitted with sensors that can deliver not only high-res- olution visual aerial images, but also hyperspectral images, and can analyze them automatically. Our main focus in this project is to develop a hyperspectral oil detection method. This involves identifying the ways in which the material-specific reflective behavior of crude oil differs from that of uncontaminated backgrounds (e.g. grass or gravel). The difficulty lies in detecting crude oil with a high degree of reliability while also avoiding false alarms triggered by other crude oil-based products such as plastics to the great- est possible degree. In order to achieve this, a test scenario involving oil sumps, var- ious ground substrates and crude oil types/products has been constructed to be as realistic as possible and the latest sensor and drone technology (Headwall Hyperspec Co-Aligned VNIR/ SWIR combined with a DJI Matrice 600 Pro drone) has been flown over it. The results obtained so far indicate extensive detection of crude oil with only a small number of false alarms. These promising findings have resulted in further development work in an attempt to roll this out for additional scenarios. Dipl.-Ing., Dipl.-Wirt.-Ing. Henning Schulte, business unit developer, Phone + 49 721 6091-275 www.iosb.fraunhofer.de/autoinspect Simon Schreiner M.Sc., project manager, Phone +49 7243 992-319 www.iosb.fraunhofer.de/hypod 53

Departments Optronics Laser Technology | LAS ....................................................................56 Optronics | OPT ................................................................................58 Signatorics | SIG ...............................................................................60 Visual Inspection Systems | SPR ........................................................62 System Technologies Cognitive Energy Systems | KES ........................................................64 Digital Infrastructure | DIS ................................................................66 Embedded Intelligent Systems | EIS ..................................................68 Information Management and Production Control | ILT ....................70 Interoperability and Assistance Systems | IAS ...................................72 Machine Intelligence | MIT ............................................................... 74 Systems for Measurement, Control and Diagnosis | MRD .................76 Underwater Robotics | UWR ............................................................78 Image Exploitation Interactive Analysis and Diagnosis | IAD ...........................................80 Object Recognition | OBJ .................................................................82 Scene Analysis | SZA ........................................................................84 Video Exploitation Systems | VID ......................................................86 Variable Image Acquisition and Processing | VBV .............................88 55

Departments View of the new high-power laser lab. High-power active fiber with end cap. 57 signed in 2021 a specific collaboration agree- ment with HENSOLDT, a high-tech pioneer in defense and security electronics, to develop future-proof lasers for laser-based countermea- sures for self-protection systems and reconnais- sance purposes. According to Countermeasures Product Manager Harald Knapp, HENSOLDT “made a very conscious decision to go with the Fraunhofer Institute of Optronics, System Tech- nologies and Image Exploitation IOSB in Ettlin- gen and their Laser Technology Department, as their expertise in the laser field and their ambitious laser roadmap are the best fit for our product portfolio. The institute has an excellent reputation in our industry.” The Programme Area Manager Self-Protection Land Vehicles, Dr. Oliver Rudow, added: “With laser technol- ogy, we are advancing into a new wavelength range and higher power classes, which will enable us to develop various applications and novel products in the land, sea and air sectors. Our German customer is showing great interest in these product developments. A concrete selection decision has already been made for a first application, and we are in promising dialogue for two others.” laser splicers dedicated to the realization of specific fiber-optical components, crystal- growth equipment, and sophisticated X-ray crystal analysis devices. In collaboration with our central workshop, high-precision opto- mechanical components are realized to allow for utmost precision and robustness needed from research to fieldable laser systems. Highlight topics New high-power solid-state laser lab and office building We have inaugurated a new lab specifically dedicated to high-average-power and high- pulse-energy solid-state lasers. Together with this new space providing a down-to-class 1000 clean-room environment, new advanced equipment like multiple pump lasers, optical spectrum analyzers, and high-speed metrology has been installed. The lab serves the investi- gation of SWIR solid-state lasers and related research on novel laser architectures for rugged miniaturized and efficient sources. It also allows investigation and performance test- ing of optical components (like LIDT, thermal effects etc.). In parallel, a new office building has been officially opened to serve the staff increase in laser research while providing additional lab space for fiber investigations and propagation experiments across the southern experimental field at the Ettlingen site. Cooperation agreement on laser development with HENSOLDT Based on a long-term relationship we have

Departments The AR/VR device test station consists of a three-dimensional gimbal bearing an imaging system that is positioned in a way that mimics the human eye. It enables us to investigate AR/ VR imaging performance for operational use. counter-measures (OCM). Our equipment includes a fast, high-precision IR-Scene projector with 1024x1024 pixels for calibrated camera and algorithms evaluation, test stations for performance characterization of thermal imagers and night vision devices and a laser cross-section measuring track over 500 m, 2 km and 2.6 km. Highlight topics Laser Safety Assessments for High- Energy Lasers Each application of a laser source requires a detailed laser safety assessment. This holds especially true for the application of high-en- ergy lasers in the field. Even if the beam path is secured, unintended reflections from a hit object must be considered, as they can be dan- gerous for uninvolved persons further away. For this reason, we perform reflection measurements on various metallic surfaces and investigate the temporal development of the reflection patterns, generated by the interaction with high-energy infrared lasers [1]. When the surface starts to melt due to the absorbed irradiation, complex reflection structures, so-called caustics, can form, which show rapidly changing intensity patterns with local hotspots. For a better understanding of these reflection patterns, and to improve laser safety assessments, we created a simulation tool to calculate these caustics at various distances from an object hit by a high-energy laser. Additionally, we use neural networks to improve the input parameters of our simu- lations using experimental data. The goal of these efforts is to generate a better under- standing of hazards from laser reflections serving as a basis for risk assessments regard- ing unwanted eye damage before applying a high-energy laser in the field. AR/VR devices: assessment of imaging quality and impact on laser safety studies Near-eye displays are a technology steadily gaining significance in industrial and defense applications, e.g., for virtual and augmented reality (AR) as well as digital night vision. Con- sidering the growing usage of such displays and their ongoing technological development, we designed a specialized measurement test- bed to evaluate the imaging quality of these types of displays [2]. Our testbed enables automated measurement procedures analyzing different properties of near-eye displays, e.g., different methods of measuring such a device’s modulation transfer function. Results of these measurements can support device choices for specialized applica- tions as well as deliver input for the advance- ment of the simulation of the optoelectronic imaging chain with TRM4 and OSIS. Apart from the assessment of imaging quality, we also investigate potential new applications of such devices. For example, AR devices allow for a new way to simulate laser dazzling with- out having to use real laser dazzlers [3]. This is especially useful to investigate the deteriora- tion of human performance, e.g., in terms of the ability to drive a vehicle or task perfor- mance, like shooting precision. Additionally, it offers a lot of potential for training on how to react to being dazzled. Augmented and virtual environments are an innovative field with an important future that we want to help shape. 1 2 3 4 Time-dependent reflection characteristics of a near-infra- red laser beam after hitting a metallic target. Initially, there is a geometrically confined reflection in a defined direc- tion (1). Then, ring structures can emerge due to physical stress on the target (2) and when the target starts to melt, pronounced caustics form in the reflection pattern (3-4). 1 M. Henrichsen et al, “Laser safety assessments supported by analyses of reflections from metallic targets irradiated by HEL light,” Appl. Opt. 60, F71-F87 (2021) 2 C. Günther, M. Henrichsen, and S. Kessler, “Prototype measurement setup to assess near-eye display imaging quality,” Proc. SPIE 11866, 118660J (2021) 3 M. Henrichsen et al, “Simulating laser dazzling using augmented and virtual reality,” Proc. SPIE 11867, 118670F (2021) 59

Departments method for underwater communications, with a low data transmission rate. A higher bandwidth can be achieved with visible light, as other wavelengths of electromagnetic radi- ation get severely absorbed by water. How- ever, the underwater environment presents challenges for the propagation of laser beams. In clean ocean water, extinction due to absorp- tion and scattering limits the useful range of underwater communications. Additionally, underwater optical turbulence stemming from minute differences in water’s refractive index and salinity perturbs the laser beam. The new Underwater Communications Labo- ratory has been set up at the Ettlingen site to study and compensate for this effect. Exper- iments have been carried out in a 1.5 meter long water tank. For the first time, real-time correction of underwater turbulence through the use of adaptive optics has been demon- strated in scenarios emulating propagation over approx. 100 meters in clear ocean water. By using adaptive optics, one can expect to decrease the number of errors in transmission by several orders of magnitude. behavior under arbitrary illumination sources and configurations. The measurement of this high-dimensional function requires special equipment, so-called gonioreflectometers, which are used to determine the BRDF accord- ing to the traditional pointwise sampling method. However, this is a difficult and very time-consuming procedure. We implemented a new measurement method in a robot-based gonioreflectometer. It is based on imaging capture of the sample reflectance. This method is over 300,000 times faster than the sequential sampling method. Nevertheless, to take full advantage of this, a few drawbacks must be compensated. The directional information must be mapped using a mirror reference measurement. This allows for the same angular representation as with the sampling method. Possible cross-reflec- tions within the hemispherical sector must also be quantified and compensated. These two points are research topics that are currently being investigated and explored. Underwater optical wireless communications The oceans are Earth‘s final frontier. Their exploration will require not only R&D activities in robotics, material science, and navigation, but also in communications. Underwater exploration will generate a vast amount of data that should be reliably transferred between unmanned underwater vehicles, be-tween submarines and between either of the former and their ground stations. For many years, sound has been used as the primary Above: Image of the hemi- spherical sector placed around a sample. Below: Image of material reflected radiance onto the dome. Our 1.5-meter long 500 liter tank with an adaptive optics experimental setup as well as heating and cooling device for controlled turbulence generation. 61

Departments Therefore, a hand-held spectral sensor is used to measure the ripeness of berries based on their reflectance spectra. During harvest, it is desirable to only pick ripe and healthy grapes, which could be automated by AI-based imag- ing systems and has already been successfully demonstrated. Spectral sensors embedded into the tank of the harvester allow for in-field quality selection. Secondly, we investigate imaging applications for yield prediction. As soon as flower buds appear in the vineyard, they could be auto- matically counted using image processing to compute an initial yield estimate. This estimate could regularly be updated until ripe berries are considered for a final prediction, result- ing in increased planning reliability for the winemaker. Thirdly, just like other organisms, grapevine is constantly threatened by diseases. In order to facilitate early detection and prompt counter- measures, we work on evaluating the spectral fingerprints of infected and healthy plants. For this, a multispectral camera specifically adapted to the disease symptoms is to be configured. While these projects focus on grapevine, the methods and algorithms could be transferred to other crops in the future and thus provide a valuable contribution to the automation of in-field phenotyping. Waste4Future: high-quality plastic recyclates from sensor-based waste sorting Plastics recycling – one of the greatest challenges of our time – is attracting huge amounts of attention in the media and soci- ety as a whole. Political regulations such as CO² pricing are providing incentives for a truly circular economy. Our sensor-based sorting technology lays the foundation for producing high-quality recyclates in the mechanical and chemical recycling process. This is because the composition of the input material has a significant impact on the final product. There are a whole range of variables to consider, for example the proportion of different polymers, material changes to the plastic due to aging, any flame retardants contained therein and even the color of the plastic. Hyperspectral camera for recording the health status of plants. Based on data captured by sensors and using fast switching pneumatic valves, our facilities therefore sort plastic waste by material and, where necessary, color. The Fraunhofer light- house project Waste4Future, which involves six other institutes, is currently focusing on a multi-sensor system to enable robust mate- rial characterization. The aim is to develop a system that can also characterize black plastics (colored using soot), which cannot be detected using NIR hyperspectral imaging in the short-wave infrared range. The potential of the terahertz sensor technology that will be used has already been demonstrated in a pre- decessor project. Machine learning methods for sensor data fusion should also allow for test objects to be sorted in real time accord- ing to their age. The end result is high-pu- rity input material, which in turn produces high-quality recyclates. Plastic waste is mostly a disordered material flow. The aim is to turn it into partial streams for a suitable processing route. 63

Departments field of signal analysis and assistance systems, enabling forward-looking technology transfer that strengthens the regional economy. As a part of the High-Performance Center, we are participating in the pilot project “Design and implementation of a cross-institutional, real-time-capable simulation platform for cross-sectoral energy systems.” Germany’s ambitious climate protection targets make the cross-sectoral consideration of energy flows more important than ever. In light of this, the pilot project will collect energy data and, by means of innovative sensor systems, environ- mental data for selected buildings at Ilmenau University of Technology and use them in the real-time simulation platform. Due to increasing electrification (e-mobility, heat pumps) in the residential sector, automat- ic detection of faults in the power grid (anom- aly detection) is becoming more and more important − another field we are researching at InSignA. The real-time simulation platform, which is currently under construction, also serves as a basis for joint research projects, as a showcase demonstrator and as a platform for implementing customer solutions. ZO.RRO: Zero Carbon Cross Energy System Germany is off to a good start on its journey to renewable power generation. As the nation’s exit from fossil fuels enters the second phase, the focus is shifting toward reducing energy consumption and carbon emissions. This is where the ZO.RRO (Zero Carbon Cross Energy System) research proj- ect comes in. The project develops approach- es and solutions for a CO²-free energy supply system using the example of the Free State of Thuringia by taking a cross-sectoral view of electricity, heat and gas supply and also considering the provision of the necessary system services. Studies have shown that these system services, which are necessary to ensure system stability, are responsible for 20 percent of carbon emissions. To bring the energy sector and industry together in this project, an interdisciplinary consortium has been formed comprising the departments of Electrical Energy Supply and Energy Use Opti- mization at Ilmenau University of Technology, Fraunhofer IOSB-AST, Nordhausen University The “InSignA” High-Performance Center combines the strengths of Fraunhofer, the Ilmenau University of Technology and the regional SMEs and industry in the field of intelligent signal analysis and assistance systems. of Applied Sciences, the Engineering Bureau for the Energy Sector (IfE), the KoCoS Tech- nology Group, the Thuringian Renewable Energies Network (ThEEN) and the municipal utilities partnership Trianel. The models that we have developed as part of this project consider an innovative scenario characterized by renewable energies, storage and hydrogen technologies, as well as a con- servative scenario that includes power-to-gas and gas power plants. An optimization model shows the technologies that will need to be installed in Thuringia in the future in order to keep greenhouse gas emissions to a minimum. Our newly developed CO² monitoring tool allows companies to monitor their CO² foot- print in detail according to their current energy mix and visualize CO² emission forecasts. The necessary data is provided by the ZO.RRO box developed as part of the project. A special piece of software makes it possible to make use of existing flexibilities to shift gas, heat and power loads to minimize carbon footprints and offer zero-carbon system services. The potential users of these hardware and software solutions include municipal utilities, accommodation services and ministries as well as businesses. In phase two of the project, which is scheduled to start in 2022, the solu- tions will be developed further and tested by participating businesses. ZO.RRO Digital User-Interface. Website https://s.fhg.de/insigna https://www.zorro- thueringen.de/en 65

Departments driving on conventional railroad tracks. These innovative vehicles require gyro stabilization and appropriate control technology, which in turn relies on real-time information from the environment and other vehicles. 5G promis- es to adequately meet the high latency and reliability requirements and still leave capacity for infotainment applications and for user interaction. The project will also investigate how supplementing public 5G networks with private 5G cells can meet the requirements without gaps. 3D lift system for automatic and reproducible testing of 5G and localization technology. Test of Ethernet TSN proto- types on a machine in the SmartFactoryOWL. test centers for digitalization in the fields of automation, products / devices, cities, mobility and production. For the first time, the research partners are now linking up across locations. In addition to remote maintenance of production machines, other scenarios could include automation from the cloud or the use of the latest technologies such as millimeter waves in industrial applications. 5G in future mobility: 5G SIMONE and MonoCAB 5G offers many possibilities for quickly ana- lyzing large amounts of data and optimizing processes in various fields of application. One of several new projects funded in the con- text of North Rhine-Westphalia’s 5G.NRW initiative, with a budget of up to 2.6 million euros, is “5G SIMONE”. Here we collaborate with the local technical university, TH OWL, and industry partners to research the potential of 5G with respect to “closed-loop control” applications in the context of public transport. 5G SIMONE is closely associated with another project: MonoCAB aims at constructing and testing self-propelled monorail shuttles 67

Departments The living lab also features comprehensive measurement and sensor technology. Our customers have at their disposal: 3D-LiDAR sensors, reference measurement technology such as DGPS and gyro systems for vehicle dynamics measurements, optical laser trackers and high-performance communication sys- tems. Other mobile working machines such as field tractors, forklifts, driverless transport systems and municipal vehicles are also be available at the living lab for the purpose of testing and validating new types of mobile working machines in combination with estab- lished, partly automated systems. Living laboratory for mobile working machines: Off-road section with all-terrain forklift and drone. The system on display at the “proVention” COVID-19 trade fair in Erfurt, Germany. downtime between the two possible uses. The disinfection process can also be per- formed on the way to the next emergency. Living lab for automated working machines The use of automation solutions is grow- ing also in the field of mobile working and utility machines. In addition to conventional assistance systems, there is a growing demand from users for additional support functions that require an increased degree of autonomy from vehicles: If a car can drive itself, why can’t a mobile working machine? As systems become more complex and new types of sensors emerge in the market, estab- lished simulation methods are quickly pushed to their limits. For this reason, we are setting up a living lab at the Ilmenau site in order to carry out tests on vehicles in typical practical applications. Roadways and movable elements have been set up near the existing integration hall on a fenced area to simulate a wide range of scenarios and validate algorithms for new autonomous vehicles. 69

Departments Historical buildings, collection, and historical parks and gardens can also be affected by extreme weather conditions. Assessing the magnitude of this risk is an interdisciplinary task that requires the collaboration of histori- ans, meteorologists, preservationists, restor- ers, and engineers. However, this discussion is currently rarely taking place in Germany, either on a scientific or societal level. The BMBF-funded KERES project aims at changing this and researches, with a focus on Germa- ny, which risks arise from extreme weather events that are increasing both in frequency and extent, and which adaptation strategies should be developed. Charlottenhof, castle and park – one of the KERES research areas. In close cooperation with the relevant players and potential users, such as the Stiftung Preussische Schlösser und Gärten (Prussian Palaces and Gardens Foundation), Fraunhofer IOSB is building a knowledge platform that shows the current and future condition of a property (buildings and parks). The aim is to create as high a level of user orientation as possible so that the knowledge platform can be used sustainably over the long term. The platform is supported by an app, which will be developed during the project and can be used in both emergency management and preven- tion. The necessary data for the knowledge platform is being contributed by the Climate Service Center Germany (HZG-GERICS) and the Fraunhofer Institutes IBP and IMW. Open source software as a successful model For over 40 years, our department has been developing innovative software for use in a range of different fields. Many fashions have come and gone in that time, but there is a clear trend toward open source software (OSS). The OSS model creates an innovative, open and collaborative software development environment. This is especially important when working with standardization bodies, as in the case of our FROST ® reference implementations for the OGC SensorThings API standard and open62541, our OPC UA implementation of the IEC 62541 standard. It is giving rise to a culture of collaboration that improves both the quality of the soft- ware and how it is distributed. This is leading to new business models beyond just the sale of proprietary software. Automated vulnerability detection for industrial IT components Networked machines and equipment are becoming increasingly common in the man- ufacturing industry. This is creating major opportunities to improve processes, but also poses new risks as industrial automation com- ponents become more vulnerable to attacks over the network. This is why we have devel- oped the industrial security testing framework ISuTest ®, a tool for finding vulnerabilities in automation components. ISuTest ® is designed to be an open, expand- able framework, setting it apart from com- mercial competitors that use closed source software. The system helps manufacturers of automation components and provides support for establishing and performing vulnerability tests as well as for isolating vulnerabilities and correcting faults on the developer’s side. ISuTest ® has already been used to detect many vulnerabilities that have subsequently been confirmed by the manufacturer. The benefits of OSS are also clearly visible in European research projects such as OpenIoT and SymbIoTe and the work being done on IVCT software in NATO working groups: ISuTest ® opens up the domain of security testing for automation experts, allowing the vision of “security by design” to be put into practice. ISuTest® consists of a distri- buted software framework and makes use of different hardware components. The hardware components are used for the automated execu- tion of vulnerability tests, taking into account any input and output of the automation components to be tested. 71

Highlight topics Concepts, architectures and services for Joint Intelligence, Surveillance and Recon- naissance (Joint ISR; JISR) In ISR operations, remote exploitation and analysis, as well as multinational collaboration, are rather the rule than the exception, thus network-enabled data and information shar- ing is a major topic. To achieve this seamlessly, interoperability is key. Actors and systems, processes and procedural instructions must be appropriately analyzed, modeled and coordinated to enable consis- tent, reliable, timely and secure exchange of data and information. Legacy systems need to be connected in an overall architecture. These aspects influence which technological concepts, patterns and solutions are appli- cable and selected. Artificial intelligence (AI) can support the human decision-maker if the appropriate algorithms are chosen, and suffi- cient information models can be created. We are adressing these topics since many years, resulting in various contributions to interoperability standards, test procedures and systems (e.g., at the JISR Testcenter at WTD 81), as well as to systems and services for projects like VJTF 2023. For STANAG 4559 AEDP-17 compliant data exchange, storage, and retrieval architectural concepts are devel- oped, and the Fraunhofer IOSB CSD PLUS Server and clients are integrated into complex systems of systems also connecting different security domains. Our research here addresses new concepts, architectures and solutions for data and information sharing in complex environments, as well as data and information analysis by AI concepts and tools. Intelligent mission planning for hetero- geneous groups of mobile autonomous systems To use heterogeneous groups of mobile autonomous systems effectively and efficient- ly for reconnaissance, the mission has to be ”understood” by the group, and the corre- sponding tasks have to be analyzed to select the most suitable resources (sensors, actuators, and mobile carriers, etc.). Next up, the mission in itself has to be planned out. To enable the complete and resource-efficient reconnaissance of an area, the relevant parameters of the individual systems, such as detection ranges, sensor resolutions, speeds, power reserves, must also be considered at path planning. We developed and implemented new algo- rithms for mission and path planning. Our PathPlanner software allows planning all trajectories that are necessary for performing reconnaissance by groups of mobile systems: approach from take-off to target position, trajectories for complete reconnaissance of complex areas and departure. Obstacles and prohibited areas with the necessary safety distances are taken into account during planning, as well as many parameters of the mobile systems, such as their turning radius or whether border crossings at the edges of the reconnaissance area are permitted. Path- Planner possesses a GUI, but it can also work as middleware and transmit path planning tasks and planned trajectories via different interfaces including ROS, MQTT. The soft- ware can get adapted to specific customer requirements, e.g., by adding interfaces and data formats. Departments Data and information dis- semination in multinational operations. Path planning for a photo flight with six UAVs (Unmanned Arial Vehicles) for reconnaissance of a complex area (boundary in red) includ- ing approach and departure in the presence of several prohibited zones or obstacles (brown). 73

Departments Intelligent object recognition is the first step of intelligent traffic lights. utility company, we recently installed the tech- nology at the first traffic light intersections. The goal is to improve traffic flow through adaptive, “learned” traffic light behavior, for example during rush hour. As the potential of this research project and our technology is considered high, a follow-up project address- ing traffic lights for pedestrians already has been launched successfully. Production layout of the CUNA reusable cups. 75 75 Highlight topics CUNA: Next level production at SmartFactoryOWL In summer 2021, Fraunhofer IOSB-INA and the startup CUNA Products GmbH started the real-scale production of cups ready for the market inside our research facility and with close accompaniment of our AI special- ists. In total, a cooperation of ten partners who represent the whole value chain has formed around the AI Living Lab and the cup’s production line, providing valuable insight into the ifs and hows of tomorrow’s sustain- able production processes. In particular, the product demonstrates how technical innova- tion and protection of the environment can harmonize. The cups are characterized by a sustainable concept with a future-proof prod- uct idea, as they are made from CO²-neutral bio-based plastic that does not contain oil, and are both reusable and recyclable. AI and traffic lights: a worldwide novelty on the verge of breakthrough Artificial intelligence (AI), or more precisely “reinforcement learning”, is an approach that has not yet been tested in the control of traffic lights in real road traffic. In Lemgo, our project “Artificial Intelligence for Traffic Signal Systems” (KI4LSA), aiming at intelli- gent, predictive traffic light control, is gaining momentum and has generated a great deal of public interest. Together with the manufacturer of traffic technology Stührenberg and the Lemgo public

Departments The autonomous excavator ALICE loads a transport vehicle. are times when too little or too much energy is being generated. Energy needs to be stored and the use of different energy sources needs to be dynamically optimized. When it comes to sector coupling between heat and electricity in particular, an intelligent balance of supply and demand is crucial. The dynOpt-En project funded by the German Federal Ministry for Economic Affairs (BMWi) has developed an energy manager that optimizes cogeneration plants in buildings and accommodation in a holistic way. It is designed to store and use the power pro- duced by the plant as locally as possible in the form of heat or electrical energy, while also adjusting heat pumps and peak load boilers to minimize ongoing emissions and costs. It does this by using a prediction algorithm: The dynOpt energy manager uses meter and sensor measurements in combination with weather forecasts to make continuous pre- dictions. These predictions are used to match the different heating or cooling generators perfectly, including heat pumps and gas-fired boilers as well as cogeneration units, com- bined with photovoltaic or solar heat plants. The energy manager has been designed as a fully online service, and therefore requires almost no on-site installation work. All the data is available through the data interface (web API), meaning that measurement service providers will be able to offer energy man- agement as part of their service in the future. Useful figures relating to costs, energy con- sumption and CO² emissions are calculated and visualized in clear dashboard layouts [1]. The project has tested and demonstrated the functionality of the energy manager in three buildings, in which costs and carbon emissions were reduced by at least 15 percent. Autonomous construction machinery Remotely operated or autonomous robot systems can be used in harmful environments to make people’s work safer and easier. In the field of landfill and contaminated site decontamination, we are currently conducting research into an autonomous excavator that removes contaminated layers of soil on its own. To achieve this, a conventional excavator has been fitted with a large number of sensors to enable it to precisely identify its surroundings: With laser scanners and cameras, the exca- vator is able to produce a three-dimensional model of its environment. It can use this 3D model to find its own way to the operation site and avoid collisions, for example with other construction machinery. This also forms the basis for planning the excavation strategy and the specific movements of the excavator arm. Angular rate and acceleration sensors and GPS allow the excavator to orient itself precisely in its environment at all times. [1] Links to dashboards for real buildings: https://www.dynopt.de/ servlet/is/107489/ The excavator’s autonomy function is imple- mented using the “algorithm toolbox for autonomous mobile robot systems” developed at Fraunhofer IOSB, which integrates compo- nents for application-specific autonomy func- tions according to the principle of modularity. The dynOpt-En energy manager activates differ- ent energy sources in large buildings based on supply and demand to minimize costs and CO² emissions. Savings through dynOpt-En CO² € Optimizer ON OFF Consumption and sensor data Weather forecast Dynamic CO²€ boundary conditions Prediction of consump- tion and generation Cost-/CO² minimization Control signals to heat generators Monitoring and reporting 77

Departments TORTUGA ROV: The TORTUGA ROV family is a multifunctional family of agile, remotely operated underwater vehicles with diving depths of up to 500 meters. They can handle a wide range of tasks thanks to their flexible design and various interface options. Multiple sensors are available for underwater navigation and environmental sensing. DEDAVE 2000: This is an autonomous underwater vehicle with a diving depth of up to 2000 meters and a large payload range with versatile sensor equipment. C-CAT: C-CAT is a remotely controllable or autonomously operating catamaran with a payload of up to 300 kg and versatile sensor equipment. Highlight topics Underwater archeology In the field of underwater archeology, we have been cooperating successfully with the Saxony-Anhalt State Office for the Preserva- tion of Monuments and Archeology (LDA) for many years now. At the end of 2018, we were successful in surveying and 3D modeling an approx. 7000-year-old submerged burial mound in the “Süßer See” lake in Saxony-An- halt using an autonomous surface vehicle. produces high-resolution video and sonar data of objects and structures that may elude even experienced recreational divers in the murky waters. In 2020, an underwater laser scanner from the company Kraken Robotik was integrated into the vehicle. This sensor facilitates the most accurate measurement of underwater structures to date − the discrep- ancies in the 3D model produced amount to less than one millimeter. This sensor under- went its first practical tests in the IOSB-AST test tank and later in both the Süßer See and Arendsee lakes. DEDAVE 2000 The autonomous underwater vehicle DEDAVE 2000 is the “little brother” of DEDAVE. It was sold to the Canadian company Kraken Robotics in 2017 and has since been used primarily in the North Atlantic. DEDAVE 2000 is identical in structure except for some hard- ware component optimizations. The buoyancy foam used in the vehicle allows it to operate at depths of up to 2000 meters. In addition to the multibeam echosounder that is already installed, it will also be possible to equip the large payload bay with a state-of-the-art syn- thetic aperture sonar in the future, which − in addition to creating 3D profiles of the seabed − will also enable high-resolution images to be taken. Since then, our TORTUGA ROV (remotely operated vehicle) has been deployed several times in the Süßer See and Arendsee lakes to investigate objects identified by surface sonar in greater detail. To do this, the ROV In August 2021, DEDAVE 2000 was able to show what it can do at the Baltic Live Trials of the European OCEAN2020 project (see p. 44). The task was to map a 30-hectare area of the Baltic Sea in southern Sweden. ROV belonging to UWR with integrated SeaVision laser scanner shortly before it was deployed in the Arendsee lake (November 2020). Above: AUV DEDAVE on board the research vessel PLANET during the Baltic Live Trials (August 2021). 79

Departments The Digital Map Table (DigLT) is a system that uses virtual reality to bring multiple users into a virtual, three-dimensional environment in which real-world geographic data from all over the globe can be displayed and interact- ed with. The three-dimensional display gives a much better insight into a scenario than two-dimensional maps and in addition to the display of data at a virtual table, allows users to „jump into“ the map and stand on the top of buildings to get a better understanding of the real world. In contrast, we use AR for displaying overlay information in manufacturing and quality inspection use cases. Hands are recognized and tracked, and use case-depending user interfaces are presented, guiding through individual tasks. Most use cases require 3D CAD models of respected components that are to be highlighted, e.g. to overlay their digital representations at their exact locations along with additional information. The open standard OPC UA is being used, which allows to submit all required data via WIFI. Above: The Digital Map Table in virtual reality. Below: Augmented Reality for manufacturing. XAI Toolbox in the mar- itime sector, adapted to explain ship trajectories‘ classifications. society in recent years. AI models can only develop their full potential if the AI is devel- oped to high-quality standards and there- fore can be fully trusted. Our XAI toolbox is designed to make AI black box models more transparent. The toolbox can be used for data analysis, debugging and the explanation of AI models and predictions. It generates two kinds of explanations: local explanations for a single instance and global explanations for multiple instances or the entire model. This improves the user‘s ability to debug and explain the results and in addition strengthens the user’s trust in the AI methods in decision-critical sec- tors such as medical, automotive, maritime, etc. We research explanatory interfaces in the context of AI methods and use design pro- cesses to tailor them to different user groups. Explaining outputs of AI models is also rele- vant from a legal point of view. The General Data Protection Regulation sets requirements in terms of transparency for data subjects once their personal data is processed, and the draft for a European Artificial Intelligence Regulation imposes a duty of supervision for users of specific categories of AI applications. In the latter case, manufacturers of AI-based solutions are directly made accountable for providing explainable systems. Mixed Reality for Geointelligence and Quality Control Virtual and Augmented Reality have made great leaps forward in terms of available technology, such as markerless localization and high-resolution head-mounted displays. 81

Departments A helicopter demonstration platform during starting procedure. The helicopter trajectory estimated solely from live imagery during flight. 83 applications. Due to the VSLAM component, THS creates its own maps while working. These maps can be used as a reference in sce- narios where an area is visited several times. Common geo data products such as ortho- photos together with elevation models can provide the necessary information in other, more general scenarios. In the present realization, THS can provide position information for a sensor or sensor carrying platform just as GNSS-based posi- tioning systems. With its two components VSLAM and geo-localization, THS resembles the functional combination of an inertial mea- surement unit and a GPS receiver. We believe that THS is a valuable option for positioning systems in scenarios where GNSS is not available or might be not reliable. If the application of visual sensors is possible (view- ing conditions, etc.), THS can provide GNSS-in- dependent position information, which is the basis for navigation, blue force tracking, and target acquisition in military operations. objective was to deliver a complete system comprising sensors, optics, interface, and image exploitation hard- and software to be mounted to a helicopter platform and which would be able to estimate the helicopter’s pose – position and orientation – during flight. This information should be passed on to the helicopter’s aviation system via aviation certified hardware. The overall goal was to provide the position and orientation of the helicopter in GPS-denied situations, i.e., solely based on imagery. The desired functionality could be demonstrated in several test flights during which GPS was disconnected or active- ly jammed and spoofed. The system we implemented collects imagery with the purpose of geo-localizing this imag- ery. For every geo-localized image the pose of the sensor is known in a global coordinate system (e.g., UTM). Geo-localization of an image is based on recognizing certain visual landmarks both visible in the geo-data and the present image. Unfortunately, not every image from a sensor stream can be registered against geo-data in this way. To overcome this challenge, THS combines the sporadic geo-localization with estimates for the relative sensor movement on a frame-by-frame basis. This process is called visual simultaneous localization and mapping (VSLAM) and enables THS to provide the geo- pose for every single image. Concerning geo-data necessary for THS, this data can look very different in different

Departments Application of measuring marks on the wing of a Boeing 777-300ER in Zurich, 2021. The selection of appropriate components is crucial for the design of the utilized photo- grammetric system. An extended and con- stant depth of field, for instance, is required to obtain well-focused images. Furthermore, for the precise determination of optical paths, i.e., spatial directions, an in-situ calibration of the camera system is essential, also capturing the distortions caused by the window glasses. In recent years, the feasibility and usability of the approach have been demonstrated during several transatlantic flights with long- range wide-body airliners. The results verified expected temporal changes due to the fuel consumption and the resulting loss of weight. The cost-efficient and unintrusive approach has been filed as a patent application [1]. In long-term work, such a monitoring system allows for conclusions about the mechanical load and thus the service limit of an airfoil. For a correct measurement, the application must be accu- rate to the centimeter. potentially dangerous since this implies the measurement through windows glass onto reflective surfaces. To determine an airfoil’s shape in midair, we developed a tailor-made solution for our part- ner and customer Lufthansa Technik AG. The developed measurement concept allows for the deployment of a single camera observing an aircraft’s wing through a cabin window. This monocular approach paves the way to the easy application of a lightweight in-flight measurement system as a stand-alone solu- tion during scheduled flights. However, since a camera is a bearings-on- ly sensor, a model describing the wings’ deflections and additional measurements on the ground have to provide depth infor- mation. Measuring marks are stuck on the wing’s upper surface, and their positions are determined using an electronic tachymeter. These marks are later observed in the images captured during the flight. The bending model assumes the preservation of arc lengths during the lift. Thermal expansions can be taken into account on a calculative basis. An optimization process provides the distances required for the polar point determination, and the computed 3D points are utilized to bend a virtual wing model supplied by a terrestrial laser scan. Eventually, the resulting model is used for flow simulations. 1 Pfingsten, Kai-Christoph; Meidow, Jochen. Verfahren und Anordnung zur Vermessung von Flugzeugflügeln während des Fluges. Patent Application No. DE 10 2019 122 898.0 85

Departments Example results of the AI-based video exploitation. The bounding boxes visualize the persons detected by the AI and the texts above them show the recognized roles as well as the confidence. The confidence indicates how “reliably” the AI has assigned the roles, where 1 stands for the highest and 0 for the lowest confidence. medical personnel, patients and cleaning personnel. The analysis is done by the AI in a two-stage approach with the first stage detecting all people and the second stage identifying the roles of the detected people. high detection accuracy. Due to the application of an improved learning algorithm and careful hyper-parameter selection, we were able to further improve the accuracy without increas- ing the required runtime of the network. Additionally, we converted our network to the TensorRT execution environment which is highly optimized for the processing units of the Jetson platform and is capable of utiliz- ing a compact and fast 16-bit floating-point format as number representation. In total, we can achieve real-time operation at about 20 frames per second with a very high detection accuracy and a small number of missed detections while only requiring 15 watts of power. Thus, our system can be easily deployed on drones or used for other mobile applications. A major challenge in the project was the small amount of available training data due to data protection. Thus, optimizations from the field of few-shot learning like strong data augmen- tation are applied. This synthetically increases the diversity of the images by showing the images multiple times to the AI with random variations in terms of color and image range. These approaches can ensure, and possibly enhance, data protection as well as increase the efficiency of the training process of the AI. Fast and resource-saving person detection Person detection has gained massive interest recently due to the COVID-19 pandemic and the requirement to enforce distancing regula- tions. However, the presence of permanently installed surveillance cameras is limited. Thus, cameras mounted on small UAVs (Unmanned Aerial Vehicles) are a perfect solution for verifying compliance with the regulations. To enable automatic processing of the verifica- tion on drones, we developed an AI-based person detection solution capable of running on low-power embedded devices in real-time. To achieve real-time operation on the chosen Nvidia Jetson platform, we designed a custom convolutional neural network architecture that provides a low processing time while retaining An image of persons processed by our AI-based person detector with the detected persons being highlighted with bounding boxes. 87

Departments Confocal microscopy with direct-imaging DoEs: the optical principle (top left); photo of a prototype (top right); resulting images of a test target with convention- al microscopy (bottom left) and our proposed approach (bottom right). Camera Tube Lens Right-angle Mirror Point Light Source Relay Lens Beam Splitter Objective DLA Sample 4D light field acquisition and processing for the visual inspection of transparent objects Transparent objects can be affected by material defects that are hard to detect with conventional inspection methods since they are transparent themselves (enclosed air bub- bles, scratches, etc.) and only influence the propagation direction of light. We developed a novel sensor device, a laser deflection scan- ner that can acquire four-dimensional light fields containing information about the light’s direction of propagation. Through suitable processing algorithms, also challenging trans- parent material defects can be revealed with high contrast. Inspection images of a washing machine door glass: none of the material defects are made visible by a conventional bright field inspection approach (left); all defects are revealed and shown with high contrast by the novel laser deflection scanner and the light field processing methods (right). 89

Appendix 21 doctorates Twenty-one researchers working at Fraunhofer IOSB passed their doctoral examinations in 2020 and 2021. 50 R patents granted Applications for a further 39 patents have been disclosed in 2020 and 2021. In addi- tion, six product names were registered as trademarks. >550 journal and conference papers have been authored or co-authored by Fraunhofer IOSB researchers. ≈1,400 participants attended special events and conferences organized or co-organized by Fraunhofer IOSB and its scientific staff, when adding together on-site and virtual events. For all details on publications, academic teaching, lectures, technology transfer and out- reach activities, please refer to the comprehensive online appendix to this annual report: www.iosb.fraunhofer.de/annual-report-2022 91

Internal working groups Automotive und assistance systems Chair: M. Sc. Jens Ziehn | MRD Participating departments: IAD, MRD Consultants: Univ.-Prof. Dr.-Ing. habil. Pu Li, Prof. Dr.-Ing. Rainer Stiefelhagen Counter-improvised explosive device Chair: Dr. rer. nat. Ilja Kaufmann | OPT Participating departments: OBJ, OPT, SIG, SZA, VID Consultant: Prof. Dr. Maurus Tacke End-to-end software development Chair: M. Sc. Philipp Hertweck | ILT Participating departments: DIS, IAD, IAS, ILT, MIT, MRD, KES, SPR, VID Consultant: Prof. Dr. rer. nat. Sebastian Abeck Event based vision Prof. Dr.-Ing. Thomas Längle | SPR Participating departments: SPR, MRD, VBV Consultants: Prof. Dr.-Ing. Michael Heizmann, Prof. Dr. rer. nat. Bernd Jähne, Prof. Dr.-Ing. Uwe Hanebeck Machine learning Chairs: Dr.-Ing. Christian Kühnert | MRD, Dr.-Ing. Julius Pfrommer | ILT Participating departments: IAD, IAS, ILT, MIT, MRD, WMS, SPR, SZA, VID, WMS Consultants: Prof. Dr. Georg Bretthauer, Prof. Dr.-Ing. Michael Heizmann, Prof. Dr.-Ing. Stefan Hinz, Prof. Dr. rer. nat. Bernd Jähne, Prof. Dr. rer. nat. Wolfgang Karl, Prof. Dr.-Ing. Rainer Stiefelhagen Appendix Multispectral data acquisition and exploitation Chairs: Prof. Dr.-Ing. Thomas Längle | SPR, Dr.-Ing. Robin Gruna | SPR Participating departments and institutes: ILT, MRD, SIG, SPR, SZA, KIT - Institute of Industrial Information Technology, KIT - Institute for Applied Biosciences Consultants: Prof. Dr.-Ing. Michael Beigl, Prof. Dr.-Ing. Carsten Dachsbacher, Prof. Dr.-Ing. Michael Heizmann, Prof. Dr. rer. nat. Bernd Jähne, Prof. Dr. rer. nat. Wolfgang Karl, Prof. Dr.-Ing. Uwe Sörgel Self-organization of complex systems Dr.-Ing. Andreas Bunte | MIT Flexibilization of production systems for energy supply Univ.-Prof. Dr.-Ing. Dirk Westermann Autonomous systems Prof. Dr.-Ing. Andreas Wenzel Participating departments: MRD, IAS, VID, UWR, EIS 93

Addresses Fraunhofer IOSB Karlsruhe Fraunhoferstraße 1, 76131 Karlsruhe, Germany Phone +49 721 6091-0 info@iosb.fraunhofer.de Industrial Automation branch Fraunhofer IOSB-INA Campusallee 1, 32657 Lemgo, Germany Phone +49 5261 94290-22 Karlsruher Forschungsfabrik Rintheimer Querallee 2 - Geb. 70.41, 76131 Karlsruhe, Germany Fraunhofer IOSB Ettlingen Gutleuthausstraße 1, 76275 Ettlingen, Germany Phone +49 7243 992-0 Branch lab Oberkochen c/o HENSOLDT Optronics GmbH Carl-Zeiss-Str. 22, 73447 Oberkochen, Germany Advanced System Technologies branch Fraunhofer IOSB-AST Am Vogelherd 90, 98693 Ilmenau, Germany Phone +49 3677 461-103 Branch lab Görlitz Brückenstraße 1, 02826 Görlitz, Germany Phone +49 3581 7925354 Smart Ocean Technologies research group SOT Rostock Alter Hafen Süd 6, 18069 Rostock, Germany Phone +49 3581 7925354 Fraunhofer Center for the Security of Socio-Technical Systems SIRIOS Berlin c/o Fraunhofer FOKUS Kaiserin-Augusta-Allee 31, 10589 Berlin, Germany Phone +49 30 3463-7594 Fraunhofer Representative Office Beijing Representative for Production and Information Technologies Unit 0602G, Tower D1 DRC Liangmaqiao Office Building 19 Dongfangdonglu, Chaoyang District 100600 Beijing, PR China Appendix 95