Progress report 2023/2024 Digital technologies for productivity, sustainability, and security www.iosb.fraunhofer.de

We help harness the full potential of digital technologies Digital devices have become omnipresent in today’s world, and few activities remain entirely analog. Yet, this high degree of things being digitized doesn’t necessarily mean that digitalization, in a transformative sense, is happening. According to the Gartner Glossary, digitalization, as opposed to digitization, is “the use of digital technologies to change a business model and provide new revenue and value-producing opportunities.” [1] Digitalization is thus about fundamental change, about opening up new opportunities. However, beyond Gartner’s definition, it should aim not only for economic advantage and increased revenue, but for value creation in a much broader sense. In line with this, our goal is to future-proof our society with the aid of digitalization. One focus is productivity, because it is key for our economic strength and prosperity. To meet today’s needs, productivity has to go hand in hand with flexibility and resilience. It must be resource friendly and comply with environmental and social standards, which leads to the second focus: sustainability. Sustainability not only concerns industry and its supply chains, it also extends to the utilities sector, the energy transformation, and mobility. As we must acknowledge, while extreme weather events and armed conflicts are closing in on us, all the progress mentioned is pointless if security is lacking. Therefore, security, in a comprehensive sense, is a third focus. To achieve these ambitious goals, digitalization, like technology in general, is not the sole solution, but it plays a pivotal role. At Fraunhofer IOSB, we leverage our technological expertise and in-depth knowledge of application domains to contribute to the solution. Our research focuses on technologies that bridge the gap between the real and digital worlds. We use sensors to transform real-life situations into data and then analyze and exploit that data. We create digital twins of real-world entities and systems — whether it be machines, products, critical infrastructures, crowds, or even battlefields — and network stakeholders by helping to create interoper- able, federative data spaces. We apply physical models and/or create data-driven models through machine learning, to push the boundaries of experiential knowledge through simulation. We integrate the insights gained into dash- boards and situation pictures. We develop autonomously operating systems for appropriate applications. And we put the human in the loop wherever necessary, providing innovative interfaces for interaction and empowering them to make the best possible decisions. That is what we can contribute to achieving the benefits outlined above. That is what this report is about. 1 www.gartner.com/en/glossary 5

Preface 7 Prof. Dr.-Ing. habil. Jürgen Beyerer, head of institute of Fraunhofer IOSB We are tackling this at various levels. Among other measures, we have refined our most important internal funding instrument, the Technology Development Program (TEP, see p. 15). And we discuss effective innovation paths for the German Armed Forces in the interview on p. 45. Of course, you will also find many other developments from the 2022/2023 reporting period in this issue: The Karlsruhe Research Factory, whose opening we reported on in the last issue, has been established and “brought to life,” and the bridge professorship between Fraunhofer IOSB-INA and Bielefeld University on the sub- ject of cognitive automation, which had been announced for some time, has been filled. And in many publicly and internally funded projects — which are naturally the focus of this report, since industrial collaborations are generally subject to non-disclosure agree- ments — we are helping to drive forward pioneering developments. Transferring the state of the art in science into the state of the art in technology and bringing it into application: This mission remains demanding, and not just because of fundamental challenges. The cloudy eco- nomic outlook is making things even more difficult. However, even though R&D funds in many companies may be more limited than before, innovation remains the decisive lever for gaining a competitive edge and combat- ting the crisis. Let’s explore the best ideas and forms of cooperation together! We are your partner, please don’t hesitate to contact us at any time. Sincerely, Jürgen Beyerer Head of Fraunhofer IOSB

Highlights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 1. Digital technologies for resource-friendly, future-proof production . . . . . . . . . . . . . . . . 8 2. Digital technologies for security and safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3. Organizational development and infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4. Outreach events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 The institute in profile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Fraunhofer IOSB: A unique spectrum of scientific expertise . . . . . . . . . . . . . . . . . . . . . . 20 Expertise in applications and market-oriented solutions . . . . . . . . . . . . . . . . . . . . . . . . . 22 How to cooperate with us . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 Pooling expertise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Fraunhofer Strategic Research Field Artificial Intelligence . . . . . . . . . . . . . . . . . . . . . . . . 31 Fraunhofer Segment for Defense and Security VVS . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Business units . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Artificial Intelligence and Autonomous Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Automation and Digitalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Defense . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Energy, Environmental and Security Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Inspection and Optronic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 Departments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 LAS | Laser Technology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 OPT | Optronics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 SIG | Signatorics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 SPR | Visual Inspection Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 KES | Cognitive Energy Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 KIS | Cognitive Industrial Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 DIS | Digital Infrastructure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 EIS | Embedded Intelligent Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 ILT | Information Management and Production Control . . . . . . . . . . . . . . . . . . . . . . . . . 76 IAS | Interoperability and Assistance Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 MIT | Machine Intelligence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 MRD | Systems for Measurement, Control and Diagnosis . . . . . . . . . . . . . . . . . . . . . . . . 82 UWR | Underwater Robotics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 HAI | Human-AI Interaction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 OBJ | Object Recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 SZA | Scene Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 VID | Video Exploitation Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 VBV | Variable Image Acquisition and Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Appendix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Facts and figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 Advisory board . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Internal working groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Scientific consultants of Fraunhofer IOSB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Addresses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 Editorial notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

Highlights » With Manufacturing-X, German indus- try can transform itself internationally from a factory equipment supplier into a digital supplier and at the same time become a pioneer in climate-friendly industrial production.« Dr. Robert Habeck, Federal Minister for Economic Affairs and Climate Action AI systems engineering and the “AI Alliance Baden-Württemberg” To systematically and reliably bring artificial intelligence into productive use in the engineering domains: this is the objective of AI systems engineering is all about, a new discipline that Fraunhofer IOSB has helped launch in recent years and is driv- ing forward in the field of industrial production in particular. Fraunhofer IOSB contributes its AI and AI systems engineering expertise as a founding member of the AI Alliance Baden-Würt- temberg. This cooperation aims to create an internationally com- petitive AI ecosystem for Baden-Württemberg for a wide range of application domains, such as production, logistics, medicine, and smart cities. Six regions, including Karlsruhe, are pooling their expertise for this purpose. Fraunhofer IOSB is heading up two subprojects that were launched in 2023: The “data plat- form” subproject focuses on creating a low-threshold infrastruc- ture for companies, especially SMEs, and the public sector to share AI assets, i.e., data sets and AI models. The intended basis for business is a private-public partnership model. Ethical and legal aspects such as the implementation of the EU AI Act and data sovereignty are given high priority. The second subproject, “AI Challenge,” will organize a series of regional workshops that will ultimately incorporate the challenge of AI systems engi- neering into practice: Using the appropriate methodology, the aim is to harness AI in the product and service development of businesses and public authorities. The goal of the AI Challenge is thus to encourage innovative projects in accordance with the regional thematic priorities. More on the topic: “Data spaces and digital twins” (visIT thematic brochure, in German) https://services.iosb.fraunhofer.de/visIT/ datenraeume 11

Highlights “Waste4Future” — closing the plastics cycle In the Fraunhofer lighthouse project “Waste4Future,” a network of eight institutes is finding new ways to significantly increase the material recycling of plas- tics. To this end, a holistic, entropy-based evaluation model is being developed that transforms the previ- ously process-led recycling chain into a material-ori- ented one: Sorting technology that combines sensor technologies such as hyperspectral imaging, radar and active thermography recognizes which plastic fractions are contained in the waste. Digital twins of materials and systems are then used to determine the optimal recycling path. The goals are economic efficiency, high purity levels in plastics sorting and robust material characterization, which also includes black plastics and previously neglected criteria such as the degree of aging. At Fraunhofer IOSB, we are applying our expertise in the field of optical and hyperspectral sensors as well as various classification approaches based on decades of experience in bulk material sorting. We are also building a demonstrator for the advanced characterization and sorting of plastic streams based on AI-supported data analysis. At the same time, we are contributing our expertise in digital twins and the FA³ST service. FA³ST stands for “Fraunhofer Advanced Asset Administration Shell Tools,” an open source software suite that Fraunhofer IOSB has been devel- oping in other projects. FA³ST makes it easy to imple- ment standard-compliant digital twins based on the advanced asset shell (AAS) specification maintained by the International Digital Twin Association (IDTA). 13

Highlights » Regardless of whether it’s a medical crisis, an animal epidemic or a gas shortage — a standardized digital tool for modern and comprehensive staff work is at our disposal. This will significantly strengthen the state’s security architecture.« Thomas Strobl, Minister of the Interior in Baden-Württemberg In early April 2024, SIRIOS and the state of Berlin signed a cooperation agreement: New technologies will be developed, tested, and applied in close collaboration — for security situa- tions specifically tailored to the German capital. Specifically, the plan is to hold regular joint workshops and conferences, share research-relevant data, and scientifically assess operational situations. Two scenarios form the starting point for the scientific research and activities: the breakdown of critical infrastructures as a result of a natural disaster, such as a severe storm, and a human-made emergency caused by an explosion at a big event. In four pilot projects, SIRIOS researchers use coupled simulations to investigate the development and effects of such incidents, while also deriving procedures to strengthen security and societal resilience. At the first annual conference in fall 2023, the researchers showcased the results they have achieved so far: For example, simulations showed possible cascading effects on utilities following a substation failure caused by severe weather. Integrating the simulation data into a situation picture helped to decide whether an affected hospital should be evacuated. In another scenario, a big event was used as an example to simulate how crowds of visitors would behave if passageways were closed, stage performanc- es attracted particularly high numbers of people, and danger warnings were issued. 15

3. Organizational development and infrastructure Highlights still being developed for other target groups, such as those in the area of administration. Karlsruhe Research Factory and Research Campus In March 2022, the Karlsruhe Research Factory for AI-inte- grated Production opened on the Campus East of the Karl- sruhe Institute of Technology (KIT) — see IOSB Annual Report 2021/2022. The Cognitive Industrial Systems department (KIS), which was also newly established in 2022 and has close links in terms of content with the activities of the Research Factory, decided to relocate to an office building directly on site. This brings office and laboratory work in close proximity to each other and ensures daily communication with the research factory’s other partners. The Fraunhofer IOSB main site in Karl- sruhe is still an easy 15-minute walk away. There, the move has freed up space for other departments that are also growing. In addition, the former library has been converted into new offices and meeting rooms. An even bigger change for the site will be coming in the next few years: The plans for a new building at Fraunhoferstrasse 1, the “Karlsruhe Research Campus”, are taking shape. Construc- tion is scheduled to begin in 2026 and the structural design has been entrusted to the the team at the Henn architecture firm. The new building will accommodate new, additional laboratory and office space for our institute. Moreover, and most impor- tantly, the Fraunhofer Institute for Systems and Innovation Research ISI will relocate to the new building on what has been IOSB premises until now, as it has outgrown its current build- ing. Both institutes will conduct joint research in the future, which will boost professional networking and interdisciplinary cooperation. The focus will be on the responsible development and application of artificial intelligence and autonomous sys- tems as well as their economic and social potential. Fraunhofer IOSB is not only actively developing the state of the art in its various areas of expertise, it is also constantly improving as an organization. This includes reviewing and, if necessary, revising and modifying processes and structures, optimizing procedures and incentive systems, and responding to the requirements of the current times. Transfer-focused internal research funding We invest part of our base funding in in-house research, i.e. pre-competitive research projects financed from the institute’s own funds. Our Technology Development Program (TEP) has always focused on promoting ambitious, cross-departmental projects that are conducive to strategic portfolio development. We have now further developed this internal program in order to even more strongly incentivize the transfer of technology into application — and thus to the customer. To this end, the TEP has been expanded into a three-stage model. The basic concept is that R&D ideas worthy of support should be backed by a potential user of the technology right from the outset. They should be geared towards the user’s practical requirements so that the transfer will be successful once the R&D work has been completed. Such a potential user, or technology patron, is needed as an external point of reference even before the actual project plan is created and funding application for the TEP started. And the actual TEP phase, which typically lasts one to two years, is now followed by a technology proof of concept phase, which serves to pilot the solution in a real application context, on site at the tech- nology patron’s premises. Our researchers’ work in this phase can also be funded internally for up to one year. Updating the HR development concept In 2022, the decision was made to further enhance Fraunhofer IOSB’s existing HR development concept. Since then, a task force has been advancing the work on the actual content. Initially, it focused on the target groups of IT and research sci- entists. In these occupational fields, the institute faces particu- larly great challenges in recruiting personnel, making boosting employer attractiveness particularly important here. As a framework for development opportunities, the HR devel- opment concept aims to create attractive career prospects beyond the traditional management career path. The concept outlines three possible specialist career paths. By explicitly formulating the respective requirements, tasks and responsi- bilities, the roles of “senior scientist” and “senior innovation/ application expert,” which have existed for some time, are being refined and clarified. A third option, “senior manager of major projects,” has been added. Similar concepts are currently 17

Highlights Spotlight on sustainability The Executive Board of Fraunhofer-Gesellschaft has set the goal of reducing the organization’s emissions by 55 % by 2030 and offsetting the rest. The Fraun- hofer-Gesellschaft aims to be climate-neutral by 2045. In addition to a dedicated department at headquar- ters, climate protection projects and the Fraunhofer Institutes’ climate neutrality and sustainability officers (BfKNs) are key elements in achieving this goal. Fraunhofer IOSB has also had a BfKN since 2022, Oliver Warweg from Fraunhofer IOSB-AST in Ilmenau, as well as a deputy, Dr. Thomas Bernard from the Karlsruhe site. The BfKNs hold monthly meetings with the head of administration and facility management staff from all sites to discuss and prepare measures to achieve climate neutrality. They act as in-house ambassadors and points of contact for the topic and are in close contact with Fraunhofer headquarters and the BfKNs at the other institutes. All Fraunhofer IOSB sites are integrated into regional “energy effi- ciency networks,” where the participating institutes set a joint energy-saving target for themselves, which is to be implemented by the end of 2026. Climate protection projects include LamA® — charging at the workplace, which involves setting up charging stations for electric cars at Fraunhofer sites, and a funding program for photovoltaics (solar arrays). Fraunhofer IOSB has benefited from both of these: Karlsruhe and Ettlingen each have a charging station on the institute’s premises. In Ilmenau, the charging facility was put into operation on May 7, 2024, and can now be used by employees. In the long term, this is the aim for the other IOSB sites, too. Through the photovoltaic program, a solar array system was installed on the roof of the institute building in Karlsruhe, which should cover up to half of the daily electricity requirements in the future. Plans call for setting up a similar system for the site in Ilmenau, as an extension to the existing array. In Lemgo, there is a smaller, particularly innovative solar array system in the form of a “smart flower” that aligns itself with the position of the sun. Furthermore, Fraunhofer IOSB launched a sustainabil- ity award in 2022. The call is for actionable concepts that promote sustainability in various categories. All employees are invited to take part, with a total of 10,000 euros in prize money being awarded each year. In the two rounds to date, numerous suggestions have been submitted, ranging from the procurement of company bicycles and sustainable alternatives for the disposal of decommissioned computers to various mea- sures for more biodiversity on the institute’s premises. 19

Highlights Recap video of INDIN 2023 at: 4. Outreach events After only being able to report on the first tentative signs of a return to in-person events in the last progress report, which was published in 2022 while the pandemic was subsiding, things have returned to normal in the reporting period cov- ered by this issue. We have organized and successfully held a number of events ourselves: an AI Systems Engineering Day for Production at the Karlsruhe Research Factory, the Image Processing Forum 2022, the 6th Conference on Optical Characterization of Materials (OCM), the MAROS (Maritime Robotics and Sensors) Conference, Fraunhofer IOSB’s Defense Technology Day, and several more. Below you can read more about three special events. Along with our own events, we were present at numerous trade fairs and specialist events — including Hannover Messe, E-world energy & water, AFCEA trade exhibition, CONTROL, LASER, and many more, as well as numerous career fairs. A comprehensive list of our activities in this area can be found in the online appendix, see p. 96. INDIN 2023 in Lemgo Together with the Institute for Industrial Information Tech- nology at the OWL University of Applied Sciences and Arts, Fraunhofer IOSB-INA succeeded in bringing the 20th edition of the IEEE International Conference on Industrial Informatics, INDIN’23, to Lemgo. More than 250 scientists from all over the world came to the historic Hanseatic city in mid-July 2023. They presented groundbreaking research results and innovative ideas and were impressed by the mix of the picturesque historic city backdrop and Lemgo’s modern infrastructure for knowledge and technology transfer (including the Innovation Campus Lemgo and SmartFactoryOWL). A recurring topic at the conference was the importance of AI in the field of indus- trial production. Fraunhofer IOSB organized a special session on AI systems engineering in industrial automation. Formal inauguration of Building C in Ettlingen Selected senior representatives from the federal ministry of defense, city administration, municipal council, and the relevant authorities accepted our invitation and attended the official inauguration of Building C at the Ettlingen site in September 2023. The modular construction was erected for the new, dynamically growing Laser Technology department. It houses two laboratories and 18 office workplaces and has been filled with life for quite some time. After several wel- coming addresses, site director Prof. Marc Eichhorn’s keynote speech highlighted the importance of Fraunhofer IOSB’s laser research for Germany’s technological sovereignty. His speech was then followed by a presentation by the head of the Laser Technology department, Dr. Christelle Kieleck, providing a review and outlook on the development of the institute’s Ettlingen site in this context. There was a consensus among the participants that Building C should be retained beyond the five-year period of use initially envisaged and approved — at least until the planned large extension building for Fraunhofer IOSB Ettlingen is finished. 21

The institute in profile Core competences and structure Optronics System Technologies Image Exploitation Understanding and control of Ability to analyze, understand, model, Preparation and real-time processing of the generation of light, its beam develop and control complex systems images and videos as well as automatic forming, propagation and and interactive information extraction conversion into electronic signals Cognitive Energy Systems | KES Cognitive Industrial Systems | KIS Digital Infrastructure | DIS Embedded Intelligent Systems | EIS Human-AI Interaction | HAI Object Recognition | OBJ Scene Analysis | SZA Information Management and Video Exploitation Systems | VID Production Contro | ILT Variable Image Acquisition and Interoperability and Assistance Processing research group | VBV Systems | IAS Machine Intelligence | MIT Systems for Measurement, Control and Diagnosis | MRD Underwater Robotics | UWR Laser Technology | LAS Optronics | OPT Signatorics | SIG Visual Inspection Systems | SPR The competence triangle System Technologies KES KIS UWR EIS DIS ILT MIT MRD IAS SIG LAS OPT SPR SZA VBV HAI VID OBJ The competence triangle positions the departments of Fraunhofer IOSB in relation to the three core competences. Optronics Image Exploitation 23

The institute in profile Selected strategic collaborations, expert networks and platforms we are participating in CC-KING Competence Center KI-Engineering Lemgo Digital Open District Hub Für die Zukunft der Energiewende 25

The institute in profile Key figures Business expenses Funding How much money did we spend? Where did the money come from (in 2023)? Industrial sector 33 % Basic funding 25,6 % Others 3,1 % Public sector 38,2 % 80 70 60 50 40 30 20 10 '19 '20 '21 '22 '23 Investment Research expenditure Staff How many persons worked at Fraunhofer IOSB? '23 '22 '21 '20 '19 100 200 300 400 500 600 700 800 Scientists and engineers Other regular employees Research assistants, student assistants and interns 27

The institute in profile Ilmenau Lemgo Advanced System Technology branch Fraunhofer IOSB-AST Industrial Automation branch Fraunhofer IOSB-INA Prof. Dr.-Ing. Peter Bretschneider Prof. Dr.-Ing. habil. Thomas Rauschenbach Directors Ilmenau Prof. Dr.-Ing. Jürgen Jasperneite Director Lemgo EIS | Embedded Intelligent Systems DIS | Digital Infrastructure Administration Prof. Dr.-Ing. Andreas Wenzel Dr.-Ing. Sebastian Schriegel KES | Cognitive Energy Systems MIT | Machine Intelligence Prof. Dr.-Ing. Peter Bretschneider Dr. rer. nat. Oliver Niehörster UWR | Underwater Robotics Prof. Dr.-Ing. habil. Thomas Rauschenbach Berlin Görlitz SIRIOS | Fraunhofer Center for the Security of Socio-Technical Systems Dr.-Ing. Markus Müller IT Security for Critical Infrastructures Energy and Water (research group) Prof. Dr.-Ing. Jörg Lässig Oberkochen Rostock Active Laser Fibers (research group) Dr. Christelle Kieleck SOT | Smart Ocean Technologies (research group) Prof. Dr.-Ing. habil. Thomas Rauschenbach Head of administration / Commercial and Technical Management Dipl.-Betriebsw. (FH) Nicole Keller-Rau IT security M.Sc. Marco Feuchter Staff department Head of Staff depart- ment and Strategy and Innovation Management Dr. rer. nat. Frank Lorenz Press and Communi- cations Dipl.-Phys. Ulrich Pontes Beijing Representative office China Dipl.-Ing. Hong Mu 29

The institute in profile Fraunhofer-Gesellschaft — 75 years of innovation Over 6,200 customers from the private sector ... of which around 60 percent are small- and medium-sized companies. Over 30,000 employees In 2023, the recruitment rate for female scientists increased to 30 percent (overall approximately 25 percent of research staff are women). 76 institutes ... organized into nine groups. 2 patent applications every workday Fraunhofer ranks among the 10 to 20 most active patent applicants every year. Real innovations. #WeKnowHow 90 % of our institute directors hold a professorship They combine university research with applied research and development at Fraunhofer. Over 7,600 active patent families Fraunhofer is an EU-wide leader in the Approximately 30 spin-offs per year ... over 500 since 2000. Roughly 80 percent are still active after ten years. 2.6 billion euros of revenue from contract research each year Demand from the private sector and establishment of standards. society is driving growth. 31

The institute in profile Fraunhofer Strategic Research Field Artificial Intelligence In the factory, office and everyday life, artificial intelligence systems are taking on ever more routine tasks. Technologies like machine learning are trans- forming entire industries and are playing a key role in the future transformation of our society and econo- my. Independent and comprehensive AI expertise is important so as to avoid becoming dependent on play- ers with possibly lower requirements in terms of data protection or security, for example, and to guarantee Germany’s and Europe’s competitiveness in these stra- tegically and economically crucial technologies. AI technology you can trust The Fraunhofer Strategic Research Field Artificial Intelligence (FSF AI) explores key technologies for AI-based services and products. With our comprehensive methodical and applica- tion competence, we are developing trustworthy, secure and sustainable AI technologies to help our customers solve their practical problems. Fundamental questions, such as the explainability of AI meth- ods, the systematic engineering of AI-based systems or the inte- gration of expert knowledge into machine learning methods, are focal points of our developments. The FSF AI aims to take AI technologies from concept to application; our solutions will be applied in many different fields — from autonomous driving to intelligent production plants and medical engineering. At present, the Strategic Research field AI is focus- ing on seven future-oriented topics: AI Systems Engineering – Systematic engineering of AI – Specification, planning, dimensioning of AI – Energy optimized AI Embodied AI – Robots and autonomous systems – Edge and IoT systems – Hardware/software-co-design Hybrid AI – Combining data-driven learning with knowledge and models – Learn from small, relevant data – Development and optimization of algorithms AI and the Human – Speech and dialog systems – Intelligent agents – Multimodal, collaborative systems Certified AI – Explainability, traceability – AI safeguarding (safety/security), attack defense – Standardization – Human control of AI AI and Hardware – Neuromorphic computing – Hardware/software co-design – Quantum AI and -algorithms Generative AI and Large Language Models Data and sustainability underpin each of the focus topics. The sustainable use of computing resources and data over their entire life cycle (data value chain) plays an equally important role as data quality and data sovereignty. Fraunhofer Strategic Research Field Artificial Intelligence Spokesmen Prof. Dr.-Ing. habil. Jürgen Beyerer, Fraunhofer IOSB Prof. Dr. Stefan Wrobel, Fraunhofer IAIS Deputy spokesman Prof. Dr. habil. Alexander Martin, Fraunhofer IIS 33

The institute in profile Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE, Command, control and reconnaissance Fraunhofer Institute for Applied Solid State Physics IAF, Sensors for safety, security and reconnaissance Fraunhofer Institute for Chemical Technology ICT, Security, safety and energetic materials technology Fraunhofer Institute for Applied Optics and Precision Engineering IOF, From highly secure communication to tailored laser technology Fraunhofer Institute for Transportation and Infra- structure Systems IVI, Algorithms for risk analysis, situa- tion evaluation and decision-making support Fraunhofer Institute for Technological Trend Analysis INT, Planning support for state and industry Fraunhofer Institute of Optronics, System Technol- ogies and Image Exploitation IOSB, From networked sensor data to decision Fraunhofer Institute for Experimental Software Engineering IESE, Software and systems engineering Fraunhofer Institute for Integrated Circuits IIS, Communication, positioning technologies and X-ray for safety and security applications Fraunhofer Institute for Structural Durability and Systems Reliability LBF, Secure processes for secure structures ASSTA 3.1-Tornado. Fraunhofer Segment for Defense and Security VVS Chairman Prof. Dr.-Ing. habil. Jürgen Beyerer, Fraunhofer IOSB Deputy chairman Univ.-Prof. Dr.-Ing. Dr. rer. pol. habil. Michael Lauster, Fraunhofer INT Deputy chairman Prof. Dr. rer. nat Peter Martini, Fraunhofer FKIE Managing director Dipl.-Ing. Caroline Schweitzer, Fraunhofer IOSB caroline.schweitzer@iosb.fraunhofer.de vvs.fraunhofer.de 35

Business units Artificial Intelligence and Autonomous Systems ..............................38 Automation and Digitalization ........................................................42 Defense ...........................................................................................46 Energy, Environmental and Security Systems ...................................52 Inspection and Optronic Systems .....................................................56 37

Business units Business units » Vision language models take interaction to the next level« How artificial intelligence with new sensory modalities can support humans even more effectively across different tasks and situations Dr. Voit, AI chatbots have been making waves for over a year now. How import- ant is this as a factor in the AI and Auton- omous Systems business unit? Michael Voit: ChatGPT and other generative language AIs are based on so-called large language models. We do track developments in these models and how they can be used in our application domains closely, of course. But for us as an institute working in image processing, it’s especially interesting to go one step beyond and start to look at vision language models (VLMs), which are AI systems that are not limit- ed to language as a modality. They also process visual input at the same time. So you might think of us as preferring to work with “chatbots with eyes” than with purely language-based chatbots. This kind of AI can grasp situations on a more comprehen-sive basis, much like a human combines various sensory impressions to form a holistic concept. For us, that offers an opportunity to take assistance systems and their interactions with humans to the next level. What are potential applications? Our various departments research a wide range of different use cases for VLMs. In the e-learn- ing space, we would like to feed our semantic search engine for learning content with images as well in the future. Then you could search along the lines of, “I don’t understand this figure — what learning material could help me?” In the area of military reconnaissance, the goal is to automate image exploitation and subsequent preparation of reports. And for cars, our goal is to develop voice assistants that also take what they see into account. So, the assistant could do things like tell when a vehicle occupant is struggling with motion sickness and offer tips instead of waiting to respond until the person says they have an issue. What is your approach to getting these kinds of VLMs? We don’t have the resources to create a big VLM of our own, but adjustments to existing models are possible with relatively little effort and expense via few-shot learning, meaning not a lot of training data is needed. That’s why we know and test the available models, evaluate them with an eye to our application domains and use cases, and optimize the suit- able candidates. Linking existing large models with our own specialized systems is another promising approach. In the automotive seg- ment, for example, we use our Occupant Moni- toring System, which is opti-mized to recognize poses and activity based on images, and we feed that output into a language model. It is even conceivable that we might be able to add further modalities, supplementing the system with acceleration sensors, for example. Dr.-Ing. Michael Voit Business unit spokesperson Head of HAI department Phone +49 721 6091-449 michael.voit@ iosb.fraunhofer.de 39

Artificial Intelligence and Autonomous Systems Drone flight: 3D terrain modeling to estimate biomass. Demonstration of autonomous recovery of hazardous goods in a complex scenario. FarmerSpace — a space for experimentation with new and digital approaches to crop protection Converting construction machinery for effective rescue operations FarmerSpace focuses on the use of digital technologies for crop protection. In this project, Fraunhofer IOSB-AST in Ilmenau has teamed up with the Institute for Sugar Beet Research and the section of Agricultural Engineering, both at the University of Göttingen, and with the Lower Saxony Chamber of Agriculture to investigate how leaf diseases and the spread of weeds can be detected at an early stage and targeted protective measures can be taken. Sensors, robotics, and data-driven solutions offer the potential to minimize the use of chemical crop protectants without adversely affecting crop yields. In this context, we are investigating and evaluating sensor and data transmission systems, radio sensor networks, and options for satellite communication. In particular, we are developing an innovative method of estimating biomass based on remote sensing data. Drone-supported lidar and radar sensors are specifically being used for this. This approach depends to a crucial degree on the quality of the underlying digital terrain model, which means that precision 3D surveying of cropland is an area of focus as well. With this in mind, we conducted a 3D challenge within the project to evaluate the various approaches. FarmerSpace is due to run until 2025. It is one of 14 experi- mental fields throughout Germany that are receiving funding as part of the digital transformation strategy pursued by the German Federal Ministry of Food and Agriculture (BMEL). The German Federal Office for Agriculture and Food (BLE) is the project sponsor. When major incidents such as industrial accidents or natu- ral disasters occur, rescue teams are faced with significant challenges. The rapid removal of potential hazards is crucial to the efficient search and rescue of victims. This is where our project comes in: By using robotic technology, the risk to rescue workers is minimized and rescue operations can begin at an early stage. We addressed this challenge within the now completed AKIT-PRO joint project funded by the German Federal Minis- try of Education and Research (BMBF). Our innovation lies in retrofitting construction and work machinery to convert them into automated rescue equipment: Our autonomy kit quick- ly equips locally available machines, such as excavators and tractors, with sensors and other components, turning them into unmanned recovery vehicles. Featuring functions such as autonomous navigation and 3D-based object manipulation, it provides the best possible support for rescue teams in quickly eliminating sources of danger. In the bigger picture, AKIT-PRO explored a new type of recov- ery chain consisting of highly automated recovery vehicles, support shuttles, and communication components. Thanks to its modular design, there is no need to keep special equip- ment in stock all over the world, but instead, the necessary components can be procured and qualified on site as required. The project thus makes a significant contribution to improving disaster management and protection of rescue workers. Prof. Dr.-Ing. Andreas Wenzel, head of department Embedded Intelligent Systems, phone +49 3677 461-144 www.farmerspace.uni-goettingen.de Prof. Dr.-Ing. Andreas Wenzel, head of department Embedded Intelligent Systems, phone +49 3677 461-144 www.a-kit.de 41

Business units » Data truly delivers added value when companies share it for specific purposes« Manufacturing-X and associated projects: How the digital transformation is advancing the twin causes of economic efficiency and sustainability Dr. Sauer, why should an SME think about digital twins or data ecosystems? Olaf Sauer: As we strive for sustainability and the ideal of the circular economy, every com- pany that makes products will have to disclose information about their carbon footprint going forward, and soon, they will also have to create digital product passports, if at all possible based on standards. An asset administration shell, or AAS, is one way to do this. But regula- tory matters alone aren’t a great motivation to use digital twins. Data can’t truly deliver added value until companies share it with customers, contractors, and suppliers for specific purposes. This unlocks potential we can only dream of today. For example, a material manufacturer has data on the production process and quality of their materials, which a processing company could use to set the parameters for their pro- duction equipment more efficiently and effec- tively, potentially reducing waste or accelerating the start-up process. Within data ecosystems, companies can share data according to defined rules so everyone benefits in the end. What is Manufacturing-X’s role in this? Manufacturing-X is an overarching program to build industry-specific data ecosystems for diverse sectors. Examples include mechanical engineering, with the Factory-X flagship proj- ect, Aerospace-X for the aerospace industry, and Silicon-X for semiconductors. Because all of them build on the same infrastructure, they’re interoperable, so companies only have to sign up with one of these platforms to share data or get data from others. Digital twins, or more precisely, submodels of digital twins help to standardize the sharing of data, whether that’s horizontally, meaning along the supply chain, or vertically, which refers to runtime data of machinery and equipment. On the other hand, every industry has specific use cases stemming from factors like internal industry regulations, and those require proprietary software ele- ments in the form of business applications. How does IOSB come into play? Fraunhofer is involved in building these data ecosystems on two crucial points. We develop some of the business applications, on the basis of open standards, so companies can easily incorporate them into their IT infrastructure. And we develop software services for the base infrastructure. Here, all partners rely on open source solutions because although the infra- structure is crucial to building and operating the data ecosystems, it isn’t relevant in terms of competition or customers. So everybody shares in the effort and expense of developing these services. And then we can take what we learn from these activities and put it to work in transfer and customer projects as well. Dr.-Ing. Olaf Sauer Business unit coordinator Phone +49 721 6091-477 olaf.sauer@ iosb.fraunhofer.de 43

Automation and Digitalization Our extensive experience is incorporated into a new holistic Integrated cameras in our Halodome system detect any anoma- resilience monitoring approach. lies on products like this tea pot, our demonstration object. Cyber resilience monitoring in industrial automa- tion and control systems Cyber resilience is seen as the next step in IT security and focuses primarily on incident response and restoring pro- cess capability. Appropriate monitoring of process-carrying automation systems and infrastructures is necessary to achieve cyber resilience. CyReM-ICS, an internally funded pre-compet- itive research project started in 2023, addresses this need by transferring existing approaches to anomaly detection into a holistic monitoring system. The system collects various input data within process infrastructures and uses it as a basis for calculating defined metrics for the resilience assessment of the process networks. For the target system, the necessary database for a meaning- ful resilience assessment must be defined based on a system analysis. Open-source detection systems are used to collect the needed data on network traffic and devices. Initial methods to obtain additional information by using active probing have already been integrated and will be updated successively. Col- lected information is transferred to a central data management system, a security information and event management (SIEM) system. As cyber resilience goes beyond just security and also emphasizes incident response, we are also including advisories. Predefined resilience parameters and metrics are calculated on the basis of the status information obtained from the devices and are used to evaluate and quantify the resilience status of the systems. External data sources, such as vulnerability infor- mation, are evaluated and integrated into the overall system using cyberthreat intelligence approaches. Halodome — low-cost AI- and camera-based anom- aly detection for quality assurance Quality assurance (QA) is an important part of production, helping to boost production quality, ship defect-free products, and minimize the costs of rework and repairs. But automating QA can often be a challenge. Even if it is possible, it may be expensive, time-consuming, or programming-intensive. Halodome is an AI-based solution that uses integrated cameras to quickly learn what a correct component is supposed to look like, based on just a few training examples. The system subsequently detects any anomalies and displays them intu- itively in the camera image. The human operator can check the result, override any false positives or false negatives from the AI, and even feed the final evaluation back into the system for further training. Interaction takes place via a touch-based monitor — or the detection results are projected directly onto the component, where the human operator can interact using touch gestures and adjust the results intuitively. Voice recog- nition will also be an option in the future. Halodome can be used in various QA applications. Although it is an AI system, the training process is simple because it does not require an extensive training database. In this way, Halodome enables low-cost quality assurance for new components as long as anomalies are recognizable in the camera image. The system is user-friendly and learns from comments from human operators on an ongoing basis to make it more robust. The human operator retains the final authority for the checks. Dr.-Ing. Christian Haas, group manager Industrial Cybersecurity, phone +49 721 6091-605 www.iosb.fraunhofer.de/llcs-p, www.iosb-ast.fraunhofer.de/llcs-ew Dr-Ing. Frederik Diederichs, group manager Perceptual User Interfaces, phone +49 721 6091-419 www.iosb.fraunhofer.de/halodome 45

Business units » To put innovations to work quickly, we need more agile forms of cooperation« What contributions we can make as a Fraunhofer institute to advance the transfer of research findings into real-world use within the German armed forces Dr. Sander, since Chancellor Olaf Scholz announced the “Zeitenwende”, an epoch- al tec-tonic shift, dual use research has received a lot of positive attention in Ger- many. Is the Bundeswehr on the cusp of a golden age of technological innovation? Jennifer Sander: At Fraunhofer IOSB, just like Fraunhofer as a whole, we have long stood for a holistic understanding of security that takes military, criminal, technological, and natural threats equally into account. Our research deliberately taps into synergies between the various areas, and we’re delighted if this approach, which we think is absolutely rea- sonable, has won more wide-spread support than previously. Many areas are currently giving rise to research findings and resulting technological innovations with a great deal of military relevance, so it’s important to harness those capabilities for the armed forces as well. They include artificial in-telligence, autono- mous systems, and hypersonic and quantum technologies, to give just a few examples. But for the Bundeswehr to be able to derive increasing and timely benefits from these kinds of scientific advances and technological innovations, we think there is not so much a lack of targeted research as a lack of improved ways to transfer existing research findings into real-world practice. Can you expand on that? Over and over again, we see how hard it is to translate useful findings from the field of research and technology into actual use. There are a number of structural factors that make it more difficult. For example, we’ve had past experiences where a research and technology client will say, “Great, we’ve gotten where we wanted to with this subject, now let’s put it into action” — but then we hear from the procurement side that it will be a few years before the demand actually mate- rializes or can be identified with sufficient specificity. And yet, swiftly putting the results to work could help gain greater clarity around concrete use scenarios and needs, and thus to specify and update the further requirement situation early on. If and when the procurement process does get under way, it remains lengthy and diffi- cult. Formulating the exact requirements right at the start also requires excellent knowledge of the subject matter, including in technical terms, along with an understanding of the potential and the possible usage scenarios. There also seems to be a tendency to formu- late things right from the start with all the bells and whistles, detailing every little aspect. However, it’s hugely difficult to specify a system on paper in such detail that it actually Dr. rer. nat. Jennifer Sander Deputy business unit spokesperson Head of IAS department Phone +49 721 6091-248 jennifer.sander@ iosb.fraunhofer.de 47

Defense Visualizing the temporal dimension of situation evolution Time plays a key role in military situational planning. This dimension makes it possible to organize and coordinate the sequence of military operations. The ability to optimally use and communicate the time-related aspects of planning requires suit- able tools that can take the aspect of time into account from the data storage side and also in terms of visualization. Conventional situation maps, whether analog or digital, typically present a static situation. They are snapshots of a moment in time. Often, people end up using multiple separate maps for certain points in time. However, this is not an effi- cient way to show the dynamics of the situation. By contrast, Fraunhofer IOSB’s Digital Map Table (DigLT) was designed right from the start to ensure that developments taking place over time are easy to visualize. This relates to both storing data and visualization. At the same time, changes can be captured, potentially at any time interval desired. That brings huge challenges in terms of data storage: It must be possible to write the data very quickly, as both manual changes in the situation and data from external systems may be contributing factors, and some of those systems supply their inputs very frequently. At the same time, capacity for very fast data retrieval is required in order to visualize the development of the situation dynamically, without any lag. To ensure that users have intuitive access to the temporal data kept on file, the Digital Map Table features a timeline. This function can be used to set the point in time shown by the table to any time the user wishes. It might be in the past, for instance if the goal is a retrospective assessment of the planning that has taken place to date, or — in the case of position data that are fed into the system on an ongoing basis, for example — to discuss a development after the fact. Or the time for which the table is set can be in the future, which is useful for planning future developments. Additional tools also come into play to simplify use as far as possible. Routing tools have been integrated, for example. This means not all position changes have to be made manually during planning; instead, they are automatically calculated. In the current version of the DigLT software, temporal visu- alizations are limited to annotations and sensor data. Going forward, however, advances in technology are opening up the opportunity for dynamic visualization of background data such as aerial views as well, meaning that these are updated more frequently and the dimension of time will need to be factored in for these data as well, in terms of both data stor- age and visualization. Dr.-Ing. Florian van de Camp, group manager Interactive Sytems, phone +49 721 6091-421 www.diglt.de 1 3 2 Unlike with paper, the Digital Map Table can be used to pro- duce dynamic visualizations of how a situation develops over time and then work intuitively with them using the timeline 4 feature. 49

Defense A-priori information e.g. orthophoto Measured infrared image 3D digital twin 3D thermal simulation Simulated infrared image Model-based anomalie detection Change detection Anomaly detection without consideration of simulation Schematic representation of simulation-assisted detection of human activity in IR images. A new IR reconnaissance paradigm: simulation- assisted anomaly detection Warfare in the 21st century raises new challenges with regard to acquisition, consolidation, and usage of information. With more and more urban battlefields and less distinct front- lines, well-established reconnaissance techniques need to be adapted to new kinds of information stemming from the new environments and conditions. Current developments in AI-based reconnaissance appear promising, but they require huge amounts of data available beforehand, called “training data,” and they fuel the development in adapted countermea- sures such as camouflage. To tackle these challenges, we developed a new reconnais- sance concept that aims at robustness against camouflage and applicability in urban battlefields. As operability during nighttime is important in reconnaissance, we particularly focus on the thermal infrared spectrum. The concept’s foundation lies in a shift of perspective: Instead of looking for specific objects in data, such as mines or tanks (which might be camouflaged), we generally search for traces of human activity, for example trails of heat on the ground, abandoned firepits, or irregular hot spots in an urban area. To distinguish these, some kind of expected state or background has to be defined, which is where simulation comes into play: To create a digital representation of a real urban area, we build on our expertise in methods and procedures to generate fused source data from 3D data of various characteristics and origins — including 3D data derived photogrammetrically from image and video data, results of laser scans, or conserved geodata from geographic information systems (GIS) — in combination with our expertise in physical modelling of urban heat. The resulting digital twin of the urban area includes the sur- face temperature depending on given weather conditions and enables us to render single thermal infrared images. These con- stitute the simulated, expected state which, in essence, is to be compared to actual images of the area. Deviations between the two are interpreted as human activity of any kind. Initial tests under the premise of proof-of-concept have con- firmed the feasibility of simulation-assisted reconnaissance of human activity. As expected, the results depend on both the reliability of the thermal simulation, i.e., the expected state, and the details of the image comparison approach. To incorpo- rate the power of AI-based techniques, follow-on research is focusing on the generation of synthetic training data. Eva Strauß, Geo Intelligence, phone +49 7243 992-332 www.iosb.fraunhofer.de/en/competences/image-ex- ploitation/scene-analysis/geo-intelligence.html 51

Business units » Digital solutions give us more flexibility and agility for managing crises« From flooding to human-made threats, solutions from IOSB support government agencies and public safety and security organizations at a strategic level Dr. Moßgraber, the climate crisis, coronavi- rus, or armed conflicts have made people aware of how vulnerable we are. How can digital technologies help with this? Jürgen Moßgraber: In many ways! We as business unit see our mission as being at the strategic level. We develop digital tools that support those whose job it is to manage crises, coordinate actions, and ideally identify risks ahead of time and make plans to avert or counter them. You could say we’re responsible for software solutions that are as effective as possible for the situation centers and their dia- logue with each other. In Baden-Württemberg, for example, that means between the interior ministry, regional and local government, and all the way down to the municipal level. What solutions do you mean? We study ways to mitigate the impact of the climate crisis, for example by using AI to help predict disastrous flooding and give people more time to prepare. We also develop coupled simulations for complex scenarios specifically in urban settings to optimize security measures before an emergency occurs. That’s some- thing we do at Fraunhofer SIRIOS in Berlin (see p. 10). One key task is situational awareness and assessment. Tools in this area ideally bring together all the available information, visualize it in a clear and easily understood format, and are flexible and user-friendly. Fraunhofer IOSB works continuously to develop and refine two complementary systems for this: the Electronic Situation Dashboard for Civil Protection, called ELD-BS for short, and the Digital Map Table. Both have been in real-world use for years and have proven themselves many times over. What challenges do you focus on? We think a lot about how to prepare for the next “black swan” event. One important goal is to maintain flexibility in the face of new and unexpected situations, like the coronavirus pandemic when it first hit. In practice, it can often be difficult to gauge the full extent of a situation fast enough. Take flooding, for example. Will it only affect individual houses, maybe a village or two, or will the whole area be hard hit? The best way to meet both chal- lenges is through interoperable, end-to-end solutions that connect municipal and state authorities with each other and incorporate as many data sources and resources as possible. That’s the path we’re on in Baden-Württem- berg with ELD-BS. But situations typically don’t stop at political borders, and there are hardly any comparable systems, let alone com- patible ones, in other states in Germany or at the national level or with the Bundeswehr’s homeland defense forces. There’s tremendous potential for development there. Dr.-Ing. Jürgen Moßgraber Business unit spokesperson Head of ILT department Phone +49 3677 461-102 juergen.mossgraber@ iosb.fraunhofer.de 53

Energy, Environmental and Security Systems Classification results of maritime objects. Studying UV radiation sources in the incubator. Maritime object classification Within image exploitation, one of our core competences, one traditional focus of our research is the detection of persons and the evaluation of their actions, which has multiple appli- cations in the field of public security. Other aspects include the classification of objects in various environments, and the use of computer vision in the control of unmanned systems such as unmanned aerial vehicles (UAV) and unmanned surface vehicles (USV). In early 2024, a Fraunhofer IOSB team participated in an international competition at the 2nd Workshop on Maritime Computer Vision (MaCVi) and came in a close second in the challenge “UAV-based Multi-Object Tracking with Reidentifi- cation”. The challenge focused on the detection, tracking and classification of swimmers, boats, and other objects in open water. There, waves, sun reflections and the fast movement of the airborne video camera complicate things for computer vision algorithms. However, our team was able to complete all tasks, including long-term tracking of objects despite their occasional partial occlusion. These results pave the way for the development of assistance systems for search and rescue missions, such as when ship passengers fall overboard or during flood disasters. Development and investigation of LED-based UVC radiation sources for incubators Premature birth is often associated with significant health risks to the newborn. Premature babies do not yet have a func- tioning immune system, and the skin and gut microbiomes have not yet fully formed, either, so they do not serve their protective functions. This means premature infants need to be protected from harmful pathogens such as those found in hospital settings, even with the best possible hygiene practices in place. Special incubators achieve this to a very high degree with air filtration and other measures. Still, studies have shown that harmful pathogens can enter the incubator and come into contact with the newborn unintentionally during the process of caring for these babies. To lower the microbial count in the incubator and thus raise the survival rate for premature infants, we have developed a UVC radiation source for air and surface disinfection for incu- bators. It is regularly used when the infant is taken out of the incubator and placed skin to skin on a parent’s chest as part of the “kangaroo” method of care. Meanwhile, the incubator is effectively disinfected from the inside with UVC radiation. A double-blind study is currently being conducted to investigate the device’s clinical benefits. Daniel Stadler, Image-based Reconnaissance and Surveillance, phone +49 721 6091-621 https://s.fhg.de/macvi-2024 Thomas Westerhoff, group manager Smart UV Systems, phone +49 3677 461-107 www.iosb-ast.fraunhofer.de/smart-uv-systems 55

Business units Business units » Simulated defects open the door to AI applications« How synthetic image generation can solve the “chicken or egg” problem for AI used in inspection Prof. Längle, AI is causing a lot of disrup- tion. Does that include your business unit? Thomas Längle: Machine learning (ML) has played a big role in computer vision for quite some time now. It has direct applications for us in fields such as remote sensing — so, in analyzing and interpreting aerial images — and in sorting bulk goods. One exciting trend we’re seeing right now is in the area of inspecting surfaces and complex objects, where the advances in detecting and classifying defects that we might expect from ML approaches have often been stymied by lack of data. To train the AI models, they would need to be fed a large number of examples of good products, but also be supplied with precisely labeled bad ones. There just aren’t enough of those, if any at all. So we’ve turned the focus of our efforts on synthetic image generation. You’re planning to generate the data to train the AI with AI? Well, we already do have experience with ML approaches such as generative adversarial networks, variational autoencoders, and dif- fusion models for generating new versions of bad examples on the basis of existing images. But primarily, we’re working with simulations. The idea is to simulate the entire testing and inspection environment — specimen geometry, material properties, lighting, sensor technology — to produce images that are synthetic, but still sufficiently realistic. And then we can take data on defects that we have gathered in the past to add virtual errors as well and vary them in a number of ways. With a scratch, for exam- ple, we can cycle through different lengths, depths, and shapes. On this basis, we can then calculate the images that the analysis AI would ultimately face in the chosen inspection setup. What are the advantages of this simula- tive approach? It might help us solve the “chicken or egg” problem. ML-based reproduction of images of defects requires that there be at least some images already on hand, so it still depends on the quantity and quality of the input data. We can also build in any kind of error we want, and, of course, the synthetic images created in this way are always labeled perfectly. That’s because we know which defects we simulated and where. We can also change other relevant parameters. In real life, defects aren’t the only things that vary. There’s a certain amount of spread to a specimen’s position and dimen- sions, the lighting varies somewhat, etc. Ulti- mately, the simulation gives us a digital twin of the inspection setup — which we can also vary virtually. So we can test and optimize the entire system and draw conclusions about how it will perform before translating it into hardware. Prof. Dr.-Ing. Thomas Längle Business unit spokesperson Head of SPR department Phone +49 721 6091-212 thomas.laengle@ iosb.fraunhofer.de 57

Inspection and Optronic Systems m m / y 25 20 15 10 5 0 -5 -10 -15 -20 -25 nm 141 140 139 138 137 Measurement setup and result for film thicknesses of a concave mirror. x / mm Fast-switching pneumatic valves remove plant particles contam- inated with PAs. Retroreflex ellipsometry for non-planar surfaces Conventional ellipsometry is only suitable for planar or near-planar surfaces. Measurement of nonplanar surfaces involves time-consuming alignment of the sample and only specific sampling points are measured. This constraint limits the use of ellipsometry in industry. However, quality moni- toring or characterization of nonplanar surfaces is required for different applications, such as the uniformity of metallic coatings and functional coatings on lenses (e.g., anti-reflec- tion coatings, self-cleaning coatings). We propose a holistic approach to resolve this problem: retroreflex ellipsometry. Retroreflex ellipsometry uses a retroreflective sheet that returns a light beam from the sample back along the same beam path. The polarization properties of the retroreflector are the same as those of an ideal mirror, but within an angular range of ±30°. The alignment condition for the sample and the detector is automatically fulfilled, and thus retroreflex ellipsometry can be used to measure nonplanar surfaces without the need for precise manual adjustment in order to achieve angular toler- ances of less than 0.5°. The retroreflex ellipsometer we developed is capable of deter- mining optical properties or film thicknesses of nonplanar surfaces (e.g., spherical mirror) quickly and with high accuracy. Reflectance and ellipsometric data can be measured without apriori information about materials and angle of incidence. This makes retroreflex ellipsometry an ideal tool for differently shaped surfaces as well as for inline and in situ quality control systems. Detection and removal of weeds containing pyr- rolizidine alkaloids from harvested crops Pyrrolizidine alkaloids (PAs) are secondary metabolites produced by plants to protect against natural enemies. When weeds containing PAs are harvested along with other crops, they can get into foods (lettuce and other leafy greens, herbs, teas) or plant-based drugs. This kind of contamination poses a potential health risk due to the liver toxicity and genotoxicity of these compounds. Just a few plants per hectare can be enough to make the entire harvest commercially unviable. The only way to fight weeds containing PAs in the field is generally mechanically, which is costly in terms of personnel. Increasingly, there is no way for growers to do this economically. Imported dried goods often contain higher than approved levels of PAs as well. To remove plant parts containing PAs from the product after the fact, Fraunhofer IOSB has developed a prototype of a sorting system as part of a joint research project funded by the German Federal Ministry of Food and Agriculture (BMEL) and Fachagen- tur für Nachwachsende Rohstoffe e. V. (FNR). The automated system detects the toxic PA weeds based on hyperspectral imaging sensors in the short-wave infrared range. An AI-based classification model was trained using hyperspectral data. The deep neural network, which was optimized for real-time opera- tion, achieved detection rates of over 90 %. An automated sorting system was optimized for sorting small, dried plant particles with an eye to material handling and other sorting parameters. This makes it possible to efficiently clean heavily contaminated plant streams through multiple sorting passes: Just three passes reduce PA content by more than 99 %. Dr.-Ing. Matthias Hartrumpf, Visual Inspection Systems, phone + 49 721 6091-444 www.iosb.fraunhofer.de/rre Dr.-Ing. Georg Maier, group manager Sensor-Based Sorting Systems, phone +49 721 6091-649 www.iosb.fraunhofer.de/pa-detektion 59

Departments Optronics Laser Technology | LAS ....................................................................62 Optronics | OPT ................................................................................64 Signatorics | SIG ...............................................................................66 Visual Inspection Systems | SPR ........................................................68 System Technologies Cognitive Energy Systems | KES ........................................................ 70 Cognitive Industrial Systems | KIS ..................................................... 70 Digital Infrastructure | DIS ................................................................ 74 Embedded Intelligent Systems | EIS .................................................. 74 Information Management and Production Control | ILT ....................78 Interoperability and Assistance Systems | IAS ...................................80 Machine Intelligence | MIT ...............................................................82 Systems for Measurement, Control and Diagnosis | MRD .................84 Underwater Robotics | UWR ............................................................86 Image Exploitation Human-AI Interaction | HAI .............................................................86 Object Recognition | OBJ .................................................................90 Scene Analysis | SZA ........................................................................ 92 Video Exploitation Systems | VID ......................................................94 Variable Image Acquisition and Processing | VBV .............................96 61

are realized to allow for utmost precision and robustness needed from research to fieldable laser systems. Highlight topics Expansion to active fiber research for security of supply In April 2022, the city of Oberkochen (Germany) granted the Laser Technology department 10 million euros in funding for research and development of active silica fibers. The definition of the technical and scientific infrastructure has been finalized, and equipment procurement is underway. The commissioning of the equipment and the start of research activities coincide with the completion of construction work on the new Hensoldt Optronics building in Oberko- chen, where Fraunhofer IOSB will be renting around 500 m² of cleanroom space. Recruit- ment has begun. In June 2023, the city of Oberkochen renewed its confidence in the Fraunhofer IOSB laser strategy and granted a second 10 million euros in funding for the creation of an additional research laboratory, this time dedicated to fluoride fibers. These fibers enable the research and development of new MWIR lasers and non-linear sources. An important facet of the research is focused on fiber optic components needed for inte- grated lasers and systems; those available on the market often being limited in security of supply or unreliable for the energy or power ranges addressed by Fraunhofer IOSB. High-power fiber lasers and environmen- tally hardened laser designs The demonstration of a monolithic (all-fiber) continuous wave Tm:silica fiber amplifier with a narrow line width and 937 W of output power with a nearly diffraction limited beam quality of M² ~ 1.2 takes the laboratory to the very top of international research in 2 µm fiber lasers, the world record being 1.1 kW. Moreover, this IOSB fiber laser is ahead of all comparable designs in terms of compactness and simplicity and achieves a very high differ- ential efficiency of 58 %, which is important Departments for thermal management and integration aspects. Ongoing research into coherent combination of multiple sources of this type is targeting power scaling for high-energy laser (HEL) applications. Ruggedized thulium fiber laser for military applications. To address specific military needs in rugge- dized laser designs, the LAS department has also put a focus on specific realizations of fiber and solid-state lasers for extended oper- ating temperature ranges or reduced align- ment sensitivity. Therefore, passively-cooled Tm:fiber lasers with ultra-low thermal sensitiv- ity or beam-quality-preserving highly effi- cient Ho:YAG amplifiers with average output powers above 120 W have been designed and successfully implemented. Fiber spool, fiber drop, and fiber preform. 63

Departments » We develop and evaluate electro-optical systems.« Compressive sensing image of Fremersberg tower near Baden-Baden, Germany, at a distance of 26.5 km. Links to OSIS: [1] https://s.fhg.de/trm4 [2] www.ecomos.org 65 Highlight topics Image simulation for optronic systems The Optronic System Imaging Simulator (OSIS) is a software tool used for the image simulation of camera-related image degradation effects. These include optical diffraction, aberration, sensor sampling, temporal and fixed pattern noise, digital signal processing, and interpola- tion on screen. For convenience, applied filter modules for image degradations are controlled by camera specifications within the TRM4 [1] software, which is the German reference model for range performance assessment developed by Fraunhofer IOSB. OSIS is provided as a dynamic link library, which enables easy integration into other third-party software. Parts of the OSIS software’s capability are integrated in the ECOMOS-2 project [2] of the European Defense Agency. ECOMOS-2 is a novel approach for electro-optical system performance assessment that will enable eval- uation of the effect of advanced digital signal processing algorithms on imager performance. For investigating individual camera properties, users can retrieve simulated imagery at differ- ent stages of the image chain. Furthermore, there is a plug-in interface for advanced digital signal processing. It enables the application of algorithms for image and video signal processing developed by the user, such as contrast and edge enhancement, digital image stabilization, and noise reduction. Synthetic image degradation by OSIS opens up a wide range of possibilities for data aug- mentation on small datasets of input scene imagery, which can be crucial for robust AI applications such as image enhancement or automatic and aided target recognition. 3D remote sensing with structured light Imaging under adverse conditions, i.e., through fog, smoke or fire, demands imaging tech- niques capable of returning high-resolution images through scattering and turbulent media. Active imaging systems that use their own light source for illumination are ideally suited for this task. A key advantage of such systems is direct control over the light source, which allows the utilization of time-of-flight (ToF) information of light to suppress unwanted background noise and obtain depth resolution. The development of focal plane arrays has advanced significantly, featuring high counts of pixels with time resolution. However, this progress is limited by very high data transfer rates of highly resolved 3D data that are in the multi-gigabyte per second range. As a result, data compression has become a crucial tool to transfer key information close to real time. By using structured light for illumination, it is possible to compress data even before detection. Using compressed sensing meth- ods, image data can be reduced by an order of magnitude without compromising visual quality and depth precision. Limited data bandwidths can thus be used most efficiently for real-time imaging of dynamic scenes. Moreover, structured light offers the benefit of requiring only a single-element detector, allowing the use of high-end technology with low noise and time-resolution capabilities across any spectral range. This eliminates the need for developing new camera models featuring arrays of millions of identical, less specified individual pixels, an outcome often necessitated by array implementation.

Departments Free-space optical communication lab with view of Ettlingen. Main sensor systems of the I3R mea- suring system aboard a small ship during sea trial. 67 Highlight topics Free-space optical communications Based on our experience in laser beam propagation through turbulence, we design, implement, and test free-space, laser-based telecommunications concepts. Our laser communications laboratory (LCL) has two free-space paths that either originate or terminate there. The first path is an 800- meter double-pass link to a retroreflector and back to the receiver. Telecommunications experiments have been performed over this path, since 2018. Data rates have continuous- ly increased: from 5 Gbit/s in 2020 to 12.5 Gbit/s as of 2023. As a reference, top-of- the-range home Internet delivered by fiber optic cable in Germany is currently limited at 1 Gbit/s. One of our approaches is called “coher- ent modulation”. While in traditional on/ off keying only the laser’s intensity varies, we independently modulate the amplitude and phase of the signal. As a result, sev- eral bits can be sent at once with a single laser “pulse”. With polarization diversity, an additional data channel could be opened. We currently transmit four bits per symbol. At the same time, we investigate how to mit- igate the effects of atmospheric turbulence on the quality of the transmitted data. With the techniques implemented in the LCL, we have achieved error-free transmission over 95 % of the time. The goal is to increase this percentage to 99 % in the coming years to meet industry requirements. The second link to be commissioned is a single-pass laser propagation path spanning 7.2 km from a Karlsruhe Institute of Technol- ogy building to the LCL. This will be a fixed facility dedicated to research on the effects of atmospheric turbulence on propagation of Gaussian and other types of laser beams. Shipborne IR signature measurements with the I3R system Field experiments to determine an IR object signature become necessary when measure- ments in the laboratory are not possible or not sufficient; for example, to capture larger objects under real conditions in order to obtain comparative data sets for validation of IR scene simulations. This type of measurement requires accurate characterization of sensor systems, target, and environment, to ensure a high-quality result. This is a challenge at sea, particularly when investigating maritime objects from a ship. Our integrated imaging IR (I3R) measurement system makes a multi-spectral investigation at sea a solvable problem. It is equipped with two sensitive quantum detectors in the LWIR and MWIR spectral range, which are supple- mented by sensor systems for the visual and short-wave IR (SWIR) spectrum. Encased in robust housings and mounted on a stabilized pan/tilt unit (PTU), they can withstand even adverse conditions at sea. An environmental measuring station and systems for position and attitude determination are also con- nected. All data from the subsystems is fed into the system’s central control unit. This provides the user with an integrated view of all relevant data and allows them to control the system. The system supports the user by displaying deviations in the measurement process and by automatically tracking the target object. It gives the user the freedom to precisely observe and coordinate the mea- surement, which further increases the quality, efficiency, and probability of success of the measurement.

Departments controls of fresh produce − a process that is usually time-consuming and destructive. In a collaboration with Fraunhofer IVV and IZFP, we developed a multisensory scanner for non-destructive quality detection of fresh food. A cooperation with a fruit and vege- table wholesaler made it possible to adapt the scanner directly to the requirements in a goods receipt. By viewing entire crates at once, a time-saving, workflow-integrated process was established. The multimodal system incorporates various sensors that enable fast data assessment, including a multispectral near-infrared (NIR) camera. Due to the tendency of plant tissue to reflect light in the near-infrared range, certain properties become more visible with NIR imaging. Equipped with various sensors, the scanner can not only extend the range of measurable parameters, but information is also standardized and comparable over time since the scanner captures data under con- stant conditions. All of this happens with just one touch of a button. Within the wholesale trade partnership, the scanner was used to acquire real-world data. The dataset serves to train machine learning models, enabling the integrated data analysis to be flexible and data driven. This is essential in addressing the challenges posed by the high variability in fruit and vegetable charac- teristics, which are difficult to detect using traditional methods. Detecting and removing hazardous materials from construction and demoli- tion waste Manufacturing of new construction materials is responsible for about 11 % of global green- house gas emissions. One climate-friendlier way of producing construction materials is to recycle construction debris from demolished structures. Producing high-quality recycling products generally requires that the material is recovered as pure as possible, with impu- rities sorted out. In the past, we worked with Fraunhofer IBP to develop sensor-based sorting solutions to fill this need. In addition to needing materials to be sorted properly, harmful substances that could be contained in construction debris present a serious problem. These kinds of materials need to be detected, separated, and han- dled individually, all at an early stage of the process. There are also legal limits to observe in many cases. One example is asbestos, which was com- monly used in older buildings and other structures for insulation, fire prevention, and stabilization. If asbestos fibers in the air enter the human lungs, physical harm can result. With this in mind, we are developing a nondestructive, low-cost method of detecting asbestos using imaging sensors as part of the DIANA project. This makes it possible to detect asbestos fibers early on and separate them from uncontaminated material. Hazardous materials also play a key role in the recycling of road surfacing. Tar was used as a binder in Germany up until the 1980s, but it contains toxic hydrocarbon compounds. In the InnoTeer project, we are working on a method of using a hyperspectral imaging camera in the mid-infrared range to detect binders that contain tar. This will make it pos- sible to recycle non-hazardous road surfacing materials containing bitumen. State-of-the-art camera technology is used to acquire image data in the hyperspec- tral imaging lab for training AI models. The AI-based analysis of stan- dard fruit crates is performed directly on the multisensory scanner. 69

data-X” project, we will develop and test a data ecosystem for the sustainable, secure, and sovereign use of data in the energy sector. We are also involved in the Thuringian joint project “ZO.RRO 2 — Zero Carbon Cross Energy System for Glass Industry”. The research projects are funded by the German Federal Ministry for Economic Affairs and Climate Action. Self-learning AI firewalls for energy supply infrastructures In the PROTECT project, we are developing a learning and testing system to build and implement a self-learning AI firewall for energy suppliers: We expand traditional rule- based firewall systems with AI-based anom- aly detection methods to defend against IT attacks in the commercial IT domain as well as in the process IT domain. The focus is on the (semi-)automated detection of manipulations in network traces to assess the criticality of anomalies in terms of their potential negative impact on essential system structures or com- ponents (e.g., network assets, IT components, and communication paths). To this end, we develop an agent-based network simulation in GNS3 to model business services, including mail, web and metering activities, using sto- chastic working profiles. A coordinated attack simulation framework will be integrated to model multi-stage cyberattacks and create dedicated training and test datasets. The transparent attack detection system is realized with a transformer deep learning architecture and an innovative explanation algorithm in order to identify the most relevant network packets and packet information with a given network traffic sequence. EU project: Next level Quantum infor- mation processing for Science and Technology The vision of the NeQST (next level quan- tum information processing for science and technology) project, funded by the European Union, is to leverage recent advances in the control of qudits (multi-dimensional quantum information carriers) in order to generate foundational breakthroughs throughout the entire value chain of quantum computing. These include an experimentally tested qudit platform based on trapped ions, validated automated design tools, tailored certification methods, and demonstrated feasibility of practical applications. The emphasis is on two use cases of great interest for academia and industry: quantum simulation of lattice gauge theories and quantum optimization for the energy domain, respectively. We, the IOSB Quantum computing experts located in Görlitz, focus on the latter, particularly on optimizing the charging schedule of electric vehicles (EVs). Optimized bidirectional EV charging, i.e., optimized charging and discharging of the batteries of large numbers of EVs, provides a great opportunity to address two of the challenges arising from the transformation of the energy system: the integration of the growing number of EVs and the rising share of renewable energy. Both cause a sizeable increase in the load of the electricity grid as well as greater uncertainty and variability in electricity production and consumption. To address these challenges, we develop novel quantum algorithms and technical approaches for distributed operations with locally opti- mized charging including uncertainties. Departments Basic concept of agent-based network simulation and trans- parent attack detection. Quantum optimization with focus on optimizing the charging schedule of electric vehicles (EVs). 71

Departments PipesAI framework for model- ing and implementing AI data pipelines. open62541 | Professional Service Offering Professional Services Community (Free Tier) Open Source development (Issues, Pull Requests) Support on Github and the Mailing List Regular community call for the coordination of ongoing development Professional Long term support for past releases Access to certification artifacts Security Patch Management (Coordinated Vulnerability Disclosure) Access to developer training material (videos and slides) Included developer-days/year for fast-tracked Open Source development Enterprise Assistance in certification efforts using open62541 Backporting of requested features to past release families On-Site Developer Training for open62541 Included developer-days/year for fast-tracked Open Source development Full-Featured with Server, Client and PubSub Code Quality Management Open Source Access Control, Crypto (OpenSSL, mbedTLS), Information Model Storage, Historical Data, Network EventLoop Customization and Integration of Server/Client/PubSub into an overall application. open62541 Plugin API MPLv2 open62541 Server SDK Service Implementation Namespace Zero open62541 Client SDK open62541 PubSub SDK Service Calls Subscription Handling UDP, MQTT, Ethernet Runtime Configuration in Information Model Application Information Model Async API Encryption (SKS possible) OPC UA DataType Handling, Encoding (Binary, JSON), SecureChannel open62541 Core Stack open62541 Architecture Integration Processor Architecture, Clock, Networking, etc. Contact o p e n 6 2 5 4 1 . o r g Dr.-Ing. Julius Pfrommer Head of Department Cognitive Industrial Systems (KIS) Tel. +49 721 6091-286 julius.pfrommer@iosb.fraunhofer.de Andreas Ebner Head of Research Group Adaptive Production Systems Tel. +49 721 6091-532 andreas.ebner@iosb.fraunhofer.de Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB Fraunhoferstraße 1 76131 Karlsruhe www.iosb.fraunhofer.de Coverimage: iStock-1017193718 gremlin Long term support for past releases Access to certification artifacts Security patch management (coordinated vulnerability disclosure) On-site developer training Website For more information, see www.iosb.fraunhofer.de/ open62541 73 Open Source OPC UA Professional Service Offering Full Featured Open Source open62541 is an open source and free implementation of OPC UA (OPC Unified Architecture) written in the common subset of the C99 and C++98 languages. The library provides the neces- sary tools to implement dedicated OPC UA clients and servers, or to integrate OPC UA-based communication into existing applications. The open62541 library is platform independent. All platform-specific functionality is implemented via exchange- able plugins. The SDK is licensed under the Mozilla Public License v2.0 (MPLv2). This allows the open62541 library to be combined and distributed with any proprietary software. Only changes to the open62541 library itself need to be licensed under the MPLv2 when copied and distributed. The open62541 project is main- tained by Fraunhofer IOSB. Many companies and individuals are active users and contributors of the open62541 SDK. Product Features Server Support for all OPC UA services (except the Query service – not implemented by any SDK) Support for generating information models and types from standard XML definitions (Nodeset Compiler) Highly configurable with default plugins e.g. for encryption (OpenSSL, mbedTLS), access control, historizing, logging Support for editing nodes and references at runtime Support for subscriptions (data-change and event notifications) Portable C99 Client Support for all OPC UA services Support for asynchronous service requests Background handling of subscriptions Multi-Platform PubSub Support PubSub message encoding (binary and JSON) Transport over UDP-multicast, Ethernet, MQTT Runtime configuration via the information model Large Developer Community C99 for efficiency and speed Drift detection and long-term productive use of AI components In collaboration with our industrial partners, we not only validate the use of AI methods in prototypes, but also turn them into industri- alized applications for long-term productive use. This experience led to the development of our PipesAI tool suite which is a light- weight framework to structure and build AI training pipelines with the aim to consistently track changes intermediate results. Further- more, our AutoDrift package empowers users to detect drifts during long-term operations of AI and automatically take appropriatemea- sures for model adaption. open62541 OPC UA open62541 is a fully featured, open-source, free implementation of OPC UA (OPC Unified Architecture). It is written in the C language, making the software extremely resource efficient and suitable for all types of hardware, including small embedded systems. The library provides the necessary tools to implement dedicated OPC UA clients and servers or to integrate OPC UA-based communication into existing applications. The KIS department leads the development of open62541. As core maintainers of this open-source initiative, we also review and support many contributions from industry. For commercial users, we provide professional services, including:

Departments the identification of weak spots and facilitate quick and efficient improvements. Under- standing the correlation between a machine’s operating status and energy usage allows for more efficient machine control and better decision-making with regard to when to power it down. The instruments have been applied success- fully in medium-sized companies including IWN GmbH & Co. KG as well as in large companies such as Phoenix Contact GmbH & Co. KG. In 2020, VDMA e. V. — the German mechanical and plant engineering association — selected us to write the “Guideline Retrofit for Industrie 4.0” publication, based on our expertise in the field. The guideline aims to disseminate the knowledge on enabling sys- tematic data acquisition and analysis for aged machinery. Testing of interoperability and migration of convergent IT/OT real-time communi- cation networks Communication in industrial automation is changing from the classic hierarchy auto- mation pyramid to the Industrial Internet of Things (IIoT). With the associated interopera- bility between IT and OT, convergent real-time communication networks can be realized that reach a new level of complexity in terms of the required range of functions and the nec- essary configuration based on IEEE standards, for example. Deterministic communication in factory auto- mation with TSN and 5G. The Fraunhofer IOSB-INA 5G Application Center (www.5g-anwendungszentrum.de, headed by Timo Siekmann) and the TSN Test Lab (www.tsn-testlabor.de, headed by Alexander Biendarra) address these topics and support their partners and customers in topics such as 5G, 6G, wireless local area networks (WLAN), PROFINET, OPC UA FX, time-sensitive networking (TSN) and any other kind of real-time communication solution. In our test labs and the SmartFac- toryOWL, we can conduct interoperability tests, performance evaluations of implemen- tations or implementations of new protocol standards. Demonstrators in the SmartFac- toryOWL show the potential of industrial Ethernet protocols via 5G routes and the convergent use of several industrial real-time Ethernet protocols with simultaneous net- work load and transmission of video streams in a single communication network. Functional safety for dynamic production environments With its research focus on machine safety for adaptive modular production, Fraunhofer IOSB-INA addresses new challenges in the field of functional safety. Cross-cutting topics include: INArice traverse module equipped with sensors, mon- itoring an injection molding machine, as an example of contactless data acquisition using optical sensors. the automation of hazard analysis and advanced risk assessment to define safety functions and ensure safe operation with plug-and-produce. The necessary funda- mental procedure was researched in the AutoS² project financed by our regional cluster of excellence “it’s OWL”. The application and use of innovative tech- nologies in functional safety, for example, the functionally safe use of AI in sensor technology for object classification for safe person detection and object classification for AGVs, HRC (CoBots), and perception systems. However, we are also interest- ed in using known technologies in new applications, such as gyro stabilization for innovative autonomous monorail vehicles (MONOCAB). Upcoming standards and regulations — we stay up to date on changes and new possibilities in order to help our customers develop their product strategy. If you want to use machine safety as a competitive advantage in the future, please contact our functional safety engineer Philip Kleen, philip.kleen@iosb-ina.fraunhofer.de. The protective devices of functional safety are automatically analyzed and linked at the adaptive matrix production in the SmartFactoryOWL. 75

Departments Retrofitting mobile machines with environmental sensing capabilities: the Environment Perception Kit in use. Innovative UV disinfection solutions can reduce the use of antibiotics in poultry farming. 77 and manipulation. This compact all-in-one solution incorporates the sensors crucial for this purpose, such as a multilevel 3D laser scanner, 2D Full HD camera, and dual antenna GNSS receiver with inertial measurement unit (IMU), and is easy to add on to existing vehicles. Bio- und Umwelttechnologien e. V. (GMBU), we are developing a compact device that combines UV disinfection, photocatalysis, and particle filtration. It is able to remove both germs and harmful chemical substances (ammonia, hydrogen sulfide, volatile organic compounds) from the air inside barns. The device is designed to allow for rapid, flex- ible integration into existing poultry farming operations. It uses UVC LEDs to continually disinfect the air in the barn. The ultra-short- wave radiation has a powerful microbiocidal effect, and the relatively new LED technol- ogy offers advantages over mercury vapor lamps in that it is non-toxic and resistant to vibrations and emits an especially effective wavelength spectrum. It is suited to closed barns and can therefore also be used in piggeries — and unlike antibiotics, it could also prove useful against viral diseases. One of the research questions is how to optimize photocatalysis to reduce harmful compounds, which has thus far mainly been the purview of UVA. The project is receiving funding from the German Federal Ministry of Food and Agriculture (BMEL). The ENVP Kit stands out for its rugged construction, flexible energy supply (9–36 V), IP67 protection class, and compact dimen- sions (830 x 1,100 x 320 mm and just 35 kg in weight). External sensors can easily be integrated via Ethernet and supplied with electricity to adjust the kit to specific research questions. An optional industrial PC, including 5G modem and Wi-Fi, can be added to the kit to maximize connectivity and processing power. A number of existing algorithms can be used with the ENVP Kit, enabling efficient implementation and adjustment directly on the platform. This means the ENVP Kit offers researchers and developers a fast, convenient entry point for development of autonomous driving and working functions. Poultry farming: UV disinfection reduces antibiotic use Improving air quality for enhanced animal wel- fare, achieving lasting reductions in the use of antibiotics — and thus also the emergence and spread of multidrug-resistant microbes: Those are our goals in the “ DesGefUV” project. In partnership with PURION GmbH and Gesellschaft zur Förderung von Medizin-,

Departments AI for water management with SensorThings API. 79 standards such as the OGC SensorThings API (STA) as a basis for the flexible integration of machine learning algorithms in the water domain. With the FROST® Server, we offer an open-source implementation of the STA for sensor data management and investigate the integration of AI methods via the STA “Tasking Core” extension of the STA using our PERMA® toolkit. In the NiMo 4.0 project, AI methods are used to predict nitrate levels in groundwater. A data infrastructure based on the FROST® Server was created using data from five German states. The project partners developed sev- eral AI algorithms related to nitrate, such as nitrate regionalization, network optimization, and event detection. These algorithms were successfully integrated into the NiMo 4.0 data infrastructure via PERMA®. Measurement data and prediction results are accessible through open standards and can be integrated into various end-user platforms. The TETRA project develops methods that simplify and accelerate the use of AI in water protection, with a focus on protecting water resources and hydrological ecosystems. The data infrastructure is tested in two different use cases in the German-French region of the Rhine: First, the quality of the river will be measured using new image-based sensors that employ integrated AI procedures. Second, integrated AI models will support the gener- ation of recommendations for the restoration of hydrological ecosystems. Certificate management for OPC UA networks The secure operation of modern industrial networks is a major challenge. Communica- tion protocols that allow for protected and authenticated connections are needed to secure these networks. OPC UA, for example, is one such secure, industrial communication protocol. With OPC UA, as with many other protocols, these connections are secured through the use of certificates that enable secure authentication of communication participants. In practice, however, their use requires configuration and management efforts. To keep this effort as low as possible, the OPC UA standard already includes functions for managing these certificates, even integrated into the protocol. A “global discovery server” (GDS) with certificate management support is required to provide this in practice. Our open62541-based GDS is an implemen- tation of such a global discovery server with certificate management. It offers the man- agement of registered applications in an OPC UA network, as well as the management and distribution of certificates and trust lists. The GDS is based on the open-source open62541 implementation of the OPC UA standard and will be published as open-source software.

Departments Highlight topics Demand-driven provision of information for Joint ISR/Multi-Domain Operations Intelligence, Surveillance, and Reconnais- sance (ISR) across different armed forces and countries helps to provide key information on time for a joint picture of the situation. This is done using (NATO) standards. This approach makes it possible to ensure that information reaches the right people, as required, and that those people can interpret the infor- mation and process it in line with their tasks using software applications. The IAS department works at several levels in this field. It supports entities that require assis- tance in aligning operational processes with technical standards and provides feedback on technical feasibility. IAS is also actively involved in various committees in the field of standard- ization and develops formats, interfaces, and services. One essential part of this is keeping an eye on current trends and developments, including IT security, and evaluating and incor- porating them for future solutions. According- ly, IAS conceptualizes and develops software solutions for data and information distribution, report generation, and ISR management, which are also incorporated into operational overall systems. For a clearer picture of the work done by IAS, the graphic below shows how Coalition Shared Data can be used to provide informa- tion in a distributed reconnaissance network. Data from different sensors (and reconnais- sance disciplines) are provided for evaluation and analysis at different locations and by different countries. Plans call for effectively integrating the findings on interoperable data and information provision into other projects and initiatives, such as Multi-Domain Operations. Unmanned autonomous vehicles With threats on the rise internationally, pro- tecting critical infrastructures is growing more and more important. Ensuring long-term, across-the-board protection for maritime infrastructure such as liquid natural gas ter- minals and offshore wind farms is an area of special attention. This is the objective for the HUGIN (“Heterogeneous Unmanned Group of Intelligent Watercraft”) demonstrator, which was jointly realized by Fraunhofer IOSB and the Fraunhofer Center for Maritime Logistics (CML) in Hamburg. Intelligent, heterogeneous unmanned surface vehicles organized into a team and equipped with different sensors are responsible for surveillance. In addition to the unmanned vehicles themselves, HUGIN consists of intelligence-generating software made up of several layers and components. The planning AI is responsible for situ- ational planning and for allocating the available resources to the tasks at hand, autonomously analyzing the situation, and responding to changing requirements, along with flexible planning of missions (patrols, inspections, etc.). The responsive AI ensures intelligent control of the heterogeneous collaborative platforms (of the unmanned surface vehi- cles) on a multi-agent basis. The control system includes a number of robust control units that keep the vehi- cle on its planned trajectory. The control system also allows the operator to monitor the mission from the control station. The evaluation layer is responsible for automatically detecting and classifying ships and boats and for correlating the classified ships with the data from the general public automatic identification system (AIS). HUGIN makes it possible to protect critical maritime infrastructures 24/7, lowers costs in comparison to manned vehicles, and saves time due to the distributed use of multiple platforms and parallel processing of multiple tasks. HUGIN team in action. Provision of information in a CSD network. CSD CSD Reachback Location D CSD User User Location B Location F Location A CSD Location C CSD Location E Location G 81 User User User User User

Departments Dr. Jürgen Jasperneite, Direc- tor of Fraunhofer IOSB-INA in Lemgo, Dr. Korbinian von Blanckenburg, Dean of the Department of Economics at TH OWL, and Dr. Oliver Nie- hörster, Head of the Machine Intelligence department at IOSB-INA, are delighted about the collaboration. and TH OWL are combining an economic perspective with more than ten years of expertise in intelligent automation to enable SMEs to generate more benefits from this data by developing service-centered business models — for example with the aid of artificial intelligence. Software development for industrial digital twins Our department has been contributing to the development of industrial digital twins based on international standards such as the asset administration shell (AAS) and OPC Unified Architecture (OPC UA) for more than ten years. We model necessary semantic descrip- tions of production systems, components, and their communication, e.g., using OPC UA technology, and work on open-source soft- ware in a development project together with the Industrial Digital Twin Association (IDTA). Here, the specific focus is on the further development of an open-source server, as well as the official IDTA Engineering tool for the AAS based on C# .NET. The goal of the project is to utilize reinforce- ment learning for production planning. One of the project’s highlights was the successful application of reinforcement learning to a complex real-world use case: a two-stage permutation flow shop scheduling problem (PFSSP) with a finite buffer and sequence-de- pendent setup efforts in a household appli- ance production operation at Miele. The objective was to minimize the idle times of the final assembly lines and the setup effort. The results obtained were excellent, with rein- forcement learning outperforming all other methods [1, 2]. The next step in the project is to combine reinforcement learning with auto- mated generated simulation models, which will enable the scalability of this approach. Data-driven business: new customer benefits through data utilization Data-based value creation models are becom- ing increasingly important as the amount of data generated in the automated industrial production environment grows. Therefore, Fraunhofer IOSB-INA and the Department of Economics at the OWL University of Applied Sciences (TH OWL) have established a new research area called “data-driven business” at the Innovation Campus in Lemgo. The aim is to develop new business models based on the added value of product and production data, primarily from SMEs. Fraunhofer IOSB-INA 1 A. Müller, F. Grumbach, and F. Kattenstroth, “Reinforcement Learning for Two-Stage Permutation Flow Shop Scheduling — A Real-World Application in House- hold Appliance Production,” in IEEE Access, 2024, doi: 10.1109/ACCESS.2024.3355269 2 Grumbach, F.; Müller, A.; Vollenkemper, L.: Robust Human-centered Assembly Line Scheduling with Reinforcement Learning. In: International Conference on Dynamics in Logistics (LDIC). 2024 83

Departments The autonomous excavator ALICE loads a transport vehicle. AVEAS — situation analysis based on aerial images. TAPS — autonomous sensor carrier. Websites www.aveas.org www.octane.org 85 systems are equipped with numerous sensors and actuators and offer extensive possibilities for data collection and the development of automated driving functions. Highlight topics TAPS — underwater and surface mapping of rivers and lakes Accurately surveying water bodies such as rivers or quarry ponds is a challenging task. Authorities and private port operators need an inventory of current maps, such as of river- beds or port facilities. This typically requires specialized mapping vessels and trained personnel. This is a costly and time-consuming process, and in many cases mapping is done too infrequently. Autonomous surface vessels (ASVs) offer a solution to this problem, significantly reducing costs and the need for on-board personnel. As part of a three-year in-house research project, we have developed an autonomous surface vessel that is capable of autonomous- ly surveying a body of water to generate a corresponding 3D map. It uses sonar to scan the riverbed, and optical systems to scan the shoreline and overwater structures. This information is then fused and transferred into a common 3D model of the environment. The knowledge gained from the project can be applied in a variety of ways. Watercraft can be used for passenger and freight transport on the high seas, on inland waterways, or for logistics chains involving waterways. This also includes the excavation of waterways and the autonomous surveying of shipping channels. AVEAS — survey, analyze, and simulate traffic situations relevant to safeguarding The AVEAS project (www.aveas.org) aims to develop scalable methods for data-driven virtual testing of automated and autonomous driving functions. For the quantitative safety validation of automated and AI-based driving functions in simulations, these simulations must model the real world accurately in critical scenarios. This includes simulating the behav- ior of human traffic participants in risky and near-accident scenarios based on real-world behavioral data. The goal of AVEAS is to demonstrate how to systematically acquire such real-world data of critical scenarios at diverse accident hotspots across Germany, how to create data-driven behavior models based on the acquired data, and how to enable the quantitative virtual safety testing of automated driving functions. To this end, AVEAS brings together partners from AI and data analysis (UnderstandAI, KIT, EDI), the automotive industry (Porsche Engineering, Continental, dSPACE), simula- tion (PTV, Fraunhofer EMI, Fraunhofer IOSB), insurance and safety (Allianz Center for Technology, Fraunhofer IVI), human-machine interaction (GOTECH, Spiegel Institut), and standardization/regulation (ASAM, ADAC, TÜV). Fraunhofer IOSB contributes the traffic data acquisition from aerial images and its OCTANE simulation framework (www.octane. org). The acquired traffic data is scheduled to be released as FAIR data by the end of the project. It can be used for the development of simulation models, testing automated driving functions, and traffic safety studies.

unit consisting of a stereo camera system and an imaging sonar. This approach combines optical data from the near field with acoustic data from the far field into a coupled hybrid system. On this basis, all three UFO variants should be used with standardized automatic pattern recognition algorithms and later be optimized so that the system can determine species, size, and weight with at least 95 percent certainty. The UWR department was responsible for designing, building, and evaluating the mobile UFO. One main feature of the new underwa- ter vehicle is its ability to be used both as a remotely operated and an autonomous vehicle due to the integration of a new, compact, and highly precise navigation sensor in the vehicle. Additionally, we developed new con- trol algorithms to optimally use the high-end sensor and enable new vehicle modes for fish monitoring, such as slowly following a species after detection in the hybrid algorithm. The developed software also includes an intuitive planning tool for autonomous missions with different mission modes and the definition of an exploration area that must not be left, in order to prevent system loss. The basic functions were tested on board several research and fishery vessels on the Baltic Sea in 2022 and 2023. The mobile UFO collected a high volume of sensor data to be processed by the AI algorithms for species detection, classification, and size determina- tion. The system has already garnered interest from industry customers with regard to biodi- versity monitoring of their offshore facilities. Infrastructure and lab facilities We have a 12 x 8 x 3 meter water tank and a 1,200-bar pressure chamber for testing our maritime systems and components. Our diving drones as well as our infrastructure are available for client-specific projects such as integrating new sensors, collecting data in real environments, or validating sensors and com- ponents in the test tanks of the department. The department runs two fully operational ROVs. The ROVs are very agile, remotely oper- ated underwater vehicles with diving depths of up to 150 meters. They can handle a wide range of tasks thanks to their flexible design and various interface options. Multiple sensors are available for underwater navigation and environmental sensing. The DEDAVE 2000 is an autonomous under- water vehicle (AUV) with a diving depth of up to 2,000 meters and a large payload range with versatile sensor equipment. Highlight topic Automated monitoring of fish stocks Fish stock monitoring for commercially important fish populations typically relies on data from industrial fishing and highly invasive net sampling with research ships, entailing enormous expense for personnel and time-consuming ship operations. Moreover, this approach results in discontinuous data acquisition, both temporally and spatially. The UFOTriNet project aimed to create an automated, cost-effective, non-invasive, and flexible system as an alternative to ship-based monitoring of fish stocks, and then to test it under real conditions. Such a system would contribute to a sustainable and evidence-based fisheries policy. During the project, both a mobile and a portable UFO (underwater fish observatory) were developed, built, and combined with an already existing stationary observatory. The newly connected trilateral monitoring network was tested at two locations in the southwestern Baltic Sea. All three systems are modular and expand- able with additional sensors, with the core Departments The mobile UFO, aka ROV BETTA, is being tested onboard the “Littorina” research vessel shortly before deployment in May 2023. 87

countries. However, operators responsible for monitoring the Arctic maritime areas will have to monitor considerably more activities. To address this issue, 13 partners from research, industry and potential future users have joined forces in the EU project AI-ARC, with the aim of improving the conditions for maritime surveillance in a way which ensures that the cognitive load for operators does not increase and their situational awareness even improves despite higher traffic volumes. Our central contribution is a virtual control room (VCR) based on the technologies of the Digital Map Table (DigLT). This collaborative platform for visualizing and analyzing situations seamlessly integrates the results of various arti- ficial intelligence (AI) algorithms for detecting anomalies in shipping traffic. It also serves as a central hub for effective communication and coordination between Arctic stakeholders. This not only simplifies decision-making when mon- itoring maritime shipping traffic, but also helps to overcome the challenges posed by climate change and increasing human activity in the region. In addition, the DigLT can be used via web interfaces or virtual reality (VR). In addition, Fraunhofer IOSB contributes AI-based anomaly detection algorithms that autonomously detect ships involved in critical situations such as illegal fishing, staying close to critical infrastructure, and deviating from shipping routes. An algorithm that warns of ice floes has also been developed. These algo- rithms combine knowledge and data-driven AI approaches and offer an editor so that users can describe new situations of interest without the need for any IT support. operated on various devices — from tablets and desktop PCs to VR glasses. Highlight topics Advanced Occupant Monitoring System Occupant monitoring systems are becoming increasingly important for vehicle safety and comfort. Our state-of-the-art system surpasses conventional methods by using optical sensors that detect all occupants within the vehicle. It not only identifies the driver but also recogniz- es the 3D body poses and movement behav- iors of all individuals present. This advanced technology allows for precise differentiation between various activities and levels of distrac- tion. Consequently, it enhances both safety systems and in-car comfort. Real-time 3D body pose detection, achieved without collecting biometric data and thus without compromising privacy, is the core of this system. It uses individual 3D cameras or multiple 2D cameras, enabling accurate tracking of body joints. This includes eye, elbow, and wrist positions, making it possible to interpret pointing gestures with centimeter precision. The system can recognize gestures from any direction, not just traditional seating positions or prescribed interaction areas. As vehicles become more automated, driv- ers and passengers gain more freedom for secondary activities. Our system is instrumental in recognizing these activities in 3D space, facilitating safe and efficient in-car interactions. It can differentiate up to 35 activities, such as eating, sleeping, reading, or making phone calls, thanks to machine learning processes. Fraunhofer IOSB’s Advanced Occupant Mon- itoring System provides key insights into the driver’s state and in-vehicle context, enabling a tailored approach to assistance functions. By recognizing activities, the system can predict the driver’s intentions and improve the overall driving experience, while ensuring that safety remains a top priority. Virtual Control Room for the Arctic Climate change makes the Arctic seas more navigable, significantly shortening the shipping route between Europe and Canada and other Departments Seating buck with integrated Advanced Occupant Monitor- ing System. 89

Departments Comparison of a synthesized image (right) with the corre- sponding ground truth (left). representation. For instance, point clouds inherently lack information in unoccupied areas, while other explicit representations such as (textured) meshes are difficult to compute for fine grained structures. model representations. A comparison of the reconstructed 3D structures with results obtained by state-of-the-art SFM and MVS pipelines demonstrated comparable quantita- tive results. Currently, the computation of NeRF models is relatively heavy compared to modern SFM and MVS implementations — both in terms of memory and computation requirements. However, based on the fast evolvement in this domain, NeRF-based models hold the poten- tial to substantially alter the 3D reconstruction of satellite imagery. 3D point clouds based on two implicit scene representations reconstructed with a NeRF model. Neural radiance fields (NeRF), which use an implicit representation of the scene, have emerged as a promising alternative. They offer a novel way to model complex scenes based on deep learning. NeRF-based models capture not only the intricate details of the scene’s geometry, but also cover specific appearance features such as the scene’s material prop- erties and the scene’s radiance, accounting for specific lighting conditions. NeRF-based models thus especially excel when synthesiz- ing realistic images for novel viewpoints. In recent years, we developed a framework based on modern structure-from-motion (SfM) and multi-view stereo (MVS) methods. The framework is capable of reconstructing multi-date satellite images and yields highly competitive results in the form of textured meshes. Now we have enhanced the frame- work with a component that generates NeRF-based models, in order to leverage the potential of current NeRF models in the domain of satellite imagery. Since the satellite domain poses specific challenges to the reconstruction task (such as large distances between cameras and scene, complex camera models, and specific radiance conditions), this is not a straightforward pro- cess. Our framework supports the adjustment of several key aspects, including the adapta- tion of camera models, ray generation, coordi- nate reference systems, and model structures. This allows us to substantially simplify the integration process. Since many use cases rely on metric measure- ments between different scene structures, we also investigated the derivation of explicit 91

Departments Visualization of PSDefoPAT results generated from displacement time series by the European Ground Motion Service. In a project, we provided a proof-of-concept for detecting airborne dust in open air mines using standard camera surveillance technolo- gy combined with advanced deep neural net machine learning techniques. For this purpose, a ground truth annotation workflow based on the given requirements of the data and the specifications of the final dust detection algo- rithm was researched, developed, and real- ized. Using the knowledge gained during the annotation phase, Fraunhofer IOSB designed, trained, and evaluated a dust detector based on a density estimation neural network archi- tecture. The dust detection algorithm showed promising results on the test dataset. As part of the proof-of-concept, the core components “dust detection” and “dust opacity estimation” were included in our “DustCamVision” framework and successfully deployed at the client’s facilities. The system is able to detect and track dust clouds, thus providing the insights needed to avoid dust emissions to the best possible extent. time series model based on its displacement time series. The tool separately determines the periodic and long-term trend components of the displacement time series using different regression models and significance tests. The image above shows a visualization tool that displays selected features of the best-fitting time series models and their goodness-of-fit as colored maps. The visualization facilitates an intuitive interpretation of the spatial distri- bution of the selected features and, conse- quently, the kinematics of different deforma- tion processes. AI-based dust cloud detection helps reduce cryosphere pollution Surface mining, such as open-cast mines, releases dust emissions that not only negatively impact nearby workers and their surrounding environment. Especially in the case of high-alti- tude copper mines in Chile, the dust can settle down on nearby glaciers, which causes them to retreat faster due to the lowered ice-albedo of the dust cover, therefore accelerating global warming. The operators of these mines use various methods to control dust emissions and reduce them to an acceptable level, such as water spraying before and after blasting, covering open areas, and reducing air currents by adapting the order of ore mining. How- ever, to optimally apply such measures, close monitoring and a better understanding of dust emissions is necessary. Open-pit copper mine in the Chilean highlands with dust cloud detection result. The opacity estimate is color-cod- 1 Evers, M., Thiele, A., Hammer, H., Hinz, S. (2023): PSDefoPAT—Persistent Scatterer Deformation Pattern Analysis Tool. ed or displayed in 3D as Journal of Remote Sensing, Vol. 15, No. 19, 4646, https://doi.org/10.3390/rs15194646, p. 26 elevation. 93

Departments with regard to protection against physical attacks, against mass panic at major events, whether protection against drowning in maritime applications or in the case of flood disasters, or, in this case, the protection of critical infrastructures against assaults. Military field camps are an example of such critical infrastructures. Methods developed for their protection will be applicable to airports, port facilities, railway stations, government buildings, major events, or religious institu- tions. In each case, the aim is to detect the relevance of static objects (e.g., abandoned suitcases), objects moving in unwanted places (e.g., vehicles), and the behavior, or equip- ment (e.g., weapons) of people. In 2024, together with two other Fraun- hofer IOSB departments, we will realize an all-weather, day-and-night capable system to record and evaluate the surroundings of critical infrastructures over a wide area. This system will be equipped with high-end sen- sors, control panels, processing hardware, and sophisticated AI algorithms. The fundamental software, however, can be transferred to “smaller” situations. For example, for monitor- ing the entrances to vulnerable facilities. generated by the AI system. This makes it possible to intervene at an earlier stage. At the same time, maximum data protection is ensured. This is primarily due to the fact that the AI system first reduces people to stick figures. A stick figure has no identity, no gender, no skin color, no origin, etc. Never- theless, by analyzing its movement patterns, attacks can be detected. During last year’s six week test run at Ham- burg’s Hansa-Platz the system detected eleven police-relevant incidents, including one case of dangerous bodily injury: a person already lying on the ground was kicked in the head. Only our system reported this critical incident to the command center, while none of the passers-by called the police. A police spokes- man summarized: “The added value, i.e. the early detection of dangerous situations including rapid police intervention, can be considered as given.” Since in Hamburg, unlike Mannheim, an existing camera network (cameras, instal- lation locations, detection areas) was used, the test run shows that our system can also be successfully used with an existing camera infrastructure. A two-year follow-up project is planned in Hamburg, starting in 2024. The project in Mannheim has been extended. Protection of critical infrastructures The main topic of the VID department is the protection of people with the help of video analysis methods. Whether in public spaces Digital skeletons of a scene staged by the Hamburg police, generated with our AI-based pose estimator. Contact Intelligent video analysis in public spaces II juergen.metzler@ iosb.fraunhofer.de Protection of critical infra- structures jochen.ring@ iosb.fraunhofer.de 95

Input inspection image showing Variances estimated on original set Variances updated using automated material defects of defect-free images uncertainty propagation Departments Experimental results of the implemented anomaly detec- tion framework: Without using AutoUncertainty to cal- culate the correct pixel-wise 176 226 2.88 73.85 53.02 363.65 statistics, several false events Results of thresholding, potential anomalies in white would have been detected (lower left image). The correct statistics allowed to only detect the actual material defects (lower right image). Using the unprocessed inspection image Using the processed inspection image and the original variances Using the processed inspection image and the updated variances process — turns out to be intractable for complex algorithms as they are used in image processing, which might even involve deep neural networks with millions of parameters. Fortunately, in many cases our AutoUncer- tainty [1] framework can completely automate the process of uncertainty propagation. It combines the classical method of Gaussian uncertainty propagation with the modern concept of automatic differentiation. Gauss- ian uncertainty propagation provides a formal way to propagate uncertainties through arbitrary functions, but requires to calcu- late the gradient with respect to all input parameters. In image processing, where the input data (images) has millions of parame- ters (pixels), this cannot be done manually, but by formulating the targeted algorithm in a programming language or framework that supports automatic differentiation (e.g., PyTorch or JAX). of checking the deviation from the intended appearance. Degraded input image DeepSNR When capturing scenes with a camera, the resulting images are often degraded due to various reasons, such as motion blur, imper- fections of the employed camera system (e.g., available space limiting the number of lenses), or because the object was out of focus. One of the image processing methods that try to mitigate these effects is the well- known Wiener filter, which tries to invert the image degradation in the Fourier space (i.e., the spatial frequency domain). To avoid the amplification of unwanted noise components, the intensity of the filtering for each spatial frequency is adapted depending on the signal- to-noise ratio (SNR). Although the Wiener filter is theoretically optimal, until now, the required knowledge of the SNR has greatly hindered its practical applicability. Deep SNR Estimated SNR Wiener filter Reconstructed image We have used AutoUncertainty to realize a visual inspection system based on anom- aly detection. For a sample of defect-free instances of the produced object, we calculat- ed pixel-wise statistics, treated the standard deviations as input uncertainties, and used AutoUncertainty to propagate them through our image processing pipeline. By compar- ing the processed inspection images of yet unseen objects with the propagated output uncertainties, we realized an interpretable way To mitigate this drawback, we researched a machine-learning approach, called Deep- SNR, which is capable of estimating the SNR of a degraded input image on a per-spatial frequency basis. We train DeepSNR with pairs of ground-truth SNRs and magnitude spectra of degraded input images. Tests on the widely used non-blind image deconvolution dataset of Sun et al. [2] show the achievable recon- struction performance of the Wiener filter when used in concert with DeepSNR. Example application of Deep SNR-based Wiener filtering of a degraded input image from [2]. 1 J. Meyer, M. Hartrumpf, T. Längle, and J. Beyerer, “Visual inspection via anomaly detection by automated uncertainty propagation”, in Unconventional Optical Imaging III, M. P. Georges, G. Popescu, and N. Verrier, eds., Strasbourg, France: SPIE, May 2022, p. 81. doi: 10.1117/12.2621146 2 Libin Sun, Sunghyun Cho, Jue Wang, and J. Hays, “Edge-based blur kernel estimation using patch priors”, in IEEE International Conference on Computational Photography (ICCP), Cambridge, MA: IEEE, Apr. 2013, pp. 1–8. doi: 10.1109/ICCPhot.2013.6528301 97

Appendix 22 doctorates Twenty-two researchers working at Fraun- hofer IOSB passed their doctoral examinations in 2022 and 2023. 21 R patents granted Applications for a further 44 patents have been disclosed in 2022 and 2023. In addi- tion, five product names were registered as trademarks. >600 journal and conference papers have been authored or co-authored by Fraunhofer IOSB researchers. >1,500 participants attended special events and conferences orga- nized or co-organized by Fraunhofer IOSB and its scientific staff. For all details on publications, academic teaching, lectures, technology transfer and out- reach activities, please refer to the comprehensive online appendix to this progress report: www.iosb.fraunhofer.de/progress-report-2024 99

Internal working groups Automotive und assistance systems Chair: M. Sc. Jens Ziehn | MRD Participating departments: IAD, MRD, VID Consultants: Univ.-Prof. Dr.-Ing. habil. Pu Li, Prof. Dr.-Ing. Rainer Stiefelhagen Autonomous systems Chairs: Dr.-Ing. Janko Petereit | MRD, Prof. Dr.-Ing. Andreas Wenzel | EIS Participating departments: DIS, EIS, IAS, KES, MRD, VID Consultant: Prof. Dr. Georg Bretthauer End-to-end software development Chair: M. Sc. Philipp Hertweck | ILT Participating departments: depending on topic Event based vision Chair: Paul Bäcker | SPR Participating departments: OPT, SPR Consultants: Prof. Dr.-Ing. Uwe Hanebeck, Prof. Dr.-Ing. habil. Björn Hein, Prof. Dr.-Ing. Michael Heizmann, Prof. Dr. rer. nat. Bernd Jähne, Prof. Dr. rer. nat. Wolfgang Karl, Prof. Dr.-Ing. Rainer Stiefelhagen Large language models Chair: Dr. rer. nat. Almuth Müller | IAS Participating departments: DIS, EIS, IAS, ILT, KES, KIS, LAS, MIT, MRD, OPT, SIG, SPR, SZA, UWR, VID Machine learning Chairs: Dr. rer. nat. Constanze Hasterok | KIS, Dr.-Ing. Christian Kühnert | MRD Participating departments: IAD, IAS, ILT, MIT, MRD, EIS, SPR, SZA, VID, KES, KIS Consultants: Prof. Dr. Georg Bretthauer, Prof. Dr.-Ing. Michael Heizmann, Prof. Dr. Oliver Niggemann Surface inspection Chair: Christian Kludt | SPR Participating departments: MRD, OPT, SIG, SPR, SZA Consultants: Prof. Dr. rer. nat. Bernd Jähne, Prof. Dr.-Ing. Michael Heizmann Self-organization of complex systems Dr.-Ing. Andreas Bunte | MIT Participating departments: DIS, IAS, ILT, KES, MIT Smart Farming Chair: Dr.-Ing. Robin Gruna | SPR Participating departments: DIS, EIS, MIT, MRD, SPR, SZA Appendix 101

Addresses Fraunhofer IOSB Karlsruhe Fraunhoferstraße 1, 76131 Karlsruhe, Germany Phone +49 721 6091-0 info@iosb.fraunhofer.de Industrial Automation branch Fraunhofer IOSB-INA Campusallee 1, 32657 Lemgo, Germany Phone +49 5261 94290-22 Karlsruher Forschungsfabrik Rintheimer Querallee 2 - Geb. 70.41, 76131 Karlsruhe, Germany Fraunhofer IOSB Ettlingen Gutleuthausstraße 1, 76275 Ettlingen, Germany Phone +49 7243 992-0 Branch lab Oberkochen c/o HENSOLDT Optronics GmbH Carl-Zeiss-Str. 22, 73447 Oberkochen, Germany Advanced System Technologies branch Fraunhofer IOSB-AST Am Vogelherd 90, 98693 Ilmenau, Germany Phone +49 3677 461-103 Branch lab Görlitz Brückenstraße 1, 02826 Görlitz, Germany Phone +49 3581 7925354 Smart Ocean Technologies research group SOT Rostock Alter Hafen Süd 6, 18069 Rostock, Germany Phone +49 3581 7925354 Fraunhofer Center for the Security of Socio-Technical Systems SIRIOS Berlin c/o Fraunhofer FOKUS Kaiserin-Augusta-Allee 31, 10589 Berlin, Germany Phone +49 30 3463-7594 Fraunhofer Representative Office Beijing Representative for Production and Information Technologies Unit 0602G, Tower D1 DRC Liangmaqiao Office Building 19 Dongfangdonglu, Chaoyang District 100600 Beijing, PR China Appendix 103