2 0 1 9 / 2 0 2 0 A N N U A L R E P O R T F F R R A A U U N N H H O O F F E E R R I I N N S S T T I I T T U U T T E E O O F F O O P P T T R R O O N N I I C C S S , , S S Y Y S S T T E E M M T T E E C C H H N N O O L L O O G G I I E E S S A A N N D D I I M M A A G G E E E E X X P P L L O O I I T T A A T T I I O O N N

IOSB AS PART OF BUSINESS UNITS DEPARTMENTS FACTS AND FIGURES FRAUNHOFER 30 Pooling expertise Fraunhofer Groups, Fraun- 35 Profiles, current trends and curated projects 56 Portfolios and highlights 89 Facts & figures 2018-19 in numbers – Papers, teaching activities, hofer Alliances, Clusters Artificial Intelligence and Including of Excellence, Fraunhofer Autonomous Systems - research areas and patents and other perfor- Lighthouse Projects competences mance indicators Automation and - applications, products Fraunhofer Group for De- Digitalization and services Board of trustees fense and Security VVS Fraunhofer-Gesellschaft Defense - infrastructure and lab facilities - research groups Internal working groups Energy, Environmental - recent projects and Scientific consultants of and Security Systems highlight topics Fraunhofer IOSB for each department Inspection and Optronic Systems 95 Service Addresses and online channels Editorial notes

Dear readers, The Fraunhofer Institute of Optronics, System Technologies and Image Exploitation was estab- lished ten years ago, on January 1, 2010. Despite being a young institute, we can nonetheless look back on a long and successful history. This is because Fraunhofer IOSB is the product of a merger between two previously existing institutes: Fraunhofer IITB and FGAN-FOM. Our first maor milestone presents an ideal opportunity to preface this report with a special anniversary section devoted to tracing this process of development. We take a quick look at our roots, recall the events and circumstances attending the merger, and outline some of the ey ideas and success stories accompanying the first ten years of Fraunhofer IOSB Each one of these is testament to the huge potential stored within the comprehensive and unparalleled range of expertise that was created by the merger. On the strength of these resources, we have already achieved maor technological advances in many of these fields This introductory section is followed by the customary report – in a new and substantially more compact form – covering the reporting period of 2018/2019. This contains basic facts and figures about our institute, its structure and its place within the Fraunhofer-Gesellschaft Selected highlights from our wide range of activities illustrate the dynamism and versatility of our institute e are much in demand as a partner in our specialist fields and as a player who helps shape future technological developments in areas such as security, defense, Industrie 4.0 and digital infrastructure. The report concludes with a presentation of our business areas and departments. To give at least a avor of the full scope of our activities, we deliberately report on a large number of individual topics and projects. These are therefore presented in a concise and con- densed manner. We trust that you will understand this in the spirit intended: as a stimulating read – and, above all, an invitation to enter into dialogue with us! Karlsruhe, Ettlingen, Ilmenau, Lemgo and Görlitz February 2020 Prof. Dr.-Ing. habil. Jürgen Beyerer 3

2000 Relocation of FOM to the former Rheinland-Kaserne in Ettlingen 2006/2007 Evaluation by the German Science Council FOM 2009 Effective from August 17, FGAN is integrated into the Fraunhofer-Gesellschaft 1999 FfO and FIM are merged to form the Research Institute of Optronics and Pattern Recognition (FOM) 2010 IOSB With effect from January 1, FOM and IITB merge into Fraunhofer IOSB 1996 Opening of the representative Office in Beijing IITB 1995 The IV and DV divisions are merged. Also the Application Center for Advanced System Technology (AST) is founded in Ilmenau 2006/2007 Evaluation by the German Science Council 2009 The Competence Center for Industrial Automation (INA) is founded in Lemgo 10 YEARS OF FRAUNHOFER IOSB 5

growth happened to coincide with a base funding freeze. act on this recommendation. The integration took place in mid- So, unlike the two other FGAN institutes, we were already 2009 and the merger took effect at the beginning of 2010. strongly acquainted with the world of contract research with its uncertainties and need for project acquisition. What was your reaction? Tacke: owever, this was preceded by protracted debate Fraunhofer and the German Federal Ministry of Education and Research (BMBF) voiced concerns that quality might suffer as a result. I suspect that an ingrained aversion to supporting Beyerer: Straightaway, I was delighted with the idea of a military research also played a role here. Ultimately, though, merger. Together, we would be able to cover a vast range the Science Council’s verdict persuaded everybody that the of topics, from the physics of signal generation to the way humans and machines use the information con- tained in these signals. At the time, FOM’s spectrum leaned more towards physics. IITB took it up from there to Beyerer: “Straightaway, I was delighted with the idea of a merger” merger would be a good thing. On the other hand, some at FGAN feared that working along the lines of the Fraun- hofer model would bring on too much economic pressure, with the thinking being that this would strip us of our address everything from computer science to human-machine academic freedom and scientific autonomy interaction and questions about the interoperability and avail- ability of this data. Just imagine the possibilities that would What kind of relationship existed come of combining this spectrum from end to end in one in- between the institutes and yourselves? stitute! No one else in Europe offered such a broad spectrum, so we would have a unique selling proposition. Tacke: I liked the idea too. Before joining FOM, I had headed Beyerer: We were always aware of each other’s existence, of course I was at FOM with former IITB director ans-Achim Kuhr in 2004 for my inaugural visit. When the talk turned up a department at the Fraunhofer Institute for Physical Meas- to the merger, we debated it, and there were certainly some urement Techniques IPM in Freiburg. I soon saw the structural opposing viewpoints. We held many discussions about the pros disadvantages in the FGAN, which had been working exclu- and cons and grappled with the question of how it could be sively for the ministry of defense. The more open Fraunhofer done. Then, when it became clear that the integration and model was in many respects preferable. The only thing I merger were going to happen, the question was how do we found problematic about FGAN’s integration into Fraunhofer come together on an equal footing without this feeling like was that it was up in the air for so long. I had recently been a takeover? The departments had their reservations. After all, appointed chairman of FGAN, so I had to make sure that it they had been rivals to some extent in the past. would come out of the process unscathed, even if the integra- tion might not come to pass. This is why we worked separately until the moment the merger was definite Tacke: Despite there having been some collaboration and joint projects, the feeling of being competitors prevailed because both institutes were working in the defense market and in What happened after 2004? image exploitation. Beyerer: The German Science Council carried out its evaluation Beyerer: And suddenly there we were, sat at the same table, in 2006. I advocated recommending a merger for the reasons talking about how to work together in a future joint institute, mentioned earlier. In its report in late 2007, the German Science Council made an unequivocal recommendation to integrate FGAN into Fraunhofer and to merge our two institutes. Then it about how to bridge divides. In the end, it all comes down to trust It is my firm conviction that the merger was only successful because the two of us, as persons, managed to was up to the ministries and Fraunhofer to decide if they would work well together. 10 YEARS OF FRAUNHOFER IOSB 7

rule. This is to be expected when the dividing lines between departments and groups are set properly in terms of content. This way, individual units can work on a problem from end to end. It is tremendously helpful to be able to communicate in an atmosphere of trust within an interdisciplinary institute. Beyerer: I see the synergy evident in underwater vision, for example. It is a hot topic at Ilmenau, Karlsruhe and Ettlingen, even if each site addresses different aspects of it. Another example is drone detection and defense. This would not have been possible without the collective expertise of the colleagues in Ettlingen and Karlsruhe. And looking ahead, I see the min- istry of defense pursuing major long-term projects – Future Combat Air System, or FCAS, and Main Ground Combat System, or MGCS – where I believe our broad spectrum as an institute will be very useful indeed. Speaking of looking ahead, what are your hopes and expectations for the IOSB’s next ten years? Tacke: I hope that the structure of base funding for defense research will become more coherent and systematic. The fact that we have managed to handle the situation without it negatively impacting our work is a great success. And things could be even better given an optimized funding and regula- tory framework. Apart from that, I believe the institute is in a good position I expect to see some nice and profitable results in the next ten years, without being able to say what exactly they will be. Beyerer: aving a first-rate infrastructure at all sites is impor- tant. We are well on our way with various building projects. As far as content goes, I hope and expect that we will make visible and substantial contributions to the two major defense projects mentioned earlier and to the resolution of the chal- lenges facing civil society. There are a number of topics for which I hope we will be the partner of choice and be able to maintain our competitive lead. Examples include embodied This is a translated and (slightly) abridged version of the original machine intelligence, digitalization in the context of industrial interview with Prof. Beyerer and Prof. Tacke, which was led manufacturing and automation technology, and the question of how to turn artificial intelligence and machine learning into an engineering discipline. by Ulrich Pontes. The full interview in German can be found at iosb.fraunhofer.de/10-years-interview 10 YEARS OF FRAUNHOFER IOSB 9

CONNECTED AUTONOMOUS SYSTEMS IN HOSTILE ENVIRONMENTS Autonomous, networed robots such as ying drones, unmanned ground vehicles (UGV) and semi-autonomous Fraunhofer IOSB has pursued many projects over the years to advance these technologies. One result is a toolbox of machines are predestined to tackle dangerous tasks such as reconnoitering hazardous or poorly accessible environments in the wake of an earthquake or an industrial accident in order algorithms for localization, mapping, obstacle detection and motion planning. Another is a retrofit kit for machinery to equip standard tractors and excavators with autonomous to provide an accurate situation report. They can also clean functions. up contaminants, sparing humans the need to work in areas with health hazards. But these applications present a host of IOSB.BoB, the “excavator without operator” (see photo), technical challenges. The robots have to be able to navigate has been attracting attention at many demonstrations since unknown and irregular terrain, extract information from a 2017. In 2018, these efforts culminated in the founding of wide range of sensor data to provide a standardized situation the Fraunhofer IOSB-led competence center ROBDEKON on report, and plan and execute complex motion sequences to robotic systems for decontamination in environments hostile perform manipulation tasks. They should also be able to cope to humans (cf. also page 24). with unexpected events and make independent decisions. 2011 – 2011 – S S E E I I R R O O T T S S S S S S E E C C C C U U S S D D N N A A S S A A E E D D I I 10 YEARS OF FRAUNHOFER IOSB 11

2014 – 2014– 2011 – 2011 – S S E E I I R R O O T T S S S S S S E E C C C C U U S S D D N N A A S S A A E E D D I I CYBERSECURITY TRAINING LABS ENGINEERING AI METHODS FOR MANUFACTURING PROCESSES The digital transformation in our living and work environ- Impelled by digitalization and rapid advances in machine ments makes us more vulnerable to cyber attacks. Our insti- learning, data-driven methods are becoming more important tute has been investigating IT security issues for many years. for manufacturing. Often replacing or augmenting modelling As part of an in-house research project launched in 2014, we and knowledge-based approaches, these data-driven methods started building a demonstrator to simulate and investigate real-world threat scenarios using standard components for production control. The IT security lab was received with great interest at the annover Messe in serve many purposes. They can help optimize individual steps in the worow and control complex processes to minimize waste and maximize productivity. They can detect anomalies, enable predictive maintenance and optimize the logistics of complex supply chains. Computers can use sensors and actua- Soon after, we began to hear of growing demand for comprehensive training to ualify specialists in this field The Fraunhofer Academy, various Fraunhofer Institutes and associ- tors to learn how process parameters and outcomes are related. This enables engineers to develop and roll out ‘immature’ processes – that is, production processes that are not yet fully ated universities of applied sciences heeded the call by setting understood in terms of the physics or engineering. up the Cybersecurity Training Labs. Their hands-on seminars for professionals have been very successful. Fraunhofer IOSB was instrumental in developing the concept behind these Fraunhofer IOSB in Karlsruhe and Lemgo has realized artificial intelligence (AI) applications like this in many projects with labs and is responsible for two of the six subjects on the industry. Seeking to take an even more systematic approach training agenda: cybersecurity in industrial production at with an ideal infrastructure, we joined forces with Fraunhofer Fraunhofer IOSB in Karlsruhe and Lemgo together with the ICT and the Karlsruhe Institute of Technology to build the University of Applied Sciences Ostwestfalen-Lippe and cyber- Karlsruhe Research Factory. Its mission is not limited to security in energy and water supply at Fraunhofer IOSB in optimizing production steps and processes, but will go beyond Ilmenau and Görlitz with Fraunhofer IDMT and the University to establish a comprehensive methodology for applying AI to of Applied Sciences Zittau/Görlitz. industrial manufacturing. Our aim is to develop an engineer- ing-based approach to reliably predict the operational behavior of manufacturing processes even in the planning phase. 10 YEARS OF FRAUNHOFER IOSB 13

HOLISTIC INFORMATION MANAGEMENT FOR SECURITY AND DEFENSE The security context may be military or civilian, but in both lighten the operators’ workload. This system can feed in video use cases, sensors and information systems generate vast amounts of data. The relevant information has to reach the right decision makers quickly. And that requires a network of streams from many drone types, stabilize them live, show them in a freely configurable representation, and store them in a database. Features include the merger of images into a interoperable systems, connected across organizational and georeferenced image mosaic, change detection, and tracking national boundaries. It also requires smart analysis algorithms of marked objects even when they are temporarily obscured, and a user experience designed to help these people do their stereo images and super resolution. jobs effectively. We have been working on corresponding standards for many years, for example, at the NATO level. We develop concepts We commenced the development of the Digital Map Table to bring the many benefits of digital information ows to strategic and tactical operations centers. It is compatible and architectures and concrete hardware and software with all relevant data sources and is tailored to the needs of systems to put these standards into practice and operation- command staff and operation control centers. The Bundes- alize collaborative processes such as Joint ISR (Intelligence, wehr Command and Staff College, among others, puts it to Surveillance and Reconnaissance) in the best way possible. good use. We have continued to improve the Digital Map This is how the idea of Coalition Shared Data (CSD) came about. It affords rights-based access to data (such as images, reports, streams, tasks) from arbitrary sites in near real-time. Table, adding features and interaction techniques in many proects since ommand and field staff can wor not only with conventional devices such as smartphones, tablets and The system normally distributes only metadata to save net- the large physical Digital Map Table, but also in virtual space. work resources, but actual information may be retrieved on The participants can see each other with VR headsets and demand. The German armed forces use our CSD technology a 3D representation of the scenario and can interact quite on various missions. naturally despite being widely separated in reality. Our ABUL system provides automated image exploitation for unmanned aerial vehicles and has many features to 2010 – 2010 – S S E E I I R R O O T T S S S S S S E E C C C C U U S S D D N N A A S S A A E E D D I I 10 YEARS OF FRAUNHOFER IOSB 15

CORE COMPETENCES AND STRUCTURE OPTRONICS SYSTEM TECHNOLOGIES IMAGE EXPLOITATION Understanding and control of Ability to analyze, understand, Preparation and real-time the generation of light, its beam model, develop and control processing of images and videos forming, propagation and complex systems conversion into electronic signals as well as automatic and inter- active information extraction LAS Laser Technology OPT Optronics DIS IAS Digital Infrastructure Interoperability and IAD Interactive Analysis and Diagnosis SIG SPR Signatorics Assistance Systems OBJ Object Recognition Visual Inspection Systems ILT Information Management SZA Scene Analysis and Production Control VID Video Exploitation Systems MIT Machine Intelligence VBV Variable Image Acquisition MRD Systems for Measurement, and Processing research Control and Diagnosis group NRG Energy Systems WMS Water and Mobile Systems SYSTEM TECHNOLOGIES NRG WMS DIS ILT MIT MRD IAS SIG LAS OPT SPR SZA VBV IAD VID The competence triangle positions the OBJ departments of Fraunhofer IOSB in relation to the three core competences. OPTRONICS IMAGE EXPLOITATION 17

BUSINESS EXPENSES ow much money did we spend? FUNDING Where did the money come from (in 2019)? * Industrial sector 37% Basic funding 28% Others 4% Public sector 31% 70 60 50 40 30 20 10 '15 '16 '17 '18 '19* Investment Research expenditure STAFF ow many persons wored at Fraunhofer IOSB '19 '18 '17 '16 '15 Scientists and engineers Other regular employees Research assistants, student assistants and interns 100 200 300 400 500 600 700 * Preliminary figures as of February 2020 ** Headcount reporting date: March 31 19

SELECTED STRATEGIC COLLABORATIONS, EXPERT NETWORKS AND PLATFORMS WE ARE PARTICIPATING IN Lemgo Digital Open District Hub Für die Zukunft der Energiewende 21

ILMENAU / GÖRLITZ LEMGO Advanced System Technology Industrial Automation branch Fraunhofer IOSB-AST branch Fraunhofer IOSB-INA Prof. Dr.-Ing. habil. Thomas Rauschenbach Prof. Dr.-Ing. Jürgen Jasperneite B O A R D O F D I R E C T O R S Director Ilmenau Director Lemgo WMS | Water and Mobile Systems DIS | Digital Infrastructure Prof. Dr.-Ing. habil. Thomas Rauschenbach Prof. Dr.-Ing. Jürgen Jasperneite NRG | Energy Systems MIT | Machine Intelligence Prof. Dr.-Ing. Peter Bretschneider Deputy director Ilmenau Dr. rer. nat. Oliver Niehörster A D M I N I S T R AT I O N S TA F F D E PA R T M E N T Strategy and Innovation Management Head of Administration / Commercial Dr. Frank Lorenz Head of Staff department and Technical Management Dipl-Betriebsw F Nicole Keller-Rau Dr. rer. nat. Jürgen Geisler Krägelin IT-Security Press and Communications Dipl.-Phys. Ulrich Pontes Dipl.-Inform. Birger Krägelin Strategy and Innovation Management Dr. rer. nat. Frank Lorenz BEIJING Representative Office China Dipl-Ing ong Mu 23

AWARD-WINNING AI EXPERTISE FRAUNHOFER IOSB-INA CELEBRATES 10TH ANNIVERSARY One hundred locations for Industrie 4.0 In an announce- Anniversary and relocation to new premises Fraunhofer ment on December 3, 2019, Allianz Industrie 4.0 named IOSB-INA celebrated its tenth anniversary on October 1, 2019. Fraunhofer IOSB as one of the top 100 locations for Industrie Back in 2009, Professor Jürgen Jasperneite, now a member of 4.0 in Baden-Württemberg. It was the second time since the board of directors of Fraunhofer IOSB, started up in Lemgo 2015 that Fraunhofer IOSB has received the award. In 2019, the award focused on the use of artificial intelligence in man- ufacturing, with the jury of experts rewarding companies and with three young and dedicated scientists, all provisionally ac- commodated in office containers Their goal To help shape the future of industrial communications. The inaugural project? To institutions that demonstrate a high degree of innovation and develop a chip for high-performance real-time communications. market relevance. Today, a 75-strong team pursues top-level applied research un- der the motto “Empowering our partners for the digital age.” The award confirms that Fraunhofer IOSB is on the right trac, e.g. with its involvement in the Karlsruhe Research Factory. The focus is on providing knowledge and technology for digital transformation in manufacturing companies and municipalities. It will be opened in 2020 by the Fraunhofer-Gesellschaft in partnership with the Karlsruhe Institute of Technology. This Over the past ten years, Fraunhofer IOSB-INA has become a new facility will investigate AI applications for manufacturing renowned institution in the technology region of Ostwestfalen- in a realistic setting and thereby accelerate the transfer to industry of, in particular, processes that are still not completely understood. Fraunhofer IOSB has made further telling contributions to this field, including basic research for the Fraunhofer lighthouse project “ML4P – Machine Learning for Production” and con- Lippe. An early infrastructure milestone was the construction of the entrum Industrial IT in This was Germanys first science-to-business center in the field of industrial automation and home to IOSB-INA until In , with financial assistance from the State of North Rhine-Westphalia, IOSB-INA expanded to become a Fraunhofer application center – the first to be established in cooperation with a university of applied crete development contracts for industry. The institute has sciences. Since then, Fraunhofer IOSB-INA has been involved in expertise in all the major aspects of AI and machine learning in production ere, the ey challenge is to extract the relevant data from machine-control software and sensors, then to designing the BMBF cluster of excellence “Intelligent Technical Systems Ostwestfalen-Lippe (it’s OWL)”. It began broadening its research to include cognitive processes based on connected select the right machine-learning methods to generate models from that data the learning process, and, finally, to create a suitable IT system architecture that enables fast processing embedded systems as well as intuitive interaction technol- ogies for intelligent technical systems Before long, artificial intelligence and big data became part of the portfolio. A high of the data and models (edge computing). Fraunhofer IOSB performance computing cluster was established in 2016. That also develops methods and guides describing how typical ML same year saw the opening of SmartFactoryOWL, a research use cases can be implemented systematically and as easily as possible in manufacturing. and demonstration facility for Industrie 4.0, run in partnership with the Technische ochschule Ostwestfalen-Lippe Today, SmartFactoryOWL is a major attraction, drawing several thou- sand visitors a year. It is directly connected to the new institute building, to which IOSB-INA relocated at the beginning of 2020. 25

IOSB-AST IN NEW PREMISES New institute building in Ilmenau Relocation began in the fall of 2019. Little by little, the about 100 employees of Fraunhofer IOSB-AST moved to the offices and labs of the newly erected building. It is located in the industrial It provides , suare meters of office, lab and training space for the two research bodies and can accommodate up to 212 employees. The construction costs of around 25 million euros were funded to 40 percent by the European park Am Vogelherd within sight of the old building. The Regional Development Fund (ERDF), by the State of Thur- new building is a joint project of Fraunhofer IOSB-AST and ingia (30 percent) and by the German federal government the Electronic Measurements and Signal Processing depart- (30 percent). ment of the Fraunhofer Institute for Integrated Circuits IIS. 2727

HARMONIZING IIOT ARCHITECTURES SMART CITY LIVING LAB Fraunhofer IOSB builds bridges with Smart Factory Web On the international level, there are currently a number of vendor-neutral initiatives that aim to develop the Industrial Internet on the basis of open standards and trustworthy, interoperable architectures. Three of the most important are Launch of LEMGO DIGITAL ow can the Internet of Things (IoT) facilitate urban living? Fraunhofer IOSB-INA has inaugu- rated LEMGO DIGITAL to answer this question for concrete application proects The proect was officially launched in May 2018 in the presence of Professor Andreas Pinkwart, the Industrial Internet Consortium (IIC), Plattform Industrie North Rhine-Westphalia State Minister for Business and Digital 4.0 and the International Data Spaces Association Affairs. LEMGO DIGITAL is an open platform for developing (IDSA), which is committed to a trustworthy architecture and testing innovations for digital transformation in small in the data economy. As a longstanding contributor to the and medium-sized towns. In partnership with the University of development of an interoperable Industrial Internet of Things Applied Sciences and Arts, Ostwestfalen-Lippe as well as (IIoT), Fraunhofer IOSB is closely linked to all three initiatives – research institutions and partners from industry, it seeks to and is now playing a leading role in building bridges between create innovative solutions to everyday urban problems in the the different reference architectures and also in building and areas of mobility, the environment, commerce and energy leading an industrial community on behalf of IDSA. supply. Smart Factory Web (SFW), an online marketplace for produc- Many smart city projects focus on large cities. LEMGO DIGITAL, tion capacity developed by Fraunhofer IOSB (cf. p. 43), however, has been designed to answer the needs of medium- has been a recognized IIC testbed since 2016. In September size towns – with between 20,000 and 100,000 inhabitants 2018, when IIC and IDSA signed a memorandum of under- – of which there are over 600 in Germany. Moreover, unlike standing, SFW was designated as a joint testbed. This was other projects, LEMGO DIGITAL has actively encouraged the followed soon afterwards by the launch of SYNERGY, a pro- participation of both citizens and IoT startups from a very ject that gave concrete form to this collaboration by integrat- ing components of the International Data Spaces into SFW. This will enable more secure and trustworthy connections be- tween the participants of the marketplace. At the same time, it mars a first step towards technological interconnectivity and convergence between the three initiatives named above. On April , , at the annover Messe, II and I lie- wise announced closer cooperation and nominated SFW as a potential joint testbed. The photo above shows the heads of lattform Industrie and the II, enning Banthien left and Richard Soley resp., pictured in front of the SFW exhibit, provided by Fraunhofer IOSB and the SFW participants, in all of the annover Messe early stage onward. The project has turned the town of Lemgo into a living play-along labor – with the scientific and business communities working hand in hand with the general public to shape technological innovation. So far, the project has resulted in the creation of an open- access Wi-Fi network throughout the town and a sensor infra- structure to monitor the traffic situation Real-time informa- tion on public transport facilitates travel. In addition, work is progressing on a smart parking-management app. Finally, the mayor has been presented with a dashboard (web site) that displays live information on traffic, the weather, the number of visitors to the town etc. 29

Fraunhofer AutoMOBILE Production Alliance Powertrain and running gear, electric mobility, planning, cont- rol and logistics, car body, vehicle assembly, vehicle interior Fraunhofer Big Data and Artificial Intelligence Alliance Logistics and mobility, production / Industrie 4.0, life sciences and health care, data security, energy and environment, busi- ness and finances Fraunhofer Energy Alliance Renewable energy, energy in digital context, energy systems, energy storage, energy efficiency, energy in urban context Fraunhofer Traffic and Transportation Alliance Comfort and design concepts, safety systems, resource- and energy-efficient transportation, logistical structures and processes, innovative mobility and traffic systems, smart and resilient transportation infrastructure Fraunhofer Vision Alliance Industrial image processing, 3D surveying, thermography, X-ray technology, surface inspection, medical engineering, safety engineering, traffic engineering, terahertz tomography Fraunhofer Water Systems Alliance (SysWasser) Urban/periurban water management systems, processes Fraunhofer Food Chain Management Alliance and systems for the supply with drinking and service water, Food chemistry, food packaging technology, logistics, micro- wastewater treatment processes and systems, integrated systems technology water resource management (IWRM), analysis and evaluation of water-economic systems Fraunhofer Space Alliance Communication and navigation, materials and processes, energy and electronics, surfaces and optical systems, protec- tion technology and reliability, sensor systems and analysis Institutes working in related subject areas cooperate in Fraunhofer Groups and foster a joint presence on the R&D market. They help to define the Fraunhofer-Gesellschafts business policy and act to implement its organizational and funding principles. The groups‘ chairpersons are part of the Presidential Council of the Fraunhofer-Gesellschaft. The Fraunhofer Alliances facilitate customer access to the services and research results of the Fraunhofer-Gesellschaft. ommon points of contact for groups of institutes active in related fields provide expert advice on complex issues and coordinate the development of appropriate solutions. As a sort of “virtual institute” spread over multiple locations, the Fraunhofer Clusters of Excellence promote cooperative development of system-relevant topics through an inter-institute research structure. Their purpose is not simply to implement specific, short-term proects but rather to follow a roadmap for the long-term devel-opment of a complex technological trend In its Lighthouse Projects, Fraunhofer is tackling the current challenges facing industry head on. These 3- to 4-year projects with budgets of several million euros focus on topics geared toward strategic economic requirements, with a view to turning original scientific ideas into maretable products as uicly as possible 31

Our core capabilities - Coordinating large-scale projects - Delivering system solutions for complex issues Our unique features - Excellent infrastructure and laboratory equipment - Technical expertise always available - Superb network with research, industry and government - Well-founded assessment and consulting skills in defense research and technology - Interdisciplinary work and broad techno- logy portfolio VVS for a life of freedom Range of services - Feasibility studies - Strategic foresight, scenarios and roadmapping - Analyzing technological needs and trends - Developing methods, technologies, components and systems - Assessing systems, including those of third parties - Developing prototypes and processes - Pilot series production Members and associated members - Fraunhofer Institute for igh-Speed Dynamics, Ernst-Mach-Institut, EMI Protection, Security and Effects - Fraunhofer Institute for Technological Trend Analysis INT Planning Support for State and Industry - Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB - Fraunhofer Institute for igh Freuency hysics From Networked Sensor Data to Decision and Radar Techniues FR Radar – A Key Technology - Fraunhofer Institute for Experimental Software Engineering IESE - Fraunhofer Institute for Communication, Information Software and Systems Engineering Processing and Ergonomics FKIE - Fraunhofer Institute for Integrated Circuits IIS Command, Control and Reconnaissance Communication, Positioning Technologies and X-Ray - Fraunhofer Institute for Applied Solid State Physics IAF for Safety and Security Applications Sensors for Safety, Security and Reconnaissance - Fraunhofer Institute for Structural Durability and Systems - Fraunhofer Institute for Chemical Technology ICT Reliability LBF Security, Safety and Energetic Materials Technology Secure Processes for secure Structures 33

BUSINESS UNITS Artificial Intelligence and Autonomous Systems 36 – 39 Automation and Digitalization 40 – 43 Defense 44 – 47 Energy, Environmental and Security Systems 48 – 51 Inspection and Optronic Systems 52 – 55 35

”AI WILL NEVER REPLACE A HUMAN BEING“ The very specific strengths of artificial intelligence can help improve decision-making Everyone, it seems, is talking about artificial intelligence. Yet people still have reservations, particularly regarding data protection and the perceived threat to jobs. What’s the significance of AI for Fraunhofer IOSB? Where is research still needed? And what are the drawbacks of AI compared to conventional statistical analysis? Rauschenbach: ell, firstly, I should say that artificial in- telligence is not all that new. We’ve been using AI since the Rauschenbach: One of the big challenges we face with a number of deep-learning processes is that it is impossible to explain with absolute certainty just how they arrive at the 1990s. It’s what power companies use to predict the con- results they do. By contrast, we understand very well how a sumption of tens of thousands of customers, or automakers statistically based predictive model, such as the similar days to ensure quality in vehicle production. But AI doesn’t replace an actual person in these or any other areas. When we use AI in our projects, we use it to provide support with human decision-making. method, generates its results. A lot of AI is a bit like a black box if the result isnt uite right, its very difficult to say whats gone wrong. That’s why “Explainable AI” (XAI) is such a big research topic right now. And it’s one we’re looking at very closely here, too. In which areas is AI already better than humans? Rauschenbach: The human brain is extremely good at pattern recognition. For example, we have no problem distinguishing a cup of coffee from a glass of water owever, the picture changes when it comes to recognizing patterns in extremely large datasets, also nown as big data Artificial intelligence is far better at this than humans. At Fraunhofer IOSB, we use pattern recognition to analyze so-called PMU data (see overleaf: Enhancing grid stability with AI ere, every single sensor generates 150 datasets every second. The aim here is to rec- ognize malfunctions in the power grid before they even occur. This is the kind of application for which we can really exploit the strengths of AI. Other areas currently being investigated by our researchers include predictive maintenance for Industrie 4.0, automated detection of physical assaults, and object rec- ognition for autonomous underwater vehicles. In other words, all of these involve the use of AI-based systems to assist with the human decision-making process. Prof. Thomas Rauschenbach is head of the WMS department and director of the Advanced System Technologies branch AST of Fraunhofer IOSB. 37

Enhancing public security with AI AI applications for improved intelligence, surveillance and reconnaissance Despite the many changes that globalization, digital transfor- mation and permanent connectivity have brought to our lives, The success of intelligence, surveillance and reconnaissance “analog” crimes are still being committed on a “local” level (ISR) operations relies on the timely provision of accurate, every day. Yet modern technology can help guard us against relevant and coherent information generated across a range such criminal acts as well as provide better protection of our privacy In so doing, it can also increase public confidence in the work done by the law-enforcement agencies and in gov- of increasingly heterogeneous platforms, both manned and unmanned. Proper evaluation of this data is essential for a high level of situational awareness. AI-based systems have ernment policymaking. a key role to play here. Examples of such AI support include assisted image exploitation and support for ISR management. In a collaborative project with the city of Mannheim, the To optimize the assignment of ISR assets to intelligence and metropolitan police department and Baden-Württemberg’s surveillance requirements, Fraunhofer IOSB has developed an State Ministry of the Interior, Fraunhofer IOSB is demonstrat- ISR Manager application (see image). ing how smart video surveillance can help prevent crime. The system software analyses live video from a network of cam- In many cases, information has to be evaluated and dissemi- eras with the aim of identifying, in real time, acts of physical violence – for example, kicking or punching. Whenever the system recognizes an assault, it ags it so that the monitoring team in the control room can decide whether the incident is serious enough to warrant some form of intervention, such as sending a beat officer or patrol car to the scene nated to command structures while a mission is still ongoing. To ensure the most efficient and effective use of resources, mission command must be able to react according to a chang- ing situation on the ground. To meet this need, Fraunhofer IOSB develops systems to assist with dynamic mission planning and the control of heterogeneous groups of mobile systems based on different AI systems. In particular, this involves the In other words, the system merely has an auxiliary role, assisting use of software agents, genetic algorithms and the behavioral the police with routine surveillance. Moreover, it also permits control of individual systems and their groups. It is thereby the pixelation of normal, harmless scenes, which can only possible to make inferences about concrete situations on the alleviate the fears of those concerned about data protection basis of multi-sensor data such as radar and radio and then, in and privacy. line with this information, to adjust or reallocate such sensors. iosb.fraunhofer.de/servlet/is/93474 CONTACT Dr.-Ing. Dipl.-Inform. Markus Müller CONTACT Dr.-Ing. Rainer Schönbein Head of department Video Exploitation Systems Head of department Interoperability and Assistance Systems Phone +49 721 6091-250 Phone +49 721 6091-248 39

”AI EXPERTISE IS NOT ENOUGH ON ITS OWN“ In the future, manufacturing will be self-organizing. To achieve this end, each element of the production network – from sensors to secure cloud interfaces – must harmonize perfectly Dr. Sauer, everyone’s excited about self-organizing manufacturing. What’s behind the hype? right, which means issues such as simulation, sensor technol- ogy, real-time connectivity, interoperability, and suitable OT/ Olaf Sauer: For a while now, the trend in manufacturing has been towards ever an increasing product variety and cus- tomization, while also cutting delivery times. Conventional production methods – such as the sequencing concept used IT architectures based on edge and cloud computing. That said, we have a wealth of experience in these fields and have helped define the state of the art in these technologies and will continue to drive them forward in the future. in the automobile industry, whereby one car after another is processed on the assembly line – soon run into difficulties here. The work that needs to be done at individual stations You mentioned edge and cloud computing. Yet many companies don’t want to cede control over their data. is simply too varied, with the result that employee and plant Sauer: That will require trustworthy infrastructures. Besides, utilization decreases. In other words, the idea – which is by isolation is the wrong strategy for manufacturers who want to no means new – of smart, connected products being able to leverage the advantages of digitalization in production. You navigate their way through a modular production environment need to provide three things simultaneously: performance, is more topical than ever. So, instead of conveyor belts, you have autonomous material flow systems moving each car body individually from one work station to the next? data security and connectivity that extends beyond a manufac- turer’s own location so as to enable effective control of logistic chains. This demands a carefully scaled strategy. That’s why we and our partners have been involved for a number of years now in initiatives such as International Data Spaces, German Edge Cloud/ONCITE and Gaia-X. These all address precisely Sauer: That’s the most visible change. But the challenge we these concerns and also include sophisticated mechanisms to face at Fraunhofer IOSB, along with our industry partners, provide maximum data sovereignty and control. is to optimize the IT systems responsible for production control and automation. First of all, we have to be able to permanent- ly locate products, parts and material ow systems, such as AGVs, and then digitally map these along with other pro- duction resources. Essentially, we’re talking here about IIoT and real-time visualization. And then all this complex activity has to be able to self-organize in such a way that efficiency is maximized while also guaranteeing a robust response to any unforeseen events such as a uctuation in orders, supply prob- lems or the unavailability of modules. This requires expertise in AI, especially in areas such as reinforcement learning. But that’s not enough on its own. We also need to get the basics Dr. Olaf Sauer, deputy director of Fraunhofer IOSB, is an expert in automation and coordinates activities at the Automation and Digitalization business unit. 41

Smart Factory Web – a marketplace for industrial Cybersecurity training lab for industrial manufacturing production The Smart Factory Web aims to set up a web-based platform where factories can offer production capabilities and share resources This will enable them to improve order fulfillment with greater exibility than is currently possible with availa- ble technology. The Smart Factory Web seeks to provide the technical basis for new business models, especially for small lot sizes, by exibly assigning production resources across factory locations This testbed is chiey designed as a step towards establishing a manufacturing marketplace where one can loo for factories with specific capabilities and assets to meet production requirements. Factories with those capabilities can The principles and technologies of the Industrial Internet of Things IIoT are increasingly finding their way into domains such as manufacturing and critical infrastructures. This trend marks not only a technological shift that enables new and possibly disruptive business models; it also brings heightened security risks due to new vulnerabilities. No longer hidden in shielded, mostly proprietary network environments, IIoT com- ponents now come with built-in connectivity options based upon standardized communication networks in order to offer new services such as preventive maintenance. As a consequence, system architects and engineers in all dis- then register to join the marketplace. ciplines need at least basic security and security management This requires up-to-date information about the capabilities and status of assets in the factory The specifications of the products – availability, quality, price and other parameters – know-how and skills; these are essential for drafting and configuring systems and networ architectures in such a way that the security level meets the specified security protection goals of the application or organization. Standards for in- can be used to negotiate in the case of competing offers. formation security management systems such as the ISO/IEC 27000-series and IEC 62443 need to be applied here. This system hinges on making information about the factories available in a standardized, reliable way. To ensure this, the Our cybersecurity training lab for industrial manufacturing, Smart Factory Web applies international standards such as based in Karlsruhe and Lemgo, combines learning modules OPC UA and AutomationML as well as Industrie 4.0 and Inter- national Data Spaces IDS specifications Originally started as a testbed run by the Industrial Internet Consortium (IIC) with the with a tailored training and education environment. Designed for engineers and computer scientists, it features experimental hardware and software systems that are built from real-world Korean research partner KETI, it has since attracted participa- automation components (see image). Like the cybersecurity tion by global IT players such as Microsoft and SAP to jointly training lab for energy and water supply in Ilmenau and Görlitz, leverage commercial business cases. it lets users see the consequences of cyberattacks and apply and test countermeasures. smartfactoryweb.de iosb.fraunhofer.de/servlet/is/72741 CONTACT Dr.-Ing. Thomas Usländer CONTACT Dr.-Ing. Christian Haas Head of department ILT Phone +49 721 6091-480 Group manager Security in networked systems Phone +49 721 6091-605 43

”KEY CAPABILITIES FOR TOMORROW’S BATTLEFIELDS“ Research into future battle tank systems and digitalized military operations Digitalization is transforming practically every area of life. What impact is it having on defense research? Michael Arens: The battlefield is changing, though not mere- ly because certain aspects of it have been relocated to the uate information but also systems that can present even the most complex situations on a wide variety of device platforms and in a manner appropriate to specific roles, thereby enabling rapid action. realm of cyberspace. Instead, we’re also seeing a shift in the And what about areas beyond the intelligence cycle and requirements made of armed forces in the real world. With the interpretation of reconnaissance data? digital technologies available to everyone worldwide, armed forces must ensure they are physically equipped to combat so-called peer adversaries. They need to rapidly process large amounts of intelligence data from different sources so as to Arens: We’ve got a lot to contribute here as well. In the field of GS-denied environments, for example, weve been working for some time now on image-based localization and build an accurate picture that enables informed operational navigation. And we also have the ability to automatically refer- decisions. For example, they have to be able to operate in GPS-denied environments. At the same time, with weapon systems being transformed by digitalization and automation, ence other kinds of information within space and time, which means we can fuse this data to form an overall and unified vir- tual picture. This will be crucial for the future weapon system the emphasis is now on alliances of highly networked and alliances we referred to earlier, which will comprise manned partly manned/partly unmanned units. All of these considera- tions play a key role in, for example, the advance discussions on the development of a new European battle tank system. and unmanned units, and where perhaps one unit will be able to see the target, while another one has to fire at it hen it comes to creating these unmanned units, we have extensive So there’s a lot of research and development work to do. experience with autonomous systems in hostile environments. What kind of contribution will Fraunhofer IOSB be able to make here? Arens: We have expertise in a whole range of technologies that will be crucial for meeting these challenges. For example, we have been developing systems to help evaluate sensor data and other information or fully automate these processes for many years now. This process starts with data acquisition – for example, through the use of planning tools for sensor deploy- ment – and includes the requisite IT systems and architectures for data storage and data sharing between alliance partners. It also involves not only the algorithms used to extract and eval- Dr. Michael Arens, a computer scientist, is spokesperson of the Defense business unit. 45

Dazzling of optronic sensor systems Detecting alterations to linear-type infrastructure Over the past decade, the use of compact, commercially Roads, railroads, power lines, pipelines, dams or levees are available, high-power continuous wave (CW) “laser pointers” critical installations. Aerial photography is used to check has increased among the general public. There have also been the integrity of such infrastructure. In order to simplify this regular reports in the news of such lasers being misused, especially with people aiming them at low-ying aircraft dur- ing the hours of darkness. Bright light, such as from a laser process, we have developed a new module for our ABUL system, which provides automatic analysis of images recorded by unmanned aerial vehicles. This module analyses two or pointer, produces dazzle effects that interfere with the pilot’s more series of images recorded at intervals of several hours or vision, thereby risking disaster. days and detects any differences that could indicate structural damage. In order to gain a better understanding of situations where aircraft pilots are dazzled with handheld laser pointers, Fraunhofer IOSB too part in a oint NATO SET- field trial, performed at WTD 52, Oberjettenberg, Germany. This facility is equipped with a cable car and is ideal for slant-path exper- Our preliminary investigations showed that the ight pattern of a UAV can be systematically reproduced to a degree of ex- actitude that satisfies the reuirements of change detection The major challenge is to locate the exact reference images iments between the base station and the cable car, where within a large volume of data. This was accomplished with the sensors to be dazzled were mounted. Laser dazzling took the VABUL video database. We then evaluated two different place during dusk and nighttime. The investigation focused on a number of aspects concerned with laser dazzling: aiming accuracy, the inuence of light scattering from the aircraft canopy, the detection of laser irradiation, the inuence on optronic systems, and protection against laser irradiation. methods of change detection: video-based change detection, in which two video films are registered to one another and compared on the basis of their coverage of the same scene; and mosaic/single-image change detection, in which individu- al images are registered to one another and compared. Laser propagation is subject to atmospheric effects, which is one of the topics focused on by the Signatorics department at Fraunhofer IOSB. Laser sources for various applications are In terms of being able to detect changes in the images, the system is promising owever, it still has difficulty distinguishing relevant changes in the images from changes in image quality developed by our Laser Technology department. The Optronics resulting from, for example, shadows. To remedy this, we are department researches into the effects of laser dazzling on optronic sensor systems and corresponding protection meas- ures. And, last but not least, the Object Recognition depart- using deep learning to teach the system how to recognize and filter different classes of obect In the image above, the algorithm has been able to identify buildings, cars, trucks and ment investigates its impact on image exploitation. construction vehicles, visualized by different outline colors). CONTACT Dr. Bernd Eberle CONTACT Dipl.-Inform. Norbert Heinze, Group manager Group manager Countermeasures and protection Image-based reconnaissance and surveillance Phone +49 7243 992-0 Phone +49 721 6091-254 47

”A CROSS-SECTORAL VIEW OF THE ENERGY TRANSITION“ Sensor data management, data analysis, data protection: future challenges require interdisciplinary knowledge and skills Energy, Environmental and Security Systems – that seems like a broad spectrum. Where is the common ground? electricity without modern grid operation and control systems. Innovative ICT solutions will also enable the emergence of new business models for decentralized energy services such Peter Bretschneider: The sectors share many concerns, not as smart charging stations for electric vehicles and neighbour- least the energy transition. By 2050, Germany intends to cut its CO2 emissions by 80 percent compared with the 1990 baseline. This goal cannot be achieved simply by shutting down coal- fired power plants and replacing them with wind or solar ener- gy. Until now, the energy industry has provided the lion’s share of CO2 reductions. In the future, the production, transportation and building sectors will have to do more. That’s why we need hood-related power generation. These solutions involve collecting large quantities of data – something that many companies and private individuals feel uncomfortable about. Bretschneider: That’s true. Data must be protected, but to take a cross-sectoral view of the energy transition – some- we also need this information in order to perform our work. thing our business unit is eminently equipped to do. Can you quote an example? We use electricity consumption data to predict grid loading, analyze energy efficiency, and plan how to distribute energy as exibly as possible An even more sensitive issue is the exploita- tion of video data, such as that recorded by security cameras. Bretschneider: In the Bauhaus.MobilityLab Erfurt research That is why our institute takes great care to ensure that the project we are developing an innovative IT ecosystem that applications we develop comply with data protection, IT secu- combines smart mobility, logistics and energy solutions at a rity, and data privacy regulations. As well as cooperating with local urban level. It was one of the winning projects in the AI the Fraunhofer-wide Cybersecurity Lab, we have also set up a Innovation Competition of the German Federal Ministry for dedicated research group to deal with data protection and its Economic Affairs and Energy (BMWi), out of 135 entries. Solu- enforcement by means of privacy-by-design approaches. tions like this involve many of the skills that our business unit can provide, starting with IoT sensor technology, data man- agement and user interfaces and extending to issues such as artificial intelligence and IT security For instance, we can mae use of the Fraunhofer Open Source SensorThings API Server (FROST®) for sensor data management, and our established EMS-EDM ROET® solution for energy data management, energy forecasts and energy usage optimization. Smart ICT solutions will become increasingly important in the future. Already today, it would be impossible to assure the supply of Prof. Peter Bretschneider holds the Chair in Energy Usage Optimization at Ilmenau University of Technology and heads the Energy Systems department at Fraunhofer IOSB. 49

Crowd monitoring, prediction and decision support Open-source sensor data platform for analysis and semantic editing Public sporting events, cultural fairs and folk festivals attract large crowds. Dangerous situations may occur due to the In 2019, Fraunhofer IOSB launched CrossTEP-OPAL, an large number of people. Organizers of such events are often in-house research project to develop a web-based platform overwhelmed by these responsibilities, hence the need for toolbox for the recording, visualization and intelligent analysis new solutions and innovative ideas. As part of the BMBF-fund- of sensor data. This project connects and extends products ed research project “Safety in urban environments: Crowd developed by our institute, such as FROST®, our implementa- monitoring, prediction and decision support”, or S²UCRE for tion of the SensorThings API for sensor data management, and short, we have been investigating new emergency rescue and WebGenesis®, a framework for the generation and support security technologies for events that attract large numbers of of web-based information systems, their visualization and people, in collaboration with research partners and potential semantic integration. Other sources used include Industrie 4.0 users in Germany and France. knowhow (OPC UA), existing data analysis tools (EMS-EDM Prophet), and activity recognition algorithms for video material. In such complex situations, hidden dangers lurk everywhere. Our solution combines in-situ video monitoring to measure or estimate the size and density of the crowd, and the rate at The proect is structured around typical use cases in the fields of video data analysis, energy consumption predictions, the which it is moving, including the detection of anomalies, with recording and analysis of environmental data, and outdoor an ex-situ simulation of the short-term dynamics of crowd movements. We also employ a privacy-by-design approach to preserve individual rights as far as possible Efficient command and control is assured by means of an intuitive presentation localization. These use cases are representative of hypothet- ical – but realistic – customer proects in the fields of Data Engineering and Data Science As a first step, the available sensors were integrated and their measurements visualized, of situational awareness data and software that provides a using newly developed visualization formats were necessary. In dynamic analysis of emerging dangers, enabling rescue teams the second year of the project, we are now focusing on devel- to prepare escape routes and coordinate their interventions. By containing riss more efficiently, better crowd control can be provided The efficacity of the new solution was successfully demonstrated in 2019 at one of the two reference events: the festival to celebrate the anniversary of the port of amburg, during which the organizers ran the system 24/7. oping a framework for integrating algorithms and obtaining deeper insights from the collected data. Thus, the toolbox will offer a web-based data science pipeline that will enable projects involving sensor data to be implemented rapidly and at low cost – from smart city applications and environmental information systems to energy management systems and solutions to improve public safety. s2ucre.de iosb.fraunhofer.de/servlet/is/105429 CONTACT Dr. Jürgen Metzler CONTACT Dr.-Ing. Jürgen Moßgraber Video-based security and assistance systems Group manager Architecture and information systems Phone +49 721 6091-453 Phone +49 721 6091-562 51

”HELPING TO GUARANTEE WORLD FOOD SUPPLIES“ Expertise in inspection and remote sensing for agriculture and food chains Professor Längle, sorting systems for items including foodstuffs have long been a key domain of Fraunhofer IOSB. Does the institute have any other connection to food? Thomas Längle: A lot, really, when you look at our activities. In the field of visual inspection, we have a lot of experience in capturing and analyzing a whole variety of spectral data. the Agriculture Data Space, where it gets analyzed, along with other data, in order to produce, say, a fertilization and water- ing schedule tailored to the crop’s precise requirements. This is a vision our SPR, MRD and SZA departments are working on right now in the Fraunhofer lighthouse project COGNAC (Cognitive Agriculture) and other projects. What other opportunities are there further down the We’ve been collecting spectral data from plants and food- value chain? stuffs, for example, for a number of years now. We’ve also been woring for uite a while now on ways of efficiently analyzing remote-sensing data, and we have expertise in robotics and autonomous systems. Besides that, we also have expertise in overarching areas such as AI, sensor technology, Längle: Not only plants in the field but food products as well can be uicly classified using spectral sensors – whether for sorting newly harvested crops, for real-time control of specific properties, or actually in the store or at the consumer’s home data management and system architecture. So taking all that in order to check, for example, whether a foodstuff is edible into account, we’ve got all we need to drive the advance of or already spoiled. We’re seeking to develop a small, mobile digitalization in agriculture and right along the food chain. food scanner (see overleaf) – along the lines of the tricorder What does that mean exactly? food ends up in the garbage can because the best-before date has expired. from Star Wars – and thereby ensure that in the future less Längle: Agriculture is undergoing a profound change. We need new approaches to ensure we can sustainably feed humanity in the face of population growth and climate change. A major hope is that digital technology will help boost the efficiency of food production in an environmentally friendly way. To give you a concrete example: optical or, to be more precise, hyper- spectral sensors systems fitted to agricultural robots, drones or even satellites can be used to automatically monitor the condition of plants and soil on both a large and small scale. Using state-of-the-art algorithms, multivariate analysis and AI, we can determine the health and ripeness of plants, based on the inspection of an individual leaf, or analyze diverse soil parameters. This data is sent to a cloud-like resource called Prof. Thomas Längle serves as spokesperson for the Inspection and Optronic Systems business unit. 53

LDV measurement of vibrations of a wind turbine Light field technology As everyday experience shows us, the vibrations caused by appliances and machinery such as cars, washing machines or A light field display simultaneously displays different visual aspects of a scene from different angles. Some game consoles computer fans are a mine of information. The same applies to already use this principle to display two images – one for each wind turbines, where vibration sensors are employed during eye – from a slightly different perspective and thereby gener- design and development phases as well as for monitoring purposes during actual operation et the difficulty involved in operating such sensors often restricts their use to a bare min- imum. For example, permanently integrated sensors often fail ate a 3D image without the need for 3D glasses. At Fraunhofer IOSB, however, we are generating fully customizable light fields for applications in which the image varies according to the observer’s position. These can be used, for example, to before the end of life of the appliance they are monitoring. simulate 3D structures behind the plane of the monitor or to create light sources that illuminate individual areas differently. Alternatively, a laser Doppler vibrometer (LDV) can be used. When a laser beam is projected onto the surface of a moving object, this light is refracted and undergoes a Doppler effect A light field emitter is a planar light source in which both the position and direction of light emission can be varied. Our pro- dependent on the object’s velocity. Using an interferometer, totype combines a monitor with an array of lenses mounted the refracted light is compared with the original beam, thereby in front of it at a distance corresponding to the focal length of determining the frequency shift. This technology is commer- cially available and can be used for remote measurement of vibrations on surfaces accessible to laser projection. For wind turbines, we are developing optimized LDV techniques. This relates to two aspects: 1. Increased range so as to be able to conduct measurements from the ground and at a suitable angle to the turbines, which are often over 100 meters in height. Separation of the vibration signal from the significantly more intense, regular movements, not least the rotation of the rotor blades. For reasons of eye safety, we use a short-wave infrared laser The signal is filtered and analyzed with a high-freuency analog-to-digital (A/D) converter. Initial trials with the system the individual lenses. Whenever a pixel is activated behind an individual lens, the light field monitor emits a parallel bundle of rays in a direction determined by the position of the activat- ed pixel behind the individual lens. This generates a 4D light field L x,y,θ,φ). The spatial resolution depends on the number of individual lenses in the lens array; the angular resolution on the number of monitor pixels behind each individual lens. In order to be able to control the light field, it is necessary to know which monitor pixel is responsible for emitting light in a specific direction The assignment of each monitor pixel to a spatial pixel (individual lens) and a directional pixel (angle of have shown that it is possible to gather vibration measurements observation) is conducted by means of a calibration routine. of the rotor blades while the wind turbine is in operation. iosb.fraunhofer.de/servlet/is/26555 iosb.fraunhofer.de/servlet/is/104816 CONTACT Dr. Ilja Kaufmann CONTACT Christian Kludt M.Sc. Project manager laser Doppler vibrometry Phone +49 7243 992-122 Project manager Light field technology Phone +49 721 6091-659 55

OPTRONICS Laser Technology Optronics Signatorics Visual Inspection Systems SYSTEM TECHNOLOGIES Digital Infrastructure Interoperability and Assistance Systems Information Management and Production Control Machine Intelligence Systems for Measurement, Control and Diagnosis Energy Systems Water and Mobile Systems IMAGE EXPLOITATION Interactive Analysis and Diagnosis Object Recognition Scene Analysis Video Exploitation Systems Variable Image Acquisition and Processing (research group) 58 – 59 60 – 61 62 – 63 64 – 65 66 – 67 68 – 69 70 – 71 72 – 73 74 – 75 76 – 77 78 – 79 80 – 81 82 – 83 84 – 85 86 – 87 88 57

1 2 and high-energy laser weapons. The Laser Technology depart- ments current research and development into m fiber laser targets three main fields of operation 1. We are conducting innovative studies on active doped fibers with enhanced chemical and geometrical designs in order to research and develop kilowatt-class 2 µm continu- ous wave fiber lasers 2. Other applications require 2 µm pulsed emission together with high average power in different temporal regimes with pulse widths varying from microseconds up to fem- toseconds to achieve this, we test, fine-tune and com- pare different topological arrangements of fiber and fiber components. 3. e also optimize fiber lasers to pump nonlinear converters in demonstrators. As one of our most recent achievements, we believe we are the first to demonstrate a Tm-o co-doped triple-clad fiber laser in CW and Q-switched operation modes. Installation of new laser laboratories Our new laboratory boasts state-of-the art equipment for researching and developing m-emitting fiber lasers In a cleanroom environment, we use machines such as cleavers, splicers, recoaters and various analyzers to prepare and pro- cess active fibers, inspect and ualify them, and to manufac- ture dedicated fiber components The lab also features several optical tables, where we assem- ble fibers and fiber components using our own innovative laser architectures. Some tables are reserved for high-power continuous-wave lasers, while others are used for pulsed lasers delivering high average power. We use cutting-edge equipment to characterize and qualify laser beams. The laser sources efficiency, the spatial uality of the beams, their polarization states and their temporal behavior are among the output parameters we optimize to meet the specific reuire- ments of optronic applications. The department also equips other laboratories dedicated to the development of MWIR solid-state laser sources, non-linear converters in mid- and longwave-infrared wavelengths, as well as crystal growth to provide innovative high-performance active materials. In the near future, when all the labs are fully operational, the department’s research teams will be able to collaborate with other IOSB divisions, so that their R&D work can provide a global response to our stakeholders’ needs and challenges. 2 µm fiber laser research Fiber lasers based on active fibers doped with thulium or hol- mium ions are an efficient way to access the m wavelength region. They can be used in a wide array of applications, including clinical medicine, materials processing, atmospher- 1 Making fiber end caps. ic measurements, free-space optical communications, lidar sensors, longer-wavelength laser pumping, countermeasures 2 2 µm Tm-doped fiber laser in operation. 59

z (m) 80 40 0 -80 1 Laser spot -40 0 y (m) -80 -40 0 40 80 40 80 x (m) 2 Laser beam propagation in a maritime environment We are developing a detection system based on quantum Using lasers safely, especially in outdoor environments, calls By exploiting the quantum correlation of entangled photon for a complex analysis and understanding of the scene – pairs, it is possible to physically separate illumination and including identifying areas that could receive light intensities imaging into different parts of the spectrum. With our setup, technology that transcends these and other traditional limits. above a critical level. In a maritime environment, apart from the usual line-of-sight propagation, we also have to tae reections from the seas surface into account. Conducting outdoor experiments to study reections from water surfaces in a variety of weather conditions, however, is extremely expensive and difficult we illuminate objects in the eye-safe infrared region while they are detected in the visible range using low-noise single photon avalanche detectors (SPADs). Since each object’s spatial infor- mation is obtained from photons that have never interacted with the object itself, this imaging technique is referred to as quantum ghost imaging [3]. Furthermore, quantum ghost imaging exploits the temporal As an alternative, we are developing a software tool that correlation of the entangled photon pairs to separate signal simulates a maritime scene and calculates laser beam propaga- photons from noise and achieves a noise-free imaging scheme. tion through the atmosphere and its interaction with the sea’s surface. The water surface, the atmospheric turbulence and the [1] Schwenger, F.; Scherer-Negenborn, N.; Braesicke, B.: Simulation of atmospheric beam scattering are simulated at different levels the interaction of a high energy laser beam with the sea surface in of accuracy ranging from a rough approximation to highly the short wavelength infrared, Applied Optics Vol. 58, Issue 29, detailed, depending on what is requested [1]. Ultimately, we pp. 8126-8140 (2019), https://doi.org/10.1364/AO.58.008126 will be able to estimate the areas and the statistics for critical [2] Göhler, B.; Lutzmann, P.: Optical Engineering, 56(3), 031203 (2016). exposures. [3] Pittman, T. B.; Shih, Y. H.; Strekalov, D. V.; Sergienko A. V.: Phys. Rev. A 52, 3429 (1995). Quantum ghost imaging for remote sensing Modern imaging systems for military, security and disaster scenarios are generating a growing demand for imaging techniques capable of returning high-resolution images under adverse conditions [2]. 1 The hemisphere represents the reflected laser power at a distance from the laser spot (in red) on the ocean’s surface. The color code Current active imaging systems combine the illumination of – from dark blue to green to orange and finally yellow – indicates remote objects with time-resolved detection techniques to low to high energy. image objects through scattering and turbulent media. These systems rely on illumination with short laser pulses and de- 2 Our quantum ghost imaging experiment: A special crystal (on the tection of backscattered light with background-noise-limited right) converts laser light into entangled pairs of photons, each cameras. The success of this traditional approach is limited by consisting of one photon in the infrared and one in the visible eye-safety considerations as well as background noise from range. The former ones illuminate the object, the latter ones are the camera system and the environment. detected with high spatial resolution by a silicon chip (left). 61

Input IR Image Receptive field center Model excitation Receptive field surround Photoreceptor Horizontal cell Bipolar cell ON-center OFF surround ganglion cell To optic nerve 1 2 Adaptive optics for free-space communications Assessing camouflage using a conspicuity rating inspired by biology The potential of free-space optical communications in civilian and military contexts is vast owever, such systems are severely limited by atmospheric effects. We work to counter these effects and enable high-data-rate terrestrial and space- to-ground optical communications. The advantage of free-space laser communication technology is that it does not depend on fibre between the terminals This makes it a promising solution to the “last mile” problem, i.e. For military units, good camouage reduces their chances of being detected and makes it harder and more time-consuming to identify them Assessing camouage and finding a way to rate conspicuity quantitatively is vital to their very survival. Since camouage is designed to foil not only human observers but also the artificial neural networs in use as of late, we focus on both biological and artificial image processing providing bandwidth in rural regions and emergency services We model neuronal structures in the human visual pathway in remote areas, especially in combination with satellite links. that respond to local anomalies in images. This helps us rate Military applications include the ability to transfer intelligence, the visual attraction of regions in images or videos. With no surveillance, and reconnaissance data effectively. Laser-based prior knowledge of where the objects are positioned, our satellite and deep-space communications are a special case. model accounts for a range of factors – local discontinuities, Observation satellites are everywhere, and their imaging textural similarites of objects compared to the background, sensors produce vast quantities of data. So what about the data traffic bac to the ground terminals A low earth orbit satellite can produce up to 10 terabits of data per day. The typical bandwidth of an X-band channel is currently around 800 megabits per second, implying that the satellite would have to spend at least 15 percent of its time in the sightline of the ground terminal, using up all of its bandwidth just to transmit its data. and the transition between object and background – as it processes the whole image in different spatial sampling resolu- tions This lets us rate the conspicuity of camouaged obects in relation to the background, and yields local conspicuity val- ues for a large set of statistical parameters for each image that are biologically significant These can be combined to form an overall visual conspicuity map. Going forward, the model will use eye-tracing fixation maps to resemble a human observers performance more closely. The atmosphere can have a major impact on the performance of laser-communication systems. Apart from completely dis- ruptive phenomena such as rain and fog, turbulence affects the power delivered and the number of errors in the transmis- sion. Fraunhofer IOSB addresses this problem with advanced e also investigate the effect of camouage on convolutional neural networs designed for obect classification, both self- trained and pre-trained. These methods are currently being used on static images as well as ight approach videos adaptive optics (AO) technology. In the AO laboratory, we develop solutions to the most challenging issues of laser propagation through serious turbulence. We are currently investigating holographic wavefront sensing, wavefront- sensorless AO, and adaptable Shac-artmann sensing Our labs test these strategies in simulated and real environmental conditions. 1 Design of the new Laser Communications Laboratory. 2 Calculating local conspicuity (highlighted in red to yellow in the bottom left image) using a modelled neuronal structure of the human visual pathway (right). 63

1 2 Using machine learning to reliably detect foreign bodies in natural bulk goods enough to endanger the entire crop’s marketability as a medic- inal product Since monitoring fields in the continuous, regular manner necessary and removing the weeds mechanically Machine learning has the potential to master many challenges is hardly economically feasible, downstream quality control in visual inspection. One option we use is employing data-driven of these harvests is an important way to remove potential procedures to distinguish the different materials’ spectra from PA-containing plants. one another. When challenges are especially demanding, a sufficient number of data points need to be gathered to train an appropriate classifier On top of that, nowledge of the specific domain to which a given tas belongs must also be figured in To achieve this, we wor closely with our customers to integrate this knowledge and also leverage our contacts at In a joint project sponsored by the Fachagentur für Nach- wachsende Rohstoffe (FNR), we are working with the Julius Kühn-Institut Berlin (Institute for Ecological Chemistry, Plant Analysis and Stored Product Protection) to develop a viable detection method. It aims to use hyperspectral near-infrared other Fraunhofer institutes. This is the key to mastering tasks spectroscopy to detect contamination by weeds containing such as classifying wood, plastics, minerals, foodstuffs and PAs in harvests of medicinal and herbal plants. Early exper- tobacco. iments have shown that it is possible to classify varieties of plants using NIR spectroscopy. The aim is to sort out the A case in point: sorting tobacco, which calls for removing unwanted plants in an optical bulk goods sorting system with foreign bodies that look very similar to tobacco in the RGB this technology. We are currently developing a prototype to color spectrum. By using an RGBN camera (RGB + near-infra- purify harvests in real time that will make lowering the health red channel) and classic machine learning procedures, how- risks posed by PA-contaminated medicinal plant products envi- ever, we can make the foreign bodies visible. To achieve this, ronmentally friendly and economically feasible. we use a process known as dimension reduction to transform information from four channels to three. This yields an image with false colors that puts the information together in the ideal way to perform this particular task. Preprocessing hard- ware performs the transformation in real time, making it useful for industrial sorting systems. Detecting pyrrolizidine alkaloid-containing weeds in crops Pyrrolizidine alkaloids (PAs) are secondary matter that plants produce to protect against insect herbivores. PAs also cause 1 Wood and tobacco: while barely distinguishable in the RGB image, liver damage. When harvested unintentionally along with they show a clear contrast in the transformed four-channel image. products such as organic and herbal teas, they can result in problematic alkaloid exposure, which is why the PA limits 2 Blended sample of target plants and unwanted PA-containing considered safe were tightened in recent years. As a result, as few as four to five A-producing plants per hectare may be plants. In the hyperspectral image on the right (false color image), the blue tint makes the unwanted plants easy to identify. 65

1 2 Visible light communications for industrial production Making industrial connectivity more flexible and secure facilities Manufacturing today is thoroughly networked: production machinery, conveyor systems, robots, sensors, monitoring systems and mobile terminal devices all communicate with one another and exchange data. This growth in the number of field devices that are connected within the production environment is stretching conventional wireless technology to its limits. The bandwidth provided by Wi-Fi and Bluetooth is limited, and not even 5G will be able to meet every demand. To implement the paradigms of Industrie 4.0, we will need pro- duction systems that are significantly more exible than those in conventional industrial facilities. And we must ensure that these systems have real-time capability and the requisite cybersecurity specifications This reuires research into methods for configu- ring production systems on a dynamic basis. In the FlexSi-Pro project, we are applying software-based methods of network control to industrial environments so as to leverage the added exibility this technology provides In order to deliver high standards of real-time communication and cybersecurity, we are Visible Light Communication (VLC) is an alliance project fund- ed by Germany’s Federal Ministry for Economic Affairs and looking at a combination of technologies such as software-de- fined networing, time-sensitive networing and O A Energy. Launched in 2018, its purpose is to investigate the use of the visible light spectrum for wireless data transmission. In this technology, data is transmitted by an LED and received by a photodetector, which converts the light signal into electrical pulses. By examining the challenges involved, the project is seeing to define the reuirements for the use of L in an industrial setting and to describe possible scenarios. The project aims to design, evaluate and demonstrate software- defined networs for the connectivity of future production plants. Such technology will enable a network administrator to manage data traffic centrally by means of control software and without the need for manual access to network compo- nents on the production site e first defined use cases and drew up a list of what future production facilities will require While signal interference is the major issue in data transmis- sion, other possible sources of disruption include artificial light and the shadows or reections caused by walls, metallic objects and machinery. A series of measurements has shown for full industrial connectivity. On this basis, we produced a generic framework that now serves as a platform for the de- velopment of exible Industrie production systems Tech- nical implementations of this have already been demonstrat- that dust particles, people or slow-moving vehicles (0.2 m/s) do not pose any problem. Sources of ambient light such as uorescent tubes, welding lamps or optical tracing systems affect the entire optical spectrum, but only on a local level. ere, tests showed that L systems are able to adapt to the lighting conditions, thus minimizing the impact of such interference By contrast, reections are a cause of significant interference, since they lengthen the propagation time of the ed. One demonstrator automatically incorporates a camera application in the networ and configures the infrastructure in such a way that time-critical information always arrives at its destination on time. Another focuses on security aspects and, if reuired, will dynamically revoe access to data traffic for specific networ participants 1 The VLC box is used to analyze optical components, the use of light light signal and distort it in the nanosecond range, thereby modulation for light-based data transmission, and the impact of diminishing transmission quality. On the basis of quantitative interference signals. measurements, we and our project partners are now develop- 2 Flexible, Industrie 4.0-compatible systems must be real-time ing environmentally adaptive VLC systems for industrial use. capable, dynamically configurable via software and nonetheless secure. FlexSi-Pro has demonstrated how this can work. 67

1. Heterogeneous data flow 2. Representational model 3. Semantic fusion Reasoning Trend analysis 4. Augmented intelligence Visual analytics 1 5. Situational awareness Court-proof forensic evidence 2 Data analysis for enhanced crime prevention The purpose of this three-year project is to broaden the ISTAR Launched in 2018, MAGNETO (Multimedia Analysis and naissance) capabilities of naval vessels through the use of Correlation Engine for Organized Crime Prevention and Inves- unmanned systems and their integration within the vessel’s (intelligence, surveillance, target acquisition and recon- tigation) is a three-year EU research project. Its purpose is to command and control system. enhance the ability of law enforcement agencies to prevent and combat organized crime and terrorist organizations. Work will also focus on improving processes for the fusion To achieve this goal, the project involves developing tools the EU’s own C4ISR architecture as a common platform for of information with heterogeneous data and on developing designed to process and analyze large volumes of heteroge- neous data, thereby significantly expediting the investigative process. These tools perform tasks ranging from evaluating the available information, securing evidence and fitting to- gether individual items of evidence through to preparing the court file all command and control systems. The latter will be based on open architecture and standards so as to guarantee inter- operability between EU-led and NATO-led forces. A further aim is to integrate fused sensor data into the plot known as the recognized maritime picture and to generally improve situation- al awareness in maritime environments. These new capabilities will be demonstrated in three simulated exercises and two live Crucial here is the ability to fuse information from demonstrations in the Mediterranean and the Baltic. a range of sources such as video, audio, text, social media, telecommunications, surveillance systems and police databas- es. At the same time, using analytic tools and a knowledge model helps improve situational awareness. Enhancing situational awareness in maritime environments OCEAN 2020 (Open Cooperation for European Maritime Awareness), launched in 2018, is a project involving research and industry partners from a total of 15 EU member states. Together with these partners, Fraunhofer IOSB has devel- oped a technology demonstrator for maritime surveillance missions and the prevention of hostile maritime maneuvers. 1 Architecture of the Horizon 2020 research project MAGNETO With funding of 38.5 million euros, OCEAN 2020 is the (Multimedia Analysis and Correlation Engine for Organized Crime largest project within the EU’s Preparatory Action on Defense Prevention and Investigation). Research (PADR) – an initiative leading up to the introduction of a European Defense Fund for the promotion of European research in this field 2 By fusing data from different sensors and integrating it into the recognized maritime picture, OCEAN 2020 will improve ISTAR capabilities. 69

1 2 Open Sensor Data Management with FROST® Industrial Communication with OPC UA and open62541 Sensor data is undoubtedly the most important building block The German initiative known as Industrie 4.0 refers to the in the Internet of Things (IoT) – a “thing” in this context being application of IIoT principles to industrial production. The any physical or virtual object that provides information (obser- Industrie 4.0 platform aims for interoperable solutions and vations) about a “feature of interest” in its environment. Given takes an open approach. One of the core technologies is the the enormous variety of things involved, standards are crucial IEC standard 62541 OPC UA. in order to maintain consistency through the whole chain of data acquisition, validation, contextualization and information We are fully on board and have developed major components processing. SensorThings API is an Open Geospatial Consor- of open62541, an open source, free implementation of OPC tium (OGC) standard that uses long-established concepts in UA [1]. open62541 is a C-based library (which can be linked sensor data description, while also meeting modern require- ments for efficient interfaces with C++ projects) with a small footprint. It features all the tools necessary to either implement dedicated OPC UA clients and servers, or to integrate OPC UA-based communication We decided to develop an open source implementation for into existing applications. open62541 supports client/server as the SensorThings API. The parameters: it needed to combine well as publish/subscribe (i.e. brokerless) interaction patterns. It high performance with low resource consumption, and at the runs on Windows, Linux, QNX, Android and various embedded same time be open source to facilitate usability in research as well as in commercial applications. The result is FROST® – the Fraunhofer Open Source Sensor- systems. Server implementations based on open62541 have been officially certified by the O Foundation to conform with the organizations Micro Embedded Device Server profile supporting OPC UA client/server communication, subscriptions, Things API Server. As the acronym suggests, it keeps data method calls and security (encryption) with basic security poli- “fresh and available.” FROST® has attained the status of an cies and the “method server” and “node management” facets. OGC reference implementation and is widely accepted in the geospatial community. It is in use in various environmental Our current focus is on extending OPC UA’s security features projects, including BRGM’s French water information network, (especially implementing the Global Discovery Service) and air-quality data in Taiwan and as a web-based API for our technological mapping to Time Sensitive Networking (TSN) to WaterFrame® information systems, as well as smart city appli- cations in places such as amburg and Munich In our role as co-chairs of the OGC working group, we in turn drive forward both the evolution of the SensorThings API standard and its enable deterministic and real-time communication. [1] http://open62541.org open source implementation. This places us in an excellent 1 In environmental risk management we systematically analyze the position to respond to our customers’ needs. requirements of multi-stakeholder use cases and implement them based upon information models and services of the Open Geospa- tial Consortium (OGC). 2 For the Industrie 4.0 we conceptualize digitalization and implement secure networking of manufacturing processes – such as in our model factory shown here – based upon international standards. 71

ML4P architecture Capturing data, information and knowledge Interactive analysis Training appropriate ML models Using ML to improve processes and in engineering 1 2 1 2 Legacy data Online data (streaming) Description of facility Expertise, etc. Process models Process parameters Control variables (on-/offline) Process monitoring, etc. 1 2 Robotic systems for decontamination tasks ML methods, we can identify previously unknown correlations Contamination with radioactive or chemical substances can have serious consequences for humans and the environment. So thorough decontamination is vital in the decommissioning of nuclear power plants and the remediation of contaminated sites. The people carrying out this work are subject to physically demanding conditions and may be exposed to severe health risks. in process data, derive reliable, data-driven process models and establish adaptive mechanisms that make production sys- tems more exible and open for short term reconfigurations In contrast to well-established ML applications that involve significant amounts of data of similar origin of data – image processing, speech recognition, social media and the like – industrial processes generate merely a “large quantity” of rather unstructured data requiring in-depth expertise for its Getting humans out of these hazardous environments is the interpretation. Optimizing these complex systems thus calls objective of ROBDEKON, a project that was launched with the for an integrated approach which seeks to continuously aim of establishing a national competence center for “Robotic Systems for Decontamination in azardous Environments In ROBDEKON, which is coordinated by Fraunhofer IOSB, we merge process data with human expertise. While the trend topic of deep learning is of interest in this context, there are also a wide range of other specially adapted ML methods that work with partners from academia and industry to engineer can work with less data while simultaneously mining troves of practical robot systems capable of performing the tasks existing knowledge. involved in decontamination. Based on various Fraunhofer institutes’ experience, there is a Our research includes the development of mobile robots for great demand for this approach both in the processing industry rough terrain, autonomous construction machines and robot and in manufacturing of standardized products – a demand manipulators. It also extends to decontamination concepts, planning algorithms, multisensory 3D mapping of the envi- ronment and teleoperation using virtual reality. Thanks to that is often coupled with a lack of ML expertise on the part of operators ML aims to specifically address these issues by adopting ML methods for manufacturing environment and by ROBDEKON’s effective algorithms, standard heavy machinery developing a process model for their sustainable deployment. can be equipped with autonomous capabilities. These include The MRD department’s R&D focus is on developing a software basic functions like autonomous driving as well as advanced architecture that enables the application of machine learning capabilities like autonomously performing decontamination methods in the production environment. tasks, e.g., removing a layer of soil. Machine Learning for Production (ML4P) The Fraunhofer lighthouse project “Machine Learning for Production” brings together six Fraunhofer institutes under the coordination of Fraunhofer IOSB. The group is developing a tool-aided process model and implementing interoperable 1 IOSB.BoB autonomous excavator. software tools that can systematically exploit ways for machine learning (ML) to optimize production processes. By introducing 2 ML4P architecture. 73

1 2 KnowledgeBrowser: Industrial process data at a glance XTEND: Multimodal support in production, job training and maintenance Generating added value from industrial data is a challenging proposition. It is not always clear what data is available or in what form, to say nothing of what it all means. In addition, the infrastructure that can integrate the data into the appro- Assistance systems are intended to counter the increased complexity that digital transformation, enhanced exibility and decentralization have brought to production. They do this by priate services is often lacking. On top of that, the systems and technologies on the shop oor are often a heterogeneous mix, which adds an extra layer of complexity. giving workers the information and help they need to perform specific tass, right at their wor stations Our TEND software platform accomplishes this by combining the advantages The KnowledgeBrowser enables digital services and users to find essential process data on the shop oor and receive it It indexes existing data, makes it searchable and passes it on to its final destination Data is integrated primarily via O A Additional knowledge is added through the Asset Adminis- tration Shell. It is continuously checked for any changes in the data on the shop oor, which are then synchronized with the knowledge databases. The system where the knowledge or of various modes of interaction used to support industrial activities. The product family comprises: XTEND|assembly (to guide the user through assembly processes), XTEND|mobile (to enable operation and maintenance of machinery via tablet or data glasses) and XTEND|remoteSupport (to connect work stations irrespective of their location). Building these products required the use of object tracking in order to ensure proper reproduction in augmented reality metadata is stored is an elastic database. An elastic API lets (AR). We also developed a cloud database to connect the users and services query the databases. The services can com- various software modules. In order to ensure that the platform municate with the databases directly, while users can search remains useful in the long term, we developed an editing manually with a web tool. The selected data, necessary configuration information and target services are sent to a middleware program to establish the data ow It creates and maintains the containers for the OPC UA data subscription and transport. This pipeline trans- mits the data from the shop oor to its designated destination Finally, the services can use the selected data to generate system with which even users with no coding skills can create and configure content This proect included worshops with participants from industry, research and education. Aside from the technical work, another key aspect of the proect was the development of new fields of application For example, the software platform can now also be used to help people with disabilities to perform certain tasks. Similarly, the value added. Overall operation is documented to make sure scope of XTEND|mobile has been further expanded to include that no resources are wasted by duplicating existing data job training. subscriptions or transmissions. ith the KnowledgeBrowser, companies can find and inte- grate the data available on their shop oor easily, so they can put it to work to enhance production. 1 The KnowledgeBrowser makes industrial process data more trans- parent, searchable and therefore much more useful for monitoring and other purposes. 2 Hardware- and application-independent: the XTEND system is used for assembly assistance, job training, service and maintenance, and commissioning. 75

1 2 A technology platform for efficient residential neighborhoods Cybersecurity training lab for the energy and water industries The smood® (“smart neighborhood”) growth hub is a Thur- The energy and water industries rely increasingly on IT systems ingia-based alliance of 17 companies, four research establish- ments and a nonprofit organization Its purpose is to develop an innovative technology platform that will enable property and automated processes. As such, they are more vulnerable to cyberattacks. Areas that are particularly susceptible include distribution grids and components as well as certain network developers, energy companies and residential tenants to realize efficient and environmentally friendly neighborhoods offering a high standard of living. Existing neighborhoods will create their own electricity grids featuring a high degree of self-sufficiency in terms of local power supply protocols. Scenarios range from undetected data theft and individual system failures to the ongoing disruption of company operations and supply outages. Effective protection against such threats requires an analysis of vulnerabilities in planning and operation, thorough risk assessment and the implemen- tation of preventive measures. This includes not only technical Such projects will employ energy and control technologies safeguards but also training for managers and personnel in provided by the smood team while also making use of existing order to facilitate the development and implementation of infrastructure. smood offers an integrated, digitalized plan- suitable security procedures at the organizational level. In ning process, ranging from innovative neighborhood storage addition to our customized in-house training courses, we also systems for electricity and heat to solutions for control and offer a training program covering the following areas: operational management as part of a systemic value creation concept These improvements will increase the energy effi- ciency, utility and desirability of the properties being renovat- ed and may also reduce the total cost of renting (including IT security in the energy and water industries Protecting power grids against cyberattacks Secure configuration of automation systems in the energy heating). industry We are working on the joint project smoodACT, which aims to achieve energy savings of at least 15 percent through IT security management in the energy and water industries IT security officers in the energy and water industries Secure data communications in the energy market cross-sectoral management of the neighborhood energy net- Data protection for data managers in the energy industry work and to halve the costs required for network development by tapping into opportunities for greater exibility within the neighborhood. In addition, there are plans to develop and trial a whole range of cross-sectoral energy services for buildings and entire Among the first customers for our customized in-house train- ing programs in 2019 were the power companies Thüringer Energie AG and Vattenfall AG. neighborhoods. Based on the smoodACT energy manage- 1 Launch of the smood growth hub in front of the Klimapavillon ment system, these range from tenant/owner utilization of in Jena. locally generated electricity to virtual power plants and smart recharging of electric vehicles. Prof. Peter Bretschneider, head of the Energy Systems department, is scientific spoesperson of the smood alliance. 2 Our cybersecurity training lab for the energy and water industries is used also for real-time simulation of dynamic effects in power grids carrying a large share of renewable energy. 77

1 2 Autonomy kit gives conventional vehicles connectivity and capacity for assisted removal of hazards Ferdinand-Braun-Institut fr ochfreuenztechni in Berlin, to research technical solutions for practical use. hen accidents involving radioactive, chemical or highly am- mable substances occur, or structures are in danger of collaps- These solutions aim to fully exploit the benefits LEDs offer compared with traditional, mercury-based light sources: LEDs ing, their removal is a job for specialized, remote controlled vehicles. But such vehicles are very expensive and take time to arrive at the scene of an accident. feature far higher mechanical stability, can operate at low volt- age and offer great exibility in their arrangement lus, their wavelengths can be configured with relative freedom, which makes them suitable for a great many uses. Wavelengths in A research consortium led by Fraunhofer IOSB’s AST branch the UV-C range (240–280 nanometers) are very effective at has now come up with an alternative and tested it successfully in the field an autonomy it nown as AKIT that converts universally available construction machinery – common killing microorganisms because they destroy the DNA in bacte- ria and viruses. excavators, for example – into unmanned recovery equipment. UV-C LEDs with a wavelength of 265 nanometers improve dis- AKIT is a fast way to add sensors and other components that infection quality by up to two orders of magnitude compared let vehicles maneuver autonomously in unstructured environ- ments. They also optimally support 3D-based assisted object manipulation and enable adaptive connectivity. All these with conventional mercury-vapor lamps with the same emis- sions This capability maes - LEDs an efficient weapon against multiresistent germs, and they work without chemicals functions help rescue teams remove hazards quickly. Thanks to its design as a retrofit set, AKIT enables conventional vehicles that are already on site to perform these extraordinary tasks, so there is no need to keep special equipment available around the world. Our consortium partners in Germany are the Fraunhofer IOSB in Ilmenau and Karlsruhe, Kerntechnische ilfsdienst Gmb in Eggenstein-Leopoldshafen, Götting KG in Lehrte and SGE- Gmb in irna Associated partners are laas KGaA mb in arsewinel, Liebherr Gmb in Bischofshofen, and Mann- heims fire brigade and emergency services, all in Germany, along with alfinger AG in Bergheim, Austria Innovative disinfection solutions using UV-C LEDs or antibiotics. LEDs are small and emit at high intensity – which makes them ideal for medical uses, such as the targeted disinfection of liquids, surfaces and hard-to-reach areas. Since LEDs are high-intensity spotlights, when placed in the right configuration they are an efficient way to create complex light geometries. We have a great deal of experience under our belts in designing LED-based UV-C light sources and simulating and optimizing the emission field – experience our customers benefit from every day Since 2014, Fraunhofer IOSB-AST has been working as part 1 Two machines equipped with the autonomy kit known as AKIT of the Advanced-UV for Life consortium to develop UV LEDs work together in the project’s final demonstration. and use them in various real-world applications. We col- laborate with tech partners that develop LEDs, such as the 2 The light from these LEDs kills microorganisms reliably. 79

1 2 QSelect and 4Crypt Video Image Exploitation Station of the Future Quality assurance (QA) is an important part of today’s man- Modern reconnaissance and surveillance systems still require ufacturing processes. But many QA tasks, such as surface humans to interpret the images. The ”Image Exploitation inspection of pressed metal parts and painted surfaces, cannot yet be fully automated in a cost-efficient manner This leaves many companies with no alternative to inspection by employ- Station of the Future” is an experimental system that demon- strates and investigates ways to help the people who perform this task. The system comprises a video surveillance simula- ees, who scan and visually inspect the surfaces. Yet frequently, tor based on an open architecture. It makes integrating and quality assurance is limited to mere seconds per component. In these cases, documentation must be kept to an absolute minimum to prevent an stop in production or an imprecise or testing new supporting components fast and straightforward, so sensor ights and a sensor controllers worstation can be simulated. even incomplete documentation of existing defects. Video image interpretation is vital to modern reconnaissance QSelect is a unique system that enables fast, precise, digital and surveillance systems. Various investigations have shown defect documentation directly on each component. It uses an that a combination of gaze-based interaction and automatic eye-safe laser pointer to intuitively illuminate the location of motion detection, object tracking and change detection signif- a defect. A sensor pinpoints the laser’s target in real time and the system digitally stores the location of the defect as 3D coordinates. Supplementary documentation can be provided via a freely configurable input menu proected onto the com- ponent, in which various types of defects can be selected. The icantly improves performance while reducing human stress. In order to boost efficiency in surveillance and reconnaissance, we explored options that let a human control a sensor and conduct observation tasks at the same time. documented defects are visualized through precise projection The interaction system we designed reduces perceptive and onto the component. cognitive stress so that both tasks can be performed simulta- neously. To demonstrate it, we simulated two video sensors The sensor’s cameras can also record the rework of the com- (LUNA and pan-tilt) and integrated a joystick, a gamepad and ponent for documentation purposes. Whenever the personal an eye tracker. In addition, we showed that an automatic data involved in this process needs to be stored, appropriate image interpretation tool can help the sensor controller in and effective security and privacy protection mechanisms detection tasks. are essential factors in gaining the trust and acceptance of employees and their representatives. 4Crypt Video provides a cryptographically secure way to store video footage of critical production work steps, such as safety-critical screws or weld seams. In order to protect workers’ privacy, it uses double verification videos are always encrypted with at least two keys, such as the authorized worker’s key plus a works council 1 QSelect enables fast, precise, digital defect documentation key, so that video recordings can only be decrypted with the via an eye-safe laser pointer and an input menu projected worker’s consent. 4Crypt Video uses authentication features onto the component. from directory services and already includes several different methods (smart card, password, face recognition). 2 Our Image Exploitation Station of the Future. 81

1 2 Analyzing change in 3D and assessing risk in urban areas Lidar-based UAV detection from mobile platforms Mobile laser scanning can capture large-scale urban environ- In the near future, both military and civilian vehicles will be ments in just a few hours. By taking multiple measurements equipped with a variety of sensors to enhance situational at different times, changes can be detected automatically, to awareness. As part of NATO’s SET-260 program entitled identify damage to roadside infrastructure or to update 3D “Assessment of EO/IR Technologies for Detection of Small city models. Additional research focuses on risk assessment UAVs in an Urban Environment,” we are investigating how involving special transports as well as the automatic planning well commercial mobile lidar system (MLS) sensors can protect of measurement campaigns. military vehicles against UAV attacks. MLS sensors are also used to produce 3D maps of operational areas and provide Calibrating the sensor setup involved (intrinsic calibration) context information for further analysis. while determining the transformation between sensors and vehicle coordinate system (extrinsic calibration) are essential in MODISSA, our experimental platform for hardware evaluation these applications. In order to avoid having to set up complex and software development, features four lidar sensors and an calibrations, we are working on methods to derive these EO and an SWIR camera. As it is acquired, the lidar data is syn- transformations directly from measured data. We calculate chronized with an inertial navigation system, transferring each point clouds based on the individual range measurements of 3D range measurement of each of MODISSA’s lidar scanners the calibrated sensors, and register them using simultaneous straight to a 3D point in geographic coordinates. This process localization and mapping (SLAM). This includes building a pose is known as direct georeferencing. Thus, as MODISSA drives graph based on navigational data and sensor measurements, which is modified using non-linear optimization to create as accurate a 3D image of the environment as possible. Data through an area, it automatically generates an accumulated 3D point cloud representing the area’s 3D geometry. Figure 2 shows a bird’s-eye view of France’s CENZUB/Jeoffrecourt military originating from several measuring campaigns are coregistered training facility. This 3D point cloud, which was acquired in 32 using a similar method. minutes, is made up of 5 billion 3D points in total. These coregistered data sets can be used to detect change automatically The first step is obect-based analysis to eliminate moving objects, for example. The actual change detection MODISSA’s current UAV detection method uses 360-degree scans from all of its lidar sensors to monitor the vehicle’s environment. If a potential threat is detected, the pan-tilt function then applies a hierarchical occupancy grid, which head is pointed in that direction to verify and identify the is adapted to the measurement data in an iterative way that UAV. This approach was published at the 2019 SPIE confer- optimizes the ratio between spatial resolution and required ence, Laser Radar Technology and Applications XXIV, DOI: memory. One particular advantage to this approach is that 10.1117/12.2518427, and extensively tested during the it maps unoccupied and unseen parts of the environment in CENZUB measurement campaign. addition to surfaces. In one of our measuring campaigns, we collected data in and an occupancy grid of the type used to detect change automatically. around the Technical University of Munich’s city campus and made it available to the public (Data MLS 1 - TUM City Campus). 2 Jeoffrecourt city center: screenshot of MODISSA’s onboard display with 3D lidar data and real-time detection. 1 Laser measurements of a street scene, iteratively approximated by 83

CAD model Simulation Reality 1 2 Scene simulation in thermal infrared SAR image exploitation supported by SAR simulation Machine learning with deep artificial neural networs has great potential to improve the way images are processed and understood. When large amounts of training data are available, this can deliver benefits in imaging reconnaissance, protecting critical infrastructure, and in environmental monitoring – as long as the images remain within the spectrum of visible light. Applications that depend on thermal infrared, however, often fall short, since producing labeled training data from infrared SAR image exploitation comprises signature analysis and fea- ture extraction, as well as change detection and classification based on single images, image pairs and image stacks. Two of the main challenges in this field are interpreting obect signa- tures in SAR data, which is more difficult than in optical data, and the availability of SAR data for functions such as collecting training data for deep learning classification In these cases, simulated SAR image signatures can provide interpretation images is far less common. aids and a large number of different object signatures. In seeking a workaround for this, we are exploring how far we The core technology of our department’s CohRaS® (Coherent can push the limits of thermal infrared, or long-wave infrared Raytracing-based SAR) simulator is an extended ray-tracing (LWIR) image simulation. LWIR is highly relevant in military approach that simulates both the amplitude and the phase of vision systems, since owning the night is a key capability. So the returned signal coherently. The simulation uses a 3D model virtual simulation systems for training and mission rehearsal of the object or a whole scene that can contain up to one should also enable LWIR observation of the scene. This is billion triangles. Performing the simulation requires some SAR indeed possible, but in the interest of achieving an acceptable sensor parameters and some information about the image frame rate, physical quality suffers. Physically derived emission simulation does not appear to work with real-time systems. creation process in the system being simulated. These include the wavelength, ight path, system resolution, final pixel size and windowing function used in the system. For radiometric In order to help training simulators generate more realistic correctness, different materials, such as vegetation, asphalt, LWIR images at a faster rate, we developed a thermodynamic water and concrete need to be assigned to different parts of model that describes 3D objects as layered surfaces. Relevant the model. surface features are reconstructed from remote sensing data derived from aerial images, satellite images and multi- or Exemplary applications of SAR simulation include the generation hyperspectral imaging. A processing pipeline reconstructs of training data for ATR, the analysis of critical infrastructure, geometry and surface materials simultaneously, and converts the results into the layered model eat can be transferred between these layers according to physical models. We also and change detection in complex environments. added a simple model of atmospheric effects, and more so- 1 Simulated thermal infrared image (left) in comparison with a phisticated features are underway. real thermal infrared image taken from a helicopter (right). Data acquired and pre-processed in cooperation with the University of The simulation results correlate well with the measurements, Perth, Melville, Australia. which means that using the simulated data as a source for training data to support machine learning classification methods is an option. This, however, is a subject of ongoing research. 2 Comparison between CohRaS® simulator’s SAR signatures and real SAR sensors: example of an ATR signature (top), of critical infra- structure (center) and a complex urban scene (bottom row). 85

1 2 Fusion of images and videos for 4D crime scene Operational products for civil security and defense investigations In forensic investigations, images and video data from witness- es and surveillance cameras are used to reconstruct the crime scenes and the course of events Nowadays, police officers typically view the individual camera recordings one at a time. ivisX is a software platform for law enforcement agencies. It is designed for criminal investigations involving mass video data. The modular system offers highly efficient processing and analysis capabilities, implementing tools like a map-based over- view of all sources, advanced algorithms such as image-based In cases involving huge amounts of data, the workload is split search methods and superresolution, and a handy tool for among several police units, which is very time-consuming. case-related documentation and preserving evidence. In an effort to improve and speed up the investigative work, ABUL is a real-time optimized system for online surveillance we partnered with the University of Konstanz in a Federal and tactical reconnaissance using live images from drones. Ministry of Education and Research-funded project known as Tailored to the needs of image interpreters, it boasts a large “Flexible, semi-automatic Analysis System for the Evaluation of Mass Video Data” (FLORIDA) to develop software that array of features designed to make their work easier and more efficient see page calculates the positions of all images (videos) from all cameras, fuses the results and computes a 3D scene. A temporal recon- NEST is a versatile suite of products that enable intelligent struction (4D) from all camera perspectives is also calculated video monitoring. It consists of activity recognition and crowd and merged with the 3D scene, allowing it to be played back monitoring/prediction capabilities and complies with privacy like a single 3D video, viewable from any angle [1]. The advan- regulations. Potential applications include crime prevention in tage here is that all video and image data can be viewed with public spaces, perimeter surveillance, detection of industrial a temporal reference, giving a clear view of the entire scene/ espionage and safety measures for large public events (see incident. page 51). It is also possible to interact directly with the 4D result in order to view trajectories of particular individuals, for example, or to tag special objects such as suitcases. The user can also go back to the original material at any time, choosing the best view of the current virtual camera perspective. Thanks to a virtual reality (VR) connection, investigators and witnesses can also access the crime scene. [1] “Computer Vision meets Visual Analytics: Enabling 4D Crime Scene Investigations from Image and Video Data,” 9th Int. Conference on 1 A simulated crime scene, viewable from any angle at any given Imaging for Crime Detection and Prevention in London. time (4D), computed from various camera recordings. 2 A developer is testing a variant of the ABUL system for UAV- based image exploitation in real time; it is to be used for tactical reconnaissance. 87

FACTS & FIGURES 2018–2019 8989

2 spin-offs are taking technology developed at Fraunhofer IOSB to the market (see page 26). 30 doctorates Thirty researchers working at Fraunhofer IOSB passed their doctoral examinations in 2018 and 2019. ≈ 2,500 participants attended special events and conferences organized or co-organized by Fraunhofer IOSB and its scientific staff. R 14 patents granted Applications for a further 33 patents were filed in 2018 and 2019. In addition, six product names were registered as trademarks. For all details on publications, academic teaching, lectures, technology transfer and outreach activities, please refer to the comprehensive online appendix to this annual report: iosb.fraunhofer.de/annual-report-2020 91

INTERNAL WORKING GROUPS Automotive und assistance systems Maritime technologies Chair: Dr.-Ing. Michael Voit | IAD Participating departments: IAD, MRD Consultants: Univ.-Prof. Dr.-Ing. habil. Pu Li, Chair: Dr. Dieter Willersinn | MRD Participating departments: MRD, OPT, SIG, SZA, WMS Prof. Dr.-Ing. Rainer Stiefelhagen Machine learning Big data Chair: Dr. rer. pol. Ljiljana Stojanovic | ILT Participating departments: IAD, IAS, ILT, MIT, NRG, VID Consultant: Prof. Dr. Michael Beigl Counter-improvised explosive device Chair: Dr. rer. nat. Ilja Kaufmann | OPT Participating departments: OBJ, OPT, SIG, SZA, VID Consultant: Prof. Dr. Maurus Tacke End-to-end software development Chair: Dr.-Ing. Jürgen Moßgraber | ILT Participating departments: DIS, IAD, IAS, ILT, MIT, MRD, NRG, SPR, VID Chairs: Dr.-Ing. Christian Kühnert | MRD, Dr.-Ing. Julius Pfrommer | ILT Participating departments: IAD, IAS, ILT, MIT, MRD, NRG, SPR, SZA, VID, WMS Consultants: Prof. Dr. Georg Bretthauer, rof Dr-Ing Michael eizmann, rof Dr-Ing Stefan inz, Prof. Dr. rer. nat. Bernd Jähne, Prof. Dr. rer. nat. Wolfgang Karl, Prof. Dr.-Ing. Rainer Stiefelhagen Surface inspection Chair: rof Dr-Ing Michael eizmann MRD Participating departments: MRD, OPT, SPR, VID Consultant: Prof. Dr.-Ing. Carsten Dachsbacher Consultant: Prof. Dr. rer. nat. Sebastian Abeck Multispectral data acquisition and exploitation Cognitive robotic systems Chair: Dr.-Ing. Christian Frese | MRD Participating departments: IAD, IAS, MRD, VID Chairs: Prof. Dr.-Ing. Thomas Längle | SPR, Dr.-Ing. Robin Gruna | SPR Participating departments and institutes: ILT, MRD, SIG, SPR, SZA, KIT - Institute of Industrial Information Technology, Consultants: Prof. Dr. Georg Bretthauer, KIT - Institute for Applied Biosciences Prof. Dr.-Ing. Rainer Stiefelhagen, rof Dr-Ing habil Brn ein Consultants: Prof. Dr.-Ing. Fernando Puente León, Prof. Dr.-Ing. Uwe Sörgel, Prof. Dr.-Ing. Carsten Dachsbacher 93

ADDRESSES Fraunhofer IOSB Karlsruhe Fraunhoferstraße 1 76131 Karlsruhe, Germany Telefon + 49 721 6091- 0 info@iosb.fraunhofer.de Fraunhofer IOSB Ettlingen Gutleuthausstraße 1 76275 Ettlingen, Germany Telefon + 49 7243 992- 0 Advanced System Technologies branch Fraunhofer IOSB-AST Am Vogelherd 90 98693 Ilmenau, Germany Telefon + 49 3677 461-103 Branch lab Görlitz Brückenstraße 1 02826 Görlitz, Germany Telefon + 49 3581 7925354 Industrial Automation branch Fraunhofer IOSB-INA Production and Information Technologies Fraunhofer Representative Office Beijing Representative for Unit 0610 Landmark Tower II 8 North Dongsanhuan Road Chaoyang District 100004 Beijing, PR China Phone + 86 10 6590-0621 Campusallee 1 32657 Lemgo, Germany Telefon + 49 5261 94290-22 Online channels iosb.fraunhofer.de twitter.com/FraunhoferIOSB linkedin.com/company/fraunhofer-iosb facebook.com/FraunhoferIOSB s.fhg.de/Youtube-IOSB 95

3