Data Engineering

The Challenge

With the advent of digitization into mobility, the car has become an information and communication platform. New challenges arise in data processing and data transmission, within the framework of this paradigm shift from driver to autopilot with the goal of fully automated and networked driving. The modern car collects information on location, speed, road users and traffic signs, among other things. In order to process this high volume of data and to play back analysis results in the car, an almost real-time system is necessary. Therefore, the technical solution space for such an analysis system should be evaluated and implemented in the project. The goal was to identify the most powerful stream processor and design a virtualizable environment that simplifies deployment of stream processors and optimizes their resource utilization in the Hadoop cluster.

German Car manufacturer

Data processing of vehicle and traffic data

The Solution

To identify the most suitable stream processor, a local environment was set up in the first step, which allows the different stream processors to be configured and scaled. The individual process steps of data processing were implemented as independent, reusable components. The advantage is that the stream processors can be used unchanged and thus remain comparable. Appropriate performance indicators (KPI) and user stories were defined as prerequisites for the evaluation. The KPIs refer to customer-specific requirements, such as latency, throughput or functionality in monitoring and the provision of streaming tasks. The user stories also cover the “soft skills” of the stream processors. The subsequent evaluation provided information on weak points and possibilities for optimizing data processing with the various stream processors and their individual components. In the next step, the developed proof-of-concept was established on the customer’s own Hadoop cluster. TIQ Solutions implemented this on the basis of a virtualized environment using containers and an easily scalable big data architecture.

The Result

A big data platform was provided to test and analyze stream processors. After completion of the evaluation, the customer received a concrete recommendation for the most suitable stream processor and the optimal configuration of the streaming data processing path. Our customer can now independently create and test data processing lines between his vehicles and the analysis system according to the application. In particular, it is possible to configure and scale the process steps with regard to validation, transformation and process flow (dispatching). By virtualizing the individual components with Docker, the user can easily create new test routes or copy and modify existing routes.

Graphic

Streaming Hadoop Cluster
Autonomous vehicles à Hadoop cluster and container environment à external services and analysis system
Wie wir Sie unterstützen können

Unsere Data Services

Data Engineering

Business Intelligence & Data Analytics

Data Strategy