Big Data platform for the evaluation of streaming technologies

BIG DATA

Big Data Streaming

With the advent of digitization into mobility, the car has become an information and communication platform. New challenges arise in data processing and data transmission, within the framework of this paradigm shift from driver to autopilot with the goal of fully automated and networked driving. The modern car collects information on location, speed, road users and traffic signs, among other things. In order to process this high volume of data and to play back analysis results in the car, an almost real-time system is necessary. Therefore, the technical solution space for such an analysis system should be evaluated and implemented in the project. The goal was to identify the most powerful stream processor and design a virtualizable environment that simplifies deployment of stream processors and optimizes their resource utilization in the Hadoop cluster.

To identify the most suitable stream processor, a local environment was set up in the first step, which allows the different stream processors to be configured and scaled. The individual process steps of data processing were implemented as independent, reusable components. The advantage is that the stream processors can be used unchanged and thus remain comparable. Appropriate performance indicators (KPI) and user stories were defined as prerequisites for the evaluation. The KPIs refer to customer-specific requirements, such as latency, throughput or functionality in monitoring and the provision of streaming tasks. The user stories also cover the “soft skills” of the stream processors. The subsequent evaluation provided information on weak points and possibilities for optimizing data processing with the various stream processors and their individual components. In the next step, the developed proof-of-concept was established on the customer’s own Hadoop cluster. TIQ Solutions implemented this on the basis of a virtualized environment using containers and an easily scalable big data architecture.

A big data platform was provided to test and analyze stream processors. After completion of the evaluation, the customer received a concrete recommendation for the most suitable stream processor and the optimal configuration of the streaming data processing path. Our customer can now independently create and test data processing lines between his vehicles and the analysis system according to the application. In particular, it is possible to configure and scale the process steps with regard to validation, transformation and process flow (dispatching). By virtualizing the individual components with Docker, the user can easily create new test routes or copy and modify existing routes.

Graphic

Streaming Hadoop Cluster

autonomous vehicles à Hadoop cluster and container environment à external services and analysis system

Project Overview

Customer
German Car manufacturer

Customer Objective
Data processing of vehicle and traffic data

Tasks

  • Developing a big data platform
  • Virtualizing the platform environment
  • Comparing and evaluating the stream processors

Aim

  • Streaming route comparison and operation system

Customer Benefits

  • Independent creation and optimization of streaming routes
  • Easy provisioning of streaming components by containers
  • Scalability through big data architecture
  • Optimal use of cluster resources
  • Transparent evaluation of the efficiency of streaming

Technology

Spark, Samza, Storm, Docker, Kafka, Hadoop, HDFS, YARN, ZooKeeper

You might also be interested in these services

Data protection
We, the TIQ Solutions GmbH (Registered business address: Germany), would like to process personal information with external services. This is not necessary for the use of the website, but allows us to interact even more closely with them. If desired, please make a choice:
Data protection
We, the TIQ Solutions GmbH (Registered business address: Germany), would like to process personal information with external services. This is not necessary for the use of the website, but allows us to interact even more closely with them. If desired, please make a choice: