Category: Big Data

Big Data platform for the evaluation of streaming technologies

[vc_row el_class="container"][vc_column][vc_btn title="Succes stories" style="classic" shape="square" color="btn-light-orange-blue" i_icon_fontawesome="fa fa-angle-double-left" add_icon="true" link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2F|title:Succes%20stories||"][/vc_column][/vc_row][vc_row el_class="container reference-mobile" el_id="reference-detail-layout"][vc_column el_class="reference-detail-layout-left .reference-mobile-order-2" offset="vc_col-lg-8 vc_col-md-8"][vc_row_inner][vc_column_inner][vc_single_image image="4179" img_size="750x495" el_class="reference-detail-layout-left-image"][/vc_column_inner][/vc_row_inner][vc_tta_accordion shape="square" gap="20" c_position="right" active_section="1" el_class="reference-detail-left-accordion"][vc_tta_section title="The Challenge" tab_id="1508250369004-11fd70c1-c50a"][vc_column_text]With the advent of digitization into mobility, the car has become an information and communication platform. New challenges arise in data processing and data transmission, within the framework of this paradigm shift from driver to autopilot with the goal of fully automated and networked driving. The modern car collects information on location, speed, road users and traffic signs, among other things. In order to process this high volume of data and to play back analysis results in the car, an almost real-time system is necessary. Therefore, the technical solution space for such an analysis system should be evaluated and implemented in the project. The goal was to identify the most powerful stream processor and design a virtualizable environment that simplifies deployment of stream processors and optimizes their resource utilization in the Hadoop cluster.[/vc_column_text][/vc_tta_section][vc_tta_section title="The Solution" tab_id="1508250369051-4a626d72-344d"][vc_column_text]To identify the most suitable stream processor, a local environment was set up in the first step, which allows the different stream processors to be configured and scaled. The individual process steps of data processing were implemented as independent, reusable components. The advantage is that the stream processors can be used unchanged and thus remain comparable. Appropriate performance indicators (KPI) and user stories were defined as prerequisites for the evaluation. The KPIs refer to customer-specific requirements, such as latency, throughput or functionality in monitoring and the provision of streaming tasks. The user stories also cover the "soft skills" of the stream processors. The subsequent evaluation provided information on weak points and possibilities for optimizing data processing with the various stream processors and their individual components. In the next step, the developed proof-of-concept was established on the customer's own Hadoop cluster. TIQ Solutions implemented this on the basis of a virtualized environment using containers and an easily scalable big data architecture.[/vc_column_text][/vc_tta_section][vc_tta_section title="The Result" tab_id="1508252697978-ef0a40d1-ea09"][vc_column_text]A big data platform was provided to test and analyze stream processors. After completion of the evaluation, the customer received a concrete recommendation for the most suitable stream processor and the optimal configuration of the streaming data processing path. Our customer can now independently create and test data processing lines between his vehicles and the analysis system according to the application. In particular, it is possible to configure and scale the process steps with regard to validation, transformation and process flow (dispatching). By virtualizing the individual components with Docker, the user can easily create new test routes or copy and modify existing routes.[/vc_column_text][/vc_tta_section][/vc_tta_accordion][vc_row_inner el_class="reference-detail-left-heading"][vc_column_inner][vc_custom_heading text="Graphic" font_container="tag:h3|text_align:left" use_theme_fonts="yes"][/vc_column_inner][/vc_row_inner][vc_row_inner el_class="reference-detail-left-image-text"][vc_column_inner offset="vc_col-lg-3"][vc_single_image image="4181" img_size="165x165" onclick="link_image"][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][vc_column_text]autonomous vehicles à Hadoop cluster and container environment à external services and analysis system[/vc_column_text][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][/vc_column_inner][/vc_row_inner][/vc_column][vc_column offset="vc_col-lg-4 vc_col-md-4" el_class=".reference-mobile-order-1"][vc_row_inner el_class="background-color-grey reference-detail-right-sidebar-one"][vc_column_inner][vc_column_text]Project Overview Customer German Car manufacturer Customer Objective Data processing of vehicle and traffic data Tasks Developing a big data platform Virtualizing the platform environment Comparing and evaluating the stream processors Aim Streaming route comparison and operation system Customer Benefits Independent creation and optimization of streaming routes Easy provisioning of streaming components by containers Scalability through big data architecture Optimal use of cluster resources Transparent evaluation of the efficiency of streaming Technology Spark, Samza, Storm, Docker, Kafka, Hadoop, HDFS, YARN, ZooKeeper[/vc_column_text][/vc_column_inner][/vc_row_inner][/vc_column][vc_column offset="vc_col-lg-4 vc_col-md-4" el_class="reference-mobile-order-3"][vc_row_inner el_class="reference-detail-right-sidebar-one-heading"][vc_column_inner][vc_custom_heading text="THESE SUCCESS STORIES MIGHT ALSO INTEREST YOU:" font_container="tag:h6|text_align:left|color:%23193d66" use_theme_fonts="yes"][/vc_column_inner][/vc_row_inner][vc_row_inner el_class="background-color-grey reference-detail-right-sidebar-two"][vc_column_inner][tiq_reference_link_box tiq_reference_link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2Fstate-analysis-platform-for-the-bi-support-of-an-automobile-manufacturer%2F|title:State%20Analysis%20Platform%20for%20the%20BI%20Support%20of%20an%20Automobile%20Manufacturer||"][tiq_reference_link_box tiq_reference_link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2Fintelligent-controlling-with-qlikview%2F|title:Intelligent%20Controlling%20with%20QlikView%C2%AE%20at%20the%20Karosseriewerke%20Dresden||"][/vc_column_inner][/vc_row_inner][/vc_column][/vc_row][vc_row el_class="container"][vc_column][tiq_start_service_boxes start_service_box_1_image="413" start_service_box_2_image="412" start_service_box_3_image="410" start_service_box_4_image="414" start_service_box_5_image="415" start_service_convert_as_slider="true" start_service_title="You might also be interested in these services" start_service_box_1_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fconsultancy%2F|title:Consultancy||" start_service_box_2_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fbusiness-intelligence%2F|title:Business%20Intelligence%20Solutions||" start_service_box_2_title="Business Intelligence" start_service_box_3_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fadvanced-analytics%2F|title:Advanced%20Analytics||" start_service_box_3_title="Advanced Analytics" start_service_box_4_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fgraph-data%2F|title:graph%20Data||" start_service_box_4_title="Graph Data" start_service_box_5_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Ftraining%2F|title:Training||" start_service_box_5_title="Training" start_service_box_6_url="|||" start_service_box_1_title="Consultancy"][/vc_column][/vc_row] ...

Big Data for digital television and telephony at Deutsche Telekom

[vc_row el_class="container"][vc_column][vc_btn title="Succes stories" style="classic" shape="square" color="btn-light-orange-blue" i_icon_fontawesome="fa fa-angle-double-left" add_icon="true" link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2F|title:Succes%20stories||"][/vc_column][/vc_row][vc_row el_class="container reference-mobile" el_id="reference-detail-layout"][vc_column el_class="reference-detail-layout-left .reference-mobile-order-2" offset="vc_col-lg-8 vc_col-md-8"][vc_row_inner][vc_column_inner][vc_single_image image="4131" img_size="750x495" el_class="reference-detail-layout-left-image"][/vc_column_inner][/vc_row_inner][vc_tta_accordion shape="square" gap="20" c_position="right" active_section="1" el_class="reference-detail-left-accordion"][vc_tta_section title="The Challenge" tab_id="1508250369004-11fd70c1-c50a"][vc_column_text]Digitization is well advanced in communications services. Today, landline telephony, television and internet are being offered from a single source via the same network infrastructure (triple play). Sound, image and data are transmitted IP-based via data packets. Telekom Deutschland has a nationwide, complex communication network. As a result of digital progress, new technologies and hardware manufacturers are constantly being introduced into the existing network. In order to keep pace with this dynamic, Deutsche Telekom uses a software system to analyze the quality and availability of its services and to monitor the utilization of the network infrastructure. In addition, Deutsche Telekom also planned to further expand the scope and sound quality of its HD programs and the functionalities of its services. The resulting increase in data volume would no longer have been controllable for the existing analysis system. TIQ Solutions, as a long-term consultant for the existing solution, therefore recommended switching to a new big data system.[/vc_column_text][/vc_tta_section][vc_tta_section title="The Solution" tab_id="1508250369051-4a626d72-344d"][vc_column_text]The architecture of the new big data system and its technological implementation were developed in workshops lasting several days together with the IT and the specialist department. For the first time, big data's powerful processing mechanisms made it possible to separate the technical data model from technical barriers during loading. TIQ Solutions also specified the necessary applications, their rights and the routing between the nodes in the cluster and the external QlikView server providing the visualization. The Oracle database has been replaced by Hive. Here, TIQ Solutions developed a generator that creates the various Hive table objects including their attributes for the more than 30 heterogeneous data sources, thus generating the database semi-automatically. Data processing was implemented with the Oozie Workflow Scheduler and replaced Informatica. For the Hadoop cluster, a rights and roles concept based on Kerberos was then designed and implemented. The historical data was initially migrated to the Hadoop cluster. For the business users, the usual access to the raw data via SQL statements was established via the HUE web frontend hue. During the migration of the business intelligence application QlikView®, TIQ Solutions primarily advised on optimizing the loading logic.[/vc_column_text][/vc_tta_section][vc_tta_section title="The Result" tab_id="1508252697978-ef0a40d1-ea09"][vc_column_text]The existing classic, purely relational database system was successfully transferred to a big data system. The new options eliminate aggregations and filters for limiting data volumes. This greatly increased the resolution of the data (granularity), the scope of the history and the consistency of the observable periods. This gives Deutsche Telekom a more precise picture of the state of its network and the current quality of the IPTV and VoIP services offered (Quality of Service). The resulting long-term observations are now more comprehensive and help to detect creeping state changes. The distributed processing in the cluster has made it possible to increase the load of raw data during the day and gives the department a quicker overview of the current transaction data. The use of the Cloudera Hadoop distribution and the integrated applications for data integration and data retrieval provided a consistent system. The operating costs of the software solution and the hardware could thus be reduced many times over.[/vc_column_text][/vc_tta_section][/vc_tta_accordion][vc_row_inner el_class="reference-detail-left-heading"][vc_column_inner][vc_custom_heading text="Graphic" font_container="tag:h3|text_align:left" use_theme_fonts="yes"][/vc_column_inner][/vc_row_inner][vc_row_inner el_class="reference-detail-left-image-text"][vc_column_inner offset="vc_col-lg-3"][vc_single_image image="4134" img_size="165x165" onclick="link_image"][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][vc_column_text]Migration of a relational database system (RDBMS) to a big data management and analysis system[/vc_column_text][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][/vc_column_inner][/vc_row_inner][/vc_column][vc_column offset="vc_col-lg-4 vc_col-md-4" el_class=".reference-mobile-order-1"][vc_row_inner el_class="background-color-grey reference-detail-right-sidebar-one"][vc_column_inner][vc_column_text]Project Overview Customer Telekom Deutschland GmbH / T-Systems International GmbH Case of application Migration of an analysis system for monitoring the image and sound quality of IPTV and VoIP on a big data platform Tasks Conception of the new system architecture Implementation of PoC Migration of business logic Migration of historical data Implementation of the security concept Aim Processing of large amounts of data More up-to-date, flexible and broadened view of the data Improved analysis possibilities on historical data Reduction of operating costs Generic processing processes More efficient data analysis Customer Benefits Reduce subscription costs Reduction of costs for data storage Improved informative value of the analyses Minimum expansion effort Efficiency for growing data volumes Technology Hadoop, Cloudera, Hive, HDFS, YARN, ZooKeeper, Oozie, Hue, Parquet, Kerberos, individual extensions with Java, Bash Scripting, Enterprise Architect, QlikView®[/vc_column_text][/vc_column_inner][/vc_row_inner][/vc_column][vc_column offset="vc_col-lg-4 vc_col-md-4" el_class="reference-mobile-order-3"][vc_row_inner el_class="reference-detail-right-sidebar-one-heading"][vc_column_inner][vc_custom_heading text="THESE SUCCESS STORIES MIGHT ALSO INTEREST YOU:" font_container="tag:h6|text_align:left|color:%23193d66" use_theme_fonts="yes"][/vc_column_inner][/vc_row_inner][vc_row_inner el_class="background-color-grey reference-detail-right-sidebar-two"][vc_column_inner][tiq_reference_link_box tiq_reference_link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2Fstate-analysis-platform-for-the-bi-support-of-an-automobile-manufacturer%2F|title:State%20Analysis%20Platform%20for%20the%20BI%20Support%20of%20an%20Automobile%20Manufacturer||"][tiq_reference_link_box tiq_reference_link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2Fintelligent-controlling-with-qlikview%2F|title:Intelligent%20Controlling%20with%20QlikView%C2%AE%20at%20the%20Karosseriewerke%20Dresden||"][/vc_column_inner][/vc_row_inner][/vc_column][/vc_row][vc_row el_class="container"][vc_column][tiq_start_service_boxes start_service_box_1_image="413" start_service_box_2_image="412" start_service_box_3_image="410" start_service_box_4_image="414" start_service_box_5_image="415" start_service_convert_as_slider="true" start_service_title="You might also be interested in these services" start_service_box_1_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fconsultancy%2F|title:Consultancy||" start_service_box_2_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fbusiness-intelligence%2F|title:Business%20Intelligence%20Solutions||" start_service_box_2_title="Business Intelligence" start_service_box_3_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fadvanced-analytics%2F|title:Advanced%20Analytics||" start_service_box_3_title="Advanced Analytics" start_service_box_4_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fgraph-data%2F|title:graph%20Data||" start_service_box_4_title="Graph Data" start_service_box_5_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Ftraining%2F|title:Training||" start_service_box_5_title="Training" start_service_box_6_url="|||" start_service_box_1_title="Consultancy"][/vc_column][/vc_row] ...

Traceability of the semiconductor-production by building a Big Data database

[vc_row el_class="container"][vc_column][vc_btn title="Succes stories" style="classic" shape="square" color="btn-light-orange-blue" i_icon_fontawesome="fa fa-angle-double-left" add_icon="true" link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2F|title:Succes%20stories||"][/vc_column][/vc_row][vc_row el_class="container reference-mobile" el_id="reference-detail-layout"][vc_column el_class="reference-detail-layout-left .reference-mobile-order-2" offset="vc_col-lg-8 vc_col-md-8"][vc_row_inner][vc_column_inner][vc_single_image image="4099" img_size="750x495" el_class="reference-detail-layout-left-image"][/vc_column_inner][/vc_row_inner][vc_tta_accordion shape="square" gap="20" c_position="right" active_section="1" el_class="reference-detail-left-accordion"][vc_tta_section title="The Challenge" tab_id="1508250369004-11fd70c1-c50a"][vc_column_text]Infineon Technologies AG is one of the largest global suppliers of semiconductor- and system solutions in the fields of energy efficiency, mobility and security. For its semiconductor production, Infineon produces semiconductor chips from silicon wafers for further processing. While passing through the production chain interferences may occur, which lead to defects on the chips. In the case of customer claims, the determination of error sources is rather complicated, due to the high number of resources and process steps involved. The determination of which semiconductor was build in any intermediate or final product behaves similarly. Since the productive traceability of products as well as of individual components from the origin through to completion and vice versa, the so-called forward- and backward tracing, is not possible straightaway, necessary actions cannot always be conducted. Within the context of a tender, Infineon has therefore sought for concepts to increase the transparency of the production process, which shall contribute to a better tracking of the semiconductor manufacturing and consequently also to a more proactive customer care.[/vc_column_text][/vc_tta_section][vc_tta_section title="The Solution" tab_id="1508250369051-4a626d72-344d"][vc_column_text]With the previous use of a classic relational database the capturing of the high number of production stages can be handled only on a limited level. Whereas the use of a column-oriented Big Data database will make it possible to store and process all the data of productions orders without restrictions. New options for the tracking of the production steps will be offered as well. Infineon decided to build an Apache HBase database. This database will now archive all the data from the semiconductor production as well as their history on an optimized level and provide them to analysis systems. As a result, all the process steps involved and all the resources can be tracked up to the starting product (the chip) at any time. A web-based analysis front-end with various specialist search masks allows permanent access to the Big Data database.[/vc_column_text][/vc_tta_section][vc_tta_section title="The Result" tab_id="1508252697978-ef0a40d1-ea09"][vc_column_text]The Big Data concept developed by TIQ Solutions was taken over by Infineon for the development of a prototype. An additional check by T-Systems Multimedia Solutions GmbH confirmed the efficiency of this solution as well. This Big Data concept enables Infineon to build a productive forward and backward tracing, which shall contribute to a faster and more straightforward error decoding. Due to the simplified traceability of the individual production stages internal and external sources of error can be clearly identified. Based on that, automatisms, which can prevent faulty semiconductor chips and will also allow a generalization of error patterns, can be developed. This prevents further processing of faulty chips and allows a customer and employee-friendly organization of complaints processes.[/vc_column_text][/vc_tta_section][/vc_tta_accordion][vc_row_inner el_class="reference-detail-left-heading"][vc_column_inner][vc_custom_heading text="Graphic" font_container="tag:h3|text_align:left" use_theme_fonts="yes"][/vc_column_inner][/vc_row_inner][vc_row_inner el_class="reference-detail-left-image-text"][vc_column_inner offset="vc_col-lg-3"][vc_single_image image="6437" img_size="165x165" onclick="link_image"][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][vc_column_text]Replacement of the classic relational database with a Big Data solution[/vc_column_text][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][/vc_column_inner][vc_column_inner offset="vc_col-lg-3"][/vc_column_inner][/vc_row_inner][/vc_column][vc_column offset="vc_col-lg-4 vc_col-md-4" el_class=".reference-mobile-order-1"][vc_row_inner el_class="background-color-grey reference-detail-right-sidebar-one"][vc_column_inner][vc_column_text]Project Overview Customer Infineon Technologies AG Customer Objective Transparency of all production stages Our Solution Integration of a Big Data database into the production process Customer Benefits Traceability of all production stages Faster error decoding Consumer satisfaction Support of the process automation Increased productivity [/vc_column_text][/vc_column_inner][/vc_row_inner][/vc_column][vc_column offset="vc_col-lg-4 vc_col-md-4" el_class="reference-mobile-order-3"][vc_row_inner el_class="reference-detail-right-sidebar-one-heading"][vc_column_inner][vc_custom_heading text="THESE SUCCESS STORIES MIGHT ALSO INTEREST YOU:" font_container="tag:h6|text_align:left|color:%23193d66" use_theme_fonts="yes"][/vc_column_inner][/vc_row_inner][vc_row_inner el_class="background-color-grey reference-detail-right-sidebar-two"][vc_column_inner][tiq_reference_link_box tiq_reference_link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2Fstate-analysis-platform-for-the-bi-support-of-an-automobile-manufacturer%2F|title:State%20Analysis%20Platform%20for%20the%20BI%20Support%20of%20an%20Automobile%20Manufacturer||"][tiq_reference_link_box tiq_reference_link="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fsuccess-stories%2Fintelligent-controlling-with-qlikview%2F|title:Intelligent%20Controlling%20with%20QlikView%C2%AE%20at%20the%20Karosseriewerke%20Dresden||"][/vc_column_inner][/vc_row_inner][/vc_column][/vc_row][vc_row el_class="container"][vc_column][tiq_start_service_boxes start_service_box_1_image="413" start_service_box_2_image="412" start_service_box_3_image="410" start_service_box_4_image="414" start_service_box_5_image="415" start_service_convert_as_slider="true" start_service_title="You might also be interested in these services" start_service_box_1_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fconsultancy%2F|title:Consultancy||" start_service_box_2_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fbusiness-intelligence%2F|title:Business%20Intelligence%20Solutions||" start_service_box_2_title="Business Intelligence" start_service_box_3_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fadvanced-analytics%2F|title:Advanced%20Analytics||" start_service_box_3_title="Advanced Analytics" start_service_box_4_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Fgraph-data%2F|title:graph%20Data||" start_service_box_4_title="Graph Data" start_service_box_5_url="url:http%3A%2F%2Fwww.tiq-solutions.de%2Fen%2Fservices%2Ftraining%2F|title:Training||" start_service_box_5_title="Training" start_service_box_6_url="|||" start_service_box_1_title="Consultancy"][/vc_column][/vc_row] ...