Spark Steaming Framework for Large-Scale Multi-Stream Data Analytics

Abstract

Real-time streaming applications with multiple heterogeneous data streams have become increasingly popular especially in IoT applications where huge amount of sensors produce large amount of data in the form of data streams. However, many issues still exist, especially in deploying and maintaining these large amounts of data streams. Using Spark Structured Streaming, this research introduces a Spark Streaming framework for multiple heterogeneous data streams which focuses on the ease of deployment and proper scheduling. Our proposed framework is a library that allows the deployment of multiple heterogeneous data stream processing in a single Spark application. Our framework can reduce deployment difficulties, coding redundancy, monitoring difficulties, and solve the problem of inefficient job queueing in multi-stream applications.