Streaming report -- Functional Comparison and Performance Evaluation

2020-02-27 165浏览

  • 1.Functional Comparison and Performance Evaluation 王华峰 毛玮 张天伦 2016/10/27 Intel亚太研发中心 Intel 软件与服务部 Big Data Technology Department
  • 2.Overview  Streaming Core  MISC  Performance Benchmark Choose your weapon ! 2 *Other names and brands may be claimed as the property of others.
  • 3.Execution Model + Fault Tolerance Mechanism
  • 4.Continuous Streaming Apache Storm* Twitter Heron* Source Aapche Flink* Operator Micro-Batch Apache Gearpump* Apache Spark Streaming* Apache Storm Trident* Sink Source Operator Sink 4 *Other names and brands may be claimed as the property of others.
  • 5.Continuous Streaming Continuous Streaming Micro-Batch Ack per Record Checkpoint “per Batch” Checkpoint per Batch Apache Storm* Twitter Heron* Aapche Flink* Apache Gearpump* Storage Source Operator id Acker This is the Apache Storm Trident* Storage Sink id Apache Spark Streaming* Source Operator offset state str JobManager/ HDFS Storage Sink ack Source offset Operator state str Driver Sink job status HDFS critical part, as it affects many features 5 *Other names and brands may be claimed as the property of others.
  • 6.Continuous Streaming Continuous Streaming Micro-Batch Ack per Record Checkpoint “per Batch” Checkpoint per Batch Apache Storm* Twitter Heron* Aapche Flink* Low Latency Apache Gearpump* Apache Spark Streaming* Apache Storm Trident* High Latency High Overhead Low Overhead Low Throughput High Throughput 6 *Other names and brands may be claimed as the property of others.
  • 7.Delivery Guarantee Apache Storm* Twitter Heron* Aapche Flink* At least once • Ackers know about if a record is processed successfully or not. If it failed, replay it. • There is no state consistency guarantee. Apache Gearpump* Apache Spark Streaming* Apache Storm Trident* Exactly once • State is persisted in durable storage • Checkpoint is linked with state storage per Batch 7 *Other names and brands may be claimed as the property of others.
  • 8.Native State Operator Apache Storm* Twitter Heron* Yes* •Storm: KeyValueState •Heron:X User Maintain Aapche Flink* Apache Gearpump* Yes • Flink JavaAPI:'>API: