华为 时金魁 - Flink技术栈及其适用场景

2020-02-27 210浏览

  • 1.Flink技术栈及其适用场景 @෸ᰂṮ 2017/05 ܏ԅ๺Ꮈಅ shijinkui@huawei.com
  • 2.Who am i • @෸ᰂṮ • Sohu -> Alibaba -> Huawei • Work onғhigh performance computing҈Spark҈Flink 2
  • 3.Flink༷ᥦ
  • 4.Streaming Compute Frameworks Amazon Kinesis Streams Aliyun StreamCompute -Twitter Flink Azure stream-analytics Apache Edgent (incubating) -IBM Apache Gearpump (incubating) -Intel 4
  • 5.Flink &RQQHFWRUV redis Kafka casandra elasticsearch flume rabbitmq kinesis twitter CEP complex event process Table & SQL Gelly ML graph machine process learning Table & SQL Library DataStream API Stream Processing DataSet API Batch Processing API Runtime Distributed Streaming DataFlow Core Local Single JVM Cluster Yarn, Mesos,Standalone Cloud GCE, EC2 Deploy 5
  • 6.Open source community Status Spark Flink 2000 1,945.1 1500 1000 1,064 1,090.3 1,254 1,177.6 500 50 26 305 0 493 231 Committer Contributers Commits PR 224.1 Star 157.3 Fork ಢ॒ቘғ‫ي‬ፐ ၞᦇᓒғ‫ي‬᩸
  • 7.Flink example case class WordWithCount(word:String,count:Long) val windowCounts = env .socketTextStream(hostname, port, '\n') .flatMap { w => w.split("\\s") } .map { w => WordWithCount(w, 1) } .keyBy("word") .timeWindow(Time.seconds(5)) .sum("count") env.execute("Socket Window WordCount") Data Source Transform Function Execution 7
  • 8.Flink architecture
  • 9.Flink Inside
  • 10.build graph sources sinks streamNodes StreamGraph ExectutionGraph streaming graph 1 transform 2 note 1 slot group, tasks resources node 2 slot group, tasks node 3 slot group, tasks 3
  • 11.Source 1 Runtime Source 2 Simple Runtime of Flink from XVHUSURGXFWparall:4'>parall:4