滴滴 梁李印 实时计算在滴滴出行的实践之路
2020-03-01 193浏览
- 1.
- 2.
- 3.H • H • • 20 1 6 A4
- 4.4 2 & & 3104 4
- 5.StreamSQL &CEP 1. 2. 3. 2018 2017 2016 20+ 1. 2. Spark Flink 3. I 1. 2. 3. 2019 StreamSQL II CEP StreamSQL 1. Flink 2.StreamSQL 3.Meta
- 6.StreamSQL IDE YARN Router/ Flink Flink Flink YARN YARN YARN HDFS HDFS HDFS DC1 DC2 DC3
- 7.2 BU2 1 BU1 GroupA GroupB GroupC GroupD GroupE NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager LabelA LabelB NoLabel LabelE
- 8.• APP1 APP2 APP4 APP5 APP6 APP7 APP3 APP8 Queue1 • Queue2 • Queue3 C ( • NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager NodeManager LabelA LabelB NoLabel )
- 9.• % • ResourceManager % • heartbeat update vcore heartbeat • update vcore NodeManager update monitor Container monitor % NodeManager update monitor Container update Container update monitor Container 5 0 3 C C
- 10.R • N • Y A • • A
- 11.4 2 & & 3104 4
- 12.POI • • • • • • -->ES/HDFS • binlog->Druid -->HBase/GeoMesa • • • • • • •
- 13.Join/ Flink Druid
- 14.Flink KV Store
- 15.xx /Join CEP xx
- 16.
- 17.• •
- 18.4 2 & & 3104 4
- 19.DDL UDX Join SQL IDE
- 20.HBase DDMQ StreamSQL Kafka GeoMesa • ElasticSearch • Druid • KV Store HDFS
- 21.UDF JSON UDX UDTF UDX UDAF MAP UDX Hive UDX UDF UDTF UDAF
- 22.source1 Join Operator Sink source2 TTLState1 TTLState2 • • • Join Operator source Sink TTLState cache
- 23.il • D • H • kS F J L 7 e m 7XQU 7 %% 0 Tm 7a
- 24.,0 Sa • 1 6 _ • B 1 11 2 && = = =B 3( B =3 3 ) 3 5 = . 2 = = =B 3( M • && • P T Fb c E
- 25.4 2 & & 3104 4
- 26.SQL IDE
- 27.Killed/Failed NodeManager JobManager TaskManager Flume Druid Kafka
- 28.DB Kafka Flink Kafka Druid Dashboard
- 29.
- 30.
- 31.DB Kafka table1 topic1 table1-stream table2 topic2 table2-stream table3 topic3 table3-stream StreamSQL
- 32.Flink StreamSQL IDE StreamSQL Service MetaStore SQL Flink Spark MetaStore Hive Presto
- 33.Q • E • P • • S L C
- 34.Thank You
- 35.
