腾讯游戏 周东祥 大数据分析系统在游戏领域的迭代与实践

2020-03-01 171浏览

  • 1.xk I
  • 2.
  • 3.u 2 E Te 2 c F c - 2 D 2 c
  • 4.M t“
  • 5.+ , j l , o
  • 6.2
  • 7.M t“
  • 8.s ü ü ü o ü ü G ü ü ü N2G G 3SLNS ü o –” a–” ü ü t –” ü o ü t ü t ü t d
  • 9.i p –” t o 3 l w - s o 1
  • 10.t
  • 11.o o o t 1 o o G d –” t t - o 3 d –” t
  • 12.c d 52V NJ Kafka Storm TGDruid –” o d MQ t –” MySQL d o HDFS RPC&HTTP o SQL RDS CloudDB 59GVW 54GI
  • 13.M c d x Ø 59GVW Ø 54GI Ø 52V NJ t“
  • 14.o 1 c d j HDFS RDS h CloudDB o au b –” o r –”
  • 15.o 1 c d j 2019.03 2019.04 LSLJ JNW NSI NS MY T IG LY OLYL H L IL LLU a% , b HU a% LVT U ] NS MG NSL RG Q Q HU W R UG]R S 2019.05 Fliter r ‘201903’ - ‘201905’ Merge max(level) Fliter max(level)>30 sum(payment) sum(payment)>100 group by uin , (b Distinct distinct(uin)
  • 16.o 1 c d j 2019.03 2019.04 LSLJ W R F W V UG]R S F W V JG MY T IG YHJRGH SLM Q PU U IG LY U F VGIPFG NS. F W V NS OLYL IG LY H L IL LLU a% , bHU a% , (b LVT U ] NS JG 2019.05 Walk through A Fliter a% a% a% , b , b , (b Merge Merge group by uin group by uin sum(payment) sum(payment) a% a% a% , b , b , (b
  • 17.o 1 i t“ m == Filter 1 2 3 Fliter Distinct 20 10 4 Merge group by uin max(level)>30 ‘201903’ - ‘201905’ DataNode RPC max(level) sum(payment)>100 sum(payment) r 1 2 10 20 n g n 2G G TJ 1001001 1001002 1001003 1001004 ----------- 2G G TJ % 2001001 2001002 2001003 2001004 ----------- 2G G TJ 1101001 2101002 3101003 4101004 ----------- 2G G TJ % 1201001 2201002 3201003 4201004 ----------- g n n n n 9<< distinct(uin)
  • 18.o 1 t“ %i + s x n <1 n , s n n n v ÷
  • 19.o 1 t“ %i n g n h n 0 && (o ),+ 000010000100 0000100001000010…0101 . I MM 1001 n -+ g
  • 20.o 1 59GVW c ((& Ø d t“ c l d f(+.C %%W Ø ( W Ø [ ([ Ø Ø u % d e 30 7 4 15 5 7 3 10 2 0 iData 2000 4000 TDW Spark 6000 TDW MR 8000
  • 21.o 1 t“ 59GVW Y JL Y 7Y _ WBJOL SLY i v9<< h n IBJOL ( SLY = n n ( CH R5 LJ 1WW Y n %i 1WW 4H H= = n n L i L H= L UP Y LLH= LH= H= LLL h Dt Eh ÷h h n n m m
  • 22.M c d x Ø 59GVW Ø 54GI Ø 52V NJ t“
  • 23.yw l j m t 1 2-10 11-20 l e m e t 1 2-10 11-20 e
  • 24.yw r g+&&& f l j q ~ +&&& *&&& &# + cl +&&& *&&& +&&& *&&& )&&& +&&& *&&& )&&& vk d )&&& (&& & )& &&& % +&&& ,&& (&& *&&& )&&& (&& &&&
  • 25.- t“ v 54GI DataNode-1 1 Storage Scheduler Data Stats Gather 9<< Block 1 9> n … n DataNode-2 n Block 1 … SQL Parser Query Optimizer DataNode-n Executor-n Block Execution Plan Bitcode Emitter Execute Scheduler 1 … Data Mapper Col1 Col2 Col n Bitmap Index Generator Bitmap Filter Builder Execute Engine Dynamic Bitmap Index Cache Aggregate Merger
  • 26.i i c t“ d 1 DataNode A B Executor DataNode C Executor Block1 ,&& ) … [_ B A =8 % h [ % 2G G TJ C [ % [
  • 27.- t“ %i )W Executor-n B j Data Mapper Col1 Col2 Col n Bitmap Index Generator Bitmap Filter Builder Execute Engine B5 53C IP THWGPU L 7A D 2F J U 6 A &&( (& ,& ( Dynamic Bitmap Index Cache E85A5 L SJ V . a b 0 2 GL 13 33 GL gender d e d e &d e d e + 0 2 %) 0 2 GV G . a a d(&e 0 2 &d e d((e d /e . pb b0 2 + %) & & GV G d me 0 2 &d te &d e &di e Aggregate Merger
  • 28.yw j A1 B1 C1 0-10 D2 C2 11-20 B2 A1∩B1∩ j C1∩D1 D1 D1 f B1 B2 j ( j ) j * , j - j * j + j j . j / j & j j * j + j ( D1 A2 o j ) j
  • 29.i 9> 76 SELECT mean(BKey_1 / 8) / 6 FROM #1412923.20151019 WHERE BKey_2 = 1 AND BKey_1 BETWEEN 10 AND 100 =8 Ø B B LSLJ Ø 9C 5TP LY_ A n d 9Ce y LSLJ AL TLYNL SUM S SUM SUM BKey_1 / 8 SUM COUNT COUNT COUNT COUNT MEAN mean(BKey_1 / 8) / 6
  • 30.54GI t“ m l l 30 Ø ) CB+f(* f,*7 Ø u • , j • , • &o h (g 20 10 %(W % W 1.058 0.919 0 2 1.292 1.489 10 15 5 W iData spark h 9> n 35 30 25 20 15 10 5 0 0.855 0.879 1.364 1.243 1.182 . - =8 , 76 + *& o & )( o (. o n (* o (& o n n 30.902 28.854 28.23 26.54 25.385 (g c n 27.758 24.283 24.129 23.911 iData spark
  • 31.M c d x Ø 59GVW Ø 54GI Ø 52V NJ t“
  • 32.t o 3
  • 33.t o u 3 ~ 3,174,023 2,661,884
  • 34.t x o c 3 d v2V NJ n n <00 AB n n n 42 2V NJ TJ OW n n v35.M t“36.237.M c t“ d Ø 59GVW Ø 54GI Ø 52V NJ g38.
  • 35.M t“
  • 36.2
  • 37.M c t“ d Ø 59GVW Ø 54GI Ø 52V NJ g
  • 38.