知乎 王雨舟 - 知乎大数据平台架构实践

2020-02-27 58浏览

  • 1.ᎣԒय़හഝଘ‫ݣ‬ຝ຅ਫ᪢ ሴᵨᛪ ᎣԒහഝଘ‫ݣ‬ᨮᨱՈ
  • 2.य़ᕐ  හഝଘ‫ݣ‬ຝ຅  හഝ‫ݢ‬ᥤ۸'HPR  ๅग़ਫᴬଫአ
  • 3.ຝ຅ හഝଘ‫ݣ‬ෆ֛ຝ຅
  • 4.ଘ‫ݣ‬ຝ຅ හഝრ App Web ங‫מ‬ੜᑕଧ ‫ݸ‬ᒒ෭ப හഝପ ളත੶ Log Server Maxwell փᬌ੶ Kafka ਂ‫੶ؙ‬ HBase HDFS Redis Kudu ᦇᓒ੶ ଫአ੶ Druid Hadoop Spark Hive ‫ݢ‬ᥤ۸‫ړ‬ຉଘ‫ݣ‬ හഝՙପ ଫአፊഴଘ‫ݣ‬ A/B Testing ӱ‫ۓ‬ᔮᕹ Ⴠ᭲ᓕቘଘ‫ݣ‬ ईᅩ‫ݎ‬ᇇၞᑕ ᓕቘ޾ၥᦶ ‫௳מز‬ᓕቘ ׁᩢ޾᧣ଶ ๦ᴴᓕቘ
  • 5.ईᅩ ईᅩ ईᅩ‫ݎ‬ᇇၞᑕ ᓕቘ޾ၥᦶ
  • 6.ईᅩၞᑕ 1. Ծߝᕪቘ൉‫ڊ‬ईᅩᵱ࿢ 2. හഝ‫ړ‬ຉ૵ٟईᅩ෈໩ 3. Ի՞ૡᑕ૵୏‫ݎ‬ 4. ईᅩࢧ୭ၥᦶ 5. ਮಁᒒ‫ݎ‬ᇇ
  • 7.ईᅩຽ‫ٵ‬۸ᥢ᝜ ֵአ Protobuf ؉ईᅩຽ‫ٵ‬۸
  • 8.ईᅩ SDK Өଘ‫ݣ‬ :HE :HFKDW$SS $QGURLG L26 ‫ݸ‬ᒒ๐‫ۓ‬ -66'. -DYD6'. 2EMHFWLYH& 6'. 3\WKRQ-DYD 6'.
  • 9.3URWREXIጱսᅩ 1. ईᅩૡᑕ૵ӧ਻ฃٟᲙ 2.‫ץ‬ද൉ Code Review 3.ᕹӞ޸‫ݷ‬ᓕቘ 4.ଧ‫ڜ‬۸හഝጱଘ‫ݣ‬෫‫ى‬௔ 5.փᬌ֛ᑌੜ҅፜ၞᰁ 6.ඪ೮ग़᧍᥺҅ ‫਻ّݸݻ‬
  • 10.ईᅩ໐ஞ௏మ :KR :KHQ Ŏ ,',QIR Ŏ &OLHQW,QIR Ŏ 7LPH,QIR Ŏ 1HWZRUN,QIR :KHUH :KDW Ŏ $FWLRQ Ŏ 8UO Ŏ (OHPHQW Ŏ 0RGXOH Ŏ 1DPH Ŏ ([WUD,QIR
  • 11.What • ٖ਻ኧӱ‫ݸۓ‬ᒒଧ‫ڜ‬۸অ PB҅Base64 ౮ string ‫ݸ‬ӥ‫ݎ‬ • ਮಁᒒփࢧ‫ݸ‬ᒒӥ‫ ݎ‬string҅හഝଘ‫ݍݣ‬ଧ‫ڜ‬۸ •ਮಁᒒԆۖතᵞٖ਻Ӥಸֺ҅ইᶭᶎ‫ے‬᫹෸ᳵᒵ
  • 12.ईᅩ໛ຝ - Hybrid ਮಁᒒ Hybrid ໛ຝईᅩጱ໐ஞᥝᔰ
  • 13.Hybrid ໛ຝईᅩጱ໐ஞᥝᔰ Ŏ ‫ڹ‬ᒒ-6ପ᭗ᬦ1DWLYH൉‫׀‬ጱᒒᚆ‫ێ‬ᬰᤈ಑ᅩᕹᦇ Ŏ +\EULG໛ຝ‫ݎݝᦤכ‬Ӟེᶭᶎ઀ሿ Ŏ +\EULG໛ຝ॒ቘ5HIHUUHU
  • 14.ईᅩ6FKHPD‫ץ‬ද&RGH5HYLHZ 2016.02.16 - 2017.07.28 Ӟ‫ ํو‬1187 ེ൉Ի ईᅩ෭ப໒ୗ෈կᤈහ 2568
  • 15.௔ᚆፊഴईᅩ'HPR message MonitorInfo { ... // App ᶭᶎ‫ے‬᫹‫௳מ‬ optional AppPerformancePageLoadInfo app_performance_load = 7; // App ‫ܜ‬ᶷ‫௳מ‬ optional AppPerformanceBlockInfo app_performance_block = 8; }
  • 16.ളත੶ ෭பളත Log Server ၾ௳ፊ‫ލ‬ Maxwell
  • 17.෭பളත • ളත Protobuf̵Json ޾ String ᔄࣳ໒ୗහഝ • හഝٟ‫ ف‬Kafka • ٟ‫ ف‬Kafka ०ᨳ෸ਂ‫ف‬๜ࣈ Leveldb • ‫ݎ‬ᭆᴚ‫ڜ‬؋଼෸҅‫ݎ‬ᭆ Leveldb හഝ‫ ک‬Kafka
  • 18.ၾ௳ፊ‫ލ‬ ֵአ Maxwell ള Mysql Binlog ٟ Kafka
  • 19.ᦇᓒ੶ ᦇᓒ੶
  • 20.හഝၞୗࢶ Kafka Spark Kafka HDFS Druid Mysql Sqoop Kafka Spark Kudu Hive Impala
  • 21.හഝಢ॒ቘ ಢ॒ቘ
  • 22.හഝಢ॒ቘ Ŏ ᛔᎸಢ॒ቘᔮᕹ҅᧛.DINDٟ+')6 Ŏ ᧛+')6ٟ+')6 Ŏ 6TRRSಢᰁ੕‫ڊ‬0\VTOහഝ‫ک‬+LYH Ŏ ᧛+')6ٟ'UXLG
  • 23.හഝਫ෸॒ቘ ਫ෸॒ቘ
  • 24.හഝਫ෸॒ቘ 6SDUN6WUHDPLQJ(7/ٟ.DIND
  • 25.ਫ෸(7/ Ŏ ,3ࣈ࣎ᥴຉ Ŏ 8VHU$JHQWᥴຉ Ŏ ӱ‫ۓ‬හഝ‫ړ‬ၞ
  • 26.හഝਫ෸੕‫'ف‬UXLG 7UDQTXLOLW\ၾᩇ.DINDٟ'UXLG
  • 27.ਫ෸੕‫ف‬.XGX 6SDUN6WUHDPLQJၾᩇ.DINDٟ.XGX
  • 28.ັᧃ੶ ັᧃ੶
  • 29.᯿ଶֵአ'UXLG • Druid • Hive • Impala
  • 30.ັᧃᖨਂӨහഝᇇ๜ Ŏ ੒ग़ᖌ‫ړ‬ຉ޾ኸਂ‫ړ‬ຉ4XHU\ೲ෸ᳵೆ‫ړ‬ Ŏ 4XHU\ᕮຎೲ෸ᳵೆ‫ݸړ‬ٟ‫ف‬ᖨਂ Ŏ හഝრ᯿੕‫ݸ‬੒ଫ෸ᳵ᝜ࢱጱᖨਂᛔۖ०ප Ŏ ༄ັᖨਂᇇ๜҅‫ํݝ‬හഝ๚‫ۖݒ‬ጱັᖨਂ
  • 31.0\VTOහഝਫ෸2/$3 Mysql හഝਫ෸ OLAP
  • 32.0\VTOහഝਫ෸2/$3 Ŏ a,PSDOD.XGX Ŏ սᅩғັᧃ᭛ଶள҅ਫ෸௔ṛ Ŏ ᗌᅩғᤒᕮ຅‫ݒ‬ๅ‫ݸ‬ᵱᥝ᯿੕ Ŏ a๚๶7L6SDUN Ŏ ์ବ3LQJ&$3വ‫ڊ‬7L6SDUN҅୏তၥᦶ7L6SDUN௔ᚆ
  • 33.ᤈӱዳᅩ य़හഝጱਫ෸ OLAP • ᶋࢴ۸ᵱ࿢ • ᛔਧԎग़ᖌ‫ړ‬ຉ • ᛔਧԎኸਂ‫ړ‬ຉ
  • 34.හഝ‫ݢ‬ᥤ۸'HPR හഝ‫ݢ‬ᥤ۸ Demo
  • 35.‫ݢ‬ᥤ۸‫ړ‬ຉଘ‫ݣ‬ $30&RQහഝ‫ړ‬ຉ  හഝრ੕‫ف‬  ᛔਧԎग़ᖌ‫ړ‬ຉັᧃ  ᛔਧԎኸਂ‫ړ‬ຉ
  • 36.᭗አ‫ݢ‬ᥤ۸‫ړ‬ຉଘ‫'ݣ‬HPR +LYHහഝ੕‫ف‬ଘ‫ݣ‬
  • 37.හഝ੕‫ف‬KLYHGHPR
  • 38.
  • 39.
  • 40.
  • 41.හഝრ‫ڹ‬ᗝ༄ັ ੕‫ف‬හഝრ:RUNIORZጱ‫ڹ‬ᗝ༄ັ
  • 42.ᛔਧԎ೰ຽ‫ڠ‬ୌ ‫ڠ‬ୌ೰ຽ
  • 43.੕‫ݸف‬ጱහഝრ೰ຽ‫ڜ‬ᤒ
  • 44.‫ڠ‬ୌๅग़೰ຽ
  • 45.‫ڠ‬ୌ೰ຽ'HPR
  • 46.୩य़ጱᬦᄁ࢏
  • 47.‫ڠ‬ୌग़ᖌ‫ړ‬ຉಸᤒ ‫ڠ‬ୌग़ᖌ‫ړ‬ຉಸᤒ
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.‫ڠ‬ୌኸਂಸᤒ ‫ڠ‬ୌኸਂಸᤒ
  • 55.᭗አ‫ݢ‬ᥤ۸ኸਂ‫ړ‬ຉ
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.ኸਂᓀᭌ 8.10 ‫ލ‬ᬦ CDN ጱՈࣁ 8.11 ݈‫ލ‬ԧᎣԒጱํग़੝ՈҘ
  • 61.
  • 62.
  • 63.ๅग़ਫᴬଫአ ๅग़ਫᴬଫአ
  • 64.ଫአፊഴଘ‫ݣ‬ ଫአፊഴଘ‫ݣ‬
  • 65.ଫአፊഴଘ‫ݣ‬೰ຽӨᖌଶ Ŏ ᶭᶎ‫ے‬᫹෸ᳩ Ŏ $SS‫ۖސ‬෸ᳩ Ŏ ᔮᕹ௔ᚆ Ŏ $SSၞᰁᕹᦇ Ŏ ᶭᶎ‫ܜ‬ᶷ‫௳מ‬ Ŏ ଘ‫ݣ‬ Ŏ ᔮᕹᇇ๜ Ŏ ଫአᇇ๜ Ŏ ๢ࣳ Ŏ ᬩ០ࠟ Ŏ ᗑᕶᔄࣳ
  • 66.ӱ‫ݸۓ‬ᒒ੒ള ӱ‫ݸۓ‬ᒒ੒ള
  • 67.ӱ‫ݸۓ‬ᒒ੒ള
  • 68.*URZWK Growth • ၞᰁ๶რ • ਮಁᒒෛीᦩ‫ڦ‬ • Ⴠ᭲ᓕቘ‫ݣݸ‬
  • 69.ၞᰁ๶რ ၞᰁ๶რ
  • 70.:HEᒒၞᰁ๶რ •ᛔᆐၞᰁ๶რ • ൤ᔱ୚කၞᰁ • ᐒԻၞᰁ • ፗളၞᰁ •՞ᩇၞᰁ๶რ • ֵአՈૡ utm ຽᦕ
  • 71.ਮಁᒒၞᰁ๶რ ᤩ Scheme ౲ Universal Link ࠏᯯጱ App҅ࣁ‫ۖސ‬෸Ӥ ಸࠏᯯ᱾ള҅හഝଘ‫ݣ‬ᕟು‫ݐ‬᱾ളӾጱ UTM ֢ԅ୮‫ڹ‬ ෭ப UTM҅տᦾ‫ݢܨݸۆڔ‬஑‫ک‬ਮಁᒒၞᰁ๶რ
  • 72.$%7HVWLQJ A/B Testing
  • 73.ਫḵᔮᕹ •ᯈᗝӥ‫ݎ‬ •ਮಁᒒኞප‫ݸ‬Ӥಸ •හഝ‫ݢ‬ᥤ۸
  • 74.$%7HVWLQJ ᎣԒ ಑୏ 掚꡶䲿꡼ ํߺԶ؋᫝๐ᭇ‫ݳ‬ॕॠࣁ؋᫝ ಄ᑯํߺԶ؋᫝๐ᭇ‫ݳ‬ॕॠҘ ಑୏ 掚꡶俒畎 ۹ՂํՋԍẌԅՈᎣጱ҅உঅ᭘ጱᤋ१ ੜମҘ ಑୏ -JWF ἑᨬ‫ܜ‬ғই֜Ӟੜ෸ٖ؉‫ڊ‬Ӟ ᶷਹ਷Ҙ ಑୏ 歏㶩⛼ ෭ଉᕪၧ਍ ಑୏ 湱Ⱒ䲿꡼ ํߺԶၞᤈ஑᩸๶ጱੜռॾָߝߝᇈҘ ಑୏
  • 75.4$ QA
  • 76.THANK YOU