Google 刘靖-大数据驱动下的谷歌云平台运维

2020-02-27 57浏览

  • 1.Devops Practices on google cloud Jeff Liu Google
  • 2.Jeff Liu ● SWE / Devops Google ● SRE - Twitter ● Devops - Splunk ● SRE - Dell
  • 3.Devops Evolution
  • 4.Devops Evolution
  • 5.Devops Evolution ● Cloud／Platform ● Microservices ● Big data ● Machine Learning
  • 6.Cloud and Platforms
  • 7.Microservices
  • 8.Big Data and Machine Learning
  • 9.Devops DevOps (a clipped compound of "development" and "operations") is a software development process that emphasizes communication and collaboration between product management, software development, and operations professionals. --- Wiki “DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity.” --- AWS “DevOps is where any application starts.” --- Google
  • 10.
  • 11.Dataflow evolution MapReduce Colossus Spanner BigTable PubSub Megastore Millwheel Dremel Google Cloud Dataflow Flume Apache Beam
  • 12.
  • 13.Fully-managed Deploy Tear Down
  • 14.Dynamic Work Rebalancing 100 mins. 65 mins. vs.
  • 15.Unified programming model
  • 16.Dataflow - Integrated Capture Process Google App Engine Cloud Logs Google Analytics Premium Cloud Pub/Sub Cloud Dataproc Batch Stream Cloud Dataflow Real time analytics and Alerts Cloud Monitoring Cloud DataStore Cloud Bigtable Store BigQuery Storage (tables) Cloud Bigtable (NoSQL) Cloud Storage (files) Analyze BigQuery Analytics (SQL) Cloud Dataflow Cloud Dataproc 16
  • 17.Dataflow public static void main(String[] args) throws IOException { ... Pipeline pipeline = Pipeline.create(options); pipeline .apply(PubsubIO.Read.topic(options.getPubsubTopic())) .apply(ParDo.of(new ExtractWords())) .apply(ParDo.of(new Uppercase())) .apply(ParDo.of(new StringToRowConverter())) .apply(BigQueryIO.Write.to(tableSpec) .withSchema(StringToRowConverter.getSchema())); PipelineResult result = pipeline.run();
  • 18.Dataflow java -cp ./ -Dexec.mainClass=com.example.WordCount \ -Dexec.args="--project=\ --stagingLocation=gs://'>gs://