myers nakashima qcon nyc 2018

2020-02-27 57浏览

1.Help! I accidentally distributed my system! Rachel Myers, Emily Nakashima
2.@eanakashima @rachelmyers, @eanakashima
3.@rachelmyers, @eanakashima
4.@rachelmyers @rachelmyers, @eanakashima
5.@rachelmyers, @eanakashima
6.@rachelmyers, @eanakashima
7.ModCloth Web App  (Rails) User request (from NGINX & loadbalancer) MySQL @rachelmyers, @eanakashima
8.What even is  a distributed system? @rachelmyers, @eanakashima
9.What even is a distributed system? Some  Node Another Node @rachelmyers, @eanakashima
10.Designing distributed systems • What are the “nouns” in my system? • What are the patterns for reading data? ?????? • What are the patterns for writing data? • Do diﬀerent ﬂows have diﬀerent performance, consistency, & availability requirements?    @rachelmyers, @eanakashima
11.ModCloth Web App Mobile User request (from NGINX & loadbalancer) MySQL @rachelmyers, @eanakashima
12.ModCloth Web App Mobile User request (from NGINX & loadbalancer) MySQL Voting Service MySQL @rachelmyers, @eanakashima
13.ModCloth Identity Service MySQL Web App Mobile User request (from NGINX & loadbalancer) MySQL Voting Service MySQL @rachelmyers, @eanakashima
14.ModCloth Identity Service MySQL Asset Service MySQL Web App Mobile User request (from NGINX & loadbalancer) MySQL Voting Service MySQL @rachelmyers, @eanakashima
15.ModCloth Photo Service MySQL Comment Service MySQL Identity Service MySQL Asset Service MySQL Web App Mobile User request (from NGINX & loadbalancer) MySQL Voting Service MySQL @rachelmyers, @eanakashima
16.ModCloth Photo Service MySQL Moderation Service MySQL Identity Service MySQL Comment Service MySQL Asset Service MySQL Web App Mobile User request (from NGINX & loadbalancer) MySQL Voting Service MySQL @rachelmyers, @eanakashima
17.ModCloth Photo Service MySQL Checkout Service MySQL Moderation Service MySQL Identity Service MySQL Comment Service MySQL Asset Service MySQL Web App Mobile User request (from NGINX & loadbalancer) MySQL Voting Service MySQL @rachelmyers, @eanakashima
18.ModCloth Photo Service MySQL Checkout Service MySQL Moderation Service MySQL Identity Service MySQL Comment Service MySQL Asset Service MySQL Web App Mobile User request (from NGINX & loadbalancer) ????????? MySQL Voting Service MySQL @rachelmyers, @eanakashima
19.ModCloth Photo Service MySQL Checkout Service MySQL Web App Mobile User request (from NGINX & loadbalancer) ???????????????????????????????????? Moderation Service MySQL Identity Service MySQL Comment Asset MySQL Voting Service Service Service MySQL MySQL MySQL @rachelmyers, @eanakashima
20.And this was intentional ?????? @rachelmyers, @eanakashima
21.ﬂexﬁts.com An accidental distributed system ⛈ @rachelmyers, @eanakashima
22.ﬂexﬁts.com @rachelmyers, @eanakashima
23.ﬂexﬁts.com Ecommerce SaaS User request @rachelmyers, @eanakashima
24.ﬂexﬁts.com Subscription Plugin SaaS Ecommerce SaaS User request @rachelmyers, @eanakashima
25.ﬂexﬁts.com Fulﬁllment SaaS Subscription Plugin SaaS Ecommerce SaaS User request @rachelmyers, @eanakashima
26.ﬂexﬁts.com Metrics SaaS Fulﬁllment SaaS Subscription Plugin SaaS Ecommerce SaaS User request @rachelmyers, @eanakashima
27.ﬂexﬁts.com @rachelmyers, @eanakashima
28.ﬂexﬁts.com ?????? ?????? @rachelmyers, @eanakashima
29.ﬂexﬁts.com Metrics SaaS Fulﬁllment SaaS ? Subscription Plugin SaaS Ecommerce SaaS User request @rachelmyers, @eanakashima
30.ﬂexﬁts.com Metrics SaaS Fulﬁllment SaaS Postgres Orders Service Subscription Plugin SaaS Ecommerce SaaS User request @rachelmyers, @eanakashima
31.??????ﬂexﬁts.com Metrics SaaS Fulﬁllment SaaS Postgres Orders Service Subscription Plugin SaaS Ecommerce SaaS User request @rachelmyers, @eanakashima
32.New requirements for buy vs build @rachelmyers, @eanakashima
33.ﬂexﬁts.com SaaS SaaS Home cooked on IaaS SaaS SaaS User request @rachelmyers, @eanakashima
34.Buy or Build • Will it improve availability? • Will it improve recovery? • Will you have visibility or points of control? • Will you have customizability? • Will you know when your changes take eﬀect? • Can you audit changes? @rachelmyers, @eanakashima
35.IaaS, PaaS, BaaS, & FaaS @rachelmyers, @eanakashima
36.Cloud Infrastructure @rachelmyers, @eanakashima
37.IaaS @rachelmyers, @eanakashima
38.IaaS • Will it improve availability? ✅ • Will it improve recovery? ✅ • Will you have visibility or points of control? ✅ • Will you have customizability? ✅ • Will you know when your changes take eﬀect? ✅ • Can you audit changes? ✅ @rachelmyers, @eanakashima
39.Platforms @rachelmyers, @eanakashima
40.PaaS @rachelmyers, @eanakashima
41.PaaS • Will it improve availability? ✅ • Will it improve recovery? ?????? • Will you have visibility or points of control? ✅ • Will you have customizability? ✅ • Will you know when your changes take eﬀect? ?????? • Can you audit changes? ?????? @rachelmyers, @eanakashima
42.Backends @rachelmyers, @eanakashima
43.BaaS (RIP Parse) @rachelmyers, @eanakashima
44.BaaS • Will it improve availability? ✅?????? • Will it improve recovery? ?????? • Will you have visibility or points of control? ?????? • Will you have customizability? ✅ • Will you know when your changes take eﬀect? ?????? • Can you audit changes? ?????? @rachelmyers, @eanakashima
45.Functions @rachelmyers, @eanakashima
46.FaaS @rachelmyers, @eanakashima
47.FaaS • Will it improve availability? ?????? • Will it improve recovery? ?????? • Will you have visibility or points of control? ?????? • Will you have customizability? ✅ • Will you know when your changes take eﬀect? ?????? • Can you audit changes? ?????? @rachelmyers, @eanakashima
48.How to choose?
49.Faster to start Harder to change Backend Platform Infrastructure @rachelmyers, @eanakashima
50.@rachelmyers, @eanakashima
51.@rachelmyers, @eanakashima
52.Surprise Distributed Systems @rachelmyers, @eanakashima
53.Surprise distributedsystem:browsers @rachelmyers, @eanakashima
54.“The browser is part of your distributed system, not just a client you support”  -@danielespeset @rachelmyers, @eanakashima
55.The one where we DDOS’d ourselves @rachelmyers, @eanakashima
56.What happened? @rachelmyers, @eanakashima
57.// if the page is long enough to scroll if (document.body.clientHeight > window.innerHeight) { // add a scroll event listener document.addEventListener('scroll', function(e) { // if within 100px of the bottom of the page if (window.innerHeight + window.scrollY + 100 > document.body.clientHeight) fetchNextPage(); }); // else fetch another page of results immediately } else { fetchNextPage(); } @rachelmyers, @eanakashima
58.Frontend complexity is only increasing @rachelmyers, @eanakashima
59.Instrumentation Rule 1: Start at the edge.Corollary:The edge may be further than you think. @rachelmyers, @eanakashima
60.Secret distributedsystem:vendors @rachelmyers, @eanakashima
61.LaunchDarkly @rachelmyers, @eanakashima
62.Secret distributedsystem:vendors @rachelmyers, @eanakashima
63.# instantiate the flag with a default value # falls back to default value if service is offline show_feature = ld_client.variation( "allow_traces", { :key => "user@example.com" }, false ) if show_feature # application code to show the feature else # the code to run if the feature is off @rachelmyers, @eanakashima
64.Secret distributedsystem:vendors @rachelmyers, @eanakashima
65.Instrumentation Rule 2: Wrap 3rd party calls @rachelmyers, @eanakashima
66.Secret distributedsystem:vendors @rachelmyers, @eanakashima
67.GitHub’s customer support application @rachelmyers, @eanakashima
68.GitHub customer support tool Customer Support Tool github.com User email @rachelmyers, @eanakashima
69.GitHub customer support tool Customer Support Tool github.com GMail User email @rachelmyers, @eanakashima
70.GitHub customer support tool Customer Support Tool github.com SendGrid GMail MailChimp User email @rachelmyers, @eanakashima
71.GitHub customer support tool Customer Support Tool github.com SendGrid GMail MailChimp User email @rachelmyers, @eanakashima
72.GitHub customer support tool Customer Support Tool github.com Outbound mail service Ingest service Kafka User email @rachelmyers, @eanakashima
73.GitHub customer support tool Outbound mail service Customer Support Tool ?????????Ingest service User email github.com Kafka @rachelmyers, @eanakashima
74.GitHub customer support tool Customer Support Tool github.com SendGrid GMail MailChimp User email @rachelmyers, @eanakashima
75.Cognitive Load @rachelmyers, @eanakashima
76.Instrumentation & Chill ❄ @rachelmyers, @eanakashima
77.Fewer Alerts ?????? @rachelmyers, @eanakashima
78.@rachelmyers, @eanakashima
79.Unknown unknown unknowns ?????? @rachelmyers, @eanakashima
80.Outsource Your Brain ?????? @rachelmyers, @eanakashima
81.Use tools that let you… • Ask, what is happening now to this user? Endpoint? Team? • Ask, how does this compare to the aggregate case? • Jump between levels of abstraction • See both breadth & depth • Understand how data ﬂows  @rachelmyers, @eanakashima
82.Observability… not just for ops thought leaders! @rachelmyers, @eanakashima
83.“The ability to ask arbitrary questions about your system from the outside, and understand what is happening on the inside.”  -@mipsytipsy @rachelmyers, @eanakashima
84.logs metrics traces / events @rachelmyers, @eanakashima
85.How to pick an observability tool? @rachelmyers, @eanakashima
86.Crawl the pipes @rachelmyers, @eanakashima
87.ﬂexﬁts.com @rachelmyers, @eanakashima
88.No really, instrumentation @rachelmyers, @eanakashima
89.@rachelmyers, @eanakashima
90.# monitoring endpoint Location ~ /ping { return 200 '200 ok'; } @rachelmyers, @eanakashima
91.Instrumentation Rule 3: Your instrumentation should support depth- *and* breadth-ﬁrst @rachelmyers, @eanakashima
92.Conclusion @rachelmyers, @eanakashima
93.TwoVendors:Two Conclusions ?????? @rachelmyers, @eanakashima
94.1. We’re all distributed systems engineers operators. ☎ ?????? @rachelmyers, @eanakashima
95.2. Trust your tools 1 ?????? @rachelmyers, @eanakashima
96.Thank you @rachelmyers, @eanakashima