eBay Architecture(Tony Ng) 2011 Tony NG

2020-03-01 73浏览

  • 1.eBay  Architecture   Tony  Ng   Director,  Systems  Architecture   October  2011
  • 2.About  Me   DRAFT • eBay  –  Systems  Architecture  and  Engineering   • Yahoo!  –  Social,  Developer  PlaEorms,  YQL   • Sun  Microsystems  –  J2EE,  GlassFish,  JSRs   • Author  of  books  on  J2EE,  SOA       2
  • 3.eBay  Stats   • 94  million  acQve  users   • 200  million  items  for  sale  in  50,000  categories   • A  cell  phone  is  sold  every  5  seconds  in  US   • An  iPad  sold  every  2.2  minutes  in  US   • A  pair  of  shoes  sold  every  9  seconds  in  US   • A  passenger  vehicle  sold  every  2  minutes   • A  motorcycle  sold  every  6  minuteshttp://www.ebayinc.com/factsheets3   DRAFT
  • 4.eBay  Scale   • 9  Petabytes  of  data  storage   • 10,000  applicaQon  servers   • 44  million  lines  of  code   • 2  billion  pictures   • 99.94%  site  availability   • A  typical  day   – – – – – 4   75B  database  calls   4B  page  views   250B  search  queries   Billions  of  service  calls   100s  of  millions  of  internal  asynchronous  events   DRAFT
  • 5.History  of  Technology   DRAFT ·∙ ·∙ ·∙ ·∙ ·∙ Java V4 Components Services Internal Cloud Platform Java XSL Layered Horizontal Scale Some APIs 2009+ 2005 2001 Perl/C++ Inline HTML Monolithic Vertical Scale Walled Garden 1999 ·∙ ·∙ ·∙ ·∙ ·∙ 1995 Innovation Potential Agility / TTM Architecture Maturity ·∙ ·∙ ·∙ ·∙ ·∙
  • 6.Quali9esA:ributesConcerns   • Scalability   • Availability   • Latency   • Security   • Manageability   • Cost     6   DRAFT
  • 7.eBay  Scalable  Architecture   • ParQQon  everything   – Databases,  applicaQon  Qer,  search  engine   • Stateless  preference   – No  session  state  in  app  Qer   • Asynchronous  processing   – Event  streams,  batch   • Manage  failures   – Central  applicaQon  logging   – Mark  downs   7   DRAFT
  • 8.Next  Challenges   DRAFT • Maintain  site  stability  but  deliver  quality  features   and  innovaQons  at  acceleraQng  paces   • Complexity  as  our  codebase  grows   • Build  on  our  architecture  maturity  to  enable  faster   Qme-­‐to-­‐market   • Developer  producQvity   8
  • 9.Scalability  with  Agility   • Strategy  1:  AutomaQon  with  Cloud   • Strategy  2:  Next  Gen  Service  orientaQon   • Strategy  3:  Modularity   • …  and  more  …   9   DRAFT
  • 10.DRAFT Automa9on  with  Cloud   10
  • 11.Improving  Automa9on   request receive & rack & wire order {nb servers, model, app } DRAFT deliver Label (app) 2-3 w “several” weeks 1w repurpose request order Receive pre-racked Pre-wired {nb servers, model } quarterly deliver to cache request deliver {nb servers, model, app } minutes 2-3 w 1 day repurpose 11
  • 12.Improving  U9liza9on   DRAFT DR Number of servers required based on utilization for 8 pools 12
  • 13.Infrastructure  Virtualiza9on   Application App App App Spare spare spare spare Infra Infra Infra Infra Application App DRAFT App Global resource pool Shared infrastructure App
  • 14.eBay  Cloud   Self Service Portal DRAFT Automation Capacity Management Resource Allocation Virtualization Spare Capacity pool provisioning in minutes Hardware Acquisition Improved Time to Market
  • 15.Design  Principles   DRAFT • Network  isolaQon  to  enable  mobility  and  isolaQon  at   scale   • Capability  to  automate  reliably   • StandardizaQon   • Private  vs.  Public   – Start  with  Private,  opQon  to  go  Hybrid     • Buy  vs.  Build   – Build  +  OSS   15
  • 16.Infrastructure  &  PlaJorm  as  a  service   DRAFT Higher developer productivity Full application level automation Enables innovation on new platforms Infrastructure level automation Platform As A Service Automated Life Cycle Management Front End, Search Back End, Generic Platform Infrastructure As A Service Automated Operations Virtualized & Common Infrastructure 16
  • 17.IaaS   DRAFT DNS! Name! Organization! * Access ! Point! * Compute Nodes OS! Images! Storage networks Compute Nodes Virtual Cluster Storage Account! networks 1 Virtual Pool Class of! Service! configurations! Compute Nodes Virtual Environment networks Storage Access! Lists! Reserved Instances Data  Centers   Racks   Virtual   machines   Physical   Load  Balancers   machines   networks   Firewalls   Network  Storage   (SAN/NAS)   Physical Infrastructure eBay Confidential DNS   Storage     Mail   Service   As  a  Service   Service   NTP   Service   Proxy   Service   Infrastructure Services 17
  • 18.PaaS   DRAFT DNS! Name! Organization! * * Account! Access! point! groups! Builds &! packages! Application Services groups! * Service Instances 1 Update Strategy Class of! Service! Profiles configurations! Generic   Front  End   Search   SOA   Virtual Environment Login   Payment   Iden9ty   Catalog   Search   Shipping   List   Pricing   eCommerce Services Offer   Access! Lists! ADs   Coupons   Messages   Cart   CS   Logging   Analy9cs   Monitoring   DB  as   Storage     A  Service   As  a  Service   Platform Services 18
  • 19.Model  Driven  Automa9on  for  Reliability   DRAFT LB Server LB Pool Server Expected State Server Server Reconciliation Server Current State Comparison Orchestration Site • Desired configuration is specified in the expected state and persisted in CMS Pool Discovery Server • Upon approval, the orchestration will configure the site to reflect the desired configuration. • Updated site configuration is discovered based on detection of configuration events • Reconciliation between the expected and current state allows to verify the proper configuration. • On going validation allows the detection of out of band changes. 19
  • 20.Open  Source  Integra9on   DRAFT IaaS/PaaS API IaaS/PaaS API orchestrat ion Resource Allocation Distribute d State orchestrat ion Resource Allocation Distribute d State AuthN/ AuthZ Applicatio n Controller Access Point Controller AuthN/ AuthZ Applicatio n Controller Access Point Controller Compute Controller Cluster Controller Pool Controller Compute Controller Cluster Controller Pool Controller Compute Mgt. Network Prov DNS Mgt. LB Mgt. Image/Pkg Repo Monitor ing Software Dist. Open  Source   SoluQon   (openstack  /  Cloudstack)
  • 21.Applica9on  Architecture   Before Ongoing “Cloud Friendly” DRAFT Future ‘Cloud ready’
  • 22.DRAFT Next  Gen  Service  Orienta9on   22
  • 23.Services  @  eBay   DRAFT • It’s a journey ! • History • One of the first to expose APIs /Services • In early 2007, embarked on service orienting our entire ecommerce platform, whether the functionality is internal or external • • • Support REST style as well as SOA style Have close to 300 services now and more on the way Early adopters of SOA governance automation (Discovery focus rather than control) 23
  • 24.Architecture  Vision   DRAFT Customer Experience Core  Experience   Channels   Custom  Experiences   Application Platform Services Login   Iden9ty   Catalog   Search   List   Pricing   Offer   ADs  Messages   Cart   Coupons  Payment  Shipping  CS   Technology Platform App   Stack   Data  Access   Layer   Dev  Tools   Presenta 9on   Messaging   SOA   Cloud   Operations Infrastructure Layer Power   Data  Center   Hardware   Network   Database   Tools   Opera9ons
  • 25.Design/Dev eBay  SOA  Stack  Overview   Dev/Arch DRAFT Life cycle/dependency mgmt Develop Services, consumers, Registry/ Repository Eclipse Dev tools Assertions svc Test tools 25 clients External routing (ESB) SOA/REST client Framework REST Mapping Tier Runtime Registry Authentication Admin console services SOA/REST Server Framework Authorization RL Monitoring Logging Policy service Monitoring console
  • 26.Challenge  1:  Mul9ple  Data  Formats   DRAFT • 2005:  Mix  of  user  preferences   – – – – SOAP   REST-­‐like:HTTP  GET  with  all  request  informaQon  in  the  URI   Plain  Old  XML  (POX):  HTTP  POST  with  XML  data  but  no  SOAP  envelope   JSON   • Shopping  API  was  our  first  “XML  Unified  Field”  web  service   – Inputformats:'>formats: