AS深圳2017 Using sagas to maintain data consistency in a microservice architecture Chris

2020-03-01 187浏览

  • 1.Using sagas to maintain data consistency in a microservice architecture Chris Richardson Founder of Eventuate.io Founder of the original CloudFoundry.com Author of POJOs in Action @crichardson chris@chrisrichardson.nethttp://eventuate.io@crichardson
  • 2.Presentation goal Distributed data management challenges in a microservice architecture Sagas as the transaction model @crichardson
  • 3.About Chris @crichardson
  • 4.About Chris Consultant and trainer focusing on modern application architectures including microservices (http://www.chrisrichardson.net/)@crichardson
  • 5.About Chris Founder of a startup that is creating an open-source/SaaS platform that simplifies the development of transactional microservices (http://eventuate.io) @crichardson
  • 6.For more informationhttp://learnmicroservices.io@crichardson
  • 7.Agenda ACID is not an option Overview of sagas Coordinating sagas @crichardson
  • 8.The microservice architecture structures an application as a set of loosely coupled services
  • 9.The microservice architecture tackles complexity through modularization * might improve scalability too @crichardson
  • 10.Microservice architecture Database per service HTML Browser Store Front UI Customer Service Customer Database Order Service Order Database … Service … Database @crichardson REST API Gateway REST Mobile Device
  • 11.Loose coupling = encapsulated data Order Service Order Database Customer Service Customer Database Order table Customer table orderTotal creditLimit @crichardson
  • 12.How to maintain data consistency?!?!?Invariant:sum(open order.total) <= customer.creditLimit @crichardson
  • 13.Cannot use ACID transactions Distributed transactions BEGIN TRANSACTION Private to the … Order Service SELECT ORDER_TOTAL FROM ORDERS WHERE CUSTOMER_ID = ? … SELECT CREDIT_LIMIT FROM CUSTOMERS WHERE CUSTOMER_ID = ? … Private to the INSERT INTO ORDERS … Customer Service … COMMIT TRANSACTION @crichardson
  • 14.2PC is not an option Guarantees consistency BUT 2PC coordinator is a single point of failureChatty:at least O(4n) messages, with retries O(n^2) Reduced throughput due to locks Not supported by many NoSQL databases (or message brokers) CAP theorem 2PC impacts availability …. @crichardson
  • 15.Agenda ACID is not an option Overview of sagas Coordinating sagas @crichardson
  • 16.From a 1987 paper @crichardson
  • 17.Use Sagas instead of 2PC Distributed transaction Service A X Service B Service C Saga Service A Service B Service C Local transaction Local transaction Local transaction @crichardson
  • 18.Create Order Saga createOrder() Order Service Customer Service Order Service Local transaction Local transaction Local transaction reserveCredit() approve order() createOrder() Order state=PENDING Customer Order state=APPROVED @crichardson
  • 19.If only it were this easy… @crichardson
  • 20.Rollback using compensating transactions ACID transactions can simply rollback BUT Developer must write application logic to “rollback” eventually consistent transactions Careful design required! @crichardson
  • 21.Saga:Every Ti has a Ci FAILS T1 T2 C1 C2 … Compensating transactions T1 T2 C1 @crichardson
  • 22.Create Order Saga - rollback createOrder() Insufficient credit Order Service Customer Service Order Service Local transaction Local transaction Local transaction createOrder() reserveCredit() Order Customer FAIL reject order() Order @crichardson
  • 23.Sagas complicate API design Request initiates the saga. When to send back the response? Option #1: Send response when sagacompletes:+ Response specifies the outcome - Reduced availability Option #2: Send response immediately after creating the saga (recommended): + Improved availability - Response does not specify the outcome. Client must poll or be notified @crichardson
  • 24.Revised Create Order API createOrder() returns id of newly created order NOT fully validated getOrder(id) Called periodically by client to get outcome of validation @crichardson
  • 25.Minimal impact on UI UI hides asynchronous API from the user Saga will usually appear instantaneous (<= 100ms) If it takes longer UI displays “processing” popup Server can push notification to UI @crichardson
  • 26.Sagas complicate the business logic Changes are committed by each step of the saga Other transactions see “inconsistent” data, e.g. Order.state = PENDING more complex logic Interaction between sagas and other operations e.g. what does it mean to cancel a PENDING Order? “Interrupt” the Create Order saga Wait for the Create Order saga to complete? @crichardson
  • 27.Agenda ACID is not an option Overview of sagas Coordinating sagas @crichardson
  • 28.How to sequence the saga transactions? After the completion of transaction Ti “something” must decide what step to execute nextSuccess:which T(i+1) - branchingFailure:C(i - 1) @crichardson
  • 29.Orchestration-based saga coordination createOrder() Order Service Local transaction createOrder() Order state=PENDING CreateOrderSaga Customer Service Order Service Local transaction Local transaction reserveCredit() approve order() Customer Order state=APPROVED @crichardson
  • 30.Complex coordination logic is centralized 😀 @crichardson
  • 31.Services expose APIs that are invoked by saga 😄 @crichardson
  • 32.CreateOrderSaga orchestrator Create Order Order Service Customer Service OrderService create() reserveCredit() CreateOrder Saga create() approve() creditReserved() Customer creditLimit creditReservations ... Order state total… @crichardson
  • 33.Saga orchestrators are state machines reply and action event [guard] / action / customerService. reserveCredit() credit reserved / order.approve() APPROVED RESERVING CREDIT Initial action credit limit exceed / order.reject() REJECTED @crichardson
  • 34.Create Order Saga code State Enum Persistent data Statelesssingleton:Behavior Eventuate Saga framework @crichardson
  • 35.Create Order Saga State machine definition @crichardson
  • 36.Initializing the saga Create order Invoke saga participant @crichardson
  • 37.Handling a reply Update Order @crichardson
  • 38.Customer Service - command handling Reserve credit @crichardson
  • 39.About Saga orchestrator participant communication command Saga Orchestrator reply Saga Saga Participant Participant Saga must complete even if there are transient failures @crichardson
  • 40.Use asynchronous messaging @crichardson
  • 41.Create Order Saga messaging Commands Order Service reserve credit Message Broker Customer Service Customer Service Request Channel Create Order Saga Orchestrator Customer Service Reply Channel Credit Reserved Credit Limit Exceeded Begin TXN Read data Send reply Commit TXN create() approve() reject() Order Replies @crichardson
  • 42.Messaging must be transactional How to make atomic without 2PC? Service update Database publish Message Broker @crichardson
  • 43.Option #1: Use database table as a message queue ACID transaction Customer Service ? Local transaction INSERT INSERT CUSTOMER_CREDIT_RESERVATIONS table ORDER_ID 99 CUSTOMER_ID 101 1234 Publish Message Broker QUERY DELETE MESSAGE table TOTAL Message Publisher ID 84784 TYPE OrderCreated DATA {…} DESTINATION … SeeBASE:An Acid Alternative,http://bit.ly/ebaybase@crichardson
  • 44.Publishing messages Poll the MESSAGE table OR Tail the database transaction log @crichardson
  • 45.Option #2: Eventsourcing:event-centric persistence Event table Service Entity id save events and publish Event Store Every state change Entity type Event id Event type Event data 101 Order 901 OrderCreated … 101 Order 902 OrderApproved … 101 Order 903 OrderShipped … event @crichardson
  • 46.Summary Microservices tackle complexity and accelerate development Database per service is essential for loose coupling Use sagas to maintain data consistency across services Use transactional messaging to make sagas reliable @crichardson
  • 47.@crichardson chris@chrisrichardson.net Questions?http://learnmicroservices.io@crichardson