【T112017 数据工程和技术分会场】TiDB+as+an+HTAP+Database

2020-03-01 147浏览

  • 1.TiDB as an HTAP Database 主讲人：PingCap CEO 刘奇
  • 2.About me • • • • • • CEO and co-founder of PingCAP Open source hacker Infrastructure engineer Founder of TiDB/TiKV/Codis Infrastructure software engineer Wandoulabs/JD
  • 3.Why a new database?
  • 4.Brief History RDBMS ● Standalone RDBMS 1970s NoSQL 2010 NewSQL 2015 Present ● NoSQL ● Middleware & Proxy ● NewSQL MySQL PostgreSQL Oracle DB2 ... Redis HBase Cassandra MongoDB ... Google Spanner Google F1 TiDB
  • 5.NewSQL Database ● ● ● ● Horizontal Scalability ACID Transaction High Availability SQL at Scale
  • 6.OLTP & OLAP OLAP OLTP ETL 8am 2pm 6pm 2am Database ERP Where is my data? Is the data out-of-date? Data Warehouse CRM
  • 7.Why two separate systems ● Huge data size ● Complex query logic ● Latency VS Throughput ● Point query VS Full range scan ● Transaction & lsolation level
  • 8.OLAP + OLTP = HTAP Hybrid Transactional / Analytical Processing ● ACID Transcation ● Real-time analysis ● SQL HTAP
  • 9.How do we build the new database
  • 10.What is TiDB ● Scalability as the first class feature ● SQL is necessary ● Compatible with MySQL, in most cases ● OLTP + OLAP = HTAP (Hybrid Transactional/Analytical Processing) ● 24/7 availability, even in case of datacenter outages ● Open source, of course
  • 11.Architecture Stateless SQL Layer TiDB ... TiDB ... TiDB Metadata / Timestamp request Placement Driver (PD) TiKV Raft ... TiKV Raft TiKV Distributed Storage Layer Raft TiKV Controlflow:Balance / Failover
  • 12.TiKV - Overview •Region:a set of continuous key-value pairs • Data is organized/stored/replicated by Regions • Highly layered TiKV Key Space RPC (gRPC) Node A Transaction 256MB MVCC [ start_key, end_key) Raft RocksDB Raft Raft Node B Node C Raft (-∞, +∞) Sorted Map
  • 13.TiKV - Multi-Raft Multiple raft groups in the cluster, one group for each region. Client RPC Raft Group RPC RPC RPC Store 4 Store 1 Store 2 Store 3 Region 1 Region 1 Region 2 Region 1 Region 3 Region 2 Region 5 Region 2 Region 5 Region 4 Region 3 Region 5 Region 4 Region 3 TiKV node 1 TiKV node 2 Region 4 TiKV node 3 TiKV node 4
  • 14.TiKV - Horizontal Scale Node B Region 1^ Region 1 Region 2 Region 3 Region 1* Region 2 Node D Region 2 Region 3 Region 3 Node C Node A Add Replica Three steps to move a leader replica Node E ● ● ● Transfer Leader Add Replica Remove Replica
  • 15.PD - Overview • Meta data management • Load balance management Route Info TiKV Client PD Node/Region Info TiKV Cluster TiKV TiKV Management Command TiKV … ... TiKV
  • 16.PD - TiKV Cluster Managment Node 1 PD HeartBeat Cluster Info Region A Region B Movement Node 2 Region C Scheduling Command Admin Scheduling Stratege Config
  • 17.TiDB - Overview • The stateless SQL layer SQL AST Logical Plan Cost Model Statistics Selected Physical Plan TiDB SQL Layer TiKV TiKV TiKV Optimized Logical Plan
  • 18.TiDB - Distributed SQL SELECT COUNT(c1) FROM t WHERE c1 > 10 AND c2 = ‘shanghai’; Physical Plan on TiDB Final Aggregate SUM(COUNT(c1)) COUNT(c1) DistSQL Scan Physical Plan on TiKV (index scan) Partial Aggregate COUNT(c1) Row Filter c2 = “shanghai” Row Read Row Data by RowID COUNT(c1) TiKV COUNT(c1) COUNT(c1) RowID Read Indexidx1:(10, +∞) TiKV TiKV
  • 19.TiDB - Cost Based Optimizer • • • • • • Predicate Pushdown Column Pruning Eager Aggregate Convert Subquery to Join Statistics framework CBO Framework – Index Selection – Join Operator Selection • Hash join • Index lookup join • Sort-merge join – Stream Operators VS Hash Operators
  • 20.OLTP + OLAP TiDB OLTP Query TiKV Jobs TiDB High Priority OLAP Query Scheduler TiKV Job Queue TiDB TiKV Worker Worker TiDB . . . . TiKV Worker Pool TiDB Jobs Low Priority Worker
  • 21.HTAP
  • 22.Ecosystem:Beyond TiDB and SQL
  • 23.Spark on TiDB PD PD Data location TSO/Data location PD PD Cluster Meta data Spark Driver TiDB TiKV Application TiDB TiDB Job TiKV DistSQL API DistSQL API TiKV TiKV Worker Worker Syncer TiDB TiKV TiKV TiDB ... TiDB Cluster ... TiKV Cluster (Storage) Worker ... Spark Cluster TiSpark
  • 24.Spark on TiDB Spark ecosystem TiKV Connector is better than JDBC connector Index support Complex Calculation Pushdown CBO – Pick up right Access Path – Join Reorder • Priority & Isolation Level • • • • •
  • 25.Future plan • • • • Code Generation MPP Engine Mixed storage engine (Columnar / Row-based) Heterogeneous computing (CPU/GPU/FPGA)
  • 26.THANKS