豆瓣架构演进

2020-02-27 144浏览

1.QCon Beijing 2012
2.
3.book models group views shire templates doumail scripts . . . tests
4.memcached nginx shire mysql beansdb
5.memcached nginx shire mysql beansdb server_name 9.douban.com; rewrite (.*) /ninetaps/$1 last;
6.
7.•
8.• •
9.• • •
10.shire Python 4000 500000 3000 375000 2000 250000 1000 125000 0 0 2006 2007 2008 2009 2010 2011 2012
11.
12.•
13.• •
14.• • •
15.• • • •
16.• • • • •
17.• • • • • •
18.
19.……
20.
21.
22.•
23.• •m.douban.com:Pylons
24.• •m.douban.com:Pylons : Django •
25.
26.•
27.• • book movie alphatown...
28.• • • book movie alphatown...
29.• • • book • IP movie alphatown...
30.• • • book • IP • movie alphatown...
31.
32.•
33.• •
34.
35.•
36.• •
37.• • •
38.
39.•
40.• •
41.• • •
42.• • • •
43.• • • • •
44.
45.•
46.• •
47.• • •
48.• • • •
49.• • • • • /
50.
51.•
52.• •
53.• • •
54.• • • •
55.• • • • • douban-corelib
56.shire douban-corelib community models book views movie templates . . scripts corelib tests
57.
58.• app server
59.• • nginx app server
60.nginx shire.main memcached shire.fm mysql alphatown beansdb . . . services
61.
62.
63.
64.
65.
66.• web
67.• web •
68.• web • •
69.• web • • •
70.• web • • • •
71.
72.•
73.• •
74.• • •
75.• • • •
76.• • • • •
77.• • • • • •
78.
79.•
80.• • fail over
81.
82.•
83.
84.• sys.path
85.• • sys.path
86.
87.
88.•
89.• •
90.• • •
91.• • • •
92.• • • • •
93.• • • • • • API
94.• • • • • • • API
95.• • • • • • • • API
96.• • • • • • • • • API
97.
98.• DAE (Douban App Engine)
99.PaaS
100.PaaS • IaaS - Infrastructure as a Service (EC2)
101.PaaS • IaaS - Infrastructure as a Service (EC2) • PaaS - Platform as a Service (GAE)
102.PaaS • IaaS - Infrastructure as a Service (EC2) • PaaS - Platform as a Service (GAE) • SaaS - Software as a Service (Gmail)
103.
104.
105.
106.•
107.• •
108.• • •
109.• • • •
110.
111.• UNIX
112.• • UNIX
113.• • UNIX •
114.
115.• Python
116.• Python • Python C
117.• Python • Python C • Python/C API
118.• Python • Python C • Python/C API • Cython/Pyrex
119.• Python • Python C • Python/C API • Cython/Pyrex • go
120.
121.• virtualenv + pip
122.• virtualenv + pip • app virtualenv
123.• virtualenv + pip • app • dae install virtualenv
124.• virtualenv + pip • app • dae install • pip install virtualenv pip-req.txt
125.
126.$ dae create daetest
127.$ dae create daetest $ dae install web.py
128.$ dae create daetest $ dae install web.py $ vim app.py
129.$ dae create daetest $ dae install web.py $ vim app.py import web urls = ('/', 'index') classindex:def GET(self): return "Hello, DAE" app = web.application(urls, globals()).wsgifunc()
130.$ dae create daetest $ dae install web.py $ vim app.py import web urls = ('/', 'index') classindex:def GET(self): return "Hello, DAE" app = web.application(urls, globals()).wsgifunc() $ dae serve
131.$ dae create daetest $ dae install web.py $ vim app.py import web urls = ('/', 'index') classindex:def GET(self): return "Hello, DAE" app = web.application(urls, globals()).wsgifunc() $ dae serve $ dae deploy
132.$ dae create daetest $ dae install web.py $ vim app.py import web urls = ('/', 'index') classindex:def GET(self): return "Hello, DAE" app = web.application(urls, globals()).wsgifunc() $ dae serve $ dae deploy openhttp://daetest.dapps.douban.com
133.app.yaml
134.app.yaml •
135.app.yaml • • wsgi apps
136.app.yaml • • wsgi apps • daemons
137.app.yaml • • wsgi apps • daemons • cron
138.app.yaml • • wsgi apps • daemons • cron • services
139.app.yaml • • wsgi apps • daemons • cron • services • protected_files
140.API
141.API • mysql
142.API • • mysql memcache
143.API • • • mysql memcache beansdb
144.API • • • • mysql memcache beansdb filesystem storage (MooseFS)
145.API • • • • • mysql memcache beansdb filesystem storage (MooseFS) task queue
146.API • • • • • • mysql memcache beansdb filesystem storage (MooseFS) task queue scheduled task
147.API • • • • • • • mysql memcache beansdb filesystem storage (MooseFS) task queue scheduled task user login
148.API • • • • • • • • mysql memcache beansdb filesystem storage (MooseFS) task queue scheduled task user login dpark
149.API • • • • • • • • • mysql memcache beansdb filesystem storage (MooseFS) task queue scheduled task user login dpark app namespace
150.
151.• LVS
152.• • LVS nginx
153.• • • LVS nginx deploy
154.• • • • LVS nginx deploy thrift
155.• • • • • LVS nginx deploy thrift onimaru - error collector based on django-sentry
156.• • • • • • LVS nginx deploy thrift onimaru - error collector based on django-sentry scribe
157.• • • • • • • LVS nginx deploy thrift onimaru - error collector based on django-sentry scribe puppet
158.• • • • • • • • LVS nginx deploy thrift onimaru - error collector based on django-sentry scribe puppet
159.
160.• gunicorn -- a fast wsgi server
161.• gunicorn -- a fast wsgi server • gevent -- coroutine library based on greenlet and libev
162.• gunicorn -- a fast wsgi server • gevent -- coroutine library based on greenlet and libev • websocket support
163.• gunicorn -- a fast wsgi server • gevent -- coroutine library based on greenlet and libev • websocket support • mesos
164.gunicorn (app A) gateway gunicorn (app B) nginx Node Node worker worker worker worker
165.
166.•
167.• •
168.
169.• setuptools (setup.py)
170.• setuptools (setup.py) • DAE freeze pip
171.• setuptools (setup.py) • DAE pip freeze • pip install -r
172.• setuptools (setup.py) • DAE pip freeze • • pip install -r
173.
174.• pip-req.txt
175.• • PyPI pip-req.txt
176.• • PyPI • pip-req.txt revision
177.• • PyPI • pip-req.txt revision -ehg+http://hghub.dapps.douban.com/doubancorelib@fb367759be2e1b37c77701d59be6514a3b39837e#egg=DoubanCoreLibdistribute==0.6.19 web.py==0.36 wsgiref==0.1.2
178.
179.• DAE Service
180.• • DAE Service thrift
181.• • • DAE Service thrift app.yaml
182.• • • • DAE Service thrift app.yaml gunicorn gevent worker
183.• • • • • DAE Service thrift app.yaml gunicorn gevent worker DNS
184.• • • • • DAE Service thrift app.yaml gunicorn gevent worker DNS •
185.• • • • • DAE Service thrift app.yaml gunicorn gevent worker DNS • • zookeeper
186.• • • • • • DAE Service thrift app.yaml gunicorn gevent worker DNS • • zookeeper fail over
187.• • • • • • • DAE Service thrift app.yaml gunicorn gevent worker DNS • • zookeeper fail over DAE
188.user.thrift struct UserProfile { 1: i32 uid, 2: string name, 3: string blurb } service UserStorage { void store(1: UserProfile user), UserProfile retrieve(1: i32 uid) }
189.user.thrift struct UserProfile { 1: i32 uid, 2: string name, 3: string blurb } service UserStorage { void store(1: UserProfile user), UserProfile retrieve(1: i32 uid) } app.yamlservices:-interface:user.UserStoragehandler:user.handler:UserStorageHandler
190.user.thrift struct UserProfile { 1: i32 uid, 2: string name, 3: string blurb } service UserStorage { void store(1: UserProfile user), UserProfile retrieve(1: i32 uid) } app.yamlservices:-interface:user.UserStoragehandler:user.handler:UserStorageHandler handler.py class UserStorageHandler(object): def store(self, user): ... def retrieve(self, uid): ...
191.DAE Service
192.DAE Service $ dae service gen_client
193.DAE Service $ dae service gen_client from daetest_client import UserStorage user = UserStorage.retrieve(1)
194.DAE Service $ dae service gen_client from daetest_client import UserStorage user = UserStorage.retrieve(1)
195.
196.• The next big issue
197.• The next big issue • Done is better than perfect
198.
199.
200.
201.app app app app Platform app app app app
202.• • • • •
203.Q &Ahttp://www.douban.com/people/hongqn/hongqn@douban.comtwitter:@hongqn : @hongqn
204.Thanks
205.
206.