微软亚洲研究院主管研究员王井东 - CNN Architecture Design：From Deeper to Wider_部分1

2020-03-04 379浏览

1.CNN ArchitectureDesign:From Deeper to Wider 主讲人：微软亚洲研究院主管研究员王井东
2.Not good as expected • ImageNet, GPU Win almost in all the applications
3.Deeper and deeper
4.ResNet, Highway, 152 layers 100+ GoogLeNet, 8 22 layers VGGNet, 19 layers AlexNet, 7 layers
5.ResNet, Highway, 152 layers 100+ GoogLeNet, 19 22 layers VGGNet, 19 layers AlexNet, 7 layers
6.ResNet, Highway, 152 layers 100+ GoogLeNet, 22 22 layers VGGNet, 19 layers AlexNet, 7 layers
7.T 1-T x H x + 100+ T 1-T x H x + T 1-T x H x + T 1-T x H x + VGGNet, 19 layers GoogLeNet, 22 layers Highway, 100+ AlexNet, 7 layers ResNet, 152 layers
8.152 VGGNet, 19 layers GoogLeNet, 22 layers Highway, 100+ AlexNet, 7 layers ResNet, 152 layers
9.
10.Unifying GoogLeNets, Highway, ResNets Jingdong Wang, Zhen Wei, Ting Zhang, WenjunZeng:Deeply-Fused Nets. CoRR abs/1605.07716 (2016)
11.Input Input C CC L01 R01 L01 R01 R02 R02 + L02 R03 = + L02 R03 R04 R04 + + L03 R05 L03 R05 R06 R06 + + FC FC FC LLL RRR
12.
13.Input C CC L01 R01 L01 R01 R02 R02 + L02 R03 = + L02 R03 R04 R04 + + L03 R05 L03 R05 R06 R06 + + FC FC FC LLL RRR Input
14.Input C L01 R01 R02 + L02 R03 = R04 + L03 R05 R06 + FC Input CC L01 R01 R02 + R03 L02 R04 + R05 L03 R06 + FC FC LRR RLL
15.Input C L01 R01 R02 + L02 R03 = R04 + L03 R05 R06 + FC Input CC L01 R01 R02 + R03 L02 R04 + L03 R05 R06 + FC FC LRL RLR
16.Input C L01 R01 R02 + L02 R03 = R04 + L03 R05 R06 + FC Input CC L01 R01 R02 + L02 R03 R04 + R05 L03 R06 + FC FC LLR RRL
17.Input 1 2 Input 3 4 C CCCCCCCC L01 R01 L01 R01 L01 R01 L01 R01 L01 R01 R02 R02 R02 R02 R02 + L02 R03 = + + + + L02 R03 R03 L02 R03 L02 L02 R03 R04 R04 R04 R04 R04 + + + + + L03 R05 L03 R05 R05 L03 L03 R05 R05 L03 R06 R06 R06 R06 R06 + + + + + FC FC FC FC FC FC FC FC FC LLL RRR LRR RLL LRL RLR LLR RRL
18.
19.Input Input C CCCCCCCC L01 R01 L01 R01 L01 R01 L01 R01 L01 R01 R02 R02 R02 R02 R02 + L02 R03 = + + + + L02 R03 R03 L02 R03 L02 L02 R03 R04 R04 R04 R04 R04 + + + + + L03 R05 L03 R05 R05 L03 L03 R05 R05 L03 R06 R06 R06 R06 R06 + + + + + FC FC FC FC FC FC FC FC FC LLL RRR LRR RLL LRL RLR LLR RRL
20.Input C L01 R01 R02 + L02 R03 R04 + L03 R05 R06 + FC Input CCCCCCCC L01 R01 L01 R01 L01 R01 L01 R01 R02 R02 R02 R02 L02 R03 R03 L02 R03 L02 L02 R03 R04 R04 R04 R04 L03 R05 R05 L03 L03 R05 R05 L03 R06 R06 R06 R06 FC FC FC FC FC FC FC FC LLL RRR LRR RLL LRL RLR LLR RRL
21.
22.Input C L01 R01 R02 + L02 R03 R04 + L03 R05 R06 + FC Input CCCCCCCC L01 R01 L01 R01 L01 R01 L01 R01 R02 R02 R02 R02 L02 R03 R03 L02 R03 L02 L02 R03 R04 R04 R04 R04 L03 R05 R05 L03 L03 R05 R05 L03 R06 R06 R06 R06 FC FC FC FC FC FC FC FC LLL RRR LRR RLL LRL RLR LLR RRL
23.Input C L01 R01 R02 + L02 R03 R04 + L03 R05 R06 + FC Input CC L01 R01 R02 3 layers L02 R03 R04 CC L01 R01 R02 2 layers R03 L02 R04 4 layers L03 R05 R06 3 layers L03 R05 R06 FC FC LLL RRR FC FC LRL RLR
24.2 7 12
25.Deep fusion Multiple paths Long and short Express way between layers Weight sharing
26.
27.Input conv conv conv + conv conv + conv conv + conv conv + conv conv + conv conv + FC 6 blocks Input conv conv conv + conv conv + conv conv + FC L 3 blocks
28.Top-1 validation error Top-5 validation error ResNet 24.94 7.46 DFN 25.10 7.85 DFN ResNet
29.Wider wider Deeper and deeper
30.Liming Zhao, Jingdong Wang, Xi Li, Zhuowen Tu, WenjunZeng:Deep Convolutional Neural Networks with Merge-andRun Mappings. (2017)Blog:深度神经网络中深度究竟带来了什么？ (http://www.msra.cn/zh-cn/news/blogs/2016/12/deep-neuralnetwork-20161212.aspx)
31.Identity mapping Input conv conv conv + conv conv + conv conv + conv conv + conv conv + conv conv + FC Residual branch
32.conv conv + conv conv + Merge-and-run conv conv + conv conv + +
33.Identity mapping Input conv conv conv + conv conv + conv conv + conv conv + conv conv + conv conv + FC Residual branch Input conv conv conv + conv conv + + conv conv + conv conv + + conv conv + conv conv + FC Merge-and-run Ours is less deep DMRNet
34.
35.conv conv + conv conv + wider conv conv + conv conv + +
36.Merge-and-run conv conv + conv conv + + Quick path from t’ to t
37.CIFAR-10 CIFAR-100 L Identity Merge-and-run Identity Merge-and-run 48 5.21 4.99 25.31 24.73 96 5.19 4.84 24.16 23.98 conv conv conv conv ++ conv conv + conv conv + +
38.#(training images) CIFAR-10 50,000 CIFAR-100 50,000 SVHN 73,257 + 531,131 #(testing images) 10,000 10,000 26,032 #classes 10 100 10
39.- - 7.97 34.57 1.92 21 38.6M 5.22 23.30 2.01 21 38.6M 4.60 23.73 1.87 110 1.7M 6.41 27.22 2.01 200 10.2M 4.35 20.42 - 16 11.0M 4.81 22.07 - 28 36.5M 4.17 20.50 - 40 1.0M 5.24 24.42 1.79 100 27.2M 3.74 19.25 1.59 DMRNet (ours) 56 1.7M 4.94 24.46 1.66 DMRNet-Wide (ours) 32 14.9M 3.94 19.25 1.51 DMRNet-Wide (ours) 50 24.8M 3.57 19.00 1.55
40.- - 7.97 34.57 1.92 21 38.6M 5.22 23.30 2.01 21 38.6M 4.60 23.73 1.87 110 1.7M 6.41 27.22 2.01 200 10.2M 4.35 20.42 - 16 11.0M 4.81 22.07 - 28 36.5M 4.17 20.50 - 40 1.0M 5.24 24.42 1.79 100 27.2M 3.74 19.25 1.59 DMRNet (ours) 56 1.7M 4.94 24.46 1.66 DMRNet-Wide (ours) 32 14.9M 3.94 19.25 1.51 DMRNet-Wide (ours) 50 24.8M 3.57 19.00 1.55
41.DSN - - 7.97 34.57 1.92 FractalNet 21 38.6M 5.22 23.30 2.01 with DO/DP 21 38.6M 4.60 23.73 1.87 110 1.7M 6.41 27.22 2.01 Multi ResNet 200 10.2M 4.35 20.42 - Wide ResNet 16 11.0M 4.81 22.07 - 28 36.5M 4.17 20.50 - 40 1.0M 5.24 24.42 1.79 DenseNet 100 27.2M 3.74 19.25 1.59 DMRNet (ours) 56 1.7M 4.94 24.46 1.66 DMRNet-Wide (ours) 32 14.9M 3.94 19.25 1.51 DMRNet-Wide (ours) 50 24.8M 3.57 19.00 1.55
42.#parameters L 0.4M 12 0.6M 18 0.8M 24 1.0M 30 1.2M 36 1.5M 48 1.7M 54 3.1M 96 ResNets 6.62 5.93 5.60 5.50 5.35 5.26 5.24 5.47 DMRNets 6.48 5.79 5.47 5.10 5.18 4.99 4.96 4.84
43.#parameters L 0.4M 12 0.6M 18 0.8M 24 1.0M 30 1.2M 36 1.5M 48 1.7M 54 3.1M 96 ResNets 29.69 27.90 27.03 26.44 26.00 25.44 24.56 24.41 DMRNets 29.62 27.80 26.76 25.87 25.41 24.73 24.41 23.98