recurrent neural networks

2020-02-27 58浏览

1.Recurrent Neural Networks Presented by Chris Foster
2.RNN Hidden States A recurrent neural network node contains a hidden state h
3.RNN Hidden States In practice, we also often have output y
4.Back Propogation Through Time (BPTT) Apply the backpropogation algorithm to the unrolled computational graph
5.BPTT is comparitively very expensive to perform Often, you can only hold limited steps in memory Training occurs in "batches" of the dataset Training is more difﬁcult to parallelize h is initialized to 0's or the most recent h
6.RNN Architectures One-to-one Vanilla processing modeEx:Image classiﬁcation
7.RNN Architectures One-to-many Sequence outputEx:Image captionining
8.RNN Architectures Many-to-one Sequence inputEx:Sentiment analysis
9.RNN Architectures Many-to-many Sequence input/outputEx:Machine translation
10.RNN Architectures Many-to-many Synced sequence input/outputEx:Video labelling
11.Other Network Types
12.Applications of RNN's
13.Applications of RNN's
14.Applications of RNN's
15.Finding Interpretable Cells
16.Vanishing Gradient Problem
17.Vanishing Gradient Problem Gradients in earlier layers are unstable This is noticable even in very deep FFN's An RNN, as we know, is like a very deep FFN The problem is made worse due to a static W
18.Gradient Clipping
19.Recall:Resnet Gradient Shortcuts
20.The LSTM Network
21.The LSTM Network
22.The LSTM Network
23.The LSTM Network
24.The LSTM Network
25.The LSTM Network
26.RNN vs LSTM
27.RNN vs LSTM
28.Gated Recurrent Unit
29.Summary RNNs allow a lot of ﬂexibility in architecture Vanilla RNNs are simple but don't work very well Practical networks should use a LSTM/GRU Backward ﬂow of gradients can explode or vanish
30.Thanks!
31.Sources Deep Learning, Chapter 10 Understanding LSTM Networks Recurrent Neural Networks Tutorial The Unreasonable Effectiveness of Recurrent Neural Networks CS231N Lecture 10 - Recurrent Neural Networks, Image Captioning, LSTM