+ - 0:00:00
Notes for current slide

Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press P again to switch presenter notes off

Press C to create a new window where the same presentation will be displayed. This window is linked to the main window. Changing slides on one will cause the slide to change on the other.

Useful when presenting.

Notes for next slide



Recurrent neural networks (RNN)

Deep Learning - Part 2



last_modification Updated:   purlPURL: gxy.io/GTN:S00090

text-document Plain-text slides |

Tip: press P to view the presenter notes | arrow-keys Use arrow keys to move between slides
1 / 22

Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press P again to switch presenter notes off

Press C to create a new window where the same presentation will be displayed. This window is linked to the main window. Changing slides on one will cause the slide to change on the other.

Useful when presenting.

Requirements

Before diving into this slide deck, we recommend you to have a look at:

2 / 22

question Questions

  • What is a recurrent neural network (RNN)?

  • What are some applications of RNN?

3 / 22

objectives Objectives

  • Understand the difference between feedforward neural networks (FNN) and RNN

  • Learn various RNN types and architectures

  • Learn how to create a neural network using Galaxy’s deep learning tools

  • Solve a sentiment analysis problem on IMDB movie review dataset using RNN in Galaxy

4 / 22

What is a recurrent neural network (RNN)?

5 / 22

What is a recurrent neural network (RNN)?

Recurrent Neural Network (RNN)

  • RNN models sequential data (temporal/ordinal)
  • In RNN, training example is a sequence, which is presented to RNN one at a time
    • E.g., sequence of English words is passed to RNN, one at a time
    • And, RNN generates a sequence of Persian words, one at a time
  • In RNN, output of network at time t is used as input at time t+1
  • RNN applied to image description, machine translation, sentiment analysis, etc.
6 / 22

One-to-many RNN

Neurons forming a one-to-many recurrent neural network

7 / 22

Many-to-one RNN

Neurons forming a many-to-one recurrent neural network

8 / 22

Many-to-many RNN

Neurons forming a many-to-many recurrent neural network

9 / 22

RNN architectures

  • Vanilla RNN
    • Suffers from vanishing gradient problem
  • LSTM and GRU
    • Uses gates to avoid vanishing gradient problem
10 / 22

Sentiment analysis

  • We perform sentiment analysis on IMDB movie reviews dataset
  • Train RNN on training dataset (25000 positive/negative movie reviews)
  • Test RNN on test set (25000 positive/negative movie reviews)
  • Training and test sets have no overlap
  • Since dealing with text data, good to review mechanisms for representing text data
11 / 22

Text preprocessing

  • Tokenize a document, i.e., break it down into words
  • Remove punctuations, URLs, and stop words (‘a’, ‘of’, ‘the’, etc.)
  • Normalize the text, e.g., replace ‘brb’ with ‘Be right back’, etc
  • Run the spell checker to fix typos
  • Make all words lowercase
12 / 22

Text preprocessing

  • Perform stemming/lemmatization
    • If we have words like ‘organizer’, ‘organize’, and ‘organized’
      • Want to reduce all of them to a single word
    • Stemming cuts end of these words for a single root
      • E.g., ‘organiz’. May not be an actual word
    • Lemmatization reduces to a root that is actually a word
      • E.g., ‘organize’
13 / 22

Bag of words (BoW)

  • If you don’t care about the order of the words in a document
  • 2D array. Rows represent documents. Columns represent words in vocabulary
    • All unique words in all documents
  • If a word not present in a document, we have a zero at row and column entry
  • If a word is present in a document, we have a one at row and colum entry
    • Or, we could use the word count or frequency
14 / 22

Bag of words (BoW)

  • Document 1: Magic passed the basketball to Kareem
  • Document 2: Lebron stole the basketball from Curry

Table showing a bag-of-words representation of sample documents

  • BoW is simple, but does not consider rarity of words across documents
    • Important for document classification
15 / 22

Term frequency inverse document frequency (TF-IDF)

  • If you don’t care about the order of the words in a document
  • Similar to BoW, we have an entry for each document-word pair
  • Entry is product of
    • Term frequency, frequency of a word in a document, and
    • Inverse document frequency, total number of documents divided by number of documents that have word
      • Usually use logarithm of the IDF
  • TF-IDF takes into account rarity of a word across documents
16 / 22

One-hot encoding (OHE)

  • Technique to convert categorical variables such as words into a vector
  • Suppose our vocabulary has 3 words: orange, apple, banana
  • Each word is represented by a vector of size 3

Mathematical vectors representing one-hot-encoding representation of words orange, apple, and banana

  • OHE problems
    • For very large vocabulary sizes requires tremendous amount of storage
    • Also, no concept of word similarity
17 / 22

Word2Vec

  • Each word represented as an n dimensional vector
    • n much smaller than vocabulary size
  • Words that have similar meanings are close in vector space
  • Words considered similar if they co-occur often in documents
  • Two Word2Vec architectures
    • Continuous BoW
      • predicts probability of a word given the surrounding words
    • Continuous skip-gram
      • given a word predicts probability of the surrounding words
18 / 22

Sentiment analysis

  • Sentiment classification of IMDB movie reviews with RNN
  • Train RNN using IMDB movie reviews
  • Goal is to learn a model such that given a review we predict whether review is positive/negative
  • We evaluate the trained RNN on test dataset and plot confusion matrix
19 / 22

For references, please see tutorial's References section

20 / 22
21 / 22

Thank You!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors!

Author(s) Kaivan Kamali avatar Kaivan Kamali
Reviewers Saskia Hiltemann avatarKaivan Kamali avatar
Galaxy Training Network

Tutorial Content is licensed under Creative Commons Attribution 4.0 International License.

22 / 22

Requirements

Before diving into this slide deck, we recommend you to have a look at:

2 / 22
Paused

Help

Keyboard shortcuts

, , Pg Up, k Go to previous slide
, , Pg Dn, Space, j Go to next slide
Home Go to first slide
End Go to last slide
Number + Return Go to specific slide
b / m / f Toggle blackout / mirrored / fullscreen mode
c Clone slideshow
p Toggle presenter mode
t Restart the presentation timer
?, h Toggle this help
Esc Back to slideshow