Recurrent neural networks (RNN) Deep Learning

name: inverse
layout: true
class: center, middle, inverse

</span></div>

</span></div>

---

# Recurrent neural networks (RNN) 
 Deep Learning - Part 2

<div class="contributors-line">
		
	
<ul class="text-list">
			
			<li>
				<a href="/training-material/hall-of-fame/kxk302/" class="contributor-badge contributor-kxk302"><img src="https://avatars.githubusercontent.com/kxk302?s=36" alt="Kaivan Kamali avatar" width="36" class="avatar" />
    Kaivan Kamali</a></li>
</ul>

</div>

<div class="footnote" style="bottom: 8em;">
  <i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated:   
  <i class="fas fa-fingerprint" aria-hidden="true"></i><span class="visually-hidden">purl</span><abbr title="Persistent URL">PURL</abbr>: <a href="https://gxy.io/GTN:S00090">gxy.io/GTN:S00090</a>
</div>

<div class="footnote" style="bottom: 5em;">

<i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a> |

</div>

<div class="footnote" style="bottom: 2em;">
    <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes
    | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides

</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

## Requirements

Before diving into this slide deck, we recommend you to have a look at:

- [Statistics and machine learning](/training-material/topics/statistics)
    - Introduction to deep learning: [<i class="fas fa-laptop" aria-hidden="true"></i><span class="visually-hidden">tutorial</span> hands-on](/training-material/topics/statistics/tutorials/intro_deep_learning/tutorial.html)
    - Deep Learning (Part 1) - Feedforward neural networks (FNN): [<i class="fab fa-slideshare" aria-hidden="true"></i><span class="visually-hidden">slides</span> slides](/training-material/topics/statistics/tutorials/FNN/slides.html) - [<i class="fas fa-laptop" aria-hidden="true"></i><span class="visually-hidden">tutorial</span> hands-on](/training-material/topics/statistics/tutorials/FNN/tutorial.html)

---

### <i class="far fa-question-circle" aria-hidden="true"></i><span class="visually-hidden">question</span> Questions

- What is a recurrent neural network (RNN)?

- What are some applications of RNN?

---

### <i class="fas fa-bullseye" aria-hidden="true"></i><span class="visually-hidden">objectives</span> Objectives

- Understand the difference between feedforward neural networks (FNN) and RNN

- Learn various RNN types and architectures

- Learn how to create a neural network using Galaxy’s deep learning tools

- Solve a sentiment analysis problem on IMDB movie review dataset using RNN in Galaxy

---

# What is a recurrent neural network (RNN)?

???

What is a recurrent neural network (RNN)?

---

# Recurrent Neural Network (RNN)

- RNN models sequential data (temporal/ordinal)
- In RNN, training example is a sequence, which is presented to RNN one at a time 
	- E.g., sequence of English words is passed to RNN, one at a time 
	- And, RNN generates a sequence of Persian words, one at a time
- In RNN, output of network at time *t* is used as input at time *t+1*
- RNN applied to image description, machine translation, sentiment analysis, etc.

---

# One-to-many RNN

![Neurons forming a one-to-many recurrent neural network](/training-material/topics/statistics/images/RNN_1_to_n.png)

---

# Many-to-one RNN

![Neurons forming a many-to-one recurrent neural network](/training-material/topics/statistics/images/RNN_n_to_1.png)

---

# Many-to-many RNN

![Neurons forming a many-to-many recurrent neural network](/training-material/topics/statistics/images/RNN_n_to_m.png)

---

# RNN architectures

- Vanilla RNN
	- Suffers from *vanishing gradient* problem
- LSTM and GRU
	- Uses *gates* to avoid vanishing gradient problem

---

# Sentiment analysis

- We perform sentiment analysis on IMDB movie reviews dataset
- Train RNN on training dataset (25000 positive/negative movie reviews)
- Test RNN on test set (25000 positive/negative movie reviews)
- Training and test sets have no overlap
- Since dealing with text data, good to review mechanisms for representing text data

---

# Text preprocessing

- Tokenize a document, i.e., break it down into words
- Remove punctuations, URLs, and stop words (‘a’, ‘of’, ‘the’, etc.)
- Normalize the text, e.g., replace ‘brb’ with ‘Be right back’, etc
- Run the spell checker to fix typos
- Make all words lowercase

---

# Text preprocessing

- Perform stemming/lemmatization
	- If we have words like ‘organizer’, ‘organize’, and ‘organized’
		- Want to reduce all of them to a single word
	- Stemming cuts end of these words for a single root
		- E.g., ‘organiz’.  May not be an actual word
	- Lemmatization reduces to a root that is actually a word
		- E.g., ‘organize’

---

# Bag of words (BoW)

- If you don’t care about the order of the words in a document
- 2D array. Rows represent documents. Columns represent words in vocabulary
	- All unique words in all documents
- If a word not present in a document, we have a zero at row and column entry
- If a word is present in a document, we have a one at row and colum entry
	- Or, we could use the word count or frequency

---

# Bag of words (BoW)

- Document 1: Magic passed the basketball to Kareem
- Document 2: Lebron stole the basketball from Curry

![Table showing a bag-of-words representation of sample documents](/training-material/topics/statistics/images/BoW.png)

- BoW is simple, but does not consider rarity of words across documents
	- Important for document classification

---

# Term frequency inverse document frequency (TF-IDF)

- If you don’t care about the order of the words in a document
- Similar to BoW, we have an entry for each document-word pair
- Entry is product of
	- Term frequency, frequency of a word in a document, and
	- Inverse document frequency, total number of documents divided by number of documents that have word
		- Usually use logarithm of the IDF
- TF-IDF takes into account rarity of a word across documents

---

# One-hot encoding (OHE)

- Technique to convert categorical variables such as words into a vector
- Suppose our vocabulary has 3 words: orange, apple, banana
- Each word is represented by a vector of size 3

![Mathematical vectors representing one-hot-encoding representation of words orange, apple, and banana](/training-material/topics/statistics/images/OHE.gif)

- OHE problems
	- For very large vocabulary sizes requires tremendous amount of storage
	- Also, no concept of word similarity

---

# Word2Vec

- Each word represented as an *n* dimensional vector
	- *n* much smaller than vocabulary size
- Words that have similar meanings are close in vector space
- Words considered similar if they co-occur often in documents
- Two Word2Vec architectures
	- Continuous BoW
		- predicts probability of a word given the surrounding words
	- Continuous skip-gram
		- given a word predicts probability of the surrounding words

---

# Sentiment analysis

- Sentiment classification of IMDB movie reviews with RNN
- Train RNN using IMDB movie reviews
- Goal is to learn a model such that given a review we predict whether review is positive/negative
- We evaluate the trained RNN on test dataset and plot confusion matrix

---

# For references, please see tutorial's References section

---

## Thank You!

This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!

<div class="contributors-line">
		
<table class="contributions">
	
	<tr>
		<td><abbr title="These people wrote the bulk of the tutorial, they may have done the analysis, built the workflow, and wrote the text themselves.">Author(s)</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/kxk302/" class="contributor-badge contributor-kxk302"><img src="https://avatars.githubusercontent.com/kxk302?s=36" alt="Kaivan Kamali avatar" width="36" class="avatar" />
    Kaivan Kamali</a>
		</td>
	</tr>

<tr class="reviewers">
		<td><abbr title="These people reviewed this material for accuracy and correctness">Reviewers</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/shiltemann/" class="contributor-badge contributor-badge-small contributor-shiltemann"><img src="https://avatars.githubusercontent.com/shiltemann?s=36" alt="Saskia Hiltemann avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/kxk302/" class="contributor-badge contributor-badge-small contributor-kxk302"><img src="https://avatars.githubusercontent.com/kxk302?s=36" alt="Kaivan Kamali avatar" width="36" class="avatar" /></a></td>
	</tr>

</table>

</div>

</div>

Tutorial Content is licensed under <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.<br/>