Given a couple of words, can we generate the next word? Or, given a sentence, can we determine whether each word expresses a positive or negative sentiment? If you were to design a neural network function to do this, what would it look like?
He's → → 0
Happy → neural network → 1
I → (function) → 0
will → → 0
be → → 0
Mad → → 1
To make the word to be input parameter, we need to convert to numbers that computer can recognize. This is called encoding.
One extreme way to let it recognize the word is a large vector that every word has its unique position place as 1 and rest are 0s. Which is called one-hot encoding. But its dimension is too high. If we have 100k words, then we need to have 100k dimension vector space.
We need one way that dimension is not high neither low, which is called word embedding.
Why Word Embedding Works
Why this method can represent the correlation between word and word meaning? We can use two vectors’ dot product or cosine similarity to represent the correlation between words.
Dot product: a · b = ∑i=1N aᵢbᵢ
Cos(θ) = (a · b) / (|a| * |b|)
Embedding Matrix
All word vectors form one big matrix. This matrix is called embedding matrix.
← vocabulary size →
word → [ 0.12, 0.11, 0.29, 0.00, 0.23 ]
vector → [ 0.00, 0.00, 0.00, 0.00, 0.00 ]
... [ ..., ..., ..., ..., ... ]
dimension → [ 0.40, 0.90, 0.50, 0.00, 0.50 ]
You can explore the visualization interactively at TensorFlow Embedding Projector.
The graph below shows how high-dimensional word vectors can be projected into 2D or 3D space for visualization.
Feeding Into Neural Networks
Now every word can encode to vector to send to neural network of input neurons.
Just like CNN issue we had: What's the solution we can solve the order of words and reduce the amount of parameters in input layer in field of natural language processing?
First, we use the classic neural network (not one sentence but single word only):
He X → g(Wx + b) → Y
// output: positive or negative value of the word
We do the same on second word with order now:
He X⁽¹⁾ → g(Wx + b) → Y⁽¹⁾
happy X⁽²⁾ → g(Wx + b) → Y⁽²⁾
Adding Sequential Context
Now we have order, but the calculation of second word doesn’t have any info from first word to involve into. What we can do is: when after first word passes through non-linear exchange, don’t output result yet — output to a hidden state H₁, then do another non-linear exchange to get result Y₁.
X⁽¹⁾ → g(Wx + b) → H⁽¹⁾ → g(Wh + b) → Y⁽¹⁾
Now we let H₁ operate with second word x₂, same as second word to third word:
H⁽¹⁾ ──┐
├─→ g(Wx⁽²⁾ + Wh⁽¹⁾ + b)
x⁽²⁾ ──┘
So we can have the last word info transfer to next word, until the last word from last sentence. But we need to distinguish W:
x⁽¹⁾ → g(WxhX⁽¹⁾ + bh)
↓
h⁽¹⁾ → g(Whyh⁽¹⁾ + by) → y⁽¹⁾
↓
x⁽²⁾ → g(WxhX⁽²⁾ + Whhh⁽¹⁾ + bh)
↓
h⁽²⁾ → g(Wxh⁽²⁾ + by) → y⁽²⁾
Simplified RNN Graph
There’s a Wxh matrix specifically for word vectors, a Whh matrix for hidden state, and a hy matrix to calculate the final output, along with the bias term b. Simplify the above graph: this is Recurrent Neural Network (RNN).
x⁽¹⁾ → Wxh → h⁽¹⁾ → Why → y⁽¹⁾
↓
Whh
↓
x⁽²⁾ → Wxh → h⁽²⁾ → Why → y⁽²⁾
X → Wxh → h → Why → Y
↺ Whh
Final RNN Formula
Now this model understands the order between words, which solves the positive or negative value of words. Given one sentence to generate next word, translation, and so on — multiple tasks of NLP.
hᵗ = g(Wxh · Xᵗ + Whh · hᵗ⁻¹ + bh)
yᵗ = g(Why · hᵗ + by)
If you compare the formula above with a classic neural network, you'll see the only difference is the addition of the previous hidden state.
This is just one way to understand how word embedding connects to RNNs — a fundamental concept that powers many breakthroughs in natural language processing. As the field evolves, more advanced architectures like LSTMs, GRUs, and Transformers have built upon these ideas. The key is to keep building your foundation, experimenting with models, and exploring how words can be turned into meaning through mathematics and code.