The power of embeddings

Natural Language Processing (NLP, also known as text mining) is an umbrella term for techniques used to extract insights from text data. These techniques are applied in practice with varying degrees of success. In recent years, rapid advances in neural networks (particularly by tech giants such as Google) have led to several major breakthroughs. How do they work? How useful are these new techniques in practice? And do they also work with Dutch-language text data? There is too much to cover in a single blog post, so this will be a two-part series. This first part takes a closer look at the concept of embeddings: a way of representing text data that underpins recent developments. The second part looks at practical applications and specific possibilities for Dutch-language text.

NLP is not a development of the past few years; it has been around for a long time. Conceptually, it dates back to the 1950s, and in practical terms to around the 1990s. Topics such as Named Entity Recognition, Part-of-Speech tagging and stemming & lemmatization are standard components of NLP toolkits. The main applications for text data are semantic search, sentiment analysis, topic modelling, classification and clustering. Common techniques include bag-of-words, tf-idf, Naïve Bayes and LDA. What we have seen in recent years, however, is a shift towards NLP techniques based on neural networks. This began with the introduction of word2vec[1] in 2013: a kind of dictionary that lets you translate a word into a series of numbers carrying semantic meaning (word embeddings). These embeddings were widely used in recurrent neural networks. The principle of attention[2] made neural networks better able to learn and retain longer-term relationships within passages of text. The arrival of the transformer[3] improved this considerably, and today the focus is on pre-trained language models from the BERT family[4]. This has finally made transfer learning a reality in NLP, just as pre-trained convolutional neural networks had done earlier in computer vision. Chronologically, the most important academic developments are therefore:

But as noted, the rapid progress began with word2vec and word embeddings. Why is this so important? Because it fundamentally changes how text is represented.

Representations for text data

A computer cannot calculate with text. It can only work with numbers. If we want to apply algorithms to text data, we first need to ‘translate’ the text into numbers. In other words, we need to find a numerical representation.

Bag-of-words

For many years, the bag-of-words (BoW) representation has been widely used. The approach is simple. Suppose you have a set of documents (pieces of text). You then:

• Identify which words occur
• Use them to build a vocabulary
• For each document, count how often each word in the vocabulary occurs
• Each document is now a long sequence of numbers
• Your ‘corpus’ (text dataset) is now a matrix of numbers
• B. Stop words are often filtered out of the vocabulary

This matrix of numbers can now be used with all kinds of algorithms for classification, clustering, topic modelling, etc. tf/idf is also often used to give distinctive words a higher weight in this matrix and irrelevant words a lower one, but this technique is not exclusive to BoW. So what is the problem with this representation? BoW’s simplicity is its strength, but it also has several limitations. To understand text, we need to:

1. Understand the meaning of individual words
2. Understand how words relate to one another

How does BoW perform on these points?

• Understanding the meaning of individual words
  • A word is represented only by an index referring to the vocabulary. No meaning is attached to it.
• Understanding how words relate to one another
  • The order of words in the text is ignored.

Two examples to illustrate:

• BoW sees no similarity between ‘animal’ and ‘creature’, even though they mean almost the same thing
• By contrast, BoW sees these sentences as exactly the same:
  • “Not as I expected, this film is great”
  • “As I expected, this film is not great”

Word embeddings

The central idea behind word embeddings is John Firth’s distributional hypothesis[5]:

“You shall know a word by the company it keeps”.

In other words, you can infer a word’s meaning from the context in which it occurs. And that is exactly how the word2vec algorithm works: it looks at passages of text to see which words occur alongside which context words.

In the The quick brown fox example, this may not seem very useful. But if you do it for millions or even billions of sentences, patterns start to emerge. This gives you plenty of data to train a model (a neural network) that tries to predict the context words given a word. This may seem like a pointless task, but it has a useful by-product: a matrix containing a set of weights (numbers) for each word. And here’s the interesting part: there is something special about these numbers. The meaning of a word is encoded in them and can be extracted. How? By comparing numbers: they can be close together (81 and 82) or far apart (2 and 81), which you can interpret as similarity or difference. Because each word is represented by several numbers, you can consider different kinds of similarity. Take the now-canonical example “king – man + woman = queen”:

Let’s look at the numbers we get (called embeddings) for the four words “king”, “man”, “woman” and “queen” in the image below.

Here’s what stands out:

• Some numbers/columns (shown in colour below) are close together when we compare “king” and “man”. This suggests a semantic similarity between the two words (which there is)
• The same applies to “woman” and “queen”
• There is a similar relationship between “king” and “queen”, but in different numbers/columns

Semantically, if you take the word “king” and replace its male aspect with a female one, you get “queen”. And that is exactly (well, almost exactly) what happens when you do the equivalent calculation with these words’ word embeddings! This suggests that some numbers in the embedding capture gender, while others capture ‘royalty’. Now imagine that our embeddings contain just three numbers (in reality, they contain hundreds or even thousands). If you treat those numbers as points in 3D space, you can visualise the relationships between words:

If you want to discover a few more interesting and surprising connections, I recommend this blog post: https://graceavery.com/word2vec-fish-music-bass/

Word embeddings as transfer learning

So where do you get those word embeddings? Generally, you don’t create them yourself; you reuse existing ones. Google researchers trained their word2vec algorithm on enormous datasets and made the embeddings publicly available in various languages. There are also other algorithms for creating word embeddings. The best-known are fastText from Facebook and GloVe from Stanford. You can download and use all of them. This was the first form of transfer learning in NLP.

From word embeddings to text embeddings

So, we’ve found a new representation for words in which meaning is more or less encoded in numbers. How can we assess this representation?

• Understanding the meaning of individual words
  • It looks like we’ve done this very well! The word embeddings for “animal” and “creature” will be very similar
• Understanding how words relate to each other
  • Erm, how they relate…?

Well, as the term suggests, word embeddings represent words, but not yet sentences or passages of text. With BoW, we could at least represent passages of text, so where do we go from here? A common approach was to combine recurrent neural networks (e.g. LSTM, GRU) with word embeddings. The text is split into words and converted into a sequence of word embeddings, which are read one step at a time. At each step, the network’s internal ‘memory’ is updated, so you still end up with an embedding for the entire text. This embedding is then used for the task at hand (prediction, text generation, etc.).

From here, we can broadly distinguish two approaches:

1. ‘direct’ text embedding techniques
2. improvements focused on sensitivity to context

Direct text embeddings

There are several techniques for creating text embeddings. They are generally used to embed sentences or paragraphs. Embedding entire pages of text at once does not yet work well. The simplest method (though not necessarily the worst) is to use average word embeddings: take the word embeddings of all the words in your text and calculate their average. The drawback is that this ignores word order and includes irrelevant words. You can address the latter by weighting the words with tf/idf beforehand and taking a weighted average. A series of other approaches followed. The doc2vec[6] algorithm is, in effect, an extension of word2vec that includes the text as context alongside the individual words. Words and texts are thus encoded in the same embedding space and can be compared. With skip-thought vectors[7], the word2vec algorithm is scaled up by replacing words with sentences.

So we move from: “the meaning of a word is determined by the words around it” to: “the meaning of a sentence is determined by the sentences around it”. This is done using RNNs (GRU) with word embeddings in an encoder-decoder model.

Context sensitivity

Another line of research focused more on helping neural networks use word embeddings to handle context better. The concept of attention began as a technique for focusing on particular parts of a text during a given task. Within the text as a whole, a specific selection of the text (the context) is relevant at certain points. With the arrival of the transformer architecture, earlier architectures based on recurrent and convolutional neural networks were replaced by combinations of different attention mechanisms. This was followed by a series of large-scale language models based on transformers. The best-known models, those in the BERT family, offer pre-trained, context-sensitive word embeddings. From there, it is a relatively small step to text embeddings. Sentence-BERT[8] (better known by the term sentence transformers) is a good implementation geared towards practical applications. At this point, we have reached a significant milestone: transfer learning for NLP is finally a reality!

Wrap-up

Now that we have reached text embeddings, we have found a way to create a powerful representation of a piece of text that captures both the meaning of the words and the relationships between them (the sentence structure). So do we just need to download a pre-trained model, use it to generate embeddings for our text, and get started? Depending on the task, that is true to some extent. At least, in theory.

In part 2, we will see how well these promises hold up in practice…

References

[1] [Mikolov et al. 2013] https://arxiv.org/abs/1301.3781
[2] [Bahdanau et al. 2014] https://arxiv.org/abs/1409.0473
[3] [Vaswani et al. 2017] https://arxiv.org/abs/1706.03762
[4] [Devlin et al. 2018] https://arxiv.org/abs/1810.04805
[5] https://en.wikipedia.org/wiki/Distributional_semantics
[6] [Le, Mikolov 2014] https://arxiv.org/abs/1405.4053
[7] [Kiros et al. 2015] https://arxiv.org/abs/1506.06726
[8] [Reimers, Gurevych 2019] https://arxiv.org/abs/1908.10084