Pre-trained Word embedding using Glove in NLP models

In modern Natural Language Processing (NLP), understanding and processing human language in a machine-readable format is essential. Since machines interpret numbers, it's important to convert textual data into numerical form. One of the most effective and widely used approaches to achieve this is through word embeddings.

What is GloVe?

GloVe (Global Vectors for Word Representation) is an unsupervised learning algorithm designed to generate dense vector representations also known as embeddings. Its primary objective is to capture semantic relationships between words by analyzing their co-occurrence patterns in a large text corpus.

GloVe approach is unique as it effectively combines the strengths of two major approaches:

• Latent Semantic Analysis (LSA) which uses global statistical information.
• Context-based models like Word2Vec which focuses on local word context.

At the core of GloVe lies the idea of mapping each word into a continuous vector space where both the magnitude and direction of the vectors reflect meaningful semantic relationships.

For instance, the relationship captured in a vector equation can be like: king - man + woman = queen.

To achieve this, GloVe constructs a word co-occurrence matrix where each element reflects how often a pair of words appears together within a given context window. It then optimizes the word vectors such that the dot product between any two word vectors approximates the pointwise mutual information (PMI) of the corresponding word pair. This optimization allows GloVe to produce embeddings that effectively encode both syntactic and semantic relationships across the vocabulary.

Glove Data

Glove has pre-defined dense vectors for around every 6 billion words of English literature along with many other general-use characters like commas, braces and semicolons. The algorithm's developers frequently make the pre-trained GloVe embeddings available. It is not necessary to train the model from scratch when using these pre-trained embeddings which can be downloaded and used immediately in a variety of natural language processing (NLP) applications. Users can select a pre-trained GloVe embedding in a dimension like 50-d, 100-d, 200-d or 300-d vectors that best fits their needs in terms of computational resources and task specificity.

Here d stands for dimension. 100d means in this file each word has an equivalent vector of size 100. Glove files are simple text files in the form of a dictionary. Words are key and dense vectors are values of key.

How GloVe works?

The creation of a word co-occurrence matrix is the fundamental component of GloVe. This matrix provides a quantitative measure of the semantic affinity between words by capturing the frequency with which they appear together in a given context. Further, by minimising the difference between the dot product of vectors and the pointwise mutual information of corresponding words, GloVe optimises word vectors. It is able to produce dense vector representations that capture syntactic and semantic relationships.

Creating Vocabulary Dictionary

Vocabulary is the collection of all unique words present in the training dataset. The first dataset is tokenized into words, then all the frequency of each word is counted. Then words are sorted in decreasing order of their frequencies. Words having high frequency are placed at the beginning of the dictionary.

Dataset= {The peon is ringing the bell}
Vocabulary= {'The':2, 'peon':1, 'is':1, 'ringing':1}

Algorithm for word embedding

• Preprocess the text data.
• Created the dictionary.
• Traverse the glove file of a specific dimension and compare each word with all words in the dictionary,
• if a match occurs, copy the equivalent vector from the glove and paste into embedding_matrix at the corresponding index.

Code Implementation

Here we will see step by step implementation

1. Importing Libraries

We will import numpy and tenserflow for it.

from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
import numpy as np

2. Creatong Vocabulary

Here we will create a list of words to build a vocabulary

# Define a sample corpus
texts = ['text', 'the', 'leader', 'prime', 'natural', 'language']

3. Initialize and Fit Tokenizer

• Initializes a tokenizer and fits it on the text corpus.
• Prints the number of unique words and their assigned indices.

# Create and fit the tokenizer
tokenizer = Tokenizer()
tokenizer.fit_on_texts(texts)

# Output the word-index dictionary
print("Number of unique words in dictionary =", len(tokenizer.word_index))
print("Dictionary is =", tokenizer.word_index)

4. Define function to Create embedding matrix

• Defines function that loads GloVe word vectors from file.
• Creates an embedding matrix matching tokenizer word indices with GloVe vectors.

def embedding_for_vocab(filepath, word_index, embedding_dim):
    vocab_size = len(word_index) + 1  # +1 for padding token (index 0)
    embedding_matrix_vocab = np.zeros((vocab_size, embedding_dim))

    with open(filepath, encoding="utf8") as f:
        for line in f:
            word, *vector = line.split()
            if word in word_index:
                idx = word_index[word]
                embedding_matrix_vocab[idx] = np.array(vector, dtype=np.float32)[:embedding_dim]

    return embedding_matrix_vocab

5. Download GloVe File

Here we will Download and Unzip the GloVe file

# Download the GloVe dataset
!wget http://nlp.stanford.edu/data/glove.6B.zip

# Unzip the file
!unzip -q glove.6B.zip

Output:

6. Load GloVe Embeddings and Create Matrix

• Sets embedding dimension and file path.
• Generates the embedding matrix using the vocabulary.

# Set embedding dimension (match this with glove file)
embedding_dim = 50

# Path to GloVe file
glove_path = './glove.6B.50d.txt'

# Generate embedding matrix
embedding_matrix_vocab = embedding_for_vocab(glove_path, tokenizer.word_index, embedding_dim)

7. Access Embedding Vector for a Word

Prints the GloVe vector for the word with index 1 in the tokenizer.

# Print the dense vector for the first word in the tokenizer index
first_word_index = 1  # Tokenizer indexes start from 1
print("Dense vector for word with index 1 =>", embedding_matrix_vocab[first_word_index])

GloVe Embeddings Applications

GloVe embeddings are a popular option for representing words in text data and have found applications in various natural language processing (NLP) tasks. The following are some typical uses for GloVe embeddings:

• Text Classification: GloVe embeddings can be utilised as features in machine learning models for sentiment analysis, topic classification, spam detection and other applications.
• Named Entity Recognition (NER): By capturing the semantic relationships between words and enhancing the model's capacity to identify entities in text, GloVe embeddings can improve the performance of NER systems.
• Machine Translation: GloVe embeddings can be used to represent words in the source and target languages in machine translation systems which aim to translate text from one language to another, thereby enhancing the quality of the translation.
• Question Answering Systems: To help models understand the context between words and produce more accurate answers, GloVe embeddings are used in question-answering tasks.
• Document Similarity and Clustering: GloVe embeddings enable applications in information retrieval and document organization by measuring the semantic similarity between documents or grouping documents according to their content.
• Word Analogy Tasks: In word analogy tasks, GloVe embeddings frequently yield good results. For instance, the generated vector for "king-man + woman" might resemble the "queen" vector, demonstrating the capacity to recognize semantic relationships.
• Semantic Search: In semantic search applications where retrieving documents or passages according to their semantic relevance to a user's query is the aim, GloVe embeddings are helpful.