Creating a Serverless AI Bot

Introduction

In today's competitive job market, efficiently matching candidates to job openings is crucial for both job seekers and employers. At Parser, we are continually exploring alternatives to improve our existing processes to address challenges like this one. With this idea in mind, we've developed a model using BERT (Bidirectional Encoder Representations from Transformers) and Weights & Biases (WandB) to enhance our matching capabilities and provide insightful results. This article explores the architecture of our model, the tools used for training, and the strategies we implemented to improve its accuracy.

Understanding BERT Architecture

BERT is a transformer-based machine learning technique for natural language processing developed by Google. It is designed to understand the context of words in search queries by considering the full context of a word by looking at the words that come before and after it. This bidirectional approach allows BERT to understand the full context of a word based on all of its surroundings.

Unlike traditional language models, BERT learns context from both the left and right sides of a token simultaneously. This ability to comprehend complex language patterns and contextual nuances is crucial for matching job requirements with candidate qualifications.

Understanding Weights & Biases (WandB)

WandB is a Machine Learning Operations (MLOps) platform that helps in tracking experiments, managing, and visualising machine learning models. It offers several key benefits:

Experiment Tracking: WandB logs hyperparameters, metrics, and model performance, creating a detailed history of our iterations.

Visualisation: The platform generates real-time, interactive visualisations of our training progress, helping us quickly identify and address issues like overfitting.

Model Versioning: We use WandB to maintain version control for our models, making it easy to compare performance across iterations or revert to previous states when needed.

By leveraging these features, we maintain a structured, data-driven approach to developing and refining our CV-Job Description matcher, including logging our training progress, such as loss and evaluation metrics for each epoch.

What Do These Values Mean?

Accuracy measures the proportion of correct predictions out of all predictions. For example, if your model correctly classified 500 images out of a total of 1000 images, your accuracy would be 500/1000 = 0.5 for 50%.

Recall, also known as sensitivity or true positive rate, measures the proportion of correct positive predictions over the actual positive instances. For example, if your task is to classify images as either "cat" or "not a cat", and your model correctly identifies 500 cat images but also misidentifies 100 non-cat images as "cat", your recall would be 500/600 = 0.83 or 83%.

Epoch refers to a complete cycle of training on a dataset. The number of epochs you train for determines how many times your model has seen the complete dataset and had the opportunity to adjust its parameters based on the error signals.

Precision measures the proportion of correct positive predictions out of all predicted positives. Continuing with the animal classification example, if your model confidently predicts 600 images as "cat" but only 250 of those are actually cats, your precision would be 250/600 = 0.4167 or 41.67%.

F1 score is a measure generally used in the field of binary classification problems. It is the harmonic mean of the precision and recall of the model. A high F1 score suggests that the model accurately identifies relevant CVs for a given job description (high precision) while capturing a large proportion of the relevant CVs (high recall).

Initial Implementation (Step by Step)

Here is an overview of our process for implementing the model:

1. We started with a dataset containing pairs of CVs and job descriptions, labelled as matches or non-matches.

2. We used a pre-trained BERT model (bert-base-uncased) and added a classification layer on top for our binary matching task.

3. We tokenized the CV and job description pairs using BERT's tokenizer, with a maximum sequence length of 512 tokens.

4. Our initial training process ran for 5 epochs, using AdamW optimiser with a learning rate of 2e-5. AdamW is an optimization algorithm commonly used to train deep-learning models. It incorporates a technique called decoupled weight decay regularization, which, in simpler words, "avoids our model getting too complicated." This means the BERT model can learn to match CVs and job descriptions accurately, without becoming too complex or specific to the training data.

Initial Results

While our baseline model showed promising results, we identified several areas for improvement, particularly regarding some false positives — matches that were not necessarily accurate. As we observed, even though the Job description and the CV were not a potential match, our model was giving 38.6% which might not be accurate.

How Is the Match Calculated?

Our model calculates the match between a CV and a job description by outputting a probability score, from 0 to 1. A score closer to 1 means it thinks it is a great match, while a score closer to 0 means the opposite. We use three main ways to check how well our matchmaker is doing:

Precision: It is like checking how many times it correctly picks the right person for a job.

Recall: It is like making sure we do not miss any good candidates.

By adjusting the threshold for what we consider a "match" (e.g., probability greater than 0.5), we can balance between precision and recall. The threshold is a post-training decision. It is a tool for interpreting the model's output rather than a part of the model's learning process.

Also when training our model, we use something called a loss function. This is like a teacher that tells the AI how wrong it is. During the process of training this "wrongness score" needs to be as small as possible, helping improve its precision and recall.

Problems We Detected

The model sometimes struggled with longer CVs and job descriptions due to the 512-token limit. Initially, we used just a few epochs (only 5). We used a small dataset for training; we definitely needed more data. These observations led us to implement several strategies to improve the model's efficiency and accuracy.

Improving Model Accuracy

Here are some strategies we employed to improve the accuracy of our model:

1. Increased training epochs.

2. Tested with larger pre-trained models (bert-large-uncased).

3. Increased maximum sequence length from 512 to 1024.

4. Early stopping to prevent overfitting.

Most Effective Strategies

While all strategies contributed to improving our model, increasing the number of training epochs and extending the maximum sequence length had the most significant impact on the model's performance.

Extending the maximum sequence length from 512 to 1024 tokens was particularly impactful, allowing the model to process longer CVs and job descriptions more effectively, capturing more relevant information and context. It is important to note that while increasing epochs improved performance, we had to be cautious of overfitting, and implementing early stopping was crucial to prevent it.

Good Data is always the key! Inadequate data preprocessing can introduce noise and bias into the model. Data cleaning, standardising formats, and removing irrelevant information is key to getting good results. Using limited or not properly balanced datasets can hinder the model's ability to generalise.

Conclusion

The BERT architecture provides a strong foundation for understanding the nuanced language of both CVs and job descriptions, while WandB offers invaluable tools for tracking, visualising, and improving our model's performance.

Through systematic fine-tuning and various optimisation strategies — such as increased training epochs, larger pre-trained models, and early stopping mechanisms — we have significantly enhanced the accuracy and reliability of our matcher. This project showcases the potential of AI in streamlining human resource processes.

As we look to the future, it is clear that the intersection of advanced NLP techniques and human resource management holds immense promise for creating more efficient, effective, and equitable hiring processes.