Here’s what we will cover:

1. Introduction
2. Decomposing the accuracy score
3. Balanced Classification Accuracy or Unweighted Average Recall
4. How this is computed
5. Conclusion

Introduction

Given a set of samples \(\mathcal{X}\) with corresponding labels \(\mathcal{Y}\). Let us assume we have a binary classification task with only two labels, \(y_1\) and \(y_2\).

We can train a classifier using binary cross-entropy loss, hinge loss (or whatever loss is fit) to get the best model for our task. Afterwards, we evaluate this model on unseen data to determine how well it generalizes. This is commonly done by calculating a score like accuracy.

Decomposing the accuracy score

Accuracy can be computed from the confusion matrix, which gives a breakdown of prediction scores such as true positive \((tp)\), true negative \((tn)\), false positive \((fp)\) and false negative \((fn)\). The goal of this article is not to explain these terms, since there are other great articles online on the subject of confusion matrix.

This can be shown to correspond the following from the confusion matrix;

where \(p\) is the total no. of positives and \(n\) is the total no. of negatives.

The class label we use as positive or negative is arbitrary here. We can further decompose the equation above into two parts;

Multiplying the first part by \(p\) and second part by \(n\) clearly becomes;

Let’s take the individual elements of the last equation to motivate the UAR.

• \(\dfrac{tp}{p}\) is known as the Recall on the positive class and it is the ratio of the total correctly predicted positives to the total number of positives. It is also known as Sensitivity
• \(\dfrac{tn}{n}\) is known as the Recall on the negative class and it is the ratio of the total correctly predicted negative to the total number of negatives. It is also known as Specificity

The accuracy score can then be written as follows;

\(\dfrac{p}{p + n}\) and \(\dfrac{n}{p + n}\) are weights applied to the sensitivity and specificity and both sum to 1. These weights apply a higher score to the recall with more class samples and lower to the other, so it does not weigh the two classes equally. This makes it generally unfit for understanding how well the model is performing for very skewed datasets.

Balanced Classification Accuracy or Unweighted Average Recall

To mitigate the bias in weighting, we can simply replace the weights with 0.5 or \(\dfrac{1}{\text{no. of classes}}\) for the multiclass scenario.

The balanced accuracy then becomes;

This balanced accuracy is known as the Unweighted Average Recall, the average of the recall on the positive class and recall on the negative class. There is a correlation between the accuracy and UAR but the UAR gives the correct expectation on class predictions.

How this is computed

This can be computed from a confusion matrix. Below is a function that computes specificity, sensitivity, accuracy and uar in python given the confusion matrix.

def compute_metrics(confusion_matrix):
    """Calculates specificity, sensitivity, accuracy and uar from confusion matrix
    Confusion matrix of form [[tp, fp]
                              [fn, tn]]
    args:
      confusion_matrix: 2 by 2 nd-array
      output: tuple of float (specificity, sensitivity, accuracy, uar)
    """
    cm = confusion_matrix
    tp, tn = cm[0, 0], cm[1, 1]
    fn, fp = cm[1, 0], cm[0, 1]
    
    sensitivity = tp / (tp + fn)
    specificity = tn / (fp + tn)
    
    uar = (specificity + sensitivity)/2.0
    
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    
    metrics_dict = dict(sensitivity=sensitivity, specificity=specificity, 
                       accuracy=accuracy, uar=uar)
    return metrics_dict

Let us look at some cases of how accuracy and UAR differ in their metrics

Case 1: Balanced class Here, we have 50 samples from the positive class and 50 samples from the negative class. The model predicts 45 of 50 samples correctly for the positive class and 39 of 50 samples correctly for the negative class.

>> import numpy as np
>> cm = np.array([[45, 11],
                  [5, 39]])
>> print(compute_metrics(cm))

# Note that values have been rounded up to 2 decimal places
output: {'sensitivity': 0.90, 'specificity': 0.78, 'accuracy': 0.84, 'uar': 0.84}

Since the classes are balanced, the UAR and accuracy both give similar information

Case 2: Imbalanced class In this case, we have 20 samples from the positive class and 80 samples from negative class. The model predicts 4 of 20 samples correctly for the positive class and 75 of 80 samples correcly for the negative class.

>> cm = np.array([[4, 5],
                  [16, 75]])
>> print(compute_metrics(cm))

output: {'sensitivity': 0.20, 'specificity': 0.94, 'accuracy': 0.79, 'uar': 0.57}

Here, the accuracy is over optimistic and indicating that we are doing well on about \(79 \%\) of the samples. This is because of the size of samples of the negative class. However, the UAR gives a truer result showing that we still have work to do. Looking at the specificity and sensitivity closely, they show where the problem lies. The model is doing poorly on the positive class (as indicated by the sensitivity, \(20 \%\))

You can play with different formulations of the confusion matrix to better understand how class imbalance affects the scores.

I first saw the UAR score on Charles Onu’s Master thesis in the section “Balanced Classification Accuracy and Misclassification Loss”. This is a decomposition of the equation.

Update Sep 11, 21: Fixed in a bug in computing the UAR in an earlier version of this article, and added clearer examples

Conclusion

Accuracy is the most used metric for evaluating machine learning classification tasks. In this article, we decomposed the accuracy into individual ratios composed of the sensitivity and specificity weighted by a class ratio. Then, we fixed the bias in the accuracy by giving equal weights to both scores. This led to the Unweighted Average Recall.

Here’s what we’ll cover:

1. Introduction
2. Getting Started
   1. Sampling Frequency
   2. Audio Format and Encoding
   3. Length of Recordings
   4. Labels
   5. Number of Speakers
3. Annotator Characteristics
   1. Gender
   2. Age Groups
   3. Accents
   4. Other Metadata
4. Other important details to note
   1. Size of the data
   2. Noise and Artefacts
   3. Unlabelled audio
   4. Data Split
   5. Text Preprocessing
   6. Data Augmentation
5. Recording details
   1. Recording tools
   2. Source of Text
6. Conclusion

Introduction

A speech corpus is a database containing audio recordings and the corresponding label. The label depends on the task. For ASR tasks, the label is the text, for TTS, the label is the audio itself, while the input is text. For speaker classification, the label will be the speaker id. Therefore, the label and data depends on the particular task. For ASR, the audio samples and text require that they correspond to the same entity. There is a large amount of recorded audio which can be sourced from podcasts, streaming platforms like Youtube, and even talk shows (if permission is granted to you to use them). While the data is available, there are major problems that should be considered when using them for speech tasks. These include;

• they may contain artefacts/noise which are not important to the task and machine learning models may find it hard separating those artefacts from the actual signal,
• recordings may/may not have corresponding transcription which may be needed,
• multiple speakers talking simultaneously in the recordings,
• audio recordings may have to be split into short durations, with alignment performed with the corresponding text,
• Podcasts may have music playing at the background, multiple simultaneous speakers etc.

With these problems in mind, you may need to determine if the audio is well suited for your task. In this article, we will focus on read speech for creating our own corpus, instead of relying on pre-recorded audio as explained above.

Getting Started

Since 2015, we have seen advances in using deep neural networks for ASR tasks [Papers with code], surpassing previous works using Hidden Markov Models (HMM) with Gaussian Mixtures (GMM) or their ensembles on various speech related task. Also, the introduction of the Connectionist Temporal Classification [A Graves, 2006] Loss to alignment has given a major boost to machine learning tasks like speech where alignment between the audio and text is cumbersome. Using the CTC Loss enables the model to maximize the objective over all possible and correct alignments between the audio and text. With these advancements, creating speech data has become significantly easier than previously imagined, with corpus requiring no alignment between the text and the read speech.

For a detailed introduction to CTC Loss, checkout my blog post on [Breaking Down the CTC Loss]

Sampling Frequency

44.1kHz is the most common sampling frequency used to produce most digital audio. This ensures that the audio can be reconstructed for frequency below 22.05kHz, since it covers all frequencies that can be heard by a human. ASR experiments do not require that high sampling rate; more common frequencies are 8kHz and 16kHz, with sampling frequency of 16kHz becoming the de facto for speech recognition both in production and research, as there is no significant improvement in using a sampling frequency higher than that, although a lower sampling frequency may reduce accuracy Also, increasing the sampling frequency beyond that just increases the overhead during preprocessing and training, with some training procedures taking twice as much time, without any improvements.

On the converse, modern production quality TTS often use 22.05kHz, 32kHz, 44.1kHz, or 48 kHz sampling rate, as 16kHz is too low to achieve high quality TTS [LibriTTS - Heiga Zen et al.] though many research works still use 16kHz. For TTS, the acoustic model needs to learn the fine-grained acoustic characteristic of the audio to be able to reproduce the same form of signal from text.

During the signal preprocessing, the audio can be downsampled to its required sampling rate.

Audio Format and Encoding

Audio format and encoding are two different things. Most popular file format used for speech-to-text experiments is the “.WAV” format. Since ‘wav’ is just a file format, it will have to be encoded during recording using one of the different encoding formats available such as Linear PCM Encoding.You do not need to worry about the details since this would be taken care of for you during your setup. Encodings can be lossy or lossless, taking up different file sizes and quality.

If your read speech corpus is saved in the MP3 file format, you may have to convert them to “.wav” during the preprocessing stage.

For a brief overview of encodings and audio formats, checkout the article [Introduction to audio encoding - GCP ]

Length of Recordings

For ASR tasks, the length of the audio samples should be smaller than about 30 seconds. Typically, for the ASR tasks I have worked on, the average length of the recordings range between 10 seconds and 15 seconds. The shorter the duration the better for the model, especially for models that use recurrent networks (RNN) for decoding. There is also the problem of long-range temporal dependency that needs to be addressed with long duration. It is also advisable to ensure the variance between the audio duration is small likewise.

In the case of TTS, recordings should be splitted on sentence stops instead of silence intervals, to learn long-term characteristics of speech such as the sentence-level prosody for given a text [LibriTTS - Heiga Zen et al.]

Other speech classification tasks such as gender identification and speaker identification don’t require long duration of samples. Typical duration of audio is 2 to 4 secs, which is enough to learn the signal characteristic for each class.

Labels

As previously mentioned, the task determines the label for the audio. For example, Automatic Speech Translation (AST) requires text in the target language which may differ from the source language/language of the audio.

It is good practice to have a balanced sample-to-label ratio, with every label well represented. For instance, speaker identification tasks will require that the number of samples assigned to each speaker should be balanced. If one speaker is over represented, the acoustic model may learn unimportant characteristics of the speaker neglecting important signals. There are sampling methods though to forestall this situation, and some loss functions can be used to cater for the imbalance.

In the case of ASR tasks, the text should contain all alphabets of the target language in considerable proportion. Even for phoneme recognition tasks, all phones should be well represented in the labels. An example of a good phoneme recognition corpus is the TIMIT Acoustic-Phonetic Continuous Speech Corpus

Number of Speakers

The more the number of speakers, the better for the acoustic model, as it will also have to hear variations of speakers in the wild when deployed. It also ensures that we have a significant sample of speakers in the validation and test set.

Annotator Characteristics

There are some characteristics of the speaker which are desirable for a balanced and unbiased data set. Some of these will be discussed here. The final task sometimes will determine where to focus on these characteristics. For example, if we can determine beforehand the target age group, we can easily focus on getting more data for them or even optimze for better predictions.

Gender

The two gender groups (Male and Female) should be well represented in the data as the prosodic characteristics of males and females differ. It is ideal to have a 50-50 split in gender or close to it whenever possible.

Age Groups

For general ASR tasks, all age groups should be represented, but may be difficult to accomplish for small ASR projects. Children under the age of 9 years, speak in a different way from adults. Their vocal characteristics begin to change at adolescence. All these should be put into consideration.

For audio recordings, not particular to humans, age may not be a requirement. For instance, in recordings involving animal sounds.

Accents

Most cultures have moved across shores to other countries taking with them their language and tongue. The accents of those nations can affected how the language is spoken or communicated. For example, the Nigerian English differs significantly in pronunciations from the Indian English or the American English. Some production quality ASR learn different models for the different accent, but this is expensive. As humans, we easily adapt to accents after learning from a few examples in our environment.

Another idea may be to feed in an accent identifier into the acoustic model during training to adapt to different speaker accents.

Other Metadata

Some metadata pertaining to the speaker should be collected during the recording. Speaker ID, age, country, text domain, Signal-to-Noise Ratio (SNR), time of recording etc. can be collected for each speaker. It is good practice to inform the speakers/annotators of the metadata that is been collected from them and how it may possibly be used.

Also, depending on the task, these metadata can be used to properly sample from the corpus to avoid the imbalance we discussed earlier.

Other important details to note

Size of the data

As with all machine learning tasks involving deep neural networks, more data is better. The task can be split across many speakers to have a large sample size. Tasks like ASR and TTS require a lot of audio samples for good performance. The best models in English ASR are trained on about 60 thousand hours of speech [Jacob Kahn et al - LibriLight]. That is equivalent to about 7 years of speech. This particular data was created from the LibriVox database of audio books.

In low resource audio settings, audio samples of this size may not be possible. We may then have to result to domain adaptation or training a self-supervised acoustic model from raw speech, if audio is available without transcriptions. There have been significant advances in unsupervised and self-supervised speech representation learnings enabling SOTA performance with limited data [Alexei Baevski et al. - Wav2Vec 2.0]

Noise and Artefacts

Noise in all forms are a bane to good acoustic model performance, as they significantly affect the learning process. Significant research has been done to learn from noisy audio or noisy texts but it is still preferable to have clean text and well recorded audio. We need to ensure that the recording environment is devoid of background noise, music, animal sounds and even noise from electric devices such as Air conditioners.

Modern microphones and devices have noise filtering or noise cancelling mechanisms, giving better recording performance. It is a good idea to check if the recording device has this feature turned on. If affordable, recording studios can be created for the task.

On the converse, training with noisy audio can make the acoustic model robust to noise. The downstream task should determine the amount of noise permissible in the audio.

Unlabelled audio

It may be difficult and more cumbersome to get large amounts of labelled data. Recent research has shown that clean unlabelled audio can also be useful for pretraining acoustic models. Unlabelled data may be easier to collect in many cases and that can be put to use in a self-supervised way. These methods have been shown to be competitive with their labelled counterparts for downstream tasks such as ASR. Two popular acoustic models for representation learning are [Aaron van den Oord et al - Contrastive Predictive Coding] and [A Baevski, 2020 - Wave2Vec2.0] are two popular unsupervised methods of learning speech representations for downstream tasks.

Data Split

The examples should be splitted across the speakers. Speaker identity in the training set should not be represented in the validation and test set, and vice versa. This ensures that we can measure performance of the model on speakers and audio samples it has never seen during training.

The 80/10/10 rule of train/validation/test splitting can be applied when more data is available e.g 100 hrs. For low resources settings, this might be inapplicable, and therefore revert to ensuring that the test set is a good sample to test generalization. Given a total audio duration of 5 hours for instance, the audio can be split into 3/1/1 hours for train, validation and test respectively.

Text Preprocessing

Typically for ASR tasks, the text may need to be cleaned and preprocessed to eliminate ambiguity in words and spellings. Digits can be spelled out in words where required, depending on the task. Also,in low resource settings, it is typical to convert all characters to their non-accented versions, reducing the character vocabulary size.

When preprocessing for ASR, punctuations are eliminated from the corpus as they are not typically read out during recitation, but just denoted with stops or gaps in the recording sequence. Words joined together by hyphens can be separated into two words. The apostrophe (') character is left in the corpus for languages such as French, which use them for conjoining words.

In a similar vein to checking length of audio duration, the length of recorded text should be also be short when possible, to prevent errors in recordings. Long sentences can be split on words or stops. It is typical to use between 10 and 30 words for a single audio sample. The length should ensure that recordings do not exceed the 30 seconds mark as discussed above. All these help to prevent unnecessary gaps and stops or loss of attention while recording.

For ASR, a different text corpus from that used for recording will be needed to create an Language Model (LM). Language models are usually integrated into the decoding process of Speech-to-text systems for better performance. The performance gap decreases with the amount of training data though.

A language model with lower perplexity gives better decoding results. Transformers are becoming the de facto for modelling sequences such as text, and should be considered as the language model of choice if enough text is available to train it.

Data Augmentation

Data Augmentation is an important technique in generating more data than available. For low resource settings, it is essential to augment the data with other versions of the audio recording. Augmentation can make the acoustic model less susceptible to overfitting.

Augmentation can be done on the raw speech or on the audio spectrogram. Some interesting augmentation methods like SpecAugment can be applied on the fly during preprocessing. It is a good practice to experiment with different augmentation strategies.

For a brief introduction to Audio augmentation, checkout the blog post by Edward Ma, Data Augmentation for Audio

Recording details

This section talks about equipments and setup tools that may be used for audio annotation

Recording tools

• Mobile phones: Modern mobile devices have very good microphones for recording sound. They can be paired with a recording application like ligAikuma for annotation. Ensure you have a large storage space on the device to save recordings. One advantage of using mobile devices is that multiple annotators can record simultaneously and/or at their convenience, inasmuch as noise is being eliminated.

  The ligAikuma app is an app I recommend for recording, elicitation and translation. It was used in collecting my previous speech project on Yoruba Language.

• Computer and Microphone: More sophisticated recording desktop applications are available for audio recordings. They can be paired to ensure noise-free recordings with great quality. The recording sampling rate, audio codec and audio format can be varied to give the desired output.

• Online recording platforms: There are also online recording platforms that do not even require any setup for recording. You can provide text and start recording almost immediately for free. Examples of such are Common Voice platform and Speech Annotation Toolkits for Low Resource Languages. Ensure you read their SLA to determine how your data might be used by the platforms in the future.

In general, it is a good starting point to check online repositories like Open SLR and Common Voice for speech samples recorded by others. It gives a perspective on what to expect and how annotations should be done.

Source of Text

Text is freely and openly available for high-resource languages like English, Mandarin, French etc. Some other languages of the world do not have large amount of text available for annotation. More often, texts are sourced from textbooks, news and media, religious publications e.g Ž Agić et al - JW300 and the Bible. Wikipedia is also a good source of text for many languages and should be the first place to go, for clean text.

The acoustic model may be biased towards text from the specific domain it was trained on. So, care should be taken when using the acoustic model.

The most appropriate text is that which mimics the domain where the model will be used.

Conclusion

This article explained in detail the various aspects of data collection that needs to be considered when creating a speech corpus, specifically for ASR.

African Masters in Machine Intelligence (AMMI) is a master’s programme offered by the African Institute for Mathematical Sciences, sponsored by Facebook and Google, with campuses currently in Rwanda and Ghana.

Check out a subsection:

1. Introduction
2. Technical Advice
   1. DO NOT QUIT! I repeat DO NOT QUIT!!
   2. Learn the foundations
   3. Learn by intuition
   4. Learn to code properly
   5. Join a Discussion Group
   6. Find an interest early (Research/Engineering/Entrepreneurship)
3. Non Technical Advice
   1. Make lasting friends
   2. Utilize your network of AMMI lecturers
   3. Apply for ML Jobs, Internships and PhD positions
   4. Try, fail, break things, then learn the right way
4. Conclusion

Introduction

Congratulations! You have been admitted into the AMMI programme. This is just the beginning of the hurdle ahead and may be full of uncertainties for you as it was for everyone of us. You just have to take a leap of faith and begin the journey. You may wonder how you will be able to cope with the programme; what the courses will look like; how you will adjust to the new environment; how you will get better at programming. Well, everyone of us had some or all of those fears, but we were able to overcome them with support.

We will dive deep into these advice in a second as I believe you are eager to read them. You are free to bookmark or revisit this page during your days at AMMI for inspiration on what to do next. Let’s dive in.

Technical Advice

DO NOT QUIT! I repeat DO NOT QUIT!!

Well, this might sound funny but it is still the best advice to give. A lot of us faced trying times with deadlines of assignments and presentations approaching, struggling with the basics and needing time to absolve them. All I can say is hang on there friend and don’t contemplate dropping the baton. You will be faced with this decision at some point too and I hope you will decide not to quit on yourself, instead put in the hardwork and reap the rewards.

Also, remember you submitted a statement of purpose where you wrote beautiful things about your passion for AI, solving societal problems with machine learning, becoming a world-renowned researcher etc. Well, now is the time to show that passion and let your goals drive you to keep working hard and pushing forward.

Learn the foundations

Every field including machine intelligence requires a solid foundation in the mathematics and science used to build up concepts and ideas. For instance, the field of machine intelligence requires a solid foundation in Linear Algebra, Probability, Statistics, Calculus, Optimization, Algorithms and Data structures. Some of this knowledge, you may have been at school, others you may have forgotten or you had a shaky intuition of the course. In any case, it is a great idea to brush up on those foundations and get ready for the real deal. You will have introductory courses to help solidify the foundations but it is still a good idea to study on your own and internalize these. Below are some online courses I watched to brush up on these foundations.

1. Linear Algebra by Gilbert Strang, MIT or if you want a recent course, you may prefer Matrix Methods in Data Analysis, Signal Processing, and Machine Learning by Gilbert Strang, MIT. These will help with your LA basics
2. Engineering Probability by Rich Radke for your statistics and probability basics

If you prefer written text, Mathematics for Machine Learning co-authored by Marc Peter Deisenroth, is your go to text. There is also a coursera course with the same title which you can review.

Learn by intuition

Machine learning is an interesting field if you let the concepts sink in to become reality in you. It can also seem unbreakable if you look from the surface. In your journey, you will surely encounter many mathematical formulations and proofs. My advice is; get an intuitive understanding of what you learn, understand how it is applied in a larger context, then dive deeper on the proofs. Try to rederive the proofs yourself, or implement the algorithm to get similar results. This I believe is a great way to push forward your learning.

It is also a good idea to explain these concepts to other colleagues. This will aid in solidifying your understanding and birth deeper questions from colleagues which you will have to answer.

Learn to code properly

The most important skill required to excel in your AI journey is programming. You will be required to convert mathematics to codes. No running away from it my friend. It is in your best interest to have a good background in Python. Other major libraries you will require are Numpy (for vectorized computations), Pytorch (for running code no GPU) and some others which you can easily acquire competence in if you already know Python. If you already know a programming language such as Javascript or C++, it will be easy to catch up with learning Python.

Apart from just programming, you need to develop algorithmic thinking. This will be useful in implementing ideas and turning mathematics into reality. As a hint, be rest assured that you will be required to code ML algorithms from scratch.

There are lots of online platforms such as Leetcode or Hackerrank where you can solve problems to improve your skills. Intro to Data Structures and Algorithms is a free course on Udacity to refresh your DS skills if you need one. In summary, look for a platform you are comfortable with and solve problems consistently. You never know how good you can become until you start and put in the work.

Join a Discussion Group

Small sized discussion groups are a great way to share ideas, help one another and collaborate. You can set up a permanent discussion group with your friends to work on assignments and review lectures. Members of your discussion group can also serve as your go-to when you need a clearer explanation of concepts you have read about.

It is a great idea to form groups with members having strength in different areas of ML. For instance, a group can consist of a mathematics major, computer scientist, a statistician etc. It balances the group and ensures everyone can contribute to each others’ development. Join or form a group

Furthermore, there is a large space of research in AI such as Natural Language Processing, Computer Vision, Optimization, Multimodal Learning, Self-supervised learning etc. To form a deeper knowledge of these fields, you can form groups where you read and summarize papers, implement ideas and discuss further research directions concentrated on these specific fields.

Find an interest early (Research/Engineering/Entrepreneurship)

With the intense curriculum and activities of the AMMI programme, it will be a great idea to know what you are set out to achieve during and after the AMMI programme. Even though many interesting activities and ideas will be brought to your doorstep, you have to prioritize and know what you really want to learn well and internalize. For instance as a clue, a researcher may be more interested in learning the mathematical foundations of machine learning, while an Engineer will be more interested in converting the algorithms into production level code. On the other hand, an entrepreneur may be willing to learn all these but also look out for areas that can be used to improve or create business solutions.

Also, as an entrepreneur, you may want to start a venture with colleagues where you create products using ML. ML Engineers can also form groups where they build end-to-end solutions together (even for fun) to strengthen their skills. Researchers will more likely want to read more papers, follow research trends and attend conferences.

It is also a good idea to discuss with your tutors/Teaching Assistants on these and map out a path to achieve the best in your career.

Non Technical Advice

Make lasting friends

AIMS is a place where you will meet people from different African countries with diverse cultures. You will be required to live under the same roof, eat the same meal, engage in team-bonding activities and trust one another. Use this opportunity to make lasting friends, learn about others’ cultures, food, dance and language. I can tell you for a fact that Africa and African culture is beautiful, when you discover them.

Also, your colleagues will be your life-long partners and friends. They will be your support during and after the programme whenever and wherever you need them. Be their friend and enjoy every moment of your partnership.

Utilize your network of AMMI lecturers

Okay, I don’t know if I have to say this again but, hey friend! You are lucky to be here. AIMS instructors and lecturers are world-standard. You have a lot to learn from them. Check out the AMMI website for a peek of some of your instructors. Well, it is up to you t0 utilize this opportunity, listen to instructions, do your labs and show them you are committed.

As most of the lecturers will be on campus with you, you have the freedom to approach them for discussions, ask questions on the course and discuss possible research directions with them. Your lecturers are always excited to interact with students and provide insight from their wealth of experience.

Apply for ML Jobs, Internships and PhD positions

As your course progresses, you will get opportunities for internships, PhD positions and jobs. The opportunities will come your way but you will be required to prove yourself. Nothing will be served on a platter, you will have to put in the work and show that you are worth it.

It is also a good idea to determine the path you will like to take early so you can focus on the specific opportunities you want whether it is a research internship, going for a PhD, looking out for residency programmes etc. It is an open call. Rest assured that the programme gives you the platform to seek out these opportunities and get them.

Therefore, prepare a great CV and tailor your CV to your interests.

Try, fail, break things, then learn the right way

The best way to learn is to try things out yourself. Be it coding up a machine learning algorithm or proving that a kernel is an inner product (: , the best way to learn machine learning is to try, run into errors, debug, retry and fail many times. That’s the only way to learn what is required for your success. Do, do not be afraid to fail.

Also, do not be afraid of rejections. Rejection mails may come from your applications or you may even fail technical interviews. I failed a lot too. Take it as an opportunity to get to work and get better at acing those interviews.

Conclusion

Thank you for taking out time to read these bullets of advice gathered from my colleagues. I hope they will be of help to you on your AMMI journey. I wish you the very best in your AMMI journey.

Special thanks to all my AMMI colleagues who contributed to this article by providing their advice in words and in deeds.

We will not be discussing the decoding methods used during inference such as beam search with ctc or prefix search. For an introductory look at CTC, you can read Sequence Modeling With CTC by Awni Hannun.

Here’s what we will cover:

1. Introduction
   1. The CTC Model
2. Getting into ctc details
   1. Setting up constrains on the alignment
   2. Composing the graph
   3. Scoring the paths
3. Forward-Backward Algorithm
   1. Forward Algorithm for computing \(\alpha_{(s, t)}\)
   2. Backward algorithm for computing \(\beta_{(s, t)}\)
   3. Computing the probabilities efficiently
   4. CTC Loss calculation for each timestep
   5. Note
4. Conclusion

Introduction

Let’s look an automatic speech recognition task where we have to predict the words spoken from the audio data.

Looking at the speech segment, how can we align the words to where they are spoken in the speech segment? Even if it is possible to manually do it for this task, it is not feasible for a large corpus of audio data.

With the CTC alignment, we do not require alignment between input and output sequence (in terms of location). CTC tries all possible alignments of the ground truth to the prediction.

The CTC Model

Let’s get concrete with what we have been talking about by designing a task and apply the CTC loss.

In the model above, we convert the raw audio signal into its spectrum or apply melfiterbanks (this is optional and can be performed by a CNN layer), which is then passed through a convolution neural network (CNN). CNNs enable us extract features, by looking at a window of the data while performing strided convolutions along the feature dimension of the audio.

The features are then passed through a Recurrent Neural Network (RNN) for decoding. At the decoding stage, if we perform a max decoding at each timestep, we will get tokens of a much longer length than the input, which naturally implies that redundant tokens will be decoded to fill up some of the timesteps. How do we contract the decoded output to represent our predictions? How should we deal with silences in the audio? How should we indicate repetitions of tokens as in “d-oo-r”?

Well, Instead of decoding characters, we can decode phonemes, subwords, or even words depending on the task. Let’s consider the instance of character decoding for this article.

We can solve these problems by explicitly introducing a blank token into our vocabulary to cater for these dynamics. We further include a separator token to indicate spaces between each word.

Thus, “a door” split into [“ε”, “a”, “_”, “d”, “o”, “o”, “r”] tokens is then transformed into [“ε”, “a”, “ε”, “_”, “ε”, “d”, “ε”, “o”, “ε”, “o”, “ε”, “r”, “ε”] where the blank token is included. With this, we know that we can only have repeating tokens only if they are separated by a blank token, “ε” e.g. “d”, “o”, “ε”, “o”, “ε”, “r” is allowed and not “d”, “o”, “o”, “ε”, “r”. The latter contracts into “dor”.

In general, given an initial sequence of length \(M\), the length of the expanded sequence is \(2M + 1\)

Getting into ctc details

At the output of the RNN, we get a vector, which has the length of vocabulary, for each time step of RNN computation. The softmax function is applied to it to get a vector of probabilities. The number of output labels cannot be more than the number of features from the CNN, so the features has to be estimated accordingly (by taking the maximum length of sequence in vocabulary or some other heuristic).

We will consider a smaller label “door” which should be enough to explain the entire concept succinctly. Let’s generate our vocabulary as the standard lowercase alphabets, including our special tokens.

[“ε”:0, “_“:1, “a”: 2, “b”:3, … ,”z”:28]

We denote the total number of timesteps by \(T\), length of the expanded target output by \(S\), and length of label by \(M\). So, \(S = 2M + 1\), e.g. for “door”, \(S = 2 \cdot 4 + 1\)

Given these vectors of probability distributions, how do we learn the alignments of the probable predictions? We need a structured way to traverse from the first softmax distribution to the last to represent the word.

Setting up constrains on the alignment

In principle, we exclude all rows that do not include tokens from the target sequence and then rearrange the tokens to form the output sequence. This is done during training only. At inference, a beam search can be performed on the distribution. So, we copy the required output for the target into a secondary reduced structure and decode on the reduced structure assuring us that only appropriate tokens will be used for selected for computing loss and gradients.

If a token occur multiple times in the label, we repeat the rows for similar tokens in their appropriate location. This becomes our probability matrix, \(y_{(s, t)}\)

Composing the graph

Now that we have our full grid, we can begin traversing the grid from top-left to bottom right in such a way that; (a) the first character in the decoding must be a blank token, ‘ε’ or the first sequence token ‘d’ (b) the last token is either a blank token or the last sequence token ‘r’ (c) the rest of the sequence follows a sequence path that monotonically travels down from the top-left to bottom-right.

To guarantee that the sequence is an expansion of the target sequence, we can only traverse the grid through these valid paths from top-left to bottom-right. I have attempted to trave all paths in the grid, and you can do it as an exercise too. Two valid paths where both collapse into “door” are shown below;

It is easy to trace these paths if we consider the following traversal rules;

• The sequence can start with a blank token or the first character token and end with a blank token or the last character token. So we have to consider both paths.
• Skips are permitted across a blank token only if the tokens on either side of the blank token are different because a blank is required to distinguish repetition of a token but not required between distinct tokens

Scoring the paths

The score of a path is the product of probabilities of all nodes along the path. For the two paths considered in the examples above.

We are required to trace out all the possible paths that contract into “door” and there are an exponential number of such valid paths as can be seen from the graph. The complexity is of the order \(\mathcal{O}(\lvert V \rvert^T)\) where \(\lvert V \rvert\) is the length of vocabulary.

Can we find a dynamic programming algorithm for solving this problem? Well, the viterbi algorithm can generate the most likely path, and does not guarantee we get the most likely sequence of labels. It finds the best path to a node by extending the best path to one of its parent nodes. Any other path would necessarily have a lower probability. But, the viterbi algorithm commits to a path or initial alignment early (without exploration) which can lead to suboptimal results.

Forward-Backward Algorithm

Instead of only selecting the most likely alignment, we find the expectation over all possible alignments during training. This allows us to also exploit the existence of subpaths in the graph.

To compute this effectively, we need a forward variable \(\alpha_{(s, t)}\) and backward variable \(\beta_{(s, t)}\) where \(s\) is the index of the token considered. The forward variable computes the total probability of a sequence \(\operatorname{seq}[1:s]\) up to a particular timestep \(t\). The backward variable calculates the total probability of remaining sequence from token \(\operatorname{seq}(s)\) to token \(\operatorname{seq}(S)\), \(\operatorname{seq}[s:S]\) at timestep \(t\).

Forward Algorithm for computing \(\alpha_{(s, t)}\)

First, let’s create a matrix of zeros of same shape as our probability matrix, \(y_{(s, t)}\) to store our \(\alpha\) values. The forward algorithm is given by;

Initialize:

alpha_mat = zeros_like(y_mat)

\(\alpha_{(0, 0)} = y_{(0, 0)}\), \(\alpha_{(1, 0)} = y_{(1, 0)}\)

\(\alpha_{(s, 0)} = 0\) for \(s > 1\)

Iterate forward:

• for \(t = 1\) to \(T-1\):
  • for \(s = 0\) to \(S\):
    • \(\alpha_{(s, t)} = (\alpha_{(s, t-1)} + \alpha_{(s-1, t-1)})y_{(s, t)}\) if \(\operatorname{seq}(s) = \text{“ε”}\) or \(\operatorname{seq}(s) = \operatorname{seq}(s-2)\)
    • \(\alpha_{(s, t)} = (\alpha_{(s, t-1)} + \alpha_{(s-1, t-1)} + \alpha_{(s-2, t-2)})y_{(s, t)}\) otherwise

Note that \(\alpha_{(s, t)} = 0\) for all \(s < S-2(T-t) - 1\) which corresponds to the unconnected boxes in the top-right. These variables correspond to states for which there are not enough time-steps left to complete the sequence.

\(\operatorname{seq}(s)\) - token at index \(s\) e.g. \(\operatorname{seq}(s=1)=\text{“d”}\)

Backward algorithm for computing \(\beta_{(s, t)}\)

Let’s also create a matrix of zeros of same shape as our probability matrix, \(y_{(s, t)}\) to store our \(\beta\) values.

Initialize:

• \(\beta_{(S-1, T-1)} = 1\), \(\beta_{(S-2, T-1)} = 1\),
• \(\beta_{(s, T-1)} = 0\) for \(s < S-2\)

Iterate backward:

• for \(t = T-2\) to \(0\):
  • for \(s = S-1\) to \(0\):
    • \(\beta_{(s, t)} = \beta_{(s, t+1)}y_{(s, t)} + \beta_{(s+1, t+1)}y_{(s+1, t)}\) if \(\operatorname{seq}(s) = \text{“ε”}\) or \(\operatorname{seq}(s) = \operatorname{seq}(s+2)\)
    • \(\beta_{(s, t)} = \beta_{(s, t+1)}y_{(s, t)} + \beta_{(s+1, t+1)}y_{(s+1, t)} + \beta_{(s+2, t+2)}y_{(s+2, t)}\) otherwise

Similarly, \(\beta_{(s, t)} = 0\) for all \(s > 2t\) which corresponds to the unconnected boxes in the bottom-left.

Computing the probabilities efficiently

From the computations, observe that we are constantly multiplying values less than 1. This can lead to underflow especially for longer sequences. We can improve these computations by performing the computations in the logarithm space. Products become sums, divisions become subtraction. For instance;

becomes

CTC Loss calculation for each timestep

Now that we have the \(\alpha\) and \(\beta\) probabilities(or log probabilities), we will compute the joint probability of the sequence at every timestep. This we will call \(\gamma_{s,t}\).

Afterwards, we compute the posterior probabilities of the sequence at every timestep by summing along columns. This is the total probability of all paths going through a token \(\operatorname{seq}(s)\) at timestep t.

Total loss of the model is then;

Derivatives can then be calculated for back propagation using Autograd. Modern deep leaning libraries such as Pytorch, and TensorFlow have this feature.

Note

1. The CTC loss algorithm can be applied to both convolutional and recurrent networks. For recurrent networks, it is possible to compute the loss at each timestep in the path or make use of the final loss, depending on the use case.
2. Some forms of the loss use only the forward algorithm in its computation i.e \(\alpha_{s, t}\). I was only able to reproduce the Pytorch CTC loss when I used the forward algorithm in my loss computation, and ignoring the backward algorithm. However, the algorithm explained in this blog post is the one proposed in the seminal paper by Graves et al..

Conclusion

In this article, we explained the connectionist temporal classification loss and how it can be applied in many-to-many input/output classification tasks without alignments. Then, we showed the computations for the forward and backward algorithm used for the training the model.

To understand how the self-attention mechanism is applied in Transformers, it might be intuitive from a mathematical perspective to build-up step-by-step from what is known, i.e. Recurrent Neural Networks such as LSTMs or GRUs to a self-attention network such as Transformers. Blog posts such as Jalammar, The Annotated Transformer, Vandergoten have attacked the explanation of Transformers from different perspectives but I believe this article will give another perspective and help engineers and researchers understand Self-Attention better, as I did.

For a beautiful explanation of everything Attention, check out Lilianweng post on Attention

Here’s what we will cover:

1. Introduction
2. Gated Recurrent Neural Networks
   1. What are these shortcut connections
3. Gated Recurrent Units to Causal Attention
   1. Let’s free the dependent weights
   2. Let’s free up candidate vectors
   3. Let’s separate Keys and Values
   4. Let’s have multiple attention heads
4. Gated Recurrent Units to Non-Causal Attention
   1. Let’s look at the entire input sequence
   2. Let’s give a sense of position to the attention mechanism
   3. Let’s use Non-Linear Attention
5. Full Self-Attention Layer
6. Conclusion

This article is based on a lecture given by Kyunghyun Cho at AMMI, 2020. Special thanks to him and his team for a beautiful NLP course.

Introduction

Recurrent neural networks with gates such as Long-Short Term Memory (LSTM) and Gated Recurrent Units (GRU) have long been used for sequence modelling with the advantage that they help to significantly solve the vanishing problem and long-term dependency problems popularly found in Vanilla RNNs. Attention mechanisms have also been used together with these gated recurrent networks to improve their modelling capacity. However, recurrent computations still persist.

Given a sequence of input tokens, \((x_1, x_2, \dots, x_T)\), where T is the total number of tokens. At time step \(t\), we can calculate a hidden vector \(h_t\) which is a representation of information gotten from tokens from time step \(1\) to \(t\).

Gated Recurrent Neural Networks

A key idea behind LSTM and GRU is the additive update of the hidden vector, \(h_t \in \mathbb{R}^d\) with dimension, d

where \(\tilde{h}_t\) is the candidate context vector for current time-step, \(t\) which is gated and added to the previous context vector in a linear way. This allows information to be propagated from previous time-steps to the current time step, if necessary. As observed, the update gate, \(u_t \in \mathbb{R}^d\)

• With \(u_t \approx 0\) (zero vector), \(h_t = \tilde{h}_t\) implying the candidate vector represents the new context vector, \(h_t\), ignoring information from previous time-step.
• With \(u_t \approx 1\), (vector of 1s), \(h_t = h_{t-1}\) implying the previous context vector is copied to the new time-step, discarding the candidate vector information
• In most cases, \(u_t\) will take values between \(0\) and \(1\), allowing some information depending on their values.

\(\tilde{h}_t\) is a function of the current input, \(x_t\) and the previous hidden vector, \(h_{t-1}\).

where \(\textbf{W}\), \(\textbf{U}\) are weight matrices, and \(b\) is a vector.

Note that we have simplified the GRU update equations ignoring the reset gate.

An interpretation of the additive updates is that they help to create linear shortcut connections between the hidden vectors of the current state and previous states (similar to residual connections found in popular neural network architectures such as ResNet).

What are these shortcut connections

If we begin to unroll the hidden vector equation, moving step by step backwards, to extract the computations done to arrive there, we notice that it forms a weighted combination of all previous hidden vectors.

for \(t\) steps of GRU update. The breakdown of \(h_t\) shows the computation involving weighted combinations of all GRU’s previous states.

Gated Recurrent Units to Causal Attention

In causal attention as in GRUS, we will only have access or look at previous hidden states. This will allow us to proceed with our decomposition, but will be relaxed later to give a general non-causal attention.

Looking at the expanded version of the GRU update, we see dependencies between a lot of parameters and components. We will attempt to free these dependencies one-by-one given rise to a disentangled unit.

Let’s free the dependent weights

Recall that the update gate, \(u_t\) is calculated thus in GRUs;

where \(W_x\), \(U_h\) are weight matrices of the Update gate computation, \(b_u\) is a bias vector and \(h_t\), \(x_t\) are hidden and input vectors respectively.

From both equations, we can observe that \(u_t\), the current update gate is dependent on \(h_{t-1}\), the previous hidden vector and vice-versa. To disentangle \(u_t\) from \(h_{t-1}\), we can learn the current hidden context, \(h_t\) as a weighted combination of candidate vectors, \(h_i\).

where \(\alpha_i \propto \exp\left(\operatorname{ATT}\left(\tilde{h}_i, x_t\right)\right)\) and \(i\) ranges from time-step \(1\) to the current time-step, \(t\), implying that it uses the candidate vectors of all previous and current state to evaluate the hidden vector.

Let’s free up candidate vectors

Recall that \(\tilde{h} = f(x_t, h_{t-1})\)

where \(\tilde{h}_t\) depends on \(h_{t-1}\); \(h_{t-1}\) depends on \(\tilde{h}_{t-1}\) and \(h_{t-2}\) and so on - check unrolled \(h_t\) above.

This implies that \(\tilde{h}_t\) still depends on all the previous \(\tilde{h}_{t-N}\) candidate vectors.

To break these dependencies in candidate vectors, \(h\), Recall that;

we replace the candidate vector by an input function \(f(x_i)\). This input function takes in \(x_i \in \mathbb{R}^d\) and map it into a space of \(\tilde{h}_i \in \mathbb{R}^d\), without having to explicitly use previous candidate vectors.

The input function \(f(x_i)\) which have been used to disentangle the candidate vectors for each time-step can serve different purposes as we see in Transformers.

1. It is sometimes used to query which of the previous hidden states are important, i.e.

   \[\alpha_i \propto \exp\left(\operatorname{ATT}\left(f(x_i), f(x_t)\right)\right)\]

   where \(i\) ranges from \(1\) to \(t\), \(f(x_i)\) represents the Key vector and \(f(x_t)\), the Query vector of the attention function, \(\operatorname{ATT}(., .)\). This attention function provides relatively high \(\alpha_i\) values for \(f(x_i)\) values associated with current token, \(f(x_t)\).

2. As seen, it is also used to calculate the candidate vectors for the content update i.e.

   \[h_t = \sum_{i=1}^t \alpha_i f(x_i)\]

   where \(f(x_i)\) serves as the Value vector here.

   In summary, pass in a vector of input \(x_i\) or \(x_t\) to function \(f(.)\) depending on what is required to calculate Query, Key and Value.

• What is \(f(x_t)\) or \(f(x_i)\) ?

  \(f(.)\) is a function that processes the current input \(x_t\) or previous hidden vectors, \(x_i\). At the input to the encoder or decoder, if \(x_t\) or \(x_i\) is a one-hot vector representation of a token, \(f(.)\) is a lookup table or embedding layer. If \(x_i\) is a hidden state from the lower layer, \(f(.)\) can either be an identity function or a MLP.

Even though we have performed a lot of disentanglement, notice that Key, Value and Query vectors will be similar as they are derived from the same function.

Let’s separate Keys and Values

Instead of using a single linear function, let’s apply independent but similar linear functions to each of Keys, Values and Queries. These will just be 3 neural networks, \(K\), \(Q\), \(V\) with independent weights. So we have;

• the Key vector network, \(K\) and Query vector network, \(Q\) used in the attention function, \(\operatorname{ATT}\) to calculate the attention weights, \(\alpha_i\)

  \[\alpha_i \propto \exp(\operatorname{ATT}(K(f(x_i)), Q(f(x_t))))\]

• the value vector network, \(V\) used to calculate \(h_t\),

  \[h_t = \sum_{i=1}^t \alpha_i V(f(x_i))\]

Putting it another way, we compute the attention weights, \(\alpha_i\) by comparing the query vector of the current position, \(Q(f(x_t))\) against all the key vectors of the previous inputs, \(K(f(x_i))\), then compute the weighted sum of the Value vectors of all the previous inputs, \(V(.)\) to get the hidden vector, \(h_t\) at time-step \(t\).

At this stage, we have pretty much built a disentangled model but ehrmm, we have only a single attention mechanism. Will this be enough to model all the dependencies in context/hidden vectors? Maybe, it will be a good idea to have multiple attention heads. What do you think?

Let’s have multiple attention heads

We can create N multiple possible \(Q\), \(K\) and \(V\) functions/neural networks. Since each of them takes in the same \(x_i\) or \(x_t\), we can have parallel computation performed by each \(Q\), \(K\) and \(V\) functions.

For each attention head, \(n \in \{1, 2, 3, \dots, N\}\), we calculate \(h_t^n\). Each \(h_t^n\) is concatenated together to form the new \(h_t\) i.e.

where

\(h_t^n = \sum_{i=1}^t \alpha_i^n V^n(f(x_i))\) and \(\alpha_i^n \propto \exp(\operatorname{ATT}(K^n(f(x_i)), Q^n(f(x_t))))\)

Questions?

• Why concatenate the multiple attention heads instead of adding them, or use some other methods? Well, the concatenation gives a vector with a representation that provides information about different aspects of the inputs, and allows each head to specialize in attending. I don’t have an answer for that, if you do, please leave it in the comment section.

Gated Recurrent Units to Non-Causal Attention

We have previously only attended to previous and current hidden states in our network. In non-causal attention, we relax this assumption and we are allowed to also look at positions \(t+1\) to \(T\) of the sequence.

Let’s look at the entire input sequence

Since we have broken all the dependencies of candidate vectors and attention weights of all hidden states from \(t = 1, \dots, T\), at time-step/position, \(t\), we can utilize the previous hidden states, current hidden state as well as future hidden states. I prefer to call \(t\) position now that there are no recurrent computations really.

\(h_t\) then becomes;

where \(h_t^n = \sum_{i=1}^T \alpha_i^n V^n(f(x_i))\)

and \(\alpha_i^n \propto \exp(\operatorname{ATT}(K^n(f(x_i)), Q^n(f(x_t))))\)

Great!!, now we have a more robust attention mechanism. One problem still persists though. With this mechanism, we can just permute the order of the hidden states and nothing changes. Is that the behavior we want? Hmm, No!

Let’s give a sense of position to the attention mechanism

We can do this by adding a position encoded vector, \(p(i)\) (usually of same dimension) to each input. Each \(p(i)\) comes from a positional embedding \(p\). This positional embedding is independent of the actual token embeddings.

So \(h_t\) becomes;

where

and \(\alpha_i^n \propto \exp(\operatorname{ATT}(K^n(f(x_i)), Q^n(f(x_t) + p(i))))\)

\(p(i)\) is the position encoded vector for position \(i\) from positional embedding \(p\).

Learned positional embedding and function-based positional embedding (such as sinusoidal positional embedding) are the common positional embeddings. The Transformer uses the sinusoidal positional embedding due to the property that it can generalize to lengths not seen during training.

To this point, we have only been using linear projections as our functions. Do you think a sprinkle of non-linearity will help our model?

Let’s use Non-Linear Attention

As seen from our discussion, we can extract the following about the linearity of our Attention. With

• the hidden vectors calculated through attention are inherently linear and are just weighted sum of input vectors.
• Also, \(f(.)\) is often an identity function (especially for intermediate layers)
• p, the positional embedding does not depend on the input.
• the Values vector network, V is often a linear transformation.

With the following observations, it will be difficult for the attention to manipulate the attention weights to find a complicated combination. The solution will be to apply a post-attention non-linear function.

Let’s define \(g(.)\) as the post-attention non-linear function, which is a feed-forward neural network in our case, applied to each time-step independently.

For higher efficiency, \(g\) may be applied to each head independently. \(h_t\) becomes;

This gives us the Non-Linear, Non-Causal, Positional Attention used by Transformer.

It has been a long ride. If you have got this far, you are a Genius!! You must have gotten something from the article. Let’s bring all together in one place.

Full Self-Attention Layer

In summary,

• the context vector with a single attention mechanism is calculated thus;

  \[h_t^n = \sum_{i=1}^T \alpha_i^n V^n(f(x_i) + p(i))\]

• which are then concatenated together, either before or after applying a nonlinear function

  \[h_t = g\left(\left[h_t^1;~ h_t^2;~ \dots;~ h_t^N \right]\right)\]

  or

  \[h_t = \left[g(h_t^1);~ g(h_t^2);~ \dots;~ g(h_t^N) \right]\]

• then, the attention weight are calculated using the Key and Query vectors as well as positional encoding for the input

  \[\alpha_i^n \propto \exp(\operatorname{ATT}(K^n(f(x_i) + p(i)), Q^n(f(x_t) + p(i))))\]

Conclusion

In this article, we showed how we can move from a recurrent-based neural network with gates such as GRU to a self-attention based model such as Transformer with disentangled hidden states and weights, enabling parallel computations.

Here’s what we will cover:

1. Introduction
2. K Nearest Neighbor
3. Expressing KNN as a Neural Network
   1. Layer 1: Computing the distance function
   2. Layer 2: Softmax Layer
   3. Layer 3: Prediction Layer
4. Implementation
5. Future Work
6. Conclusion
7. Reference

Introduction

k-Nearest Neighbor is a non-parametric model that uses a distance function to evaluate the label of a new test point. It involves taking the average of predictions of k nearest points to the given test point. It often serves a base model for many predictions tasks and often difficult to beat.

Given a set of train data-points,

with \(x_i \in \mathbb{R}^d\), \(y_i \in \mathbb{R}\) and \(i = 1, 2, \dots, n\). There are \(n\) training examples each with d number of features. Given a new datapoint \(x^t\), how can we classify the point using k-nearest neighbor into its correct class?

K Nearest Neighbor

To classify a new test point into its correct label, we perform the following steps:

• Calculate the distance between each datapoint in the training example and the test datapoint
• Assign the test data point the label of the k datapoints with minimum distance. for a classification task, we can take the mode of the k classes, while for regressing task, we average the predictions to get the prediction.
• k can range from 1 to n, but usually between 1 and 10.
• The distance function takes in each \(x_i\) and \(x^t\) and outputs a scalar. The distance function to use may depend on the task but a popular and common distance is the l2-distance. The l2-distance can be expressed as:

Using the matrix notation,

where \(X = (x_1, x_2, \dots, x_n)^T \in \mathbb{R}^{n \times d}\) and \((x^t)^T \in \mathbb{R}^d\). To simplify notation, I will assume that \(x^t\) will be broadcasted along the matrix, which becomes simply:

Expressing KNN as a Neural Network

With the understanding of the distance function, we can break it apart to derive the parameters of our neural network.

From the distance function,

Since optimizing \(d^2\) is equivalent to optimizing for \(d\), we work with \(d^2\) instead, which we will call \(d'\)

Layer 1: Computing the distance function

Expanding the equation, we get:

Note that: \(\odot\) is a hadamard product, i.e. element-wise product between the two matrices.

since the \(-2X_jx^t\) does not depend on j.

At this point we can easily extract our first layer, \(Z_1 = W_1x_1 + b\) where \(W_1 = -2X_j\), \(x_1 = x^t\) and \(b = \sum_{j=1}^{d} \left(X^2_j + (x^t_j)^2\right)\)

Layer 2: Softmax Layer

After the previous step, we then need to find the datapoint with the closest distance to the test datapoint. This can be performed in two steps using the softmax layer:

• Negate the vector, so the datapoint with minimum distance, then have the maximum value.
• Multiply the vector by a large positive constant \(\lambda \rightarrow \infty\). This has the effect of shrinking small values and increases already large values. The intention is to have the neuron turned on for only the minimum value of the input \(Z_1\). This is equivalent to applying the softmax temperature on the vector. \(\lambda\) is an hyperparameter.

For a refresher on the softmax temperature, check my previous post on Softmax temperature

Layer 3: Prediction Layer

Before now, we have not really talked out the labels of the training examples. It comes in at this layer to support in prediction.

• For a regression task, this computation is almost done. We take the vector of distances and find the prediction of the class, with the minimum distance (or maximum value in this case, as we have performed inversion).
• For a classification task, we can also take the label of the datapoint with the minimum distance or go a step further.

For a classification task, where \(Z_3 = W_3x_3 + b\), firstly, we perform one-hot encoding on the train labels. \(y_{\text{onehot}} \in \mathbb{R}^{n \times d}\)

where \(W_3 = X^T\), \(x_3 = z_2\), \(b = 0\)

Implementation

I created a jupyter notebook to show predictions on the iris dataset. You can access the notebook via this link. Feel free to drop comments, and possibly give area for clarification or improvement. Let me try to explain the major parts of the implementation.

• Data: Loaded the iris dataset via the sklearn load dataset api.
• Preprocessing: Normalized the dataset (often a good thing to normalize) and converted each label into one-hot encoded vectors.
• Built a 3 layer neural network using the datapoints as parameters.

Future Work

A 1-Nearest Neighbor implementation was discussed here. This can be extended to k- nearest neighbors. How will you go about it?

Conclusion

In this article, We showed how a k-nearest neighbor classifier can be transformed into a neural network using the datapoints as parameters. This is also a non-parametric model and the weights of the model increases as the number of training points increases. Finally, neural network is a universal function approximator and can therefore be an exercise to represent other models in terms of a basic neural network model.

Reference

1. Yan Qiu Chen, R. I. Damper and M. S. Nixon, “On neural-network implementations of k-nearest neighbor pattern classifiers,” in IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, vol. 44, no. 7, pp. 622-629, July 1997, doi: 10.1109/81.596943.
2. O. J. Murphy, “Nearest neighbor pattern classification perceptrons,” in Proceedings of the IEEE, vol. 78, no. 10, pp. 1595-1598, Oct. 1990, doi: 10.1109/5.58344.

Here’s what we will cover:

1. Introduction
2. Why Reproducibility in Training?
   1. Structuring Hyperparameters
   2. Setting a global random seed
3. Cross validation applied to neural network
   1. The (Stratified) K-fold cross validation
   2. The model initialization
   3. Train on the entire dataset
4. Conclusion

Introduction

Cross-validation is a resampling technique that assesses how the results of a statistical analysis will generalize to an independent data set. Three commonly used types are; i) K-fold cross validation, ii) a variant called Stratified K-fold cross validation and iii) the leave-one-out cross validation.
Given data samples \(\{(x_1, y_1), (x_2, y_2), \dots (x_n, y_n)\}\) where n is the total number of examples, \(\textbf{x}_i\) is a d-dimensional vector or a tensor (as in images), and \(y_i\) is the class or label of example, \(i\)

• The k-fold cross validation is the standard type. The training data is split into k different parts. k is an integer (usually between 5-10), and depends on the size of data). k < total number of examples.
• If k = total number of examples, then, the k-fold becomes leave-one-out cross validation, as only one example is placed in the validation set in each validation run.
• The stratified k-fold cross validation is suitable for instances where there is imbalance in the frequency of the classes. If we use a random sampling, as is the case in k-fold, some examples might not have enough contribution to some folds.

I think I have only talked about cross validation on a high level, and I assume you are already familiar with the concept in machine learning. You might need to check out other sources if you require more details.

Why Reproducibility in Training?

Reproducibility ensures that we can recreate experiments. Deep learning models have been notoriously known to have many parameters and hyperparameters, with randomness in initializations and sampling. For reproducibility sake, we have to put things in order to validate our experiments. Two main places to ensure this are; i) hyperparameter initializations and ii) random seed settings

Structuring Hyperparameters

Firstly, we will define our hyperparameters and other settings in a structured way, so that hyperparameter changes can only be applied only at one single point. We can use a dictionary for this purpose or the argparse library. The argparse library is more preferred as it is an argument parsing tool which helps to easily translate from hyperparameters in notebooks to arguments on the command line. For an in-depth tutorial on this check out this argparse tutorial by Jeff Hale on Medium.

The setup is simple and looks like this:

from argparse import Namespace

args = Namespace(
        size=448,

        # Model Hyperparameters
        learning_rate=1e-4,
        batch_size=8,
        num_epochs=10,
        early_stopping_criteria=10,
        momentum=0.9,

        # CV parameter
        num_folds=5,

        seed=0,

        # Runtime hyper parameter
        cuda=True,
        )

Then, you can easily call or modify the parameters, like a normal python dictionary. For example, to add a device argument

# Check CUDA
if not torch.cuda.is_available():
    args.cuda=False
    
# Add device for training
args.device = torch.device("cuda" if args.cuda else "cpu")

Setting a global random seed

Our computers only generate pseudo-random numbers. This means that we can make them generate the same set on random numbers continuously if we set a starting seed. Recall that neural network training requires different libraries such as numpy, pytorch, pandas, cudnn etc interfacing. They also make use of random number generators which all require seeds. Also, many parts of the neural network model itself - such as weights, biases, dropout - require sampling.
How can we account for these randomness in our experiments?
The simple solution is to set a single seed and consistently apply this seed across all the libraries requiring it.

def setup_seed(seed, cuda):
    # Creates global random seed across torch, cuda and numpy 
    np.random.seed(seed)
    torch.manual_seed(seed)
    if cuda:
        torch.cuda.manual_seed_all(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic=True

Then, we can easily call the function to set up the seeding across all libraries.

setup_seed(args.seed, args.cuda)

That’s it!! Now that you understand reproducibility, let me walk you through a cross validation process in neural networks. I will be applying it to a classification task.

Cross validation applied to neural network

Cross validation can be used to select the best hyperparameters for training a neural network. If the folds have good performance on their validation sets when a set of hyperparameters is applied, then that set of hyperparameters is believed will help generalize the model to unseen data. In the same vein, the Cross validation models can likewise be ensembled in several different ways for prediction;

• the weights of all the folds can be averaged to get a more robust model. This is popularly known as Polyak Averaging.
• the model of each fold can be saved to make predictions on an unseen data. Then, softmax predictions of examples of each fold can be averaged to predict the correct class. This is popularly known as model ensemble.

I will be using the Cassava Disease Challenge on Kaggle as a running example to explain the underlying concept. The challenge entailed classifying pictures of cassava leaves into 1 of 4 disease categories or healthy.

I will only highlight parts of the code that performs the cross-validation and will not try to show all codes here. For the full code, you can check the cassava disease classification Kaggle kernel I created for this tutorial.

The (Stratified) K-fold cross validation

Every task has its own peculiarity and understanding the statistics of the data can go a long way in getting good results. For instance, the number of cassava leaf samples in each disease category differs by a large margin, creating a data imbalance. To ensure each class is involved in each fold, we can “bias” the sampling using the Stratified K-fold cross validation.

Using 5-fold cross validation splits the data into 80% training and 20% validation (which is a popular choice).

def stratified_kfold(num_folds=5, images_df=None):
    st_kfold = StratifiedKFold(n_splits=num_folds, shuffle=args.shuffle_dataset, random_state=args.seed)

    fold = 0
    for train_index, val_index in st_kfold.split(images_df['images'], images_df['labels']):
        train, val = images_df.iloc[train_index], images_df.iloc[val_index]

        train_dataset = CassavaDataset(df_data=train, transform=train_trans)
        valid_dataset = CassavaDataset(df_data=val,transform=val_trans)

        train_loader = DataLoader(dataset = train_dataset, 
                                  batch_size=args.batch_size, 
                                  shuffle=args.shuffle_dataset, 
                                  num_workers=args.num_workers)
        valid_loader = DataLoader(dataset = valid_dataset, 
                                  batch_size=args.batch_size, 
                                  shuffle=args.shuffle_dataset, 
                                  num_workers=args.num_workers)

        dataloaders = {'train': train_loader, 'val': valid_loader}
        
        dataset_sizes = {'train': len(train_dataset), 'val': len(valid_dataset)}
        print(dataset_sizes)

        print()
        print(f'Starting CV for Fold {fold}')
        model_ft, criterion, optimizer_ft, exp_lr_scheduler = prepare_model(pretrained=args.pretrained)
        model_ft = train_model(model_ft, 
                              criterion, 
                              optimizer_ft, 
                              exp_lr_scheduler,
                              dataloaders,
                              dataset_sizes,
                              num_epochs=args.num_epochs,)
        
        # Save model for the current fold to your output directory
        current_fold_full_path = args.save_dir +'/model_'+str(fold)+'_tar'
        torch.save(model_ft.state_dict(), current_fold_full_path)

        fold += 1

• A dataframe containing the name/location of the images and label is passed into the stratified_kfold function, with the number of folds
• The k-fold function splits the data into 5-folds and returns the indexes of the examples belonging to train and validation set in each fold.
• These indexes are then used to retrieve the examples to create train and validation datasets.
• The dataloaders then create batches of data for training and validation.
  Note that the definitions for train_model, Dataloaders and Datasets are not shown here, but are available in the cassava disease classification Kaggle kernel setup for this tutorial for reference.

The model initialization

One subtle problem that has not yet been considered thus far is how to initialize the model in each fold. Do we continue with the model used in a previous fold or we initialize a new model? Well, it is clear, we have to reinitialize to a new but similar model on every fold. There are 3 different initialization cases to be considered;

1. Using a pretrained model with downloadable weights. This is pretty straight forward. Just redownload the weights or reload the weight from the stored cache at the beginning of each fold.
2. Using a custom model built from scratch. Just reinitialize your model on each fold. Since we have set a seed, we are more likely to have the same set of weights and biases during reinitialization. You can also initialize once, store the weights in a cache and retrieve the stored weights for each fold.
3. Using a base model architecture such as ResNet without pretrained weights. Similar to 2 above. Just make sure you reinitialize your model on each fold.

   In any case, do not use the trained model of a previous fold to initialize a new fold. The training outcome can be very*5 disastrous and deceptive!!

Train on the entire dataset

Now that you have performed cross validation with your hyperparameters, you can then use the best hyperparameters (such as learning rate, number of epochs, batch size, optimizer, image size etc) to retrain the model on the entire training samples. This might bring about some significant improvement as the dataset can now see more examples in a single run.

Conclusion

Cross validation is one main topic less talked about in deep learning due to the time it takes to perform the process. It can also be less favoured when there is a large training set (in the range of millions of examples). In cases where the sample size is not massive, we still require some classical ideas such as cross validation to generalize to unseen data. Reproducibility ensures that on each fold of cross validation, we can reinitialize our model to the original state.

Here’s what we will cover:

1. Introduction
2. Numerical Stability of Softmax
3. Log Softmax
4. Log-Softmax Derivation
5. Softmax Temperature
6. Conclusion

Introduction

Softmax is a non-linear function, used majorly at the output of classifiers for multi-class classification. Given a vector \([x_1, x_2, x_3, \dots x_d]^T\) for \(i = 1,2, \dots, d\), the softmax function has the form

where d is the number of classes.
The sum of all the exponentiated values, \(\sum_{j=1}^{d} e^{x_j}\) is a normalizing constant which helps to ensure that it maintains the properties of a probability distribution i.e. (a) the values must sum to 1 (b) they must be between 0 and 1 inclusive \([0, 1]\).

For example, given a vector \(x = [10, 2, 40, 4]\), to calculate the softmax of each element;

• exponentiate each value in the vector \(e^x = [e^{10}, e^2, e^{40}, e^4]\),
• calculate the sum \(\sum{e^x} = e^{10} + e^2 + e^{40} + e^4 = 2.353\ldots \times 10^{17}\)
• then, divide each \(x_i\) by the sum to give \(\operatorname{sm}(x) = [9.35762297 \times 10^{-14}, 3.13913279 \times 10^{-17}, 1.00000000 \times 10^{00}, 2.31952283 \times 10^{-16}]\)

This can be easily implemented in a numerical library like numpy,

import numpy as np
def softmax(x):
    exp_x = np.exp(x)
    sum_exp_x = np.sum(exp_x)
    sm_x = exp_x/sum_exp_x
    return sm_x

x = np.array([10, 2, 40, 4])
print(softmax(x))

output: [9.35762297e-14 3.13913279e-17 1.00000000e+00 2.31952283e-16]

• Questions
  • What do you observe about the output?
  • Will the output sum to 1?

These are pointers to what we will be discussing in the next sessions?

Numerical Stability of Softmax

From the softmax probabilities above, we can deduce that softmax can become numerically unstable for values with a very large range. Consider changing the 3rd value in the input vector to \(10000\) and re-evaluate the softmax.

x = np.array([10, 2, 10000, 4])
print(softmax(x))

output: [0.0,  0.0, nan,  0.0]

‘nan’ stands for not-a-number and occurs when there is an overflow or underflow. But, why the \(0\)s and \(\text{nan}\)? Are we implying we cannot get a probability distribution from the vector?

• Question: Can you find out what caused the overflow?

Exponentiating a large number like \(10000\) leads to a very, very large number. This is approximately \(2^{10000}\). This causes overflow.

• Can we do better? Well, we can. Taking our original equation,

Let’s subtract a constant \(c\) from the \(x_i\)s

We just shift the \(x_i\) by a constant. If this shifting constant, \(c\) is the maximum of the vector, \(\max(x)\), then we can stabilize our softmax computation.

• Question: Do we get the same answer as the original softmax?
  This can be shown to be equivalent to the original softmax function:
  Consider

which produces the same initial softmax

A numpy implementation of this stable softmax will look like this:

def softmax(x):
    max_x = np.max(x)
    exp_x = np.exp(x - max_x)
    sum_exp_x = np.sum(exp_x)
    sm_x = exp_x/sum_exp_x
    return sm_x

if we apply it to our old problem:

x = np.array([10, 2, 10000, 4])
print(softmax(x))

output: [0., 0., 1., 0.]

Great, problem solved !!!

• Question: Why are all other values in the softmax 0. Does it mean they have no probability of occurring?

Log Softmax

A critical evaluation of the softmax computation shows a pattern of exponentiations and divisions. Can we reduce these computations? We can instead optimize the log softmax. This gives us nice characteristics such as;

1. numerical stability.
2. gradient of log softmax becomes additive since \(\log(a/b) = \log(a) - \log(b)\)
3. lesser computations of divisions and multiplications as addition is less computationally expensive.
4. log is also a monotonically increasing function. we get this property for free

To quote a stackoverflow answer on using log softmax over softmax:

There are a number of advantages of using log softmax over softmax including practical reasons like improved numerical performance and gradient optimization. These advantages can be extremely important for implementation especially when training a model can be computationally challenging and expensive. At the heart of using log-softmax over softmax is the use of log probabilities over probabilities, which has nice information theoretic interpretations. When used for classifiers the log-softmax has the effect of heavily penalizing the model when it fails to predict a correct class. Whether or not that penalization works well for solving your problem is open to your testing, so both log-softmax and softmax are worth using.

If we naively apply the logarithm function to the probability distribution, we get:

x = np.array([10, 2, 10000, 4])
softmax(x)

output: [0., 0., 1., 0.]

np.log(softmax(x))

output: [-inf, -inf,   0., -inf]

We are back to numerical instability, in particular, numerical underflow.

• Question: Why is this so?

The answer lies in taking the logarithm of individual elements. The \(\log(0)\) is undefined. Can we do better? oh yes!

Log-Softmax Derivation

• What if we want to get back our original probabilities? Well, we can exponentiate and normalize the log softmax or log probability values.

Let’s make this concrete via code.

def logsoftmax(x, recover_probs=True):
    # LogSoftMax Implementation 
    max_x = np.max(x)
    exp_x = np.exp(x - max_x)
    sum_exp_x = np.sum(exp_x)
    log_sum_exp_x = np.log(sum_exp_x)
    max_plus_log_sum_exp_x = max_x + log_sum_exp_x
    log_probs = x - max_plus_log_sum_exp_x

    # Recover probs
    if recover_probs:
        exp_log_probs = np.exp(log_probs)
        sum_log_probs = np.sum(exp_log_probs)
        probs = exp_log_probs / sum_log_probs
        return probs

    return log_probs
  
x = np.array([10, 2, 10000, 4])
print(logsoftmax(x, recover_probs=True))

output: [0., 0., 1., 0.]

Softmax Temperature

In the NLP domain, where the softmax is applied at the output of a classifier to get a probability distribution over tokens. The softmax can be too sure of its predictions and can make other words less likely to pre sampled. For example, if we have a statement;

The boy ___ to the market.

with possible answers, \([\text{goes}, \text{go}, \text{went}, \text{comes}]\). Assume we get logits of \([38, 20, 40, 39]\) from our classifier to be fed to a softmax function.

x = [38, 20, 40, 39]
softmax(x)

output: [0.09, 0.00, 0.6, 0.24]

If we were to sample from this distribution, \(60\%\) of the time, our prediction will be “went” but we are also aware that the answer could also be any of “goes” or “comes” depending on context. The initial logits also show close values of the words but the softmax pushes them away.
A temperature hyperparameter, \(\tau\) is added to the softmax to dampen this extremism. The softmax then becomes

where \(\tau\) is in \((0, \infty]\). The temperature parameter increases the sensitivity to low probability candidates and has to be tuned for optimal results. Let’s examine different cases of \(\tau\)

case a: \(\tau \to 0\) say \(\tau = 0.001\)

softmax(x/0.001)

output: [0., 0., 1., 0.]

This creates a more confident prediction and less likely to sample from unlikely candidates.

case b: \(\tau \to \infty\) say \(\tau = 100\)

softmax(x/100)

output: [0.25869729, 0.21608214, 0.26392332, 0.26129724]

This produces a softer probability distribution over the tokens and results in more diversity in sampling.

Conclusion

The softmax is an interesting function that requires an in-depth look. We introduced the softmax function and how it can be computed. We then looked at the problems with the naive implementation and how it can lead to numerical instability and proposed a solution. Also, we introduced the log-softmax which makes numerical computation and gradient computation easier. Finally, we discussed the temperature constant used with softmax.

Here’s what we will cover:

1. Introduction
2. What we mean by efficiency
3. A closer look at standard neural networks
4. Techniques for making efficient models
5. Conclusion

Introduction

Most research directions in AI are geared towards beating the State-of-the-art on various benchmarks. As such, researchers tend to put in all their compute power and complexity to achieve this goal. This leads to trends of building bigger models with more complex architecture, for a little gain in accuracy. We need to start thinking more in terms of efficiency of the models we build and how it impacts on further research. A model that is very big and complex will loose interest from low-resource researchers and students who want to quickly experiment.
Efficiency in the context of neural networks deal with;

• Faster training time
• Fast inference time
• Smaller model size
• Low energy consumption etc.

For example, with the introduction of the Transformer architecture in the NLP domain, the field has achieved giant strides, pushing up SOTA results i almost all NLP domains. The Transformer has an interesting architecture with parallel computations and self-attention mechanism. A major concern is its parameter size due to the multi-head attention. A recent architecture from Google called Reformer uses locality-sensitive-hashing (LSH) to address this. Similarly, in Computer vision domain, models like ResNext with repeated blocks have shown to have great performance of Image-related tasks, at the expense of speed. These models are difficult to train and experiment with on low-memory GPUs. Therefore, how can we make this models more efficient without loosing performance?

This article was written to give an introductory view on this subject, and hopefully inform Researchers and Machine Learning Engineers like me, on what to have at the back of our minds when we build models and do research.

What we mean by efficiency

Let me expand on the points I made earlier about efficiency. Efficiency is relative and we are talking here in terms of our current memory capacity for on-device AI, mobile applications, computation speed etc.

• How long will it take to train the model?
  One thing I like to scan for in AI research papers these days is the total training time of the model. Many SOTA models take like 1 week to 2 months to train. With that in mind, I know where to focus my reading as it will be difficult to replicate the same results. I entered into a NeurIPS paper implementation challenge last year but got really discouraged by the number of days it will take to verify experiments. I understand that I can track my loss values and accuracy as I train, but the final pre-trained model still takes that long. For example, the Google Meena Chatbot was trained for 30 days using cloud TPU Pods (2,048 TPU cores). 30 days of pre-training is a long-time for experimentation and reproducing results.

• How long will it take to get predictions?
  This is really important for models that require streaming predictions. For example, a model that performs auto-correction, or spellcheck must be able to give fast predictions without latency. This is really important for great user experience in AI-powered applications. For instance, Instagram filters must render image filters to users split of seconds to be considered efficient.

• Will the model fit in memory?
  This is a really important aspect of efficiency that is being overlooked by the big models. Each year, the model becomes bigger and bigger in size, ensuring that memory sizes are increased to cater for this fact. We need to consider that the amount of memory/RAM available for experimentation on platforms like Google Colab and Kaggle are limited, and I would still like to experiment. I will like to observe that these models usually have a smaller version, but most times, do not perform as well as the large counterparts. As a case study, the OpenAI GPT-2 released in 2019, was so good that the source code wasn’t going to be released at first. It contained 1.5 billion parameters. Wow!! How do I use that on a mobile platform?

• How much energy will it require for training?
  Many AI research papers do not track or report the total power used in training their models. I think it will be more useful to have these results for comparism. Neural networks are implemented as computation graphs, and support vectorized implementations with GPUs and TPU support for faster computation. This is great considering the progress in hardware design but most on-device applications still require cpu for computation. Needless to say that, if on-line training will e required by these on-device models, then the computational cost increases as these models might not have been made with this specification in mind. For energy conservation sake, we need models that can perform efficient computations.

• How much energy will it require for inference?
  Energy consumption at inference time is another great consideration, especially for on-device AI applications that utilize other power sources like batteries for inference. We would like a model that runs effortlessly without consuming a substantial amount of energy.

This is a requirement for on-device applications, as this improves the experience of users. It makes no sense to run an application for 10 minutes and have to recharge afterwards due to low power.

Our goal should therefore be to retain model performance or perform better, while aiming for more efficient models.

A closer look at standard neural networks

Neural networks as computational graphs show some distinct traits which make them powerful and able to outperform their counterparts on many tasks. The first observation is that they are universal function approximators and are therefore prone to overfitting on any task. This is both a blessing and a bane. Also, modern neural networks are build to be over-parameterized to accommodate for more model capacity. Therefore, regularization is usually required to prevent overfitting. Increasing layer cardinality is also common in these models e.g ResNet, Inception, BERT etc. This also implies that some of the layers of the network may be redundant and may be learning similar properties. Skip connections and dynamic connections have been used in these large models to cater for this effect and ensure that gradients can be back-propagated efficiently. Similarly, the hyperparameters such as number of filters, number of multihead attentions, size of layers are made bigger than they should be just to squeeze out the performance.
This is better expressed by the Lottery Ticket Hypothesis postulated by MIT researchers. The Lottery Ticket Hypothesis states that

“A randomly-initialized, dense neural network contains a subnetwork that is initialized such that — when trained in isolation — it can match the test accuracy of the original network after training for at most the same number of iterations.”

What a great discovery!!!

Techniques for making efficient models

1. Train a smaller model from scratch.
   This looks obvious at first. If you want a model with smaller size, why not train yours with smaller weight sizes, less number of layers etc. Well, it turns out that this has the advantage of training faster, with smaller model size but with great performance drops. If you consider, most big models, they increase the model sizes to get better performance on the benchmark datasets. Also, intuitively, smaller models have lower capacity and may not be able to learn all the intricacies of the task to perform. As such, these smaller models are only used as proof-of-concept. Can we do better than this? Well, that’s why we are researchers and engineers right? We have to know that any proposed methods of increasing efficiency has to perform better than just this to be so cool.

2. Sparsity Inducing Training
   The goal of sparsity inducing training is to set as many weights as possible to zero during training. This brings about faster computations especially for on-device AI where we are watchful of number of computations as we have efficient methods - such as the Compressed Sparse Row Format (CSRF) or Yale format and the Compressed Sparse Column Format (CSCF) - for storing sparse matrices. Sparsity can be induced in various forms. For example, through the model loss (as in L1 and L0 regularization), removing small magnitude weights, and some other bayesian methods. For a more concrete review of these methods, check the paper titled “The State of Sparsity in Deep Neural Networks” by Trevor Gale et al.

3. Knowledge Distillation
   In knowledge distillation, a large, accurate network teaches a smaller network how to behave. That’s cool right? The large network known as the teacher instructs the smaller network called the student. This can be done done via two training methods;
   Case a: Student model learns to mimic the output of the teacher model, getting its loss from only the output layer
   Case b: Student model also learn to mimic intermediate layers as well as the output. So the loss of the student is therefore the sum of all the intermediate losses and output loss. This was successfully applied in Patient Knowledge Distillation for BERT Model Compression
   
   The image shows the model architecture of Patient Knowledge Distillation approach to BERT model compression. In the PKD-Skip architecture, the student network learns the teacher’s outputs in every 2 layers while the PKDLast represents where the student learns the teacher’s outputs from the last 6 layers.
   A major advantage of the teacher-student setup is that it provides flexibility over size, as there is no restriction on size of teacher and student. A consequence of this smaller model is its fast inference time, with similar performance as the teacher. One thing to put in mind is that a pretrained teacher model is required in this setup and the student network inherits the biases of the teacher model.
   Knowledge distillation has been successfully applied to production models such as the HuggingFace DistilBERT with a smaller, faster, cheaper and lighter BERT model. It has also been applied in Generative Adversarial Networks, KDGAN for student training. In another area of application, knowledge distillation can be used to train a surrogate model without having knowledge of original model internals or even its training data. The paper titled “Practical Black-Box Attacks against Machine Learning” highlights the fact that this can aid adversarial attack of a machine learning model as the surrogate model just has to learn to mimic the decision boundaries of the original model.

4. Pruning
   Pruning involves training a large network at training time, but then eliminating some parts of the network at inference time. This might include heuristics such as dropping convolution layers, dropping attention layers, or convolution filters, removing portion of weights etc. at inference time. The method above usually require some for of retraining or finetuning. Can we have a method that does not necessarily need retraining at inference time?
   A recent work by Angela Fan on LayerDrop showed promising results. The layer drop is so simple that it is surprising that it works at all.
   How LayerDrop works
   LayerDrop is implemented by randomly dropping layers during training using a drop rate. Possible drop rates are 10%, 20%, 25%. Its implementation is similar to dropout but does not even require that weights are upscaled after dropping layers.
   
   Also, LayerDrop increases training speed as we do not perform forward propagation on the entire number of layers, ensuring that the model is robust to perturbations and regularized.
   At inference time, you can prune to any depth of your choice without affecting performance. This means that you can adopt any pruning strategy e.g. prune all odd layers, or prune all even layers, or prune every 3 layers. Pruning strategy that may not work well are aggressive pruning e.g pruning all the early layers or pruning all the late layers or pruning more than 50% of the model.
   LayerDrop has not seen wide success in computer vision as it has in NLP though, but more research can be done in this area.

5. Weight Sharing
   The idea of weight sharing is that different layers can reuse weights. This just requires that some or all sub-networks share the same weights. A major drawback of weight sharing is that the amount of transformations that can be learned is reduced, casing a decrease in performance. In practice, the model capacity is usually increased to cater for this. The ALBERT model utilizes this idea of tying chunks of layers to the same weights. As quoted from the blog post;

   “Another critical design decision for ALBERT stems from a different observation that examines redundancy. Transformer-based neural network architectures (such as BERT, XLNet, and RoBERTa) rely on independent layers stacked on top of each other. However, we observed that the network often learned to perform similar operations at various layers, using different parameters of the network. This possible redundancy is eliminated in ALBERT by parameter-sharing across the layers, i.e., the same layer is applied on top of each other. This approach slightly diminishes the accuracy, but the more compact size is well worth the tradeoff.”

6. Quantization
   Quantization refers to techniques for performing computations and storing tensors at lower bitwidths than floating point precision. This process compresses the model size after training and a go-to approach for model compression, especially for on-device AI applications.
   In quantization, the goal is to efficiently store the weight floating point numbers using other number types such as int8, int4 or even bits(1 and 0). This is usually more memory efficient. The popular deep learning libraries provide quantization methods out of the box and have tutorials on how to perform quantization. See Tensorflow and PyTorch libraries for their apis.
   Quantization can drastically reduce model size by up to 80% and can easily be combined with other existing techniques for even lower model sizes. The quantization method and compression size has to be considered because drastic compression can reduce model performance and accuracy.

7. More Efficient Architectures
   As at this time, we have been exploring methods that involve starting with a bigger model, then compressing it. Can we do better by consciously building architectures made out of the goal for efficiency?
   For example, this paper titled “Pay Less Attention with Lightweight and Dynamic Convolutions” replaces some multihead attention weights in transformers with convolution layers. Some other propositions might include eliminating some bottlenecks in our current networks for faster computation, if it will not affect performance. Also, application specific models can be built for better efficiency.
   Some other considerations for efficient networks which were not discussed in this article are; models for specialized hardwares and specialized memory block sizes. These are also great considerations for efficiency and important for hardware manufacturers who have their chips optimized for computation in this regard.

Conclusion

In summary, we discussed what it means for models to be efficient, looked closely at neural networks to garner intuition about their behavior, described current methods for making models more efficient and took a step forward to provide a glimpse of building more efficient architectures from the get-go.