We will be assuming you have version 3 of HTK, although everything should work with any recent version.

All HTK commands start H, for example, HList is a tool to list the contents of data files (waveforms, MFCCs, etc.). The tools use command line flags and arguments extensively, as well as configuration files. Command line flags in capital letters are common across tools, such as:

• -T 1 specifies basic tracing
• -C resources/CONFIG tells the tool to load the config file called CONFIG in the directory resources

Simple scripts are provided for building a basic speaker-dependent digit recogniser from your own data. You will need to modify them slightly to make the more advanced system later. You will need to refer to the HTK manual.

HTK is not open source, so the only way to obtain the manual for HTK is to register on the HTK website. Do that now. You don’t need to read the manual yet, but it will be useful later.

To modify the scripts you’ll need to use a text editor, such as Atom or emacs. Never use a word processor!

Everywhere in this handout that you see <username>, you should replace it with your username. For example lab/<username>_test.mlf would become lab/s1234567_test.mlf.

File formats can quickly get confusing. There are dozens of waveform formats, and various formats for model parameters, MFCCs and so on. We will record the speech data into Microsoft Wav (RIFF) format waveforms, which will have the extension .wav, for example.

Please stick to the recommended names of files and directories – it will make it easier for me to help you (Edinburgh students only).

Related forum

Installing HTK on your own computer

We don’t really recommend installing HTK yourself unless you’re familiar with compiling code.  It’s fine to use the lab computers (remote or in person) or ssh. The code is quite old and not maintained, so you’re likely to run into dependency issues.  The install instructions haven’t been tested recently.

If you do want to try installing HTK on your own computer, here are some instructions:

• instructions for installing HTK on a MacBook.
• Instructions for installing HTK on Ubuntu Linux.
• Instructions for installing HTK on Ubuntu WSL

Those gists also include info on how to download the assignment data.  I was only able to test these on somewhat older machines, so there’s a high probability that new machines like the Mac M1 will run into problem.

Downloading the scripts and data to your computer

In general, you can download the data from the PPLS AT lab servers using rsync as follows. You could also use a file transfer app like FileZilla or WinSCP, you just need to navigate to the corresponding directories.

You don’t need to download any data if you are working on the PPLS AT lab machines (in-person or remote!)

On your own computer, navigate to the directory you want to download it to. The following assumes you’ll just use the same directory structure as the instructions from the PPLS AT lab remote desktops (replacing YOUR_UUN with your actual UUN:

mkdir -p ~/Documents/sp 
rsync -avz YOUR_UUN@scp1.ppls.ed.ac.uk:/Volumes/Network/courses/sp/digit_recogniser ~/Documents/sp

This will copy the scripts in the digit_recogniser directory on the AT lab servers to the directory Documents/sp in your home directory (remember ~ is a shortcut for your home directory in unix command line).

Now let’s get the data directory (e.g. features and labels from recordings):

mkdir -p ~/Documents/sp 
rsync --exclude 'wav' -avz YOUR_UUN@scp1.ppls.ed.ac.uk:/Volumes/Network/courses/sp/data ~/Documents/sp

This will exclude the `wav` directory which as you might expect includes the recordings that features were extracted from. It’s around 4 GB. If you want to download this data too, you can just remove the `–exclude ‘wav’` flag.

Assuming you downloaded the data into ~/Documents/sp, you can go to the directory with the digit recognizer scripts from your home directory:

cd ~/Documents/sp/digit_recogniser

To go to the directory with the data from your home directory:

cd ~/Documents/sp/data

Log in

This year (2023-24), the data collection step is OPTIONAL, to reduce workload. But you still need to understand how the data are created, and you will need to describe this in your report, so make sure you understand how you would collect data. You may want to create your own test set for certain experiments, so will need to come back to this later if you decide to do that.

You need data to train the HMMs, and some test data to measure the performance. Be very careful to keep these two data sets separate! Record the ten digits, and make sure you pronounce “0” as “zero”. Please record 7 repetitions of each digit for training, and a further 3 repetitions of each digit for testing.

Record all the training data into one long audio file, and all the test data into another. You should randomise the order in which you record the data (the reasons for doing this with the test data will become clear in the connected digits part, later). Remember to say the digits as isolated words with silence inbetween. I find it’s easier to write down some prompts on paper before recording.

Make a note of the device you used and edit the make_mfccs script to include this information. You should take a look at the existing info.txt file and use compatible notation :

1. username
2. gender (m or f)
3. mobile phone make and model written in lowercase without spaces (e.g., apple_iphone_8, or samsung_galaxy_a10)
4. your accent: native speakers should specify their accent, non-natives should specify “NN” plus their native country)

For example:

INFO=${USER}" m apple_iphone_8 NN France"

This allows your data to be combined with other peoples’ later on in the practical.

Your recording will probably be in mp3 format, so you will need to convert it to wav format before proceeding. The program sox can do this – ask a tutor for help in a lab session. Name the output files something like s1234567_train.wav and s1234567_test.wav .

Label your training data

You need to label the training data. Since the training data are in a single long wav file, there will be a single label file to go with it. There will be labels at the start and end of every recorded word. You don’t need to be very precise with these labels, but do be careful to include the entire word. It’s OK to include a little bit of preceding/following silence. It’s not OK to cut off the start or end of the word.

In this video:

1. Start wavesurfer and load the file you wish to label (e.g., s1234567_train.wav) and select transcription as the display method. Create a waveform pane.
2. You now have to label the speech. The convention is that labels are placed at the end of the corresponding region of speech. So, end all words with the name of the digit (use lowercase spelling: zero, one, two, …), and label anything else, including silence as junk. You must label the silence because these labels are use to determine the start times of the digit tokens. So, make sure that there is a junk label at the start of each word (the very first label in the file will therefore be junk).
3. Right click in the labeling window and select properties in the window that opens: change the label file format to WAVES and click ok
4. Right click in the labelling window again and select save transcription as to save your labels into a file called <username>_train.lab in the lab directory. During labelling, use save all transcriptions to save your work regularly.

NOTE: You can do this labelling in Praat instead, but you’ll have to convert the TextGrid to xwaves .lab format and also extract the individual wav files of the digits. See the .lab files in the assignment data.

Prepare your test data

We need isolated digit test data (each digit in a separate file) and can use Wavesurfer to split the waveform file into individual files, based on labels.

1. Use wavesurfer to create labels as before. However, do not label the speech with the digit being spoken but instead use the labels <username>_test01 , <username>_test02, <username>_test03 and so on. Place one of these labels at the end of each spoken digit and place a label called junk at the start of every digit. As for the training data, leave a small amount of silence around each spoken digit.
2. To split the file into individual files, right-click and select split sound on labels.
3. After this, you should have lots of files called <username>_test01.wav, <username>_test02.wav, <username>_test03.wav and so on, in the wav directory. You will also have one or more files with junk in their names. You don’t need those and can delete them.

Some versions of Wavesurfer create a subdirectory containing the split files, and add a numerical prefix to them. If that happens, rename your files (e.g., manually, or using a script if you know how) so they are called something like s1234567_test01.wav and move them up to the wav directory.

Listen to some or all of the files to check they are OK.

Create a master label file for the test data

Although all our test data are now in individual files, for convenience we can use a single MLF (master label file) for the test data. This label file contains the correct transcription of the test data, which we will need when we come to measure the accuracy of the recogniser. You will need to listen to the test wav files in order to write this label file.

Use your favourite plain text editor, such as Atom, but not a word processor, to create this file.

It needs to look something like this (note the magic first line, and the full stop (period) marking the end of the labels for each file; make sure there is a newline after the final full stop in the file):

#!MLF!#
"*/s1234567_test01.lab"
zero
.
"*/s1234567_test02.lab"
nine
.

and save the file as lab/<username>_test.mlf

The format of the MLF file is as follows: The first line is a header: #!MLF!#. Following this are a series of entries of filenames and the corresponding labels. "*/s1234567_test01.lab" is the filename: make sure you include the quote marks and the * because these are important. After the filename follows a list of labels (there is only one label per file in our case) and then a line with just a full stop, followed by a newline, which denotes the end of the labels for that file.

This year (2023-24) this data collection and feature extraction steps are OPTIONAL. Do not execute any of the commands in this section unless you have recorded your own data. However, you still need to understand the feature extraction process, so you can describe it in your report.

Each waveform file must be parameterised as MFCCs. This is done using the HTK command HCopy. The file called CONFIG_for_coding file, which is provided, specifies the settings used for the MFCC analysis. You should keep these settings for this assignment.

Do not run HCopy by hand – instead, you should use the make_mfccs script. The script runs HCopy like this:

HCopy -T 1 -C resources/CONFIG_for_coding wav/file.wav mfcc/file.mfcc

Run the make_mfccs script as soon as you have finished preparing and checking the data – it will copy your data and labels into a shared directory, making it available to the rest of the class.

If you make any changes to your data (e.g., correcting a label) then you must run the script again.

Make sure you are in the digit_recogniser subdirectory, then run:

./scripts/make_mfccs

Scroll back up through the output from this command to see if there were any errors. If there were, correct the problems (e.g., move files to the correct places) and run the command again.

Sharing your data

The make_mfccs script copies your data to a shared folder for use by other students. If you discover and fix any errors in your data, you should re-run this script.

Related forum

This year (2023-24), the preceding data collection steps are optional and should be skipped initially. Start here, at this step, to build your first speaker-dependent digit recognizer.

The training algorithms for HMMs require a model to start with. In HTK this is called the prototype model. Its actual parameter values will be learned during training, but the topology must be specified by you, the engineer. This is a classic situation in machine learning, where designing or choosing the type of model is made using expert knowledge (or perhaps intuition).

Select your prototype HMMs

In this video:

1. We need a model to start with – in HTK this is called a “prototype model”
2. A prototype model defines the:
   • dimensionality of the observation vectors
   • type of observation vector (e.g., MFCCs)
   • form of the covariance matrix (e.g., diagonal)
   • number of states
   • parameters (mean and variance) of the Gaussian pdf in each state
   • topology of the model, using a transition matrix in which zero probabilities indicate transitions that are not allowed and never will be, even after training

You can experiment with different topologies – although for isolated digits, the only sensible thing to vary is the number of states. Models with varying numbers of states are provided in models/proto (remember, a 5 state model in HTK actually has only 3 emitting states). In your later experiments, modify the initialise_models script to try different prototype models. You might even want to create additional prototype models.

Now train the models

In this video:

1. Training consists of two stages:
   • the model parameters are initialised using HInit which performs a simple alignment of observations and states (uniform segmentation), followed by Viterbi training
   • then HRest performs Baum-Welch re-estimation
2. A close look at the initialise_models script

In the simple scripts that you have been given as as starting point, each of the two stages of training is performed using a separate script. You can run them now:

./scripts/initialise_models
./scripts/train_models

Note: If you haven’t recorded your own data, running these scripts is going to throw an error. Can you see why from the error message? You can fix this by editing the scripts to use data from the user simonk rather than calling the command `whoami`. We’ll go over this in the lab, but there’s an example of this in Atli Sigurgeirsson’s extremely helpful tutor notes for this part of the assignment.

In your later experiments, you will want to automate things as much as possible. You could combine these two steps into a single script, or call them in sequence from a master script.

A finite state language model very easy to combine with HMMs. In your report, explain why this is.

The language model computes the term P(W) – the probability of the word sequence W. We’re actually going to use what appears to be a non-probabilistic model, in the form of a simple grammar. To think of this in probabilistic terms, we can say that it assigns a uniform probability to all allowed word sequences, and zero probability to all other sequences.

Why do we need a model of word sequences when we are doing isolated digit recognition?

HTK has tools to help write grammars manually, then convert them into finite state models. A grammar for isolated digits, and the corresponding finite state model, is provided for you in the resources directory, called grammar; have a look at it. The finite state model is in grammar_as_network.

The HTK manual contains all the information you need to understand the simple isolated word grammar, and how to extend that to connected digits.

Later, if you do the connected digits experiment, you will write a new grammar, from which you can automatically create a finite state network version using the HParse tool.

Now we are ready to run the recogniser on some test data. You should run this on some existing isolated test digits (one digit per wav file) and the MFCC files from them. We run the Viterbi decoder using HVite. The script recognise_test_data does this. The output is stored in the rec directory. Look at the recogniser output and compare it to the correct answers. Calculate the accuracy using the results script.

Again, you’ll need to edit the scripts to use a specific user (e.g. simonk) and the full data directory, rather than `whoami` and the data_upload directory.

The data from over 400 students is available for you to use in your experiments. You can find it in on the PPLS AT Lab servers in the directory: /Volumes/Network/courses/sp/data

In that directory, you will find a file called info.txt that describes the available data.

If you record your own data this year (which was optional), we can also (eventually) add that to the collection, and to info.txt. But, you don’t need to wait: all the data from previous years is already available.

The format of info.txt is quite simple. If you already have some coding experience, then you’ll probably want to automatically parse this file and automatically pull out subsets of speakers with the characteristics you want for each experiment. If you don’t have coding experience, try doing this in a spreadsheet.

The data are not perfect and you might want to think about how to detect bad data, so you can exclude those speakers from your experiments.

Many HTK functions use the -S option which allows you use input a script file: a text file that includes a list file names (1 per row). For example, using a script file will allow you to train your models on data from multiple speakers. See this thread for an example:

• https://speech.zone/forums/topic/training-on-a-list-of-users-intialise_models-works-but-train_models-does-not/

See this thread for an example of how to get test results from multiple speakers:

• https://speech.zone/forums/topic/how-do-i-get-the-results-for-multiple-speakers-at-once/

You can also read Atli Sigurgeirsson’s extremely helpful tutor notes for this part of the assignment.

Experimental design

You need to think carefully about each experiment, and what it is designed to discover. Do not simply run random, ill-thought-out experiments and then try to make sense of the results. Instead, form a hypothesis, and express it as a falsifiable statement, like this:

When training models only on speaker A, the word error rate will be lower for speaker A’s test data, than for test data from speaker B.

and then design an experiment to test whether that is true.

The key is to control any factors that you are not interested in, and only investigate a single factor (e.g., speakers’ gender) per experiment.

Shell scripting

A few fairly simple shell scripting techniques will help you enormously here. You can completely automate your experiments. This not only makes them more efficient for you, it also makes it easier to reproduce your results.

There are several resources in the “Intermission” lab tab if you need some help getting started with the shell, particular this intro to shell scripting. You may also find this bash primer by Jason Fong helpful!

To get started, use the already-trained models from the speaker-dependent experiment.

If you use models trained only on your own speech to recognise the Test set of another speaker, what Word Error Rate do you expect?

That will give you some clues, but isn’t a very sophisticated experiment: no real-world system would attempt to do that. Your main experiments should investigate speaker-independent systems trained and tested on larger numbers of speakers.

The testing speakers must be distinct from the training speakers. You need to control all factors that are not of interest: (including: accent, gender, microphone type, amount of training data). Here are some possible experiment designs:

1. The effect of gender, with simplistic control over accent and microphone
   • Training set: the training data of 20 female UK English speakers using headset microphones
   • Test set A: the test data of 20 female UK English speakers not in the training set, also using headset microphones
   • Test set B: the test data of 20 male UK English speakers (obviously not in the training set), also using headset microphones
2. The effect of gender, with more sophisticated control over accent and microphone (version 1)
   • Training set A: the training data of 50 female speakers, with a mixture of accents and microphones
   • Training set B: the training data of 50 male speakers, with a mixture of accents and microphones in the same proportions as training set A
   • Test set: the test data of 50 female speakers not in training set A, with a mixture of accents and microphones in the same proportions as training set A
3. The effect of gender, with more sophisticated control over accent and microphone (version 2)
   • Training set: the training data of 50 female speakers with a mixture of accents and microphones
   • Test set A: the test data of 50 female speakers not in the training set, with a mixture of accents and microphones in the same proportions as  the training set
   • Test set B: the test data of 50 male speakers, with a mixture of accents and microphones in the same proportions as the training set

Some of these designs are better than others. Can you work out the pros and cons of each design?

What effect does microphone type have?

Design an experiment to discover whether the microphone type is important. This might involve discovering if some microphones give lower Word Error Rate than others, or finding out the effect of mismatches between the Training and Test sets. Remember to control all the other factors.

You can perform equivalent experiments to investigate the gender and accent factors too.

What effect does the amount of training data have?

In machine learning, it’s often said that more training data is better. But is that always the case? Design some experiments to explore this. Include cases where the Training and Test sets are well-matched (e.g., in gender and/or accent, etc) and cases where there is mismatch. What is more important: matched training data, or just more data?

These questions are very important in commercial systems: it costs a lot of money to obtain the training data, so we want to collect the most useful data we can.

You should write a lab report about this speech recognition practical. Keep it concise and to the point, but make sure you detail your findings using HTK, including both the theory and how it is implemented in practice. You should also report your experimental work, clearly explaining your experimental design and your results. Use tables and graphs to present the results. Do not cut and paste Terminal output!

What exactly is meant by “lab report”?

It is not a discursive essay. It is not merely documentation of commands that you ran and what output you got. It is a factual report of what you did in the lab that demonstrates what you learnt and how you can relate that to the theory from lectures. You will get marks for:

• completing all parts of the practical, and demonstrating this in the report
• a clear demonstration that you understand what each HTK tool used does, and that you can link that to the underlying theory
• clear and concise writing, and effective use of diagrams, tables and graphs

How much background material should there be?

Do not spend too long simply restating material from lectures or textbooks without telling the reader why you are doing this. Do provide enough background to demonstrate your understanding of the theory, and to support your explanations of how HTK works and your experiments. Use specific and carefully chosen citations. Cite textbooks and papers. You will get more marks if you cite better and more varied sources (e.g., going beyond the essential course readings). If you only cite from the main textbook, this will not get you top marks. Avoid citing lecture slides or videos, unless you really cannot find any other source (which is unlikely). Make sure everything you cite is correctly listed in the bibliography.

Writing style

The writing style of the report should be similar to that of a journal paper. Don’t list every command you typed! You do not need to include your shell scripts. Use diagrams to illustrate your report, and tables and/or graphs to summarise your results. Do not include any verbatim output copied from the Terminal: you will not receive any marks for this.

Additional tips

You do not need to list the individual speakers in each of your data sets, but do carefully describe the data (e.g., “20 male native speakers of Scottish English using laptop microphones”). You might use tables to present this in a compact form, and perhaps gives short names or acronyms to each set, such as “20-male-SC-laptop”.

Please be sure to check general turnitin submission guidance on the PPLS hub. 

You must:

• submit a single document in PDF format. When submitting to Learn, the electronic submission title must be in the format “exam number_wordcount” and nothing else (e.g., “B012345_2864”)
• state your exam number at the top of every page of the document
• state the word count below the title on the first page in the document (e.g., “word count: 2864”)
• use a line spacing of 1.5 and a minimum font size of 11pt (and that applies to all text, including within figures, tables, and captions)
• If you work with a partner, you should also note their exam number either where you list your exam number in the title, or in a footnote from the introduction.

Marking is strictly anonymous. Do not include your name or your student number – only provide your exam number!

Structure

Length limits

• Word limit: 3000 words, excluding bibliography and text within figures and tables but including all other text (such as headings, footnotes captions, examples). Numerical data does not count as text.
  • The +10% rule applies: use it sparingly and wisely!
  • Text in figures and tables doesn’t contribute to the word count but, again,  use this wisely! Don’t just shove things into tables because they don’t fit in your text!
  • Note: You should assume that the markers will only read up to the word limit (i.e. 3300 words).  We have to enforce this because some people have submitted assignments that were hugely over the word limit and it’s not fair on the markers to ask them to read so much over the word count given that we’ve already set a 10% leeway.
• Page limit: no limit, but avoid blank pages
• Figures & tables: no limit on number

Sections and headings

You must use the following structure and section numbering for your report. It corresponds to the structured marking scheme and will make your marker’s life so much easier!

• 1 Introduction
• 2 Theory
  • 2.1 Data collection and acoustic features
  • 2.2 Training HMMs
  • 2.3 Language modelling
  • 2.4 Recognition using HMMs
• 3 Experiments
  • 3.1 [Insert your first experiment title here]
  • 3.2 [Insert your second experiment title here]
  • 3.3 etc, for as many experiments as you wish
• 4 Discussion and overall conclusion

You should describe your speaker-dependent isolated digit system in Sections 2.1 to 2.4. As you don’t have to record and label your own data this year, you can describe this conceptually. If you want to use a concrete example to help illustrate, you can reference a speaker-dependent system built using an existing speakers data set (e.g. ‘simonk’). Although data collection is optional this year, you still need to describe what data are required, how they need to be labelled and so on – the marks for this are under “2.1 Data collection and acoustic features”.

You are advised to structure each experimental section (3.1, 3.2, etc) by introducing a hypothesis, describing your experimental design, reporting the results, and drawing a conclusion. You should aim to write up 3-4 experiments. You may be able to fit more in, but don’t forget you need to motivate each experiment and analyze the results. It’s fine to design your experiments based on the suggestions given in the instructions (e.g. gender, microphone type, amount of training data), but you may find there are more interesting things to explore.  Experimental designs that allow you to draw conclusions across different experiments are not required, but they are definitely appreciated.

Figure, graphs and tables

You should ensure that figures and graphs are large enough to read easily and are of high-quality (with a very strong preference for vector graphics, and failing that high-resolution images). You are strongly advised to draw your own figures which will generally attract a higher mark than a figure quoted from another source.

There is no page limit, so therefore there is no reason to have very small figures.

Your work may be marked electronically or we may print hardcopies on A4 paper, so it must be legible in both formats. In particular, do not assume the markers can “zoom in” to make the figures larger.

Marking scheme

You are strongly advised to read the marking scheme because it will help you focus your effort and decide how much to write in each section of your report.

Here is the structured marking sheet for this assignment (this is the version for 2023-24, which is the same as previous years). Although data collection is optional this year, you still need to describe what data are required, how they need to be labelled and so on – the marks for this are under “2.1 Data collection and acoustic features”. If you did collect your own data, you may obtain extra marks under the “Additional” section.