Building a Statistical Language Model

Update: I finished my script for creating custom language models.  See here: https://github.com/umhau/vmc.

There's a summary at the end with what I figured out.  Most of this is me thinking on paper.

The statistical language model is used for helping CMU Sphinx know what words exist, and what the order the words exist in (the grammar and syntax structure).  The intro website to all this is here.

I'm trying to decide between the SRILM and the MITLM packages [subsequent edit: also the logios package and the quicklm pearl script - these are referenced in hard-to-find places on the CMU website; see here and here, respectively] [another subsequent edit: looks like I found a link to the official CMU Statistical Language Model toolkit - it was buried in the QuickLM script].  S- is easier to use, apparently, and the CMU site provides example commands.  M-, however, seems more likely to stick around and be accessible on github for the long-term.  Plus, I forked it.

[sorry, blogger's formatting broke and I had to convert everything to plaintext and start over...lost the links.]

Only downside is, the main contributor to MITLM stopped work on it about 6 mos ago, and started dealing with Kaldi instead.  Guess he figured the newer tech was more worth his time.  Still, dinosaurs have their place; just watch Space Cowboys to get the picture.

MITLM

Just to be sure that the software doesn't go anywhere, code is downloaded from my repository.

Update: Thanks to Qi Wang's comment below there's an extra dependency to install:

sudo apt-get install autoconf-archive

Installation of MITLM:

cd ~/tools
git clone https://github.com/umhau/mitlm.git
cd ./mitlm
./autogen.sh
./configure
make
make install

So, turns out that there's some weird problems with the installation.  Something changed, or something isn't being installed properly.  The compilation seems to fail with these errors:

./configure: line 19641: AX_CXX_HEADER_TR1_UNORDERED_MAP: command not found
./configure: line 19642: syntax error near unexpected token `noext,'
./configure: line 19642: `AX_CXX_COMPILE_STDCXX_11(noext, optional)'

g++ wasn't installed, but even after that was added it still wouldn't work.

Update: Unfortunately, I've lost track of other dependencies involved - at some point, I'll make a list of all the stuff I've installed while working on this project.  Had to install libtool (or similar?) to get here.  Mental note:

libtoolize:   error: Failed to create 'build-aux'

But, that's because I'm trying to do this on a different Mint installation from my usual - on my default workstation, that dependency is installed (no idea what it is, except that it's probably listed somewhere on this blog).

After installing the extra dependency, the installation works! So this is a viable avenue thus far to get the LM working.  I've already made it past where I need the MITLM, though, so I'm going to let it be for now.  Might have to come back for it.

SRILM

Ok, let's see what SRILM has to offer us. It's more inconvenient to install; ya have to go through a license agreement to download it, so I can't just stick a bash command here.

...unless I put the code on my github.  In which case, it's easy to get a copy of.  Too bad there's too many files to put up an extracted version, and too bad the compressed version is more than 25mb.  Time to split up the tar.gz file again; for my own records, here's how I split it.  All I need for getting and using it is the reconstruction bit.

The splitting part, given the archive file:

split -b 24m -d srilm-1.7.1.tar.gz srilm-1.7.1.tar.gz.part-

Alright. Once the file is on github, it's just more copy-pasting.

cd ~/tools
git clone https://github.com/umhau/srilm.git
cd ./srilm
cat srilm-1.7.1.tar.gz.part-* | tar -xz

By the way, WOW.  The installation process for this software is not straightforward.  See the install file for the instructions on installation - read for background, then copy-paste below as usual.

gedit ./INSTALL

Step 2 - swap out the SRILM variable for one delimiting the root directory of the package. Source.

sed -i '7s#.*#SRILM = ~/tools/srilm#' ./Makefile

For now, assuming that the variables are all good.  I don't know if I want maximum entropy models, though it sounds useful...I'll see what happens if I don't prep them.

Installing John Ousterhout's TCL toolkit - we're past the required v7.3, and up to 8.6: hope this still works.  I'm compiling from source rather than using the available binaries 'cause they come with some kind of non-commercial/education license, which I don't like being tied down by.

cd ~/tools
git clone https://github.com/umhau/tcl-tk.git
cd ./tcl-tk
gunzip < tcl8.6.6-src.tar.gz | tar xvf -
gunzip < tk8.6.6-src.tar.gz | tar xvf -

Install TCL:

cd tcl8.6.6/unix
# chmod +x configure
configure --enable-threads
make -j 3
make test 
sudo make -j 3 install

Let's try running the rest without the TK stuff...even though John says it's needed.  Heh.  Leeeroooy Jenkins!

cd ../../../srilm
make World

...aaaaaaaand, Fail.

This is going nowhere fast.  We're in dependency hell.  Let's try the perl script CMU uses (it's the backend to the online service they officially reference).

The Perl Script

Thankfully, Mint comes with perl installed.  So, the question is how to use the script.

cd ~/tools
mkdir ./CMU_LMtool && cd ./CMU_LMtool
wget http://www.speech.cs.cmu.edu/tools/download/quick_lm.pl

The only thing left here is to figure out how to use the script...having never used perl, this could be interesting.  Dug this nugget out of the script:

usage: quick_lm -s <sentence_file> [-w <word_file>] [-d discount]

So, the idea with the LMtool is to process sentences that the decoder should recognize - it doesn't need to be an exhaustive list, however, because the decoder will allow fragments to recombine in the detection phase.  As a corpus example (from the CMU website), here's the following:

THIS IS AN EXAMPLE SENTENCE
EACH LINE IS SOMETHING THAT YOU'D WANT YOUR SYSTEM TO RECOGNIZE
ACRONYMS PRONOUNCED AS LETTERS ARE BEST ENTERED AS A T_L_A
NUMBERS AND ABBREVIATIONS OUGHT TO BE SPELLED OUT FOR EXAMPLE
TWO HUNDRED SIXTY THREE ET CETERA
YOU CAN UPLOAD A FEW THOUSAND SENTENCES
BUT THERE IS A LIMIT

We'll use this sentence collection to test the perl script:

cd ~/tools/CMU_LMtool
wget https://raw.githubusercontent.com/umhau/misc-LMtools/master/ex-corpus.txt
perl quick_lm.pl -s ex-corpus.txt

Well, it did exactly nothing.  No terminal output, no new files created in the directory, and no errors.  Time to search the script for other possible output locations.  How weird can it be?

...

Ok, solved the problem.  Thank goodness for auto highlighting in Gedit.  The authors used some kind of weird system for comments that I'm guessing was retired since this script was written.  It seems to have been throwing the compiler for a loop:

=POD
/*
[some text wrapped by those comment markers]
*/
[more text, only wrapped by the '=' things]
=END

So, I re-commented all the introductory stuff, and put the fixed version in the github repo.

Summary of the Perl script

So, here's how it works: download the fixed script, give it a sentence list, and run the command.  Simple.  And, looking at the output, the function it performs is pretty simple too.  Makes a list of all the 1, 2 and 3 - word groupings in the list.

Here's what to do:

mkdir ~/tools/CMU_LMtool && cd ~/tools/CMU_LMtool
wget https://raw.githubusercontent.com/umhau/misc-LMtools/master/ex-corpus.txt
wget https://raw.githubusercontent.com/umhau/misc-LMtools/master/quick_lm.pl
perl quick_lm.pl -s ex-corpus.txt

Still not sure what that does for me, but I have my LM!

Notes: I think the word list option in the command refers to the possibility of a limited vocabulary...not sure how that relates to words outside that list used in the sentence list.  The discount in the command, however, is fixed at 0.5.  Apparently Greg and Ben did some experiments to discover that's definitely the optimal setting.

Second Note:  based on readings from the CMU website, this LM isn't good for much more than command-and-control - it can successfully detect short phrases accurately, but not long, drawn-out sentences.  So it'll be good for most of what I want, but anything complex will need to be done with the CMULMTK package.

Hold on - the [-w <word_file>] option for a dictionary might be a request for output - not an extra input.  And given that I do need an explicit dictionary for transcription, that's probably what it does.  That would be wonderful.  I can even use that sentence list for voice training - which would be a fabulous way to ensure accuracy.

Unfortunately, that's not the case.  Oh, well.

The official CMU Statistical Language Model toolkit

Ok, maybe this'll do it for me.  Here's the link to the source.  The Perl script doesn't make all the different files I need - especially the pronunciation dictionary.

mkdir ./tools/CMUSLM
cd ./tools/CMUSLM
wget http://www.speech.cs.cmu.edu/SLM/CMU-Cam_Toolkit_v2.tar.gz
gunzip < CMU-Cam_Toolkit_v2.tar.gz | tar xv
cd ./CMU-Cam_Toolkit_v2

Wow, this is old.  You have to uncomment something if your computer isn't running HP-UX, IRIX, SunOS, or Solaris.  I'm pretty sure anything build in this decade needs uncomment, but if you're unsure the README mentions a script you can run to check for yourself:

bash endian.sh

Ok, uncomment:

sed -i '37s/#//' ./src/Makefile
cd src
make install

Hard to tell if this was successful.  I get the impression watching this compile that it was written in the 80s, and updated for compatibility with something advertising a max capacity of 512 Mb of random access memory.

Time to dive into the html documentation, and figure out usage.  The goal is to create the LM and DIC files - and a nice perk would be the other stuff produced by the online LM generator.

Turns out, there doesn't seem to be any kind of pronunciation dictionary produced by this tool.  So it's no good.

The Logios Package

This seems to be the tool CMU claims was actually used in their website - and, indeed, some of their tools within the package are designed for use in a webform.  So I might be on the right track.  The only problem is, the input is not a list of sentences: it's a grammar file built by the Phoenix tool. No idea what that is or how it works.

CMU, get your act together!  The website is nice, but I've got no recourse if it goes down.  I want an independent system!

Here goes.  Goal: LM and DIC files.  Starting point: list of sentences.

Download the package.  Even this isn't user-friendly - the folder structure is in html.  I used wget recursively to download the webpages.  See here for source on the command.

CMUDict

Actually, it seems like I could just use the dictionary directly.  The whole problem is one of how to get the entries from this file into a subset file that holds just what I want - so I'll just write a small script to do just that.  What a pain.  

wget http://svn.code.sf.net/p/cmusphinx/code/trunk/cmudict/sphinxdict/cmudict_SPHINX_40

I'll post the script soon - it's being added to a larger package that should make the process of getting a personal language model pretty painless.  That'd be nice.

[shameless copy] Offline Language Model Creation for PocketSphinx

Normally, I'd be writing these myself.  But this time, the explanation was so unusually good that I don't feel the need to simplify it.  It's fantastic for my purposes as-is.  Source.  

The purpose here is to create the statistical language model that pocketsphinx uses to convert phonetics into words.  The model is based entirely on what type of sentences it expects to encounter, as defined by the input reference text.  

I need this running as a self-contained script in order to make language model generation a seamless part of my project.  All the user should have to do is provide a ready-made reference text, and the script should generate the rest.  

ARPA model training with CMUCLMTK

You need to download and install cmuclmtk. See CMU Sphinx Downloads for details.

The process for creating a language model is as follows:

1) Prepare a reference text that will be used to generate the language model. The language model toolkit expects its input to be in the form of normalized text files, with utterances delimited by <s> and </s> tags. A number of input filters are available for specific corpora such as Switchboard, ISL and NIST meetings, and HUB5 transcripts. The result should be the set of sentences that are bounded by the start and end sentence markers: <s> and </s>. Here's an example:

<s> generally cloudy today with scattered outbreaks of rain and drizzle persistent and heavy at times </s>
<s> some dry intervals also with hazy sunshine especially in eastern parts in the morning </s>
<s> highest temperatures nine to thirteen Celsius in a light or moderate mainly east south east breeze </s>
<s> cloudy damp and misty today with spells of rain and drizzle in most places much of this rain will be 
light and patchy but heavier rain may develop in the west later </s>

More data will generate better language models. The weather.txt file from sphinx4 (used to generate the weather language model) contains nearly 100,000 sentences.

2) Generate the vocabulary file. This is a list of all the words in the file:

    text2wfreq < weather.txt | wfreq2vocab > weather.tmp.vocab

3) You may want to edit the vocabulary file to remove words (numbers, misspellings, names). If you find misspellings, it is a good idea to fix them in the input transcript.

4) If you want a closed vocabulary language model (a language model that has no provisions for unknown words), then you should remove sentences from your input transcript that contain words that are not in your vocabulary file.

5) Generate the arpa format language model with the commands:

% text2idngram -vocab weather.vocab -idngram weather.idngram < weather.closed.txt
% idngram2lm -vocab_type 0 -idngram weather.idngram -vocab \
     weather.vocab -arpa weather.lm

6) Generate the CMU binary form (BIN)

sphinx_lm_convert -i weather.lm -o weather.lm.bin

The CMUCLTK tools and commands are documented at The CMU-Cambridge Language Modeling Toolkit page.

Using PocketSphinx within Python Code

Here's the source for what I've been working on.

Looks like my installation records will have to be updated to account for a different installation source, and maybe a different version of the source code.

Ok, here's the process so far.  Install sphinxbase and pocketsphinx from GitHub - this means using the bleeding-edge versions, rather than the tried-and true alpha5 versions that I talked about in previous posts.  This just seems to work better.  Once this is all figured out, I'll go back and clean those up.

cd ~/tools
git clone https://github.com/cmusphinx/sphinxbase.git
cd ./sphinxbase
./autogen.sh
./configure
make
make check
make install

cd ~/tools
git clone https://github.com/cmusphinx/pocketsphinx.git
cd ./pocketsphinx
./autogen.sh
./configure
make clean all
make check
sudo make install

Now look inside the pocketsphinx directory:

cd ~/tools/pocketsphinx/swig/python/test

There's a whole bunch of test scripts that walk you through the implementation of pocketsphinx in python.  It's basically done for you.  Check the one called kws-test.py -- that's the one that will wait to hear a keyword, run a command when it does, then resume listening.  Perfect!

I'm going to assume that you've already created your own voice model based on the other posts in this blog, and that you've got a directory dedicated to command and control experiments.

If that's not true, then just mess with the script without moving it.  Just make a backup.  The only effective difference is that the detection will be less accurate; for the purposes of this tutorial, ignore the rest of the code down to where I've pasted my copy of the python script.  The only thing you should change has to do with reading from the microphone rather than an audio file; change the script to match what I've got here.  You're done now.  The rest of this tutorial is for those who have already created their own voice model.  See others of my posts for how to do that.

# Open file to read the data
# stream = open(os.path.join(datadir, "test-file.wav"), "rb")

# Alternatively you can read from microphone
import pyaudio
 
p = pyaudio.PyAudio()
stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=1024)
stream.start_stream()

Ok.  For the rest of us, let's get back to messing with this script.  While still in the test directory,

mkdir ~/tools/cc_ex
cp ./kws_test.py ~/tools/cc_ex/kws_test.py
cd ~/tools/cc_ex/
gedit kws_test.py

There's a few changes to make in the python script.  Make sure the model directory has been adjusted.  Also, the script by default is checking in a .raw audio file for the keyword: uncomment and comment the relevant lines so the script uses pyaudio to record from the microphone.  The full text of my version of the script is below.

Note that the keyphrase it's looking for is the word 'and'.  Pretty simple, and very likely to have been covered a lot in the voice training.

Note also that there's a weird quirk in the detection - you have to speak quickly.  I tried for a long time making long, sonorous 'aaaannnnddd' noises at my microphone, and it didn't pick up.  Finally gave a short, staccato 'and' - it detected me right away.  Did it five more times, and it picked me up each time.  I don't see a way to get around that - I think it's built into the buffer, so it won't even hear the whole thing otherwise.  Or maybe I just said 'and' in the training really fast each time, though I don't think that's likely.

#!/usr/bin/python

import sys, os
from pocketsphinx.pocketsphinx import *
from sphinxbase.sphinxbase import *


modeldir = "~/tools/train-voice-data-pocketsphinx"

# Create a decoder with certain model
config = Decoder.default_config()
config.set_string('-hmm', os.path.join(modeldir, 'neo-en/en-us'))
config.set_string('-dict', os.path.join(modeldir, 'neo-en/cmudict-en-us.dict'))
config.set_string('-keyphrase', 'and')
config.set_float('-kws_threshold', 1e+1)
#config.set_string('-logfn', '/dev/null')


# Open file to read the data
# stream = open(os.path.join(datadir, "test-file.wav"), "rb")

# Alternatively you can read from microphone
import pyaudio
 
p = pyaudio.PyAudio()
stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=1024)
stream.start_stream()

# Process audio chunk by chunk. On keyphrase detected perform action and restart search
decoder = Decoder(config)
decoder.start_utt()
while True:
    buf = stream.read(1024)
    if buf:
         decoder.process_raw(buf, False, False)
    else:
         break
    if decoder.hyp() != None:
        print ([(seg.word, seg.prob, seg.start_frame, seg.end_frame) for seg in decoder.seg()])
        print ("Detected keyphrase, restarting search")
        decoder.end_utt()
        decoder.start_utt()

Anyway, that's all.  If it doesn't work, don't blame me.  That's as dead simple as I know how to make it.

Training a CMU Sphinx Language Model for Command and Control

CMU Sphinx is advanced enough to use its understanding of grammar to help it figure out the likelihood that a particular word was spoken.  To do this, it needs to have a predefined concept of which words tend to follow each other -- it needs to understand the format of what is spoken to it.  The context of a 'command and control' AI has a very specific type of grammar involved, where the format is predominately commands and statements.

If CMU Sphinx has been made to recognize that, it will be able to filter words that don't make sense in that context and weight more heavily words that do make sense as control words: it will know that 'play music' is more likely than 'pink music', and 'shutdown' is more likely to be a command than 'showdown'.

For now, here's the primary sources:
http://cmusphinx.sourceforge.net/wiki/tutoriallm
http://www.speech.cs.cmu.edu/tools/lmtool-new.html

Using this, it should be possible to create the grammar language model based on a big list of sentences; only problem is, I don't have a sentence list.  Once that LM has been created, the voice data I've created should be retrained - even that is done based on grammar statistics.

Improving the Accuracy of CMU Sphinx for a Limited Vocabulary

Update: I finished my tool for creating a customized voice model.  It encapsulates the best of what I described below.  See here: https://github.com/umhau/vmc.

The idea with a limited vocabulary is that the processor can deal with far less information in order to detect the words needed. You don't have to train it on a complete set of words in the English language, and you don't need a supercomputer.  All you have to do is teach it a few words, and how to spell them.  The tutorial is here.  I've created a script to automate the voice recording here, and stashed the needed files there with it.

Preparation

Alright, down to business.  You'll find it handy to keep a folder for these sorts of programs.

mkdir ~/tools
cd ~/tools

Install git, if you don't have it already.

sudo apt-get install git

Download the script I made into your new tools folder.

sudo git clone https://github.com/umhau/train-voice-data-pocketsphinx.git

Install SphinxTrain.  I included it among the files you just downloaded.  Move it up to ~/tools, extract and install it.  It's also here, if you don't want to use the one I provided.

sudo mv ~/tools/train-voice-data-pocketsphinx/extra_files/sphinxtrain-5prealpha.tar.gz ~/tools
sudo tar -xvzf ~/tools/sphinxtrain-5prealpha.tar.gz -C ~/tools
cd sphinxtrain-5prealpha
./configure
make -j 4
make -j 4 install

Record Your Voice

Enter this directory, run the script.  It'll have a basic walkthrough built-in.  This will help you record the data you need.  For experimental purposes, 20 recordings is enough for about 10% relative improvement in accuracy.  Use the name neo-en for your training data, assuming you're working in English.

cd ./train-voice-data-pocketsphinx
python train_voice_model.py

You'll find your recordings in a subfolder with the same name as what you specified.  Go there.

cd ./neo-en

By the way, if you ever change your mind about what you want your model to be named, there's a fantastic program called pyrenamer that can make it easy to rename all the files you created.  Install it with:

sudo apt-get install pyrenamer

Process Your Voice Recordings

Great!  Done with that part.  Now we're going to copy some other directories into the current working directory to 'work on them'.

cp -a /usr/local/share/pocketsphinx/model/en-us/en-us .
cp -a /usr/local/share/pocketsphinx/model/en-us/cmudict-en-us.dict .
cp -a /usr/local/share/pocketsphinx/model/en-us/en-us.lm.bin .

Based on this source, it looks like we shouldn't be working with .dmp files.   This is a point of deviation from the (outdated) CMU walkthrough.  Copy the .bin file instead.  Difference is explained below, sourced from the tutorial.

Language model can be stored and loaded in three different format - text ARPA format, binary format BIN and binary DMP format. ARPA format takes more space but it is possible to edit it. ARPA files have .lm extension. Binary format takes significantly less space and faster to load. Binary files have .lm.bin extension. It is also possible to convert between formats. DMP format is obsolete and not recommended.

Now, while still in this directory, generate some 'acoustic feature files'.

sphinx_fe -argfile en-us/feat.params -samprate 16000 -c neo-en.fileids -di . -do . -ei wav -eo mfc -mswav yes

Get the Full-Sized Language Model

Nice.  You have a bunch more files with weird extensions on them.  Now it's time to convert them.  You need the full version of the language model, which was not shared with your original installation for size reasons.  I included it in the github repository, or you can download it from here (you want the file named cmusphinx-en-us-ptm-5.2.tar.gz).  Put the extracted files in your neo-en directory.

Assuming you use the one from the github repo and you're still in the neo-en subdirectory,

tar -xvzf ../extra_files/cmusphinx-en-us-ptm-5.2.tar.gz -C .

There's an folder labeled en-us within the neo-en folder that was created when you made the acoustic feature files.  Give it an extension and save it in case of horrible mistakes.

mv ./en-us ./en-us-original

Now move the newly extracted directory to your neo-en folder, and rename it to en-us.

mv ./cmusphinx-en-us-ptm-5.2 ./en-us

This converts the binary mdef file into a text file.

pocketsphinx_mdef_convert -text ./en-us/mdef ./en-us/mdef.txt

Grab Some Tools

Now you need some more tools to work with the data.  These are from SphinxTrain, which you installed earlier.  You should still be in your working directory, neo-en.  Use ls to see what tools are available in the directory.

ls /usr/local/libexec/sphinxtrain
cp /usr/local/libexec/sphinxtrain/bw .
cp /usr/local/libexec/sphinxtrain/map_adapt .
cp /usr/local/libexec/sphinxtrain/mk_s2sendump .
cp /usr/local/libexec/sphinxtrain/mllr_solve .

Run 'bw' Command to Collect Statistics on Your Voice

Now you're going to run a very long command that is designed to collect statistics about your voice.  Those backslashes -- the \ things -- tell bash to ignore the following character: in this case, newline characters.  That's how this command is stretching over multiple lines.

./bw \
 -hmmdir en-us \
 -moddeffn en-us/mdef.txt \
 -ts2cbfn .ptm. \
 -feat 1s_c_d_dd \
 -svspec 0-12/13-25/26-38 \
 -cmn current \
 -agc none \
 -dictfn cmudict-en-us.dict \
 -ctlfn neo-en.fileids \
 -lsnfn neo-en.transcription \
 -accumdir .

Future note, for using the continuous model instead of the PTM model (from the tutorial):

Make sure the arguments in bw command should match the parameters in feat.params file inside the acoustic model folder. Please note that not all the parameters from feat.param are supported by bw, only a few of them. bw for example doesn't suppport upperf or other feature extraction params. You only need to use parameters which are accepted, other parameters from feat.params should be skipped. 

For example, for continuous model you don't need to include the svspec option. Instead, you need to use just -ts2cbfn .cont. For semi-continuous models use -ts2cbfn .semi. If model has `feature_transform` file like en-us continuous model, you need to add -lda feature_transform argument to bw, otherwise it will not work properly.

More Commands

Now it's time to adapt the model.  Looks like continuous will be better to use in the long run, but first we're just going to get this working.  The tutorial suggests that using MLLR and MAP adaptation methods together is best, but it looks like so far we're just using them sequentially.  Here goes:

./mllr_solve \
 -meanfn en-us/means \
 -varfn en-us/variances \
 -outmllrfn mllr_matrix -accumdir .

It appears this adapted model is now completed!  Nice work.  To use it, add -mllr mllr_matrix to your PocketSphinx command line.  I'll put complete commands at the bottom of this note.  

Now we're going to do the MAP adaptation method, which is being used on top of the MLLR method.  Back up the files you were just working on:

cp -a en-us en-us-adapt

To run the MAP adaptation:

./map_adapt \
 -moddeffn en-us/mdef.txt \
 -ts2cbfn .ptm. \
 -meanfn en-us/means \
 -varfn en-us/variances \
 -mixwfn en-us/mixture_weights \
 -tmatfn en-us/transition_matrices \
 -accumdir . \
 -mapmeanfn en-us-adapt/means \
 -mapvarfn en-us-adapt/variances \
 -mapmixwfn en-us-adapt/mixture_weights \
 -maptmatfn en-us-adapt/transition_matrices

[Optional; saves some space]

...I think.  Apparently it's now important to recreate a sendump file from a newly updated mixture_weights file.

./mk_s2sendump \
 -pocketsphinx yes \
 -moddeffn en-us-adapt/mdef.txt \
 -mixwfn en-us-adapt/mixture_weights \
 -sendumpfn en-us-adapt/sendump

Testing the Model

It's also important to test the adaptation quality.  This actually gives you a benchmark - a word error rate (WER).  See here.

Create Test Data

Use another script I made to record test data.  It's almost the same, but the fileids and transcription file formats are different.  The folder with the test data should end up in the neo-en directory. Use the directory name I provide, test-data.

python ../create_test_records.py

Run the decoder on the test files.  Go back into the neo-en folder.

pocketsphinx_batch \
 -adcin yes \
 -cepdir ./test-data \
 -cepext .wav \
 -ctl ./test-data/test-data.fileids \
 -lm en-us.lm.bin \
 -dict cmudict-en-us.dict \
 -hmm en-us-adapt \
 -hyp ./test-data/test-data.hyp

Use this tool to actually test the accuracy of the model.  You'll need a working pocketsphinx installation, since it's just a wrapper with a word comparison engine over the transcription engine.  Look at the end of the output; it'll give you some percentages indicating accuracy.

../../pocketsphinx-5prealpha/test/word_align.pl \
 ./test-data/test-data.transcription \
 ./test-data/test-data.hyp

Live Testing

If you just want to try out your new language model, record a file and try to transcribe it with these commands (assuming you're still in the neo-en working directory):

python ../record_test_voice.py
pocketsphinx_continuous -hmm ./en-us-adapt -infile ../test-file.wav

Or, if you'd rather use a microphone and record live, use this command:

pocketsphinx_continuous -hmm ./en-us-adapt -inmic yes

With 110 voice records and using 20 records as testing, I achieved 60% accuracy. At 400 records and a marginal mic, I achieved 77% accuracy.  There's about 1000 records available.

Achieving Optimal Accuracy

You'll want to create your own language model if you're going to be using a specialized language. That's a pain, and you have to know what you're going to use it for ahead of time.  If I do that, I'll collect the words from the tools where I specified them and automagically rebuild the language model.  For now, I think I can get away with using the default lm.

For actual use of the model, everything you need is in en-us-adapt.  That's what you use when you need to refer in a command to your language-model.

Use the following command to transcribe a file, if you've created your own lm and language dictionary:

pocketsphinx_continuous 
 -hmm <your_new_model_folder> \
 -lm <your_lm> \
 -dict <your_dict> \
 -infile test.wav

Upon testing it appears that a less controlled environment might be useful, as the transcription was almost perfect when I was able to recreate the atmosphere of the original training records and pretty bad otherwise.

Conclusions

I've made a bunch of scripts in the github repo that automate some of this stuff, assuming standard installs.  Look for check_accuracy.sh and create_model.sh.  Everything should be run inside the neo-en folder, except the original train_voice_model.py script.

TODO next -

1. set up the more accurate continuous model
2. create a script that generates words in faux-sentences based on my use case scenario.  
1. find a phonetic dictionary that covers my needs
2. figure out what my use case actually is

Speech Recognition Final Verdict

Final strategy: I'm going to use CMU Sphinx with a small vocabulary trained to my voice for most commands.  I'll use the kaldi-gstreamer-server, or maybe even an online service, for larger, arbitrary pieces of sound - stuff that I can't predict.

Which means that I'll have two separate, behemoth systems installed on the computer.  Ouch.  At least I can stream Kaldi from a different computer.  Sphinx should be small enough to not be a problem.

Here's what I need to be able to train the command and control language model.