Student StarCraft AI Tournament Tutorial for Getting Started with C++

The Student Starcraft AI tournament is an ongoing competition to write a bot to compete in playing 1v1 games of the real-time strategy game Starcraft. I found the documentation on how to get started to be scattered and lacking, and I hope to consolidate the knowledge I gained here into a mini-tutorial that will lower the barrier for entry.

PART 1 - Running the Example AI

First, you will need Starcraft Brood War patched to version 1.16.1 installed (You do not need the CD to run the game if you follow the instructions in the patch notes for version 1.15.2).

Next, you will need to download and extract BWAPI version 3.7.4 which is  not the latest version, however it is the version used by the SSCAI tournament.  I found that the included installer does not work, as confirmed by bug reports which were addressed in later versions.  However, later versions do not help us, as we need version 3.7.4 for the tournament.  Once extracted, the readme says

2. Run "install.exe". This will do the following:
    * Copies the contents of ./WINDOWS/ to the C:/Windows folder.
    * Copies the contents of ./Starcraft/ to your actual Starcraft folder.
    * Performs additional minor installation steps.

so, assuming the "additional minor installation steps" are nothing of importance, we can manually install BWAPI by copying the files according to the above instructions.  (This is what I did, and everything seems to have worked for me).

Next, you  need to install Visual Studio 2008 (the version is important, as later versions will give you errors when you try to compile).  You don't need to buy the full version, as Visual C++ 2008 Express Edition works fine, and is freely available for download.

When Visual Studio is installed, we will try using it to compile the example project and test it.  To do so, open up ExampleProjects.sln from the root directory of BWAPI in Visual Studio, and select Build>Build Solution from the menu bar.  When this is completed, check the output tab at the bottom for build errors (there should be none, if there are any errors, good luck!) .

To run the example AI module

1. Copy Release/ExampleAIModule.dll to
   <starcraft folder>/bwapi-data/AI/ExampleAIModule.dll

2. Open <starcraft folder>/bwapi-data/bwapi.ini and set ai to
   bwapi-data/AI/ExampleAIModule.dll if it isn't already.
 
3. Run Chaoslauncher with BWAPI Injector checked. You can also check W-MODE
   to have Broodwar run in a window rather than full-screen.

For the second step, this simply means editting the line in bwapi.ini which starts with

ai     =

to say

ai     =  bwapi-data/AI/ExampleAIModule.dll

You can also choose to build the debug version in Visual Studio.  In that case, you would copy the dll from Debug/ExampleAIModule.dll and modify the line starting with ai_dbg instead.  When starting up starcraft, you will have an option to choose between running debug mode or not, and the AI from the appropriate DLL will be loaded.  Nothing else in the bwapi.ini file needs to be edited for now.

Chaoslauncher can be found in the BWAPI main directory under Chaoslauncher/Chaoslauncher.exe.  When you open it, you can see the choice of checking off the BWAPI DEBUG or RELEASE options.  Check off just one, and optionally W-MODE (I suggest using debug mode and W-MODE), and click start.  You can safely ignore a warning about admin privileges if it pops up.  From here, you just need to get into a Starcraft game, and BWAPI should take over with the example AI.  The quickest way is to select a single-player expansion custom game and start the game.  The Example AI module should print some debug text to the screen and issue some basic commands to your probe and command center. If this happens, you have successfully completed the first step on getting your own BWAPI bot to run.

PART 2 - Creating your own AI Project

Installing Steam Linux Beta on 64 bit Debian

Valve recently came out with a closed beta for Steam on Linux.  For the initial beta, they are only supporting Ubuntu.   Luckily, people on reddit and other places have found workarounds to install the beta client if you're not part of the beta, and on other distros besides Ubuntu.  On amd64 Debian, there's the additional problem that the client is only 32 bit.

Anyone can download the .deb that valve released at http://media.steampowered.com/client/installer/steam.deb. unfortunately for us this is a 32-bit Ubuntu package, and it isn't even a clean deb file: it will try to install additional dependencies when you run it for the first time.  We basically want to ignore any dependency fetching it does and manually install all the dependencies on our own.

First, we just want to force the install of a 32-bit package on the wrong architecture.  To do this we run

$sudo dpkg --force-architecture -i steam.deb

 from the terminal.  It will probably complain about dependencies;  ignore this.  We can't install all the required libraries through the debian package system, so instead we create a folder in your home directory to contain the alternate versions of these libraries.  Luckily steam user cyb.org has put together a script which installs all the required libraries.  The script can be found at http://dl.dropbox.com/u/29081229/Steam/debian_install.sh. The full contents of the script are

#!/bin/sh
STEAMLIBS=${HOME}/Steamlibc/
mkdir -p ${STEAMLIBS}
cd /tmp/
wget http://security.ubuntu.com/ubuntu/pool/main/e/eglibc/libc6_2.15-0ubuntu10.2_i386.deb
dpkg -x libc6_2.15-0ubuntu10.2_i386.deb /tmp/libc/
mv /tmp/libc/lib/i386-linux-gnu/* ${STEAMLIBS}
wget http://mirror.ovh.net/ubuntu//pool/main/j/jockey/jockey-common_0.9.7-0ubuntu7_all.deb
wget http://mirror.ovh.net/ubuntu//pool/main/x/x-kit/python-xkit_0.4.2.3build1_all.deb
sudo dpkg -i jockey-common_0.9.7-0ubuntu7_all.deb python-xkit_0.4.2.3build1_all.deb
sudo apt-get install libjpeg8 libcurl3-gnutls libtheora0 libpulse0 libpixman-1-0 libcairo2 libgdk-pixbuf2.0-0 libgtk2.0-0 libpango1.0-0

 in case that link changes its contents (maliciously or otherwise).  However, because a required library for steam is installed in your home directory, you need to launch the steam client with a special command line option.  cyb.org provides the script http://dl.dropbox.com/u/29081229/Steam/debian_steam.sh

#!/bin/sh
STEAMLIBS=${HOME}/Steamlibc/
LD_LIBRARY_PATH=${STEAMLIBS} /usr/bin/steam $@

to launch steam (the $@ is my addition so that it passes through arguments to the steam script). run

$chmod +x steam_debian.sh

to make it executable.  At this point, running the script may launch steam for you. Running

$./debian_steam.sh steam://open/games

 supposedly bypasses the beta check.  For me it popped open a dialog to download an update for steam, but after steam is updated, it won't do anything else.  You can easily run it under a debugger like so:

$DEBUGGER=gdb ./debian_steam.sh

For me doing so revealed that it was quitting with an exit code 0377 (which is equivalent to error code -1).

FFMpeg Dnxhd: Could not write header for output file #0

Doing some video editing on linux, I wanted to convert the videos to a format easy to edit (and deinterlaced). I found this blog post: http://www.itc4u.net/it-chalk-board/67-linux-app-ffmpeg/115-edit-avchd-files-mts-with-blender-or-cinelerraconvert-mts-to-dnxhd-mov.html

Which seemed great, except when I ran the command he gave, I got the error "Could not write header for output file #0". (Stream 0 being the video stream). A bit cryptic. I eventually managed to figure out that this is due to the container format. Specifically, I was trying to encode it into an AVI. Apparently the Dnxhd encoder doesn't like this. If you encode it to a .mov, it works perfectly.

Project Schliemann/ GLF/ Generic Language Framework with Netbeans

I've been trying for some time to get editing support in Netbeans for my custom language. There are many different APIs for doing this, depending on the version of Netbeans and the level of integration you want. Unfortunately, for my purposes there was only one API which was not ridiculously over-complex, called Project Schliemann (or alternately ".nbs files", or the "Generic Languages Framework"). The basic idea is that you write a single file with the grammar and rules about your language, and a generic parser reads that file and then can do some basic syntax checking, etc. on your code. Unfortunately, with recent versions of Netbeans, it has been removed. In addition, with version 6.0, I encountered a bug which prevented me from using it. This bug is fixed in 6.1, which is the version I ended up using.

After many tutorials and searching online, as well as several attempts for me to write or copy an NBS file, I still had made no progress. As a last ditch effort, I decided to try to find a complete working GLF Netbeans Project, to see if perhaps the entire API was broken. I discovered a Prolog Language support module, written by Rosa Gutierrez, on this page: http://edu.netbeans.org/courses/nbplatform-certified-training/linz.html unfortunately, the link to the source was dead! So I emailed her, and she was able to send me a copy of the source, which you can now download here.

Once I had a working module, I was able to incrementally modify it to do what I wanted. However, there is one catch which I didn't notice anywhere else: Even if your grammar would accept the contents of a file, if there is something in your file which is not defined as one of the tokens, then it will give you an error, even if that exact token is listed in your grammar as acceptable. That's a little confusing, but the basic idea is that even if your grammar defines

Statement = "Hello" | "Goodbye";

if "hello" and "goodbye" do not somehow fall under the category of one of the tokens you defined, you will get an error.

Also, another quirk of the grammar is that the "root" of the grammar must be labeled "S". So for example, a simple grammar might be entirely defined like

S = "hello world" | "goodbye world";

I'll post any more quirks or tips I encounter here.

Fixing bad voices produced with festvox (or "Hey! My voice don't work!") (or "How to fix bad labellings")

Someone asked on a mailing list for possible ways to fix a bad voice the produced using festvox. I realized that my answer took my quite some time to figure out without any help, So I thought I'd post my response here.

Basically, if your voice is bad, chances are your labeling of some of the prompts is bad. (even if it isn't, it doesn't hurt to make sure they are good). You want to fix the bad labellings

To do so, copy the contents of your wav folder and the contents of your lab folder into the same directory (or setup links to make it seem that way). Once you've done that, open up the wav files with waveurfer, and choose the "transcription" view for all of them. Now you can go through one by one and check if the labellings are right. Options are: re-record the ones with bad labellings (remember to run bin/make_lab again before checking the labels again, I made this mistake once, and kept re-recording and thinking that the autolabeller sucked. Also, to save time, you can run bin/make_labs prompt-wav/test001.wav to just relabel test001.wav, instead of doing it to all the recordings, which can be time-consuming.), or hand-correcting the labels. You can literally just drag the labels from within wavesurfer (remember to copy your changes back to the lab/ directory).

Once you've got all the labels as perfect as you care to have them, just repeat all the steps after "bin/make_labs prompt-wav/*.wav" from whatever tutorial you are following and you should get the voice built with proper labeling (Come on, I know that if you knew how to do anything with festvox without a tutorial in front of you, there's no way you would need to be reading this post).

Switching between multiple grammars with pocketsphinx

I was having difficulty understanding the pocketsphinx api, specifically when it comes to switching between multiple grammars.

Here's how it works:

Pocketsphinx actually keeps track of a set of grammars at all time. Normally, this set of grammars only has one element. However, it can contain multiple grammars, while only one is switched on at a time. The basic method is

1. get this set of grammars using ps_get_fsgset()
2. Add your grammar to the set using fsg_set_add()
3. Select your grammar from the set as the active one using fsg_set_select()
4. Notify the recognizer that you have updated the grammar using ps_update_fsgset()

(this assumes that the recognizer was initially instantiated with a FSG, rather than an N-Gram model. Otherwise, you first need to switch it to an FSG model).

Example code:

ps_decoder_t * p= ...; //Decoder already initialized somehow
fsg_model_t * m= ...; //Load the model using fsg_model_read or jsgf_parse_file and jsgf_build_fsg
fsg_set_t* fsgset=ps_get_fsgset(p);
fsg_set_add(fsgset, "newgrammarname", m);
fsg_set_select(fsgset,"newgrammarname")
ps_update_fsgset(p);

NOTE: I realize that even jsgf_build_fsg is confusing. Here's how you should handle it:

jsgf_build_fsg(jsfgmodel, rule, ps_get_logmath(ps), 6.5);

where jsfgmodel is the jsgf model loaded using jsgf_parse_file, and "rule" is a rule chose from it. (use the jsgf_* functions to select the rule). Also, free the jsgf once the fsg has been created using jsgf_grammar_free.

Oh yeah, and the 6.5 just seems to be a magic number. In two places I've seen it used without any explanation. The documentation says nothing about what the number "lw" does anywhere, so I'd just stick to the value 6.5 and hope for the best...

Pointyclicky for festival/festvox

There are references scattered throughout the festvox documentation to a GUI program for recording speech prompts, called "pointyclicky". Sounds nice, right?


Well, if you try to find the sources, you find that it was last updated in 2000 (!), and it's in disrepair. (http://festvox.org/pointyclicky/)

It won't compile any more, but Kev 'Kyrian' Green at the redhat bugzilla patched it up so it's closer to compiling. I also made some more changes so it would work for me on Debian, but I don't remember exactly what I did. Anyway, you can download my patched version of the sources from here, which should be up at least for the next four years.