How to watch paid videos on Youtube/Google Play on Debian Linux with DRM

In order to watch paid videos from youtube/google play on Debian/Firefox, you need the latest flash plugin.  To to get this, run

sudo apt-get install flashplugin-nonfree

If you already have flashplugin-nonfree installed, you should run

update-flashplugin-nonfree --install

You can then run

update-flashplugin-nonfree --status

to verify that you have the latest version. Once you have the latest flash version, you need to install HAL for the flash DRM playback to work.  Unfortunately, HAL is deprecated, so you have two options: First is to install zombie hal which is a version of hal that has been kept alive for purposes of DRM support.  Unfortunately, zombie hal is for ubuntu, and my attempts to use it on debian didn't work.  The second option is hal-flash  which is a fake version of hal designed to be just functional enough that flash will accept it.  In order to install on debian, just download the source code, and run dpkg-buildpackage from the source directory. It will tell you any required packages you need to install, and when it finishes running, it will produce a .deb file in the parent directory, which you can install directly.

After installing hal-flash, I cleared the adobe cache which is suggested by many instructions online by deleting the .adobe directory.

At this point, I was still getting the message "an error occurred. please try again later".  To investigate, I used the rightclick -> copy debug info.  On inspection of this info, it looked to be url encoded, so I used an online url decoder to inspect the contents.  The key part of the decoded info was the section "debug_error=[ErrorEvent type="FlashAccess:3315:0" bubbles=false cancelable=false eventPhase=2 text=""]".  To investigate this further, I opened firebug, and looked at the net panel, to see the network activity taking place.  It turned out that the request to youtube.com/crossdomain.xml was never completing, and I eventually figured out that this was due to the plugin HTTPS everywhere I had installed.

After disabling HTTPS everywhere, it began to work on one computer, but on the other computer, the same steps still left it broken, giving me an error message "There is an error with your flash player. Click here, select reset license files, and restart your browser".  The debug info showed me that this is caused by error "FlashAccess:3329:500".  This indicates according to the flash reference that it is some sort of "application specific error", meaning I'd need google/youtube's help debugging it further, which they don't offer.  My own investigation of the network traffic just seems to indicate that nothing is going obviously wrong.

Hopefully these steps are useful to you!

Notes on implementing neural networks

Here are some notes on implementing deep neural networks.

Visualize as much as possible

No, I don't mean in the sense of simply imagining success, but in the sense of creating visual representations of your models and the training process.  When you write the code to do the training and run it, it can be hard to diagnose what's going on when it just prints out "ERROR RATE" at every iteration.  Even just graphing the error rate can help you tell if things are converging or diverging easier than printing it out in textual format.

Of course this is much more useful if you are learning a visual task, but be creative, and use tools like T-SNE  to visualize data that isn't directly visual.  It will end up being a lot of effort to do visualization, even more than the actual implementation, but it's worth it in the end.

Use automatic differentiation

The common advice I see on training neural networks is to always check whether your gradient code is correct by using the finite differences method.  I would go further and say that it is worthwhile to use automatic differentiation if possible.  You get precise answers as to what the gradient /should/ be, and when you write the optimized  implementation of the gradient calculation, you can compare the results side-by side.  If you don't care about speed, you can just run the whole algorithm using the Automatically differentiated gradient and not worry about writing any extra code at all.

How to build an LLVM pass using SCons

LLVM provides some nice documentation on how to build a simple pass using CMake, but if you want to build your project using SCons, this is not very useful. 

Luckily, there is an easy configuration tool "llvm-config" which gives you all the information you need to build against llvm.  Simply add the line

env=Environment()
env.ParseConfig("llvm-config-3.5 --cppflags --cxxflags")

to your SConstruct file, and it should handle all of the llvm compiler flags for you.

Building mlpack .deb

mlpack is a nice C++ library for machine learning, but unfortunately it doesn't have widely available Debian packages.  A few years ago someone made packaging, but his attempts to get it included in Debian is sadly documented here.  I wasn't able to find ready-built packages for Debian, and the build instructions on the mlpack website don't tell you how to build a deb.  Luckily, I did some snooping around and figured out how to do it.


First, download the latest tarball, and save it in a folder named "tarballs".  We then want to check out an svn directory (in the same folder the contains the tarballs folder).

Run 

$svn co http://svn.cc.gatech.edu/fastlab/mlpack/conf/packages/mlpack/trunk mlpack

Now make sure you have svn-buildpackage installed, along with all the other mlpack dependencies.


There is a bug in  the packaging, in that the default make target does not build the documentation, but the package tries to install it anyway, causing the package to fail to build.  I also found that the 1.08 version test suite fails, also halting the build, so I disabled it.

To fix these things, enter the mlpack directory you just checked out and edit debian/rules .  At the bottom of the file, modify it so it looks like


%:
        dh $@

override_dh_auto_build:
        dh_auto_build
        dh_auto_build -B -Smakefile -- doc

override_dh_auto_test:

Finally, from the mlpack directory run

$sh debian/build-from-svn.sh

This still might not work... I'll update this later 

Student StarCraft AI Tournament Tutorial for Getting Started with C++

The Student Starcraft AI tournament is an ongoing competition to write a bot to compete in playing 1v1 games of the real-time strategy game Starcraft. I found the documentation on how to get started to be scattered and lacking, and I hope to consolidate the knowledge I gained here into a mini-tutorial that will lower the barrier for entry.

PART 1 - Running the Example AI

First, you will need Starcraft Brood War patched to version 1.16.1 installed (You do not need the CD to run the game if you follow the instructions in the patch notes for version 1.15.2).

Next, you will need to download and extract BWAPI version 3.7.4 which is  not the latest version, however it is the version used by the SSCAI tournament.  I found that the included installer does not work, as confirmed by bug reports which were addressed in later versions.  However, later versions do not help us, as we need version 3.7.4 for the tournament.  Once extracted, the readme says

2. Run "install.exe". This will do the following:
    * Copies the contents of ./WINDOWS/ to the C:/Windows folder.
    * Copies the contents of ./Starcraft/ to your actual Starcraft folder.
    * Performs additional minor installation steps.

so, assuming the "additional minor installation steps" are nothing of importance, we can manually install BWAPI by copying the files according to the above instructions.  (This is what I did, and everything seems to have worked for me).

Next, you  need to install Visual Studio 2008 (the version is important, as later versions will give you errors when you try to compile).  You don't need to buy the full version, as Visual C++ 2008 Express Edition works fine, and is freely available for download.

When Visual Studio is installed, we will try using it to compile the example project and test it.  To do so, open up ExampleProjects.sln from the root directory of BWAPI in Visual Studio, and select Build>Build Solution from the menu bar.  When this is completed, check the output tab at the bottom for build errors (there should be none, if there are any errors, good luck!) .

To run the example AI module

1. Copy Release/ExampleAIModule.dll to
   <starcraft folder>/bwapi-data/AI/ExampleAIModule.dll

2. Open <starcraft folder>/bwapi-data/bwapi.ini and set ai to
   bwapi-data/AI/ExampleAIModule.dll if it isn't already.
 
3. Run Chaoslauncher with BWAPI Injector checked. You can also check W-MODE
   to have Broodwar run in a window rather than full-screen.

For the second step, this simply means editting the line in bwapi.ini which starts with

ai     =

to say

ai     =  bwapi-data/AI/ExampleAIModule.dll

You can also choose to build the debug version in Visual Studio.  In that case, you would copy the dll from Debug/ExampleAIModule.dll and modify the line starting with ai_dbg instead.  When starting up starcraft, you will have an option to choose between running debug mode or not, and the AI from the appropriate DLL will be loaded.  Nothing else in the bwapi.ini file needs to be edited for now.

Chaoslauncher can be found in the BWAPI main directory under Chaoslauncher/Chaoslauncher.exe.  When you open it, you can see the choice of checking off the BWAPI DEBUG or RELEASE options.  Check off just one, and optionally W-MODE (I suggest using debug mode and W-MODE), and click start.  You can safely ignore a warning about admin privileges if it pops up.  From here, you just need to get into a Starcraft game, and BWAPI should take over with the example AI.  The quickest way is to select a single-player expansion custom game and start the game.  The Example AI module should print some debug text to the screen and issue some basic commands to your probe and command center. If this happens, you have successfully completed the first step on getting your own BWAPI bot to run.

PART 2 - Creating your own AI Project

Installing Steam Linux Beta on 64 bit Debian

Valve recently came out with a closed beta for Steam on Linux.  For the initial beta, they are only supporting Ubuntu.   Luckily, people on reddit and other places have found workarounds to install the beta client if you're not part of the beta, and on other distros besides Ubuntu.  On amd64 Debian, there's the additional problem that the client is only 32 bit.

Anyone can download the .deb that valve released at http://media.steampowered.com/client/installer/steam.deb. unfortunately for us this is a 32-bit Ubuntu package, and it isn't even a clean deb file: it will try to install additional dependencies when you run it for the first time.  We basically want to ignore any dependency fetching it does and manually install all the dependencies on our own.

First, we just want to force the install of a 32-bit package on the wrong architecture.  To do this we run

$sudo dpkg --force-architecture -i steam.deb

 from the terminal.  It will probably complain about dependencies;  ignore this.  We can't install all the required libraries through the debian package system, so instead we create a folder in your home directory to contain the alternate versions of these libraries.  Luckily steam user cyb.org has put together a script which installs all the required libraries.  The script can be found at http://dl.dropbox.com/u/29081229/Steam/debian_install.sh. The full contents of the script are

#!/bin/sh
STEAMLIBS=${HOME}/Steamlibc/
mkdir -p ${STEAMLIBS}
cd /tmp/
wget http://security.ubuntu.com/ubuntu/pool/main/e/eglibc/libc6_2.15-0ubuntu10.2_i386.deb
dpkg -x libc6_2.15-0ubuntu10.2_i386.deb /tmp/libc/
mv /tmp/libc/lib/i386-linux-gnu/* ${STEAMLIBS}
wget http://mirror.ovh.net/ubuntu//pool/main/j/jockey/jockey-common_0.9.7-0ubuntu7_all.deb
wget http://mirror.ovh.net/ubuntu//pool/main/x/x-kit/python-xkit_0.4.2.3build1_all.deb
sudo dpkg -i jockey-common_0.9.7-0ubuntu7_all.deb python-xkit_0.4.2.3build1_all.deb
sudo apt-get install libjpeg8 libcurl3-gnutls libtheora0 libpulse0 libpixman-1-0 libcairo2 libgdk-pixbuf2.0-0 libgtk2.0-0 libpango1.0-0

 in case that link changes its contents (maliciously or otherwise).  However, because a required library for steam is installed in your home directory, you need to launch the steam client with a special command line option.  cyb.org provides the script http://dl.dropbox.com/u/29081229/Steam/debian_steam.sh

#!/bin/sh
STEAMLIBS=${HOME}/Steamlibc/
LD_LIBRARY_PATH=${STEAMLIBS} /usr/bin/steam $@

to launch steam (the $@ is my addition so that it passes through arguments to the steam script). run

$chmod +x steam_debian.sh

to make it executable.  At this point, running the script may launch steam for you. Running

$./debian_steam.sh steam://open/games

 supposedly bypasses the beta check.  For me it popped open a dialog to download an update for steam, but after steam is updated, it won't do anything else.  You can easily run it under a debugger like so:

$DEBUGGER=gdb ./debian_steam.sh

For me doing so revealed that it was quitting with an exit code 0377 (which is equivalent to error code -1).

FFMpeg Dnxhd: Could not write header for output file #0

Doing some video editing on linux, I wanted to convert the videos to a format easy to edit (and deinterlaced). I found this blog post: http://www.itc4u.net/it-chalk-board/67-linux-app-ffmpeg/115-edit-avchd-files-mts-with-blender-or-cinelerraconvert-mts-to-dnxhd-mov.html

Which seemed great, except when I ran the command he gave, I got the error "Could not write header for output file #0". (Stream 0 being the video stream). A bit cryptic. I eventually managed to figure out that this is due to the container format. Specifically, I was trying to encode it into an AVI. Apparently the Dnxhd encoder doesn't like this. If you encode it to a .mov, it works perfectly.