Friday 14 March 2014

x264 Performance with Different Compilers

Recently I made a post about compiling x264 natively using Visual Studio under Windows. With that post I also made available my git repository that includes all the necessary Visual Studio projects to get x264 up and compiling under Windows. Since the default build chain for x264 uses a gcc based compiler (which is MinGW on Windows) I wondered if there was any performance difference between the generated binaries.

So I set out by building the x264 command line tool using both MSVC from Visual Studio 2013, Intel Compiler XE 2013 SP1 and MinGW 4.8.2. To test I passed the first minute of Big Buck Bunny from the 1080p mp4 file found on the main site (www.bigbuckbunny.org/index.php/download/). I then did a CRF=20.0 encode using the 'Very Slow' pre-set. This may not exactly be the most exhaustive test but I wasn't feeling like waiting around for 10+ hour encodes.

The results can be seen for themselves:

  • MinGW:    3min 5sec
  • MSVC:      3min 5sec
  • ICL:           3min 6sec
  • ICL (O3):   3min 5sec

ICL shows up twice because I did the first one using the default compilation options that MSVC was using. I then did a second build using the higher compilation options that Intel compiler can support (a.k.a I set the /O3 compile option as opposed to the default /O2).

As you can see the difference is negligible (there so little to see here I didn't even bother making a nice table or graph). In fact the only variations was with ICL and that was probably within the error of the test measurement anyway so for all intents and purposes all the final builds performed identically. This is not too surprising as x264 uses a lot of hand tuned assembly in the key bits which there isn't much more that any of the compilers could do with. The result is that the code is optimised as much as possible so there is no room for the compiler to improve it. Therefore the performance is pretty much consistent across all compilers.

Of course Ill be the first to admit that this was a pretty quick and dirty test. Its quite possible that after a 10+ hour encode some larger differences may start to appear. However based on these values it is unlikely that those differences would be very substantial (a couple of minutes here and there over a 10+ hour run).

So if you were expecting (or hoping) for big differences then I'm sorry to disappoint. What can be taken away from this is that it appears that it doesn't matter what compiler you use to build x264 you will still achieve the same performance. So for those who prefer GCC they can continue to MSYS it up, while those who prefer Visual Studio can do so happily as there is no performance downside in doing so.

Thursday 13 March 2014

Building x264 on Windows with Visual Studio

For anyone who has ever done any video encoding (or just watching for that matter) will probably have heard of x264. And if you havnt then you should have at least have heard of h264 (and if you still havnt then this post is probably not for you) which is the standard name for the video codec that the x264 encoder implements. So in the world of h264 video encoding x264 is probably one of, if not the best. And it is completely free and open source. So anyone who wants to get their hands on it can do so easily.

One downside of the x264 project (much like most open source projects) is that the default build tool is a gnu make style build chain. These don't run natively on Windows and generally requires an emulated shell such as MSYS or Cygwin. And even from within these shells they only support gcc (MinGW on Windows) based compilers.

But for those wanting to compile x264 natively on Windows using the native Windows build chain (i.e. Visual Studio) then that can actually be done rather simply. For those with Visual Studio 2013 then compiling many similar open source projects becomes a lot easier due to the addition of partial C99 support. C99 is something Microsoft have been neglecting for many years but with the 2013 updates it is a lot closer. x264 however still requires some manipulation in order to get it compile under MSVC's mostly C89 world.
Luckily changing x264 for be MSVC compatiblilty is rather straightforward. Most of the issues are a result of C89 requiring all variable declarations to be altogether and the start of each logical block of code. Straight of the bat most of the errors that Visual Studio will spit out about the x264 code will be a result of this issue (although the error codes don't do you any favours in realising this). So most errors will be due to code such as the following:

int padv = PADV >> v_shift;
// buffer: 2 chroma, 3 luma (rounded to 4) because deblocking goes beyond the top of the mb
if( b_end && !b_start )
    height += 4 >> (v_shift + SLICE_MBAFF);
pixel *pix;
int starty = 16*mb_y - 4*!b_start;

This will generate an error on the variable 'pix' because it is declared mid way through a block of code. Luckily the specifications say 'block' of code, which does not mean function or something similar. Instead it essential means anything between a set of '{' or '}'s (There iss actually a bit more to it than that but for our purposes - as you'll see later - it is good enough). In the above example we can see that the declaration of 'pix' occurs after an if statement. So if all we need to separate blocks are some '{}'s then modifying the code to the following will actually work:

int padv = PADV >> v_shift;
// buffer: 2 chroma, 3 luma (rounded to 4) because deblocking goes beyond the top of the mb
if( b_end && !b_start )
{
    height += 4 >> (v_shift + SLICE_MBAFF);
}
pixel *pix;
int starty = 16*mb_y - 4*!b_start;

All we did here was add the '{' and the '}' to the if statement. This makes that statement a block and so the line following it becomes a new block which makes everything work. This is surprisingly simple fix and will work for all of the cases found in libx264. In fact a complete working libx264 can be achieved by just performing the above operation at 10 different location in code. Or if you couldn't be bothered doing it yourself you can just apply the following patch that I have already made up for you.

Download patch file:
https://github.com/ShiftMediaProject/x264/commit/d9004ba604283fb70a3b67d444f67576c00a0e2e.diff

Now for those who don't just want the lib for x264 but actually want to compile the command line x264.exe then you'll have to perform the same operation a few more times. However there are 2 additional things youll need to do.
First is related to an issue with the use of unions. Specifically the following piece of code:

return (union {double f; uint64_t i;}){value}.i;

The above is too much for MSVC to handle. But with a bit of massaging it can be made to work. Massaging such as this:

union { double f; uint64_t i; } ulld = { value };
return ulld.i

The second issue is due to initialization lists being used on an array of structs. MSVC defaults to thinking that each element in the initializer list is actually an input for each component for the actual struct. So in the following piece of code MSVC treats the initializer list as actually an initializer list for the first 'AVS_Value' in the array.

AVS_Value arg_arr[] = { res, avs_new_value_bool( info->interlaced ), avs_new_value_string( matrix ) };

This will cause some nonsensical errors such as how a type of AVS_Value can not be converted to type short (short here being the type of the first member of AVS_Value). There is pretty much no combination of additional '('s and '{' that will fix this problem. So we have to fallback to the slightly less convenient way of just specifying each array element individually.

AVS_Value arg_arr[3];
arg_arr[0] = res;
arg_arr[1] = avs_new_value_bool( info->interlaced );
arg_arr[2] = avs_new_value_string( matrix );

This is not as nice to look at but it works. Putting all these pieces together and x264cli will compile under Visual Studio without any further problems. Again for those who dont want to do all this themselves then the appropriate patch can be acquired from below.

Download patch file:
https://github.com/ShiftMediaProject/x264/commit/4c51a4fc51737d932eddb4060bcd03d861dfec7d.diff

Of course if you don't want to worry about any of the above then you can check out my git repository that has all the necessary changes already applied and even comes with a pre-built Visual Studio project file. My repo is up to date with the current upstream master and with x264 development slowing down due to the upcoming x265 then its unlikely my repo will fall behind the upstream master so can be treated as up to date.

git repository:
https://github.com/ShiftMediaProject/x264

So with all of those above fixes it should be reasonable trivial to get x264 building natively in Windows. As to whether this is worth it is a debate for another time. For those who aren't familiar with MSYS/Cygwin but are familiar with Visual Studio then this should be a big win. What will be interesting is to test the performance of the compiled binaries between the different compilers. Perhaps ill have more on that soon.....