tag:blogger.com,1999:blog-1680610827488671142024-03-14T03:01:20.442+11:00Silicon & LithiumA blog with any and all coding fixes and insights found while wading the seas of C++, OpenGL, SIMD, GPGPU, Ray Tracing and anything else I happen to stumble upon.Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.comBlogger13125tag:blogger.com,1999:blog-168061082748867114.post-91337634688859567722016-09-21T19:00:00.001+10:002016-09-21T19:01:01.753+10:00Introducing Shift Media ProjectAlthough I haven't commented much on it in this blog, but I have been slowly working away on a project surrounding FFmpeg on Windows. This first meant creating custom Visual Studio projects for FFmpeg which was mentioned in earlier blog posts. Then was building up a collection of dependencies that also have custom VS project files and have been successfully converted to compile and run on Windows. All these new projects could then be built into FFmpeg itself. This collection of additional libraries has grown to almost 50 at time of writing!<br />
<br />
The next step was to provide pre-compiled releases for these libraries so that people can download them directly from the web without needing source code or compilation time. This was completed several months ago and now almost all the projects have downloadable development libraries compiled in both Visual Studio 2013 and 2015, in static or dynamic (dll) libraries and with 32bit and 64bit versions. Each release even contains '.pdb' files for easy debugging of the libs themselves which uses GitLink to link directly to the source code hosted on GitHub so that the libraries can be debugged in when used in your own application without actually needing the libs source code. Each of the maintained projects is being constantly updated as new versions come out so that each new release contains the latest features (older releases are still also available on the site)<br />
<br />
Now all of that is done then the final step was creating a web site so that people can easily find and easily download everything. And that is what this post is all about. Today I'm introducing the Shift Media Project web page that can be used to access all the above stuff.<br />
<br />
For those who want to get straight in and check it out then here it is:<br />
<div style="text-align: left;">
Link: <a href="http://shiftmediaproject.github.io/">ShiftMediaProject.github.io/</a></div>
<div style="text-align: left;">
<br /></div>
<h4 style="text-align: left;">
FFmpeg </h4>
<div style="text-align: left;">
From the site you can grab development libraries for FFmpeg which includes all the useful FFmpeg libs such as libavcodec, libavformat etc. These libraries are built using a host of additional libraries for increased functionality which includes all the possible items from the complete project list below.</div>
<h4>
FFmpeg VS Project Generator</h4>
You can also download portable binaries of my FFmpeg VS Project generator program which can be used to create a custom Visual Studio project within a FFmpeg source code
distribution. This program allows for the created Visual Studio project
to be customised using virtually any of the options supported by FFmpegs
default configure script. This allows for selecting which dependency
libraries and codec/format support should be built into the created
project file. With the output project FFmpeg libraries and programs can
be built and debugged directly within Visual Studio. This program is
what is used to build the default project file found in the above FFmpeg
source directory and I might write a more detailed post on that at a later stage<br />
<br />
<h4>
Complete Project List </h4>
The site currently includes pre-built development libraries built using both Visual Studio 2013 and 2015, in static or dynamic (dll) libraries with 32bit and 64bit versions for the following projects:<br />
<ul>
<li>FFmpeg</li>
<li>FFmpeg VS Project Generator </li>
<li>bzip2 </li>
<li>enca </li>
<li>fdk-aac </li>
<li>fontconfig </li>
<li>freetype2 </li>
<li>fribidi</li>
<li>game-music-emu (or gme)</li>
<li>gmp </li>
<li>gnutls </li>
<li>harfbuzz </li>
<li>lame (or lame mp3)</li>
<li>libass </li>
<li>libbluray </li>
<li>libcdio </li>
<li>libcdio-paranoia </li>
<li>libgcrypt </li>
<li>libgpg-error </li>
<li>libiconv </li>
<li>libilbc </li>
<li>liblzma </li>
<li>libssh </li>
<li>libvpx </li>
<li>libxml2 </li>
<li>mfx_dispatch </li>
<li>modplug </li>
<li>nettle </li>
<li>ogg </li>
<li>openssl </li>
<li>opus </li>
<li>rtmpdump </li>
<li>SDL </li>
<li>soxr </li>
<li>speex </li>
<li>theora </li>
<li>vorbis </li>
<li>x264</li>
<li>x265 </li>
<li>xvid </li>
<li>zlib </li>
</ul>
Due to some potential licensing issues not all projects have a pre-compiled binary to download but the following projects still come with a custom Visual Studio project (like all the projects do) that allows anyone to compile their own version using the supplied source code:<br />
<ul>
<li>libaacs </li>
<li>libbdplus </li>
<li>libdvdcss </li>
<li>libdvdnav </li>
<li>libdvdread</li>
</ul>
This list may grow in the future as new dependency libraries get incorporated into FFmpeg and a complete list of current and old projects will always be available on the web site.<br />
<br />
Currently the web site is in its first version and hopefully with time permitting I will gradually add new features to it.<br />
Hopefully for those in media application development on Windows may just find this new site somewhat useful.<br />
<ul>
</ul>
Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com1tag:blogger.com,1999:blog-168061082748867114.post-63282212868127948242014-05-11T14:49:00.001+10:002014-05-11T18:15:04.622+10:00MSVC C99 math.h header.These days with Visual Studio 2013 (msvc12) being out, Microsoft has a proper C99 compliant math.h C header file. For anyone using any C99 math function I would first recommend you do so using msvc12. However recently I was working on a project that wanted to support older versions of msvc so I wrote up some compatibility code that added the additional missing functions that were added to math.h in C99. Since older msvc versions are C89 compliant they are missing many functions that were added in C99. However some of these functions aren't actually missing they were just added to the header using a different name (often with an '_' prefix). So if you know where these functions are you can make them usable in a C99 way.<br />
<br />
So here is some code that can be added under a normal math.h include to add missing C99 functions. Not all of them are here and those that are missing are identified with a simple comment. But many of the commonly used ones are provided so hopefully this may be useful for someone.<br />
<br />
<pre class="brush:c">#if _MSC_VER > 1800
// MSVC 11 or earlier does not define a C99 compliant math.h header.
// Missing functions are included here for compatibility.
#include <float .h="">
static __inline double acosh(double x){
return log(x + sqrt((x * x) - 1.0));
}
static __inline float acoshf(float x){
return logf(x + sqrtf((x * x) - 1.0f));
}
# define acoshl(x) acosh(x)
static __inline double asinh(double x){
return log(x + sqrt((x * x) + 1.0));
}
static __inline float asinhf(float x){
return logf(x + sqrtf((x * x) + 1.0f));
}
# define asinhl(x) asinh(x)
static __inline double atanh(double x){
return (log(1.0 + x) - log(1.0 - x)) / 2;
}
static __inline float atanhf(float x){
return (logf(1.0f + x) - logf(1.0f - x)) / 2.0f;
}
#define atanhl(x) atanh(x)
static __inline double cbrt(double x){
return (x > 0.0) ? pow(x, 1.0 / 3.0) : -pow(-x, 1.0 / 3.0);
}
static __inline float cbrtf(float x){
return (x > 0.0f) ? powf(x, 1.0f / 3.0f) : -powf(-x, 1.0f / 3.0f);
}
#define cbrtl(x) cbrt(x)
#define copysign(x,s) _copysign(x,s)
#define copysignf(x,s) _copysign(x,s)
#define copysignl(x,s) _copysignl(x,s)
static __inline double erf(double x){
double a1 = 0.254829592, a2 = -0.284496736, a3 = 1.421413741;
double a4 = -1.453152027, a5 = 1.061405429, p = 0.3275911;
double t, y;
int sign = (x >= 0) ? 1 : -1;
x = fabs(x);
t = 1.0 / (1.0 + p*x);
y = 1.0 - (((((a5 * t + a4 ) * t) + a3) * t + a2) * t + a1) * t * exp(-x * x);
return sign*y;
}
static __inline float erff(float x){
return erf((float)x);
}
#define erfl(x) erf(x)
// erfc
static __inline double exp2(double x){
return pow(2.0, x);
}
static __inline float exp2f(float x){
return powf(2.0f, x);
}
#define exp2l(x) exp2(x)
static __inline double expm1(double x){
if(fabs(x) < 1e-5)
return x + 0.5 * x * x;
else
return exp(x) - 1.0;
}
static __inline float expm1f(float x){
if(fabsf(x) < 1e-5f)
return x + 0.5f * x * x;
else
return expf(x) - 1.0f;
}
#define expm1l(x) expm1(x)
static __inline double fdim(double x, double y){
return (x > y) ? x - y : 0.0;
}
static __inline float fdimf(float x, float y){
return (x > y) ? x - y : 0.0f;
}
#define fdiml(x,y) fdim(x,y)
static __inline double fma(double x, double y, double z){
return ((x * y) + z);
}
static __inline float fmaf(float x, float y, float z){
return ((x * y) + z);
}
#define fmal(x,y,z) fma(x,y,z)
static __inline double fmax(double x, double y){
return (x > y) ? x : y;
}
static __inline float fmaxf(float x, float y){
return (x > y) ? x : y;
}
#define fmaxl(x,y) fmax(x,y)
static __inline double fmin(double x, double y){
return (x < y) ? x : y;
}
static __inline float fminf(float x, float y){
return (x < y) ? x : y;
}
#define fminl(x,y) fmin(x,y)
#ifndef _HUGE_ENUF
# define _HUGE_ENUF 1e+300
#endif
#define INFINITY ((float)(_HUGE_ENUF * _HUGE_ENUF)) /* causes warning C4756: overflow in constant arithmetic (by design) */
#define NAN ((float)(INFINITY * 0.0F))
#define FP_INFINITE 1
#define FP_NAN 2
#define FP_NORMAL (-1)
#define FP_SUBNORMAL (-2)
#define FP_ZERO 0
#define fpclassify(x) ((_fpclass(x)==_FPCLASS_SNAN)?FP_NAN:((_fpclass(x)==_FPCLASS_QNAN)?FP_NAN:((_fpclass(x)==_FPCLASS_QNAN)?FP_NAN: \
((_fpclass(x)==_FPCLASS_NINF)?FP_INFINITE:((_fpclass(x)==_FPCLASS_PINF)?FP_INFINITE: \
((_fpclass(x)==_FPCLASS_NN)?FP_NORMAL:((_fpclass(x)==_FPCLASS_PN)?FP_NORMAL: \
((_fpclass(x)==_FPCLASS_ND)?FP_SUBNORMAL:((_fpclass(x)==_FPCLASS_PD)?FP_SUBNORMAL: \
FP_ZERO)))))))))
#define hypot(x,y) _hypot(x,y)
#define hypotf(x,y) _hypotf(x,y)
// ilogb
#define isfinite(x) _finite(x)
#define isnan(x) (!!_isnan(x))
#define isinf(x) (!_finite(x) && !_isnan(x))
#define isnormal(x) ((_fpclass(x) == _FPCLASS_NN) || (_fpclass(x) == _FPCLASS_PN))
#define isgreater(x,y) ((x) > (y))
#define isgreaterequal(x,y) ((x) >= (y))
#define isless(x,y) ((x) < (y))
#define islessequal(x,y) ((x) <= (y))
#define islessgreater(x,y) (((x) < (y)) || ((x) > (y)))
#define isunordered(x,y) (_isnan(x) || _isnan(y))
#define j0(x) _j0(x)
#define j1(x) _j1(x)
#define jn(x,y) _jn(x,y)
// lgamma
static __inline double log1p(double x){
if(fabs(x) > 1e-4){
return log(1.0 + x);
}
return (-0.5 * x + 1.0) * x;
}
static __inline float log1pf(float x){
if(fabsf(x) > 1e-4f){
return logf(1.0f + x);
}
return (-0.5f * x + 1.0f) * x;
}
#define log1pl(x) log1p(x)
static __inline double log2(double x) {
return log(x) * M_LOG2E;
}
static __inline float log2f(float x) {
return logf(x) * (float)M_LOG2E;
}
#define log2l(x) log2(x)
#define logb(x) _logb(x)
#define logbf(x) _logb(x)
#define logbl(x) _logb(x)
// nearbyint
#define nextafter(x,y) _nextafter(x,y)
#define nextafterf(x,y) _nextafter(x,y)
// nexttoward</float></pre>
<pre class="brush:c"><float .h="">static __inline double rint(double x){
const double two_to_52 = 4.5035996273704960e+15;
double fa = fabs(x);
if(fa >= two_to_52){
return x;
} else{
return copysign(two_to_52 + fa - two_to_52, x);
}
}
static __inline float rintf(float x){
const double two_to_52 = 4.5035996273704960e+15f;
double fa = fabsf(x);
if(fa >= two_to_52){
return x;
} else{
return copysignf(two_to_52 + fa - two_to_52, x);
}
}
#define rintl(x) rint(x)<pre class="brush:c"><float .h="">static __inline double remainder(</float>double x, double y){</pre>
<pre class="brush:c"><float .h=""><pre> return (x - ( rint(x / y) * y ));</pre>
<pre>}</pre>
<pre><pre class="brush:c"><float .h=""><pre class="brush:c"><float .h="">static __inline float remainderf(</float>float x, float y){</pre>
<pre class="brush:c"><float .h=""><pre> return (x - ( rintf(x / y) * y ));</pre>
<pre>}</pre>
</float></pre>
</float></pre>
<pre class="brush:c"><float .h="">
#define remainder</float><float .h="">l(x) remainder</float>(x)</pre>
<pre class="brush:c"><float .h="">static __inline double remquo(double x, double y, int* q){</float></pre>
<pre class="brush:c"><float .h=""> double d = </float>rint(x / y);</pre>
<pre class="brush:c"> q = (int)d;</pre>
<pre class="brush:c"><pre> return (x - (d * y));</pre>
</pre>
<pre class="brush:c"><float .h="">}</float></pre>
</pre>
</float></pre>
<pre><pre class="brush:c"><float .h="">static __inline float </float>remquof(float x, float y, int* q){</pre>
<pre class="brush:c"><float .h=""> float</float><float .h=""> f = </float>rintf(x / y);</pre>
<pre class="brush:c"> q = (int)f;</pre>
<pre class="brush:c"><pre> return (x - (f * y));</pre>
</pre>
<pre class="brush:c"><float .h="">}</float></pre>
</pre>
<pre class="brush:c"><float .h="">#define </float>remquo<float .h="">(x) </float>remquo(x)</pre>
static __inline double round(double x){
return ((x > 0.0) ? floor(x + 0.5) : ceil(x - 0.5));
}
static __inline float roundf(float x){
return ((x > 0.0f) ? floorf(x + 0.5f) : ceilf(x - 0.5f));
}
#define roundl(x) round(x)
// scalbn
#define signbit(x) (_copysign(1.0, x) < 0)
// tgamma
static __inline double trunc(double x){
return (x > 0.0) ? floor(x) : ceil(x);
}
static __inline float truncf(float x){
return (x > 0.0f) ? floorf(x) : ceilf(x);
}
#define truncl(x) trunc(x)
#define y0(x) _y0(x)
#define y1(x) _y1(x)
#define yn(x,y) _yn(x,y)
static __inline long lrint(double x){
return (long)rint(x);
}
static __inline long lrintf(float x){
return (long)rintf(x);
}
define lrintl(x) lrint(x)
static __inline long lround(double x){
return (long)round(x);
}
static __inline long lroundf(float x){
return (long)roundf(x);
}
#define lroundl(x) lround(x)
static __inline long long llrint(double x){
return (long long)rint(x);
}
static __inline long long llrintf(float x){
return (long long)rintf(x);
}
#define llrintl(x) llrint(x)
static __inline long long llround(double x){
return (long long)round(x);
}
static __inline long long llroundf(float x){
return (long long)roundf(x);
}
#define llroundl(x) llround(x)
#endif
</float></pre>
<br />Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0tag:blogger.com,1999:blog-168061082748867114.post-47317484157426688382014-05-03T16:52:00.000+10:002014-10-07T19:12:23.329+11:00Building FFmpeg on Windows with in-line asm and the Intel compiler (Part 3).Previously I have posted about efforts to build FFmpeg natively under windows with inline assembly enabled and using the Intel compiler:<br />
<br />
<span style="color: #0000ee; font-family: Arial, Helvetica, sans-serif; font-size: x-small;"><u><a href="http://siliconandlithium.blogspot.com/2013/12/building-ffmpeg-on-windows-with-in-line.html">Building FFmpeg on Windows with in-line asm and the Intel compiler.</a></u></span><br />
<span style="color: #0000ee; font-family: Arial, Helvetica, sans-serif; font-size: x-small;"><u><a href="http://siliconandlithium.blogspot.com/2014/01/building-ffmpeg-on-windows-with-in-line.html">Building FFmpeg on Windows with in-line asm and the Intel compiler (Part 2).</a></u></span><br />
<br />
After many months of patching, testing and review the current upstream FFmpeg master is now fully updated to support inline asm compilation with Intel compiler. This means that my patches do not need to be explicitly applied as now the default FFmpeg repository has been updated to include all my changes. So now those people who have access to the Intel compiler on Windows can build FFmpeg using the latest source and will automatically have all the hand-tuned assembly optimizations built in without any extra effort.<br />
<br />
For those interested in testing it out they can grab the latest FFmpeg source from their git master. Normally FFmepg must be built through MSYS/Cygwin on Windows so if your not that way inclined you can also check out my repository which includes a copy of FFmpeg master with some additional visual studio project files added in. With these you can now compile FFmpeg natively directly through Visual Studio.<br />
<br />
<a href="http://siliconandlithium.blogspot.com/2014/01/building-ffmpeg-in-visual-studio.html">Building FFmpeg in Visual Studio.</a><br />
<br />
These changes are all part of work designed to improve the state of FFmpeg on Windows. Also added to upstream FFmpeg are patches to enable OpenCL support with native win32 threads as well as several patches for dependency library linking errors (libvpx, libssh to name a few). It took a little while to get all these patches approved (asm patches to upstream libmpcodecs took a particularly long time) but as of this morning the libmpcodec changes were pulled into mainstream FFmpeg which was the last change required for full icl support. A big shout out to Michael Niedermayer for reviewing and providing useful feedback on all the patches. He was a big help in pointing me in the right direction and for spotting all my mistakes (a side affect of me writing and submitting many of the patches in the wee hours of the morning).<br />
<br />
As always you can grab the code from my repository and feel free to post any bugs/errors you may encounter.<br />
<a href="https://github.com/ShiftMediaProject/FFmpeg">https://github.com/ShiftMediaProject/FFmpeg</a>Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com7tag:blogger.com,1999:blog-168061082748867114.post-30945025177631253192014-04-14T18:43:00.000+10:002014-05-07T00:05:32.033+10:00Intel and Microsoft compiler linker errors (already defined in)Recently I ran into a rather strange error that took me a little while to track down. So to make it easier for anyone else who runs into this then Ill describe the error and its resolution here.<br />
<br />
Basically the error was during linking a program and showed up as:<br />
<br />
<span style="background-color: white; color: #53575e; font-family: Arial, Helvetica, sans-serif; font-size: 12px; line-height: 18px;">LIBCMT.lib(log10.obj) : error LNK2005: _log10 already defined in libmmt.lib(log10_stub.obj)</span><br />
<br />
This was just one of may errors that decided to pop up (basically the same errors but for things like sin/cos etc.). This was a rather odd error and it pretty much doesn't matter what linker options you change there is no way to save you from this one (don't bother trying to ignore specific libs as that will just make things worse - trust me). The problem was due to a program that includes static libs where 1 of those libs was created by the standard Microsoft compiler and the other was created by the Intel compiler. The problem is that each of these input libs declared the same function but in different ways (one the Intel way the other the Microsoft way).<br />
<br />
The issue revolves around the standard C library (obviously I was writing C/C++ code). When using the Microsoft compiler the standard C library is libcXX where XX can change based on different settings. In the above example I was using the standard multi-threaded version (hence the mt appended to the end of libc). The Intel compiler also defines it own implementation of various standard C functions which it puts in its own libmXX. So the above problem arises because these two libs are clashing with each other. This would not be a problem if both input libs were compiled with the same compiler but if that is not an option then we have to find a solution.<br />
<br />
Luckily Intel is not entirely clueless here and there libmXX libraries are designed to play nice with the default Microsoft ones. So why the above error? Well the problem arises when each input library uses a different compilation option. There will always be a problem with mixing code generated against different libcXX versions even when using the same compiler but as you can see from above both are generated using the 'mt' version so all should be fine. The actual solution is due to the implementation of each of these clashing functions. Above the example was the log10 function which is only clashing because it is being used in different ways that can not be resolved between the Intel and Microsoft compiler.<br />
<br />
The culprit is a setting called "FloatingPointModel". If building through Visual Studio then this setting can be found under "C/C++->Code Generation" in the project properties. The issue occurs when 1 of the input libs was compiled using a different floating point model than the other lib. For instance if one of the libs was set to use Intel compilers "Fast2" setting then it will use the appropriate optimizations and optimized functions from within libmXX. However if the other lib was compiled using a different floating point model then it will try and use that version of the function. The problem is that they both have the same function name but have different implementations in different compiler libc implementations. All of a sudden libmXX and libcXX wont play nice together as they are both trying to use different optimized versions of a math function that has the same name. The linker doesn't know which one to use and so errors abound.<br />
<br />
The solution is simply making sure that all input libs are compiled using the same setting for the floating point model. This rules out using Intels "Fast2" option as this is never available on Microsoft's so it will always cause problems. But apart from that making sure they are all the same value (or not setting any value) should make this rather rare and frustrating bug go away.Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com2tag:blogger.com,1999:blog-168061082748867114.post-71346423129282000222014-03-14T18:44:00.002+11:002014-03-14T18:45:12.850+11:00x264 Performance with Different CompilersRecently I made a post about compiling x264 natively using Visual Studio under Windows. With that post I also made available my git repository that includes all the necessary Visual Studio projects to get x264 up and compiling under Windows. Since the default build chain for x264 uses a gcc based compiler (which is MinGW on Windows) I wondered if there was any performance difference between the generated binaries.<br />
<br />
So I set out by building the x264 command line tool using both MSVC from Visual Studio 2013, Intel Compiler XE 2013 SP1 and MinGW 4.8.2. To test I passed the first minute of Big Buck Bunny from the 1080p mp4 file found on the main site (<a href="http://www.bigbuckbunny.org/index.php/download/">www.bigbuckbunny.org/index.php/download/</a>). I then did a CRF=20.0 encode using the 'Very Slow' pre-set. This may not exactly be the most exhaustive test but I wasn't feeling like waiting around for 10+ hour encodes.<br />
<br />
The results can be seen for themselves:<br />
<br />
<ul>
<li><b>MinGW:</b> 3min 5sec</li>
<li><b>MSVC: </b> 3min 5sec</li>
<li><b>ICL: </b> 3min 6sec</li>
<li><b>ICL (O3):</b> 3min 5sec</li>
</ul>
<br />
ICL shows up twice because I did the first one using the default compilation options that MSVC was using. I then did a second build using the higher compilation options that Intel compiler can support (a.k.a I set the /O3 compile option as opposed to the default /O2).<br />
<br />
As you can see the difference is negligible (there so little to see here I didn't even bother making a nice table or graph). In fact the only variations was with ICL and that was probably within the error of the test measurement anyway so for all intents and purposes all the final builds performed identically. This is not too surprising as x264 uses a lot of hand tuned assembly in the key bits which there isn't much more that any of the compilers could do with. The result is that the code is optimised as much as possible so there is no room for the compiler to improve it. Therefore the performance is pretty much consistent across all compilers.<br />
<br />
Of course Ill be the first to admit that this was a pretty quick and dirty test. Its quite possible that after a 10+ hour encode some larger differences may start to appear. However based on these values it is unlikely that those differences would be very substantial (a couple of minutes here and there over a 10+ hour run).<br />
<br />
So if you were expecting (or hoping) for big differences then I'm sorry to disappoint. What can be taken away from this is that it appears that it doesn't matter what compiler you use to build x264 you will still achieve the same performance. So for those who prefer GCC they can continue to MSYS it up, while those who prefer Visual Studio can do so happily as there is no performance downside in doing so.<br />
<br />Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0tag:blogger.com,1999:blog-168061082748867114.post-49408395610720647782014-03-13T23:12:00.000+11:002014-10-07T19:13:54.159+11:00Building x264 on Windows with Visual StudioFor anyone who has ever done any video encoding (or just watching for that matter) will probably have heard of x264. And if you havnt then you should have at least have heard of h264 (and if you still havnt then this post is probably not for you) which is the standard name for the video codec that the x264 encoder implements. So in the world of h264 video encoding x264 is probably one of, if not the best. And it is completely free and open source. So anyone who wants to get their hands on it can do so easily.
<br />
<br />
One downside of the x264 project (much like most open source projects) is that the default build tool is a gnu make style build chain. These don't run natively on Windows and generally requires an emulated shell such as MSYS or Cygwin. And even from within these shells they only support gcc (MinGW on Windows) based compilers.
<br />
<br />
But for those wanting to compile x264 natively on Windows using the native Windows build chain (i.e. Visual Studio) then that can actually be done rather simply. For those with Visual Studio 2013 then compiling many similar open source projects becomes a lot easier due to the addition of partial C99 support. C99 is something Microsoft have been neglecting for many years but with the 2013 updates it is a lot closer. x264 however still requires some manipulation in order to get it compile under MSVC's mostly C89 world.
<br />
Luckily changing x264 for be MSVC compatiblilty is rather straightforward. Most of the issues are a result of C89 requiring all variable declarations to be altogether and the start of each logical block of code. Straight of the bat most of the errors that Visual Studio will spit out about the x264 code will be a result of this issue (although the error codes don't do you any favours in realising this). So most errors will be due to code such as the following:
<br />
<br />
<pre class="brush:c">int padv = PADV >> v_shift;
// buffer: 2 chroma, 3 luma (rounded to 4) because deblocking goes beyond the top of the mb
if( b_end && !b_start )
height += 4 >> (v_shift + SLICE_MBAFF);
pixel *pix;
int starty = 16*mb_y - 4*!b_start;
</pre>
<br />
This will generate an error on the variable 'pix' because it is declared mid way through a block of code. Luckily the specifications say 'block' of code, which does not mean function or something similar. Instead it essential means anything between a set of '{' or '}'s (There iss actually a bit more to it than that but for our purposes - as you'll see later - it is good enough). In the above example we can see that the declaration of 'pix' occurs after an <i>if</i> statement. So if all we need to separate blocks are some '{}'s then modifying the code to the following will actually work:<br />
<br />
<pre class="brush:c">int padv = PADV >> v_shift;
// buffer: 2 chroma, 3 luma (rounded to 4) because deblocking goes beyond the top of the mb
if( b_end && !b_start )
{
height += 4 >> (v_shift + SLICE_MBAFF);
}
pixel *pix;
int starty = 16*mb_y - 4*!b_start;
</pre>
<br />
All we did here was add the '{' and the '}' to the <i>if</i> statement. This makes that statement a block and so the line following it becomes a new block which makes everything work. This is surprisingly simple fix and will work for all of the cases found in libx264. In fact a complete working libx264 can be achieved by just performing the above operation at 10 different location in code. Or if you couldn't be bothered doing it yourself you can just apply the following patch that I have already made up for you.<br />
<br />
Download patch file:<br />
<a href="https://github.com/ShiftMediaProject/x264/commit/d9004ba604283fb70a3b67d444f67576c00a0e2e.diff">https://github.com/ShiftMediaProject/x264/commit/d9004ba604283fb70a3b67d444f67576c00a0e2e.diff</a><br />
<br />
Now for those who don't just want the lib for x264 but actually want to compile the command line x264.exe then you'll have to perform the same operation a few more times. However there are 2 additional things youll need to do.<br />
First is related to an issue with the use of unions. Specifically the following piece of code:<br />
<br />
<pre class="brush:c">return (union {double f; uint64_t i;}){value}.i;
</pre>
<br />
The above is too much for MSVC to handle. But with a bit of massaging it can be made to work. Massaging such as this:<br />
<br />
<pre class="brush:c">union { double f; uint64_t i; } ulld = { value };
return ulld.i
</pre>
<br />
The second issue is due to initialization lists being used on an array of structs. MSVC defaults to thinking that each element in the initializer list is actually an input for each component for the actual struct. So in the following piece of code MSVC treats the initializer list as actually an initializer list for the first 'AVS_Value' in the array.<br />
<br />
<pre class="brush:c">AVS_Value arg_arr[] = { res, avs_new_value_bool( info->interlaced ), avs_new_value_string( matrix ) };
</pre>
<br />
This will cause some nonsensical errors such as how a type of AVS_Value can not be converted to type short (short here being the type of the first member of AVS_Value). There is pretty much no combination of additional '('s and '{' that will fix this problem. So we have to fallback to the slightly less convenient way of just specifying each array element individually.<br />
<br />
<pre class="brush:c">AVS_Value arg_arr[3];
arg_arr[0] = res;
arg_arr[1] = avs_new_value_bool( info->interlaced );
arg_arr[2] = avs_new_value_string( matrix );
</pre>
<br />
This is not as nice to look at but it works. Putting all these pieces together and x264cli will compile under Visual Studio without any further problems. Again for those who dont want to do all this themselves then the appropriate patch can be acquired from below.<br />
<br />
Download patch file:<br />
<a href="https://github.com/ShiftMediaProject/x264/commit/4c51a4fc51737d932eddb4060bcd03d861dfec7d.diff">https://github.com/ShiftMediaProject/x264/commit/4c51a4fc51737d932eddb4060bcd03d861dfec7d.diff</a><br />
<br />
Of course if you don't want to worry about any of the above then you can check out my git repository that has all the necessary changes already applied and even comes with a pre-built Visual Studio project file. My repo is up to date with the current upstream master and with x264 development slowing down due to the upcoming x265 then its unlikely my repo will fall behind the upstream master so can be treated as up to date.<br />
<br />
git repository:<br />
<a href="https://github.com/ShiftMediaProject/x264">https://github.com/ShiftMediaProject/x264</a><br />
<br />
So with all of those above fixes it should be reasonable trivial to get x264 building natively in Windows. As to whether this is worth it is a debate for another time. For those who aren't familiar with MSYS/Cygwin but are familiar with Visual Studio then this should be a big win. What will be interesting is to test the performance of the compiled binaries between the different compilers. Perhaps ill have more on that soon.....Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com62tag:blogger.com,1999:blog-168061082748867114.post-59511836583008275872014-01-25T16:38:00.001+11:002014-11-27T15:43:23.177+11:00Building FFmpeg in Visual StudioThe default build chain for the FFmpeg project uses the standard (well standard for gnu open source projects) gnu autotools. The use of configure and make may be rather familiar to those who compile often on linux but for many Windows developers these can seem like somewhat alien tools. This is compounded by the fact that to use these tools on Windows requires setting up a MSYS/Cygwin environment which can often be easier said than done. Even after that most build chains using this environment require a gcc based compiler which on Windows is MinGW. Gcc is a good compiler but MinGW can have some issues (which in its defence is generally always around Windows specific things) which can make it less than ideal.<br />
<br />
FFmpegs default build tools do currently support compiling natively in msvc (Microsoft's C compiler) and will even convert the C99 FFmpeg code to msvc compliant C89 code. But this still requires setting up a MSYS shell and any additional FFmpeg dependencies don't offer msvc support from within the same build tools.<br />
<br />
So as powerful as the configure/make build chains can be and say what you like about the msvc compiler (I agree its standards compliance is abysmal) but for those wanting to develop natively on Windows having Visual Studio support is the simplest and most robust. Since FFmpeg wont maintain a native Visual Studio build chain (and I cant blame them for not wanting to), I got a little bored and decided to take it upon myself to provide an alternative.<br />
<br />
The result of some holiday free time is that I wrote up a FFmpeg Visual Studio project generator. This is a simple piece of software that will scan in the existing FFmpeg configure/make files and then use those to dynamically generate a visual studio project file that can be used to natively compile FFmpeg. This program not only builds the project files but will also generate the config files dynamically based on a combination of rules found in the original configure script and any passed in arguments. In fact the generator excepts many of the same configuration arguments as the configure script allowing fine-tuned control of what gets enabled/disabled.<br />
Example command line options:<br />
<br />
<pre class="brush:plain">ffmpeg_generator.exe --enable-gpl --enable-bzlib --enable-iconv
--enable-zlib --disable-encoders --enable-encoder=vorbis
</pre>
<br />
The program doesn't support all the options of the configure script but it supports many of them and in some cases actually exposes more than the original. Any option that shows up in the config.h file can be manually enabled/disabled through the command line. And even if an invalid option is set the generator uses all the inbuilt configuration checks found in the original build script to automatically validate each setting. Options such as 32/64bit are some of the options that are not supported, mainly because they are not relevant as the generated config.h file can be used for both 32/64 bit and even detects and enables inline asm if the Intel compiler is made available.<br />
<br />
<pre class="brush:cpp">#define ARCH_X86 1
#if defined( __x86_64 ) || defined( _M_X64 )
# define ARCH_X86_32 0
#else
# define ARCH_X86_32 1
#endif
</pre>
<br />
All of this happens dynamically so for those building from git master, once any commits are and added to your repo you just have to rerun the generator program and it will automatically detect any changes (new/deleted files, configuration changes, new/deleted options etc.) and generate a new project accordingly.<br />
<br />
Now I should point out that this generator was rather quickly (and half-assed) thrown together so its not exactly very brilliant code. But for something quick and dirty it gets the job done. It supports all current additional dependencies but not all have been checked and it takes certain liberties with respect to library naming. This is because many dependency libraries don't have consistent naming conventions so the link include the generated project uses may not be exactly the same as the actually file you are linking against. Should this happen then you'll just have to manually tweak the file-name in the include option as without any kind of naming consistency there's not much that can be done about it.<br />
<br />
The generator also by default generates a project with default Intel compiler settings. Those without Intel compiler may have to change a few settings in the project properties if they want to set it back to the standard msvc. Intel is chosen as default as Visual Studio 2012 doesn't support enough C99 features to be able to compile FFmpeg so only the Intel compiler can be used with 2012 to build the project. For those with Visual Studio 2013 the default compiler adds enough C99 to be able to get it to work but for the moment the generator is built to default to 2012. The same project can be loaded in both 2012/2013 and all that needs to be changed is the compiler being used. If there is enough interest I may add an option to the generator to allow for people to specify whether they want Intel support or not at generation time but in the meantime you'll just have to change the build tool in the project properties.<br />
<br />
<b>Update:</b> The current version of the generator allows for specifying the compiler that will be used as default in the output project file. Of course this can also be changed directly in the project after it is generated but for convenience the "toolchain" option is now processed by the generator. With newer patches to FFmpeg Visual Studio 2013 can compile it without problems. The toolchian parameter accepts either "msvc" for default Microsoft compiler or "icl" for the Intel compiler which also supports inline assembly.<br />
<br />
<pre class="brush:plain">ffmpeg_generator.exe --toolchain=msvc
</pre>
<br />
The project generator can be found in my git repo below. Also in the repo is a pre-built project that is built using the following command options:<br />
<br />
<pre class="brush:plain">ffmpeg_generator.exe --enable-gpl --enable-version3 --enable-avisynth
--enable-nonfree --enable-bzlib --enable-iconv --enable-zlib
--enable-libmp3lame --enable-libvorbis --enable-libspeex
--enable-libopus --enable-libfdk-aac --enable-libtheora
--enable-libx264 --enable-libxvid --enable-libvpx --enable-libmodplug
--enable-libsoxr --enable-libfreetype --enable-fontconfig
--enable-libass --enable-openssl --enable-librtmp --enable-libssh
</pre>
<br />
<b>Update:</b> The default projects now include libcdio, libbluray, opengl, opencl and sdl enabled. More will be enabled as they are tested. All of these dependencies have working Visual Studio projects found in the SMP directories in each of their repos found at the parent ShiftMediaProject repository.<br />
<br />
Your free to use this project directly as I keep it up to date and all the necessary dependency projects can also be found in my github.<br />
<br />
git repository:<br />
<a href="https://github.com/ShiftMediaProject/FFmpeg">https://github.com/ShiftMediaProject/FFmpeg</a><br />
<br />Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com82tag:blogger.com,1999:blog-168061082748867114.post-21017159095097687772014-01-05T15:48:00.001+11:002014-10-07T19:15:02.825+11:00Building FFmpeg on Windows with in-line asm and the Intel compiler (Part 2).In my last post I talked about how to get FFmpeg to compile under Windows with the Intel compiler. This has advantages in that the Intel compiler supports compilation of AT&T style inline assembly. This means that its possible to use the hand tuned optimised code found in FFmpeg while natively compiling for Windows (something that is otherwise not possible). Unfortunately the assembly support in the Windows version of the compiler is not complete and so the inline asm wont compile without some changes.<br />
<br />
The previous post on this subject had a patch that allowed for compilation with inline asm. However since then I have cleaned up the patch and fixed a few things that I wasnt entirely happy about. In fact after talking to Michael Niedermayer and others on the FFmpeg mailing list I ended up writing an entirely new patch. This patch is currently still pending review but for those who are interested in getting this working now they can grab the patch from the end of this post.<br />
<br />
The main changes in this version is the way the inline asm was changed from using direct symbol references to something defined in an asm-interface. From my previous post I mentioned that Intel compiler does not support direct symbol references in code so previously I had changed all of these to asm-interfaces. However the FFmpeg developers didn't want to change any existing code and there was some concerns over how moving variables from direct symbols into the interface may affect Position Independent Code compilation. So based on a suggestion from Michael the patch was changed to use named constraints. For those familiar with named constraints you'll know that to use these all you have to do is replace a direct symbol reference with a named constraint.<br />
<br />
Example of existing direct symbol reference:<br />
<pre class="brush:c">"movq "MANGLE(ff_pb_80)", %%mm0 \n\t"
"lea (%3, %3, 2), %1 \n\t"
</pre>
<br />
Since FFmpeg already had a macro for name mangling the direct symbol references it was rather simple to just change this macro to generate a named constraint instead. In order to do this the definition of MANGLE was changed so that with Intel compiler on Windows it generates a named constraint.<br />
<br />
<pre class="brush:c">// Determine if compiler supports direct symbol references in inline asm
#if defined(__INTEL_COMPILER) && defined(_MSC_VER)
# define HAVE_DIRECT_SYMBOL_REF 0
#else
# define HAVE_DIRECT_SYMBOL_REF 1
#endif
#if HAVE_DIRECT_SYMBOL_REF
//The standard version of MANGLE for direct symbol references
# define MANGLE(a) EXTERN_PREFIX LOCAL_MANGLE(a)
#else
//A version of mangle that instead generates named constraints
# define MANGLE(a) "%["#a"]"
#endif
</pre>
<br />
Of course using a named constraint by itself wont work as the constraint still needs to be added to the asm-interface. Since this additional interface is only required for Intel on Windows it is not desirable to have it all the time. So in keeping with not changing any existing code the asm-interfaces were added using a new macro that simply does nothing for all other build chains. Using this the above inline asm becomes:<br />
<br />
<pre class="brush:c">__asm__ volatile (
"movq "MANGLE(ff_pb_80)", %%mm0 \n\t"
"lea (%3, %3, 2), %1 \n\t"
put_signed_pixels_clamped_mmx_half(0)
"lea (%0, %3, 4), %0 \n\t"
put_signed_pixels_clamped_mmx_half(64)
: "+&r"(pixels), "=&r"(line_skip3)
: "r"(block), "r"(line_skip)
NAMED_CONSTRAINTS_ADD(ff_pb_80)
: "memory"
);
</pre>
<br />
The resulting changes are rather minimal and when compiling using any previously supported build chains there is no apparent difference in the code before and after this change. The macro NAMED_CONSTRAINTS is used to add each of the names of any directly accessed symbols. This macro just needs the name and can take a comma separated list of up to 10 values.<br />
<br />
However adding the macro NAMED_CONSTRAINTS required slightly more work than I would have liked but it at least worked as required. The difficulty was due to a bug in both the Intel and Microsoft compilers where variadic arguments where not properly expanded (see my post on the subject <a href="http://siliconandlithium.blogspot.com/2014/01/macro-variadic-argument-expansion-on.html">http://siliconandlithium.blogspot.com/2014/01/macro-variadic-argument-expansion-on.html</a>). Using the working variadic for-each from my previous post the implementation of NAMED_CONSTRAINTS is:<br />
<br />
<pre class="brush:c">#if HAVE_DIRECT_SYMBOL_REF
# define NAMED_CONSTRAINTS_ADD(...)
# define NAMED_CONSTRAINTS(...)
#else
# define NAME_CONSTRAINT(x) [x] "m"(x)
// Parameters are a list of each symbol reference required
# define NAMED_CONSTRAINTS_ADD(...) , FOR_EACH_VA(NAME_CONSTRAINT,__VA_ARGS__)
// Same but without comma for when there are no previously defined constraints
# define NAMED_CONSTRAINTS(...) FOR_EACH_VA(NAME_CONSTRAINT,__VA_ARGS__)
#endif
</pre>
<br />
Putting this all together and for the most part all existing inline asm will compile without any problems. However you'll notice I said 'most'. The Intel compiler is extremely fussy about the use of inline AT&T assembly. This is most likely because this is considered a rarely used feature and so does not see much in the way of support. Unfortunately this means that using inline asm is much harder than it needs to be. So even after the missing support for direct symbol references the compiler still has many issues. The main one is that the compiler will sporadically decide it doesn't like some particular assembly line and generate an error. The same assembly line will be working fine until you change a compilation option (or in some cases just move the assembly block somewhere else) then all of a sudden it will generate an error. Ideally it would be nice if Intel cleaned up some these inconsistencies but in the mean time the patch had to work around them.<br />
<br />
So the patch does change a couple of things. In fact there are 2 instances (in x86/motion_est.c and vf_fspp.c) where a direct symbol reference had to be removed and replaced with an asm-interface. This may be seen as changing the original code but in both instances the variable in question already existed in an asm-interface so there additional inclusion should have absolutely zero affect. In fact just to be sure I checked the generated code from gcc before and after the patch and ensured that every line was identical.<br />
<br />
The patch passes all FATE tests and compiles on Intel (under normal release compilation options) and has zero impact on any other build chain. For those interested the full patch can be downloaded below.<br />
<br />
<u><b>Update 2:</b></u> New and improved patches have been created and these are now available directly in FFmpeg master (no patching required). See my post for more information (<a href="http://siliconandlithium.blogspot.com/2014/05/building-ffmpeg-on-windows-with-in-line.html">Building FFmpeg on Windows with in-line asm and the Intel compiler (Part 3)</a>).<br />
<u style="font-weight: bold;">Update:</u> A new patch is available that adds support for an extra file. This file is only used when FAST_CMOV is enabled so was missed previously. The new patch should be grabbed from here:<br />
<a href="https://github.com/ShiftMediaProject/FFmpeg/commit/26acab2672f27b923510ec35cbbade69a7776c9d.diff">https://github.com/ShiftMediaProject/FFmpeg/commit/26acab2672f27b923510ec35cbbade69a7776c9d.diff</a><br />
<br />
Download patch file:<br />
<a href="https://github.com/ShiftMediaProject/FFmpeg/commit/6eea177a4761629df5d6f02359fccc4efb116aed.diff">https://github.com/ShiftMediaProject/FFmpeg/commit/6eea177a4761629df5d6f02359fccc4efb116aed.diff</a><br />
<br />
Note: Unlike my last patch this one is done directly against the current master (at time of writing).Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0tag:blogger.com,1999:blog-168061082748867114.post-62204354975647297032014-01-05T15:45:00.000+11:002014-01-05T15:45:23.133+11:00Macro variadic argument expansion on Windows.Recently while trying to write some code I decided to use a variadic macro. For those familiar with the concept you'll know that they can be very powerful. Unfortunately I ran into a slight problem when using them. That problem being that the Microsoft compiler (and the Intel compiler for Windows) were both incorrectly expanding the variadic arguments (__VA_ARGS__). Or more to the point they both were completely failing to expand the arguments. This meant that it was not directly possible to get each element individually as the compiler was essentially treating the whole variadic argument as one long string (commas included). This is directly in violation of what the standards require and so both compilers are being non-compliant. The Microsoft compiler has never embraced C99 so this should not be to much of a surprise but Intels failure here is a bit more egregious.<br />
<br />
Luckily there is a way to get around this. It is not quite as elegant as what works on other compilers but it will at least work on Windows while still building on other build chains. So the example I will give is a for-each macro. This uses the variadic expansion and then sends each argument element to a specific macro. The trick here was that each expansion had to be explicitly handled. This means that if you want to support a variadic macro that can take up to 10 arguments then you need to create 10 explicit variants of the expansion. This limits the maximum number of arguments that can be supported (by however many expansions you can be bothered to write) and is also not very elegant but at least it will work.<br />
<br />
<pre class="brush:cpp">#define FE_0(P,X) P(X)
#define FE_1(P,X,X1) P(X), FE_0(P,X1)
#define FE_2(P,X,X1,X2) P(X), FE_1(P,X1,X2)
#define FE_3(P,X,X1,X2,X3) P(X), FE_2(P,X1,X2,X3)
#define FE_4(P,X,X1,X2,X3,X4) P(X), FE_3(P,X1,X2,X3,X4)
#define FE_5(P,X,X1,X2,X3,X4,X5) P(X), FE_4(P,X1,X2,X3,X4,X5)
#define FE_6(P,X,X1,X2,X3,X4,X5,X6) P(X), FE_5(P,X1,X2,X3,X4,X5,X6)
#define FE_7(P,X,X1,X2,X3,X4,X5,X6,X7) P(X), FE_6(P,X1,X2,X3,X4,X5,X6,X7)
#define FE_8(P,X,X1,X2,X3,X4,X5,X6,X7,X8) P(X), FE_7(P,X1,X2,X3,X4,X5,X6,X7,X8)
#define FE_9(P,X,X1,X2,X3,X4,X5,X6,X7,X8,X9) P(X), FE_8(P,X1,X2,X3,X4,X5,X6,X7,X8,X9)
#define GET_FE_IMPL(_0,_1,_2,_3,_4,_5,_6,_7,_8,_9,NAME,...) NAME
#define GET_FE(A) GET_FE_IMPL A
#define GET_FE_GLUE(x, y) x y
#define FOR_EACH_VA(P,...) GET_FE_GLUE(GET_FE((__VA_ARGS__,FE_9,FE_8,FE_7,FE_6,FE_5,FE_4,FE_3,FE_2,FE_1,FE_0)), (P,__VA_ARGS__))
</pre>
<br />
This requires 3 pieces. The first is GET_FE_IMPL which is used to pass the variadic arguments to the explicit expansion macro (FE_0, FE_1 etc.). The second piece is GET_FE. This macro just passes directly to GET_FE_IMPL and on other compiler is completely redundant but on Windows it is used to add an extra level of indirection that forces the input parameters to be correctly 'stringified'. Remove it and everything will fail horribly. The final piece required to get this to work is the GET_FE_GLUE macro. This is perform the same task as GET_FE in that it is required to correctly pass the variadic arguments. Both of these are required in order to get around the bugs in the compilers.<br />
<br />
This new macro can be used to create any new macro that should perform a desired operation on each of the input arguments. All that is required is to call FOR_EACH_VA and pass it a macro followed by the variadic arguments. The passed macro will then be called and passed each individual element in the argument list.<br />
<br />
Example:<br />
<pre class="brush:cpp">#define DO_SOMETHING(x) /*Insert whatever code you wish to operate on the element 'x'*/
//The definition of the new variadic macro
#define DO_SOMETHING_VA(...) FOR_EACH_VA(DO_SOMETHING,__VA_ARGS__)</pre>
<br />
With the above you should be able to create any variadic macro you want.<br />
<br />
Example usage:<br />
<pre class="brush:cpp">//Perform DO_SOMETHING on each of the 3 arguments
DO_SOMETHING_VA(element1,element2,element3)</pre>
<br />
If you want to support an additional number of input arguments (the above only supports a maximum of 10) then all you have to do is add more explicit expansion macros (e.g. FE_10, FE_11 etc.) and then add these in sequence to GET_FE_IMPL and to FOR_EACH_VA.Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0tag:blogger.com,1999:blog-168061082748867114.post-10757516124711487912013-12-22T13:53:00.000+11:002014-11-25T03:48:16.135+11:00Building FFmpeg on Windows with in-line asm and the Intel compiler.Building FFmpeg for Windows is generally done using MinGW compiler in either a MSYS or Cygwin environment. But the resulting libraries don't always play all that well when you try and use them in a native Windows program (i.e. something built with the Microsoft compiler -msvc). For those who work on Windows it is often slightly more convenient to be able to use a native build chain when creating FFmpeg binaries.<br />
<br />
Unfortunately FFmpeg (much like most opensource projects) uses C99 code which will not build using the msvc compiler found in Visual Studio. This is because Microsoft in all there wisdom don't support anything past C89. With the standardisation of C++11 newer versions of msvc are slowly starting to support C99 features as the C++11 specification mandates that they must be supported. So supporting C++11 generally also means supporting C99. However this support started to feature in Visual Studio 2013 and it is still far from complete and can not compile many C99 based projects (at time of writing).<br />
<br />
Things are made worse as FFmpeg doesnt just use C99 but also has large amounts of in-line assembly. This assembly is written in the AT&T assembly syntax (or more specifically GAS - GNU Assembler) as opposed to Intel assembly syntax. Since msvc only supports Intel syntax then even with C99 support it will never be possible to compile FFmpeg with in-line assembly natively. Of-course FFmpeg has a nice config option that can be used to disable all in-line asm which would mean that it would compile on a C99 compliant compiler without needing AT&T assembly support. But then we don't get all the nicely hand tuned assembly optimisations that were written specifically to maximise performance.<br />
<br />
Luckily there is the Intel compiler to the rescue. The Intel compiler for Windows will generate a native windows library that can be easily linked and debugged like any other msvc library. It also has the added benefit of supporting AT&T assembly compilation on Windows.<br />
<br />
Unfortunately Intels AT&T support is not fully featured on Windows. On Linux the Intel compiler just uses GAS so it supports everything GAS does. But on Windows Intel had to implement the support themselves and as a result it is not as feature complete as GAS. The main issue is that the Intel compiler on Windows does not support 'Direct symbol references' which are used in various places within FFmpeg. In order to compile on Windows these lines need to be updated in order to remove the symbol references.<br />
<br />
An example of the problem and the fix can be found in libavcodec/x86/dsputil_mmx.c:<br />
<br />
<pre class="brush:diff">--- a/libavcodec/x86/dsputil_mmx.c
+++ b/libavcodec/x86/dsputil_mmx.c
@@ -115,13 +115,13 @@ void ff_put_signed_pixels_clamped_mmx(const int16_t *block, uint8_t *pixels,
x86_reg line_skip3;
__asm__ volatile (
- "movq "MANGLE(ff_pb_80)", %%mm0 \n\t"
+ "movq %4, %%mm0 \n\t"
"lea (%3, %3, 2), %1 \n\t"
put_signed_pixels_clamped_mmx_half(0)
"lea (%0, %3, 4), %0 \n\t"
put_signed_pixels_clamped_mmx_half(64)
: "+&r"(pixels), "=&r"(line_skip3)
- : "r"(block), "r"(line_skip)
+ : "r"(block), "r"(line_skip), "m"(ff_pb_80)
: "memory");
}</pre>
<br />
Here the symbol reference is replaced with an asm-interface.<br />
<br />
There are many occurrences of direct symbol references in FFmpeg (to many to list here) so I will instead direct people to check out the attached patch.<br />
<br />
The attached patch includes updates to the in-line asm that allow most of it to compile under the Intel compiler. Unfortunately there are 2 places where the asm uses slightly more complicated features than just symbol references so these cant be easily converted. These instances are in libavcodec/x86/mlpdsp.c where the use of labels (combined with a jump list) is not supported by Intel on Windows and is not easily modified. The second instance is in the BRANCHLESS_GET_CABAC macro in libavcodec/x86/cabac.h. This contains slightly more complex use of symbol references that if just swapped out like previous occurrences still doesn't compile without some additional work.<br />
<br />
However despite a couple of problem functions the attached patch will allow for FFmpeg compilation on Windows with the Intel compiler and in-line asm optimisations enabled.<br />
<br />
<u><b>Update 2:</b></u> New and improved patches have been created and these are now available directly in FFmpeg master (no patching required). See my post for more information (<a href="http://siliconandlithium.blogspot.com/2014/05/building-ffmpeg-on-windows-with-in-line.html">Building FFmpeg on Windows with in-line asm and the Intel compiler (Part 3)</a>).<br />
<u style="font-weight: bold;">Update:</u> A new patch is available in my newer post (<a href="http://siliconandlithium.blogspot.com/2014/01/building-ffmpeg-on-windows-with-in-line.html">Building FFmpeg on Windows with in-line asm and the Intel compiler (Part 2)</a>).<br />
<br />
Download patch file:<br />
<a href="https://github.com/ShiftMediaProject/FFmpeg/commit/a810d0822fd9f894a3750bdb0cbea3014229ae8b.diff">https://github.com/ShiftMediaProject/FFmpeg/commit/a810d0822fd9f894a3750bdb0cbea3014229ae8b.diff</a><br />
<br />
Lastly for those who don't want to apply the modifications themselves and would like a nicer build method than using MSYS/Cygwin then you can check out my git repository which has working Visual Studio build projects. These projects are in the 'SMP' directory and of course require a valid install of the Intel Windows compiler.<br />
<a href="https://github.com/ShiftMediaProject/FFmpeg">https://github.com/ShiftMediaProject/FFmpeg</a><br />
<br />Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0tag:blogger.com,1999:blog-168061082748867114.post-51250678658154639402013-12-21T13:23:00.000+11:002014-01-05T15:51:49.861+11:00Building FFmpeg with libssh on WindowsFFmpeg recently added support for SFTP protocol through the use of libssh. However if you have tried to use this feature on Windows you would have discovered that FFmpeg will not build without some changes. These changes are pretty simple and are yet another case of the Windows headers not including things that are commonly found in their Linux counterparts. In this case whats missing is some defines used for flagging permissions and other things on read/write operations. So in order to get them to work we just have to add the appropriate defines to the file.<br />
<br />
So if your curious as to how to get libssh support in FFmpeg on Windows then all you need to do is add the following lines to libavformat/libssh.c:<br />
<br />
<pre class="brush:diff">@@ -20,12 +20,34 @@
#include <fcntl.h>
#include <libssh/sftp.h>
+#include <sys/stat.h>
#include "libavutil/avstring.h"
#include "libavutil/opt.h"
#include "avformat.h"
#include "internal.h"
#include "url.h"
+#if defined(_WIN32)
+#ifndef S_IRUSR
+#define S_IRUSR S_IREAD
+#endif
+#ifndef S_IWUSR
+#define S_IWUSR S_IWRITE
+#endif
+#ifndef S_IRGRP
+#define S_IRGRP 0
+#endif
+#ifndef S_IWGRP
+#define S_IWGRP 0
+#endif
+#ifndef S_IROTH
+#define S_IROTH 0
+#endif
+#ifndef S_IWOTH
+#define S_IWOTH 0
+#endif
+#endif
+
typedef struct {
const AVClass *class;
ssh_session session;
</pre>
<br />
And thats it. After adding those lines then assuming you have a built libssh library then everything should work fine.Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0tag:blogger.com,1999:blog-168061082748867114.post-60457369571850203102013-12-11T18:50:00.000+11:002013-12-11T18:53:10.752+11:00Compiling LibVPX 1.3.0 using MinGWRecently Ive been building a script to compile ffmpeg (in interesting ways). As part of that I have been generating ffmpeg libraries using MinGW from within MSYS. As anyone who has tried to do this before will know that this is not exactly the quickest and simplest of tasks. Part of my ffmpeg build uses the libvpx library for VP8/VP9 encoding. Libvpx has a pretty good set of build scripts but I found that I had to change a few things in order to get it to work in my MinGW build chain.<br />
<br />
Firstly I was getting errors during compilation related to the 'strip' command. The error I was getting was from strip complaining that passed parameters where to long. While other people have run into a similar error where strip was failing due to a "Bad file number" error. Luckily the fix is the same for both problems (and is a rather simple one). Just disable strip from running. This can be done by modifying the Makefile <b>after</b> running configure.<br />
<br />
The offending line is in 'libs-x86-win32-gcc.mk' (or libs-x86_64-**** depending on whether your building 32/64 bit).<br />
<br />
<pre class="brush:bash first-line:69">HAVE_GNU_STRIP=yes</pre>
<br />
Just change the 'yes' to a 'no' and your golden. For those who dont want to have to do this manually here is a bash script that will do it for you.<br />
<br />
<pre class="brush:bash">#There is a bug that causes compilation to fail due to an error in strip
if !( sed -i -e 's/HAVE_GNU_STRIP=yes/HAVE_GNU_STRIP=no/g' $VPXFOLDER/libs-$LOCALTARGET.mk ); then
echo " Error: libvpx make update failed"
exit 1
fi</pre>
<br />
Where '$VPXFOLDER' is the location of your vpx source and '$LOCALTARGET' is either 'x86-win32-gcc' or 'x86_64-win64-gcc' depending on if you are compiling 32 or 64 bit respectively.<br />
<br />
This fixes most common MinGW errors associated with libVPX but I was also getting a linker error about missing 'strtok_r'. This was due to an inconsistency in my MinGW build with the use of <span style="font-family: inherit;">'MINGW_HAS_SECURE_API'. </span><br />
<span style="font-family: inherit;">The fix was just a simple edit in the file 'svc_encodeframe.c':</span><br />
<br />
<pre class="brush:diff">@@ -23,7 +23,7 @@
#include "vpx/vp8cx.h"
#include "vpx/vpx_encoder.h"
-#if defined(__MINGW32__) && !defined(MINGW_HAS_SECURE_API)
+#if defined(__MINGW32__)
#define strtok_r strtok_s
// proto from /usr/x86_64-w64-mingw32/include/sec_api/string_s.h
_CRTIMP char *__cdecl strtok_s(char *str, const char *delim, char **context);
</pre>
<div>
<br /></div>
<div>
Lastly there was an error when generating a 64bit version using experimental mingw-w64. This was due to an incompatible pthread implementation that caused errors when building the libvpx tests. This error is easiest to avoid by just turning of the generation of the tests.<br />
<br />
For those interested the final configure parameters looked like this:<br />
<pre class="brush:plain">./configure --disable-shared --enable-static --target=$LOCALTARGET --disable-unit-tests --disable-install-bins --disable-examples --disable-docs --enable-vp8 --enable-vp9
</pre>
<br />
Just like before the '$LOCALTARGET' variable should be set to either 'x86-win32-gcc' or 'x86_64-win64-gcc' for 32 or 64 bit compilation respectively. The above builds static libraries, if you want to generate a dll just swap the 'enable'/'disable' bits for the shared and static parameters (i.e. --enable-shared --disable-static)<br />
<br /></div>
Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0tag:blogger.com,1999:blog-168061082748867114.post-55073025058016134402013-12-10T14:07:00.001+11:002013-12-10T14:07:32.116+11:00A New BlogAs part of my day to day coding work I am constantly fixing (often after creating) new problems. After each fix I often find myself thinking that someone else out there may have had the same problem and may be interested in how to solve it. Of course previously this has been nothing more than a passing thought and on the odd occasion it is more than that by the time I get around to doing anything about I have often forgotten what the fix process was myself.<br />
<br />
So I finally made the decision to try and put some of those little fixes and insights into the public domain where maybe, just maybe they may be of use to someone. I for one have found a few solutions from other peoples blogs and decided it was probably about time I contributed a little something back.<br />
<br />
That said I cant guarantee any future fixes/insights will be any good (or exist at all) but if they are then, well, here goes...Anonymoushttp://www.blogger.com/profile/07812347642033023141noreply@blogger.com0