A description of RPATH $ORIGIN LD_LIBRARY_PATH and portable linux binaries
ldd doesn’t always give the true answer.
Linking to make portable binaries on a linux system
Portable executables means that the resultant binaries will run on different modern linux distros.
Linking to the system libc and the system libm is ok by me. You just have to make sure that you are linking to a glibc version that is older than the on all the distros you are targeting. So the build machine can link to GLIBC 2.7 and then all distros that have GLIBC >=2.7 can run the output binaries. For many arcane reasons you don’t want to statically link to glibc. I don’t statically link to glibc (libc) or libm, and I don’t include those libs either. I let the exe find the system version of those libs. Also I don’t bother to include my own pthreads lib either. These are core libs.
I’m targeting modern linux distros. The scope of the problem can’t be too big.
I build the executables that I want to be made portable in a virtualbox running an older debian. Lately I have been using debian lenny to compile ffmpeg and it’s support libs.
Considering RPATH and LD_LIBRARY_PATH
I’ve been using rpath embedded in an executable to set a high-priority library search path. Instead of using LD_LIBRARY_PATH which is very brittle, the search path gets put right into the executable, so you can do things like search the same directory where the exe is for the libs it depends on before searching system libs.
The RPATH is stored in the elf executable, in the dynamic section. It can be a relative path. So you can move the exe and it’s support libs around and it will always find it’s libraries and load those before trying the system ones, isolating your binaries.
LD_LIBRARY_PATH gets a lot of flack, and some of it is deserved. The reason I don’t like it is because you can’t just symlink the executable and run it anywhere, the executable needs the variable set, so you always have to run a shell script to bootstrap the environment with that variable and then call the executable. Also, it only takes an absolute path. So it’s brittle.
Other people don’t like it because some people have the tendency to make it a global system setting, breaking other things in subtle ways. Another reason is because any process that now has LD_LIBRARY_PATH set exports that same variable to any subprocess it starts, so again you get unwitting breakage when other processes get spawned.
Here’s some output of readelf telling it to only show the dynamic section. I trimmed most of the output.
$ readelf -d ffmpeg Dynamic section at offset 0x12574 contains 27 entries: Tag Type Name/Value 0x00000001 (NEEDED) Shared library: [libavdevice.so.52] 0x00000001 (NEEDED) Shared library: [libavformat.so.52] 0x00000001 (NEEDED) Shared library: [libavcodec.so.52] 0x00000001 (NEEDED) Shared library: [libavutil.so.49] 0x00000001 (NEEDED) Shared library: [libm.so.6] 0x00000001 (NEEDED) Shared library: [libc.so.6] 0x0000000f (RPATH) Library rpath: [$ORIGIN/../lib] 0x0000000c (INIT) 0x804a438 etc.
$ORIGIN is a special variable that means ‘this executable’, and it means the actual executable filename, as readlink would see it, so symlinks are followed. In other words, $ORIGIN is special and resolves to wherever the binary is at runtime.
You can see the RPATH there. It’s a string in the constant pool in the elf section. Because it’s an index into the string constant pool, you can only change it after the executable is built to same size or smaller. I use the great program called chrpath which is a special purpose tool to change this single string in an executable.
chrpath, and the version I am using is 0.13
So you have to compile the executable so it puts an RPATH in the header. You do this by giving a special flag to gcc which will give it to ld, the linker. It goes like this:
Getting this value into gcc is not easy. Because of quoting issues, you can’t just stick this anywhere, the $ dollar sign gets interpreted by the shell, etc, so what I like to do is just set it to this:
I replaced the dollar sign with the letter X. After the binary is compiled and made I will use chrpath to set the string to what I want it to which is the same thing with a dollar sign. Remember the constant pool, that’s why you need to reserve space in the exe. This is a trick to side-step the quoting hell that many people on the net have suffered through, myself included. Luckily I saw a neat sidestep.
Coaxing ./configure to get this in there:
LDFLAGS="-Wl,-rpath=XORIGIN/../lib" ./configure --prefix=/blabla/place
See the X? That will be replaced by a dollar sign later when you run chrpath on the resultant binaries. The configure script will see the LDFLAGS and pass it to gcc etc and the build system will incorporate that flag. See the comma between -Wl and -rpath? That’s necessary too.
Once you have done this, you will get the readelf output above, which will make the exe look in ../lib for it’s dependencies before looking in the system lib dirs like /lib and /usr/lib.
So that means you can have binaries set up like this:
place/bin/ffmpeg place/lib/libavcodec.so place/lib/libavdevice.so place/lib/libavformat.so place/lib/libavutil.so
You can just move place/ around and the ffmpeg exe will always find it’s libs before even trying to load the system ones. This is good on systems where you may actually have the libs in the system lib dirs but dont want them to be loaded, perhaps they were not compiled with the options that you want. Also good for systems where it doesn’t even have the libs at all in the system lib dirs. It effectively isolates the binaries.
Now getting back to ldd not giving the right answer sometimes. I think it is based on the fact that ldd is a shell script, and not an executable itself. Check that by doing “file `which ldd`” and seeing what type of file it is.
If we do ldd on the ffmpeg, you will get the right answer:
user@debian:~/i/bin$ ldd ffmpeg linux-gate.so.1 => (0xb77c1000) libavdevice.so.52 => /home/user/i/bin/./../lib/libavdevice.so.52 (0xb77b9000) libavformat.so.52 => /home/user/i/bin/./../lib/libavformat.so.52 (0xb779e000) libavcodec.so.52 => /home/user/i/bin/./../lib/libavcodec.so.52 (0xb769c000) libavutil.so.49 => /home/user/i/bin/./../lib/libavutil.so.49 (0xb768b000) libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7657000) libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb7510000) /lib/ld-linux.so.2 (0xb77c2000)
But now let’s make a symlink to this file and place the symlink in another directory, and run ldd on the symlink:
user@debian:~$ ldd ./symlinked-ffmpeg linux-gate.so.1 => (0xb77d5000) libavdevice.so.52 => /usr/lib/i686/cmov/libavdevice.so.52 (0xb77bb000) libavformat.so.52 => /usr/lib/i686/cmov/libavformat.so.52 (0xb76c2000) libavcodec.so.52 => /usr/lib/i686/cmov/libavcodec.so.52 (0xb6e77000) libavutil.so.49 => /usr/lib/i686/cmov/libavutil.so.49 (0xb6e67000) libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb6e41000) libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb6cfa000) etc output trimmed.
Now it’s finding the system libs, so even though the rpath was put inside the elf header, it looks like it had no effect. BUT this is WRONG in actuality, and can be verified by doing:
user@debian:~$ LD_TRACE_LOADED_OBJECTS=1 ./symlinked-ffmpeg linux-gate.so.1 => (0xb77fc000) libavdevice.so.52 => /home/user/i/bin/../lib/libavdevice.so.52 (0xb77f4000) libavformat.so.52 => /home/user/i/bin/../lib/libavformat.so.52 (0xb77d9000) libavcodec.so.52 => /home/user/i/bin/../lib/libavcodec.so.52 (0xb76d7000) libavutil.so.49 => /home/user/i/bin/../lib/libavutil.so.49 (0xb76c6000) libm.so.6 => /lib/i686/cmov/libm.so.6 (0xb7692000) libc.so.6 => /lib/i686/cmov/libc.so.6 (0xb754b000) /lib/ld-linux.so.2 (0xb77fd000)
So this command actually works.
What this command does is set an environment variable called LD_TRACE_LOADED_OBJECTS and then run the executable. When the linux loader sees this env variable has been set, instead of running the exe it will output the libs that it loads instead and exit. So you’re seeing the “real” libs that get loaded rather then some shell script fuckup, which is what I think ldd is.
tell me exactly what libs get loaded:
and with the verbose flag to see even more info:
LD_VERBOSE=1 LD_TRACE_LOADED_OBJECTS=1 ./ffmpeg
LD_VERBOSE=1 LD_TRACE_LOADED_OBJECTS=1 ./symlinked-ffmpeg
again the same output
another way is to take out the big guns and run strace on the exe:
strace output tells me that the real file (not the symlink) is found by readlink:
readlink(“/proc/self/exe”, “/home/user/i/bin/ffmpeg”, 4096) = 30
then it tries to find libavdevice and succeeds, remember this is using the real file path now, because it did readlink:
open(“/home/user/i/bin/../lib/libavdevice.so.52”, O_RDONLY) = 3
There you have it, RPATH actually works. No LD_LIBRARY_PATH needs to be set anymore.
It’s difficult to get the RPATH into the elf executable as quoting issues of $ORIGIN are rampant, but it’s worth it, especially if you sidestep the quoting and reserve the space first and then use chrpath after.
I’m not sure but you may also want to run chrpath on any shared libraries your compilation generates as well so subsequent levels of dependencies can be resolved but that may just get handled by the RPATH in the executable you are running. I don’t know.
How deep is the rabbit hole?
Filed under: Uncategorized | Leave a Comment