Running 64bit binaries on 32bit userland with 64bit kernel

The problem

Some time ago I wrote about how my Gentoo Linux is installed (blogpost in pt_BR) on my laptop. Long story short: I have a hybrid system, 32bit userland running on a 64bit kernel. I opted to do this type of install because I still have an impression that a full 64bit system still has some key problems (in my past adventures flash was one of them). Anyway, this setup works very well. I can use all 8GB (not by one same process) for example. But not long after the installation I discovered one thing that got me confused.

How come even with a 64bit kernel I still coudn’t run any 64bit ELF binary?

I thought that just having a 64bit kernel was enough to run 32 and 64bit binaries, in fact it is! But is not so trivial do run a 64bit binary in a 32bit userland.

The first attempt, no luck

The first thing I tried was obviously call the binary directly on the command line:

daltonmatos@jetta ~/src [158]$ ./m64
-bash: ./m64: No such file or directory
daltonmatos@jetta ~/src [159]$

This is, indeed, a very cryptic error message.

Enter the dynamic loader

All binaries (at least the massive majority) on your system is a dynamically linked ELF. And you can check this with the file command.

daltonmatos@jetta ~/src [179]$ file /bin/ls
/bin/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), 
for GNU/Linux 2.6.9, stripped
daltonmatos@jetta ~/src [180]$

Remember, 32bit userland. A bit further and we will see this for 64 bit binaries.

By dynamic linked you can understand that all libraries that the binary depends on are loaded at runtime. You can check which are these libraries with the ldd command.

daltonmatos@jetta ~/src [180]$ ldd /bin/ls
 linux-gate.so.1 => (0xffffe000)
 librt.so.1 => /lib/librt.so.1 (0xf776c000)
 libacl.so.1 => /lib/libacl.so.1 (0xf7763000)
 libc.so.6 => /lib/libc.so.6 (0xf7605000)
 libpthread.so.0 => /lib/libpthread.so.0 (0xf75eb000)
 /lib/ld-linux.so.2 (0xf7796000)
 libattr.so.1 => /lib/libattr.so.1 (0xf75e4000)
daltonmatos@jetta ~/src [181]$

Note: Thee ldd command is just a wrapper around the real dynamic loader, usually /lib/ld.so or /lib/ld-<version>.so. I’m running glibc-2.13 so here I have /lib/ld-2.13.so. Running ldd <some-binary> is the same as running /lib/ld-2.13.so --list <some-binary>. So for now on we will be using the second.

Here begins the differencies. The 32 bit ld.so (or the ldd wrapper) tool does not know how to handle a 64 bit binary, as we can see:

daltonmatos@jetta ~/src [229]$ /lib/ld-2.13.so --list ./m64
./m64: error while loading shared libraries: ./m64: wrong ELF class: ELFCLASS64
daltonmatos@jetta ~/src [230]$

And the ldd command:

 
daltonmatos@jetta ~/src [190]$ ldd ./m64
 not a dynamic executable

which is clearly not true, since I compiled m64 myself using a regular gcc that outputs 64bit binaries. So we need a 64bit-capable ld.so. Since I have two compilers (one that generates 32bit binaries and another that generates 64bit) I need to have two complete tool chains (basically: binutils, gcc, gdb, glibc). So my second tool chain has what I need.

Specifically on Gentoo Linux, the cross compiling infrastructure creates /usr/<arch>-pc-linux-gnu folder and stores all files under it so I have a /usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so, that is, the dynamic loader for 64bit binaries. Yes, I have a different version of the glibc for 64bit (2.14.1).

So let’s re-check our binaries, but now with this new tool:

daltonmatos@jetta ~/src [191]$ file ./m64 
./m64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), 
for GNU/Linux 2.6.9, not stripped
daltonmatos@jetta ~/src [192]$

Alright, our brand new 64bit program. Now, the shared libraries:

daltonmatos@jetta ~/src [194]$ /usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so --list ./m64
 linux-vdso.so.1 => (0x00007fffa90fb000)
 libc.so.6 => /usr/x86_64-pc-linux-gnu/lib/libc.so.6 (0x00007f103f367000)
 /lib64/ld-linux-x86-64.so.2 => /usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so (0x00007f103f6f7000)
daltonmatos@jetta ~/src [195]$

That’s awesome! You maybe asking yourself: “How is it possible to run the ld.so binary directly but not other 64bit binaries?” And you’re right, because the ld.so itself is a 64bit binary:

daltonmatos@jetta ~/src [197]$ file /usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so 
/usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), 
dynamically linked, stripped
daltonmatos@jetta ~/src [198]$

If you know this, let me know because I don’t! =) I just discovered that the dynamic loader itself is runnable directly from the command line and can be used to run other binaries.

daltonmatos@jetta ~/src [200]$ file ./m64
./m64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), 
for GNU/Linux 2.6.9, not stripped
daltonmatos@jetta ~/src [201]$ /usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so ./m64 
Hello World
daltonmatos@jetta ~/src [202]$

Like magic! =)

If you tried these exact steps I  just described you probably got an error like this one:

daltonmatos@jetta ~/src [202]$ /usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so ./m64 
./m64: error while loading shared libraries: libc.so.6: wrong ELF class: ELFCLASS32
daltonmatos@jetta ~/src [203]$

What does this mean? Remember that we have two glibc’s installed? Both binaries depend on a shared library with the name libc.so.6. So shouldn’t the SO handle this automatically? Yes, but it doesn’t and now I have to tell you that I’ve been hiding one piece of important information from you. Is the second key to be able to run 64bit binaries, being the first the discovery of ld.so as a generic binary runner.

Go back in the output of the ld.so --list command and see how the libc.so.6‘s path is resolved in both cases. See that they have different paths? 32 bit resloves to /lib/libc.so.6 and 64bit resolves to /usr/x86_64-pc-linux-gnu/lib/libc.so.6. So how do we handle this?

It turns out that we can change where the dynamic loader looks for shared objects with the LD_LIBRARY_PATH environment variable. So all commands using the 64bit loader need to be prepended with LD_LIBRARY_PATH=<new-path>, like this:

daltonmatos@jetta ~/src [203]$ LD_LIBRARY_PATH=/usr/x86_64-pc-linux-gnu/lib /usr/x86_64-pc-linux-gnu/lib/ld-2.14.1.so ./m64 
Hello World
daltonmatos@jetta ~/src [204]$

One of the key advantages of having the ability to do this is that you can compile some programs to 64bit code and this program will be able to use more that 4GB of RAM, if needed.

An easier, but not optimal way

There is another way to do this that will not require tweaking the library loading path or anythin. You can compile all your 64bit binaries as statically linked ELF’s, like this:

daltonmatos@jetta ~/src [218]$ x86_64-pc-linux-gnu-gcc -static -o m64s main.c
daltonmatos@jetta ~/src [219]$ file m64s 
m64s: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.9, not stripped
daltonmatos@jetta ~/src [220]$

This way you won’t need the dynamic loader and then you can run your binary directly from the command line:

daltonmatos@jetta ~/src [220]$ ./m64s
Hello World
daltonmatos@jetta ~/src [221]$

But this comes with a cost. Extra disk space:

daltonmatos@jetta ~/src [222]$ file m64*
m64: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, not stripped
m64s: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, for GNU/Linux 2.6.9, not stripped
daltonmatos@jetta ~/src [223]$ ls -lh m64*
-rwxr-xr-x 1 daltonmatos daltonmatos 7,7K Abr 8 17:15 m64
-rwxr-xr-x 1 daltonmatos daltonmatos 771K Abr 10 20:32 m64s
daltonmatos@jetta ~/src [224]$

The statically linked binary is 100 times bigger than the dynamic one. This is because you have all that’s needed bundled into the static binary. You can reduce the final size using the strip command but you won’t get a significant reduction.

That’s it! Thanks for reading!

Anúncios