Chrooting from a 32bit userland into a 64bit jail

Before you start

Although this work on any linux distribution, this text will talk specifically about gentoo linux. The motivation for this writing is that I run an hybrid gentoo installation (32bit userland running a 64bit kernel) and now I want to migrate to a full 64bit installation.

As a great opportunity to learn a bit more I decided to try to install the new system while still running the old one.

An alternative way to run ELF binaries

Chrooting into a different userland, in this case from 32bit into 64bit, is not trivial. This happens because a 32bit shell cannot directly run a 64bit binary. Here is an example:

daltonmatos@jetta:~ 
> file `which bzip2`
/bin/bzip2: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), 
dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped

daltonmatos@jetta:~
> file /usr/x86_64-pc-linux-gnu/bin/bzip2
/usr/x86_64-pc-linux-gnu/bin/bzip2: ELF 64-bit LSB executable, x86-64, 
version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, stripped

daltonmatos@jetta:~
 > /bin/bzip2
 bzip2: I won't write compressed data to a terminal.
 bzip2: For help, type: `bzip2 --help'.

daltonmatos@jetta:~
 > /usr/x86_64-pc-linux-gnu/bin/bzip2
 zsh: no such file or directory: /usr/x86_64-pc-linux-gnu/bin/bzip2
daltonmatos@jetta:~

as you can see, the 64bit binary could not be directly executed and the error message is everything but helpful.  We need an alternative way to execute an ELF binary. Luckly the linux dynamic linker/loader (ld.so, ld-linux.so*) can also be used to run any ELF binary. All you have to do is invoke it directly passing as the only argument the final binary you need to run. So this is essentially the same:

 daltonmatos@jetta:~
 > bzip2
 bzip2: I won't write compressed data to a terminal.
 bzip2: For help, type: `bzip2 --help'.

daltonmatos@jetta:~
 > /lib/ld-linux.so.2 /bin/bzip2
 bzip2: I won't write compressed data to a terminal.
 bzip2: For help, type: `bzip2 --help'.
daltonmatos@jetta:~

We could try to use our current /lib/ld-linux.so.2 to run na 64bit binay but unfortunately this does not work as you can see:

daltonmatos@jetta:~
> /lib/ld-linux.so.2 /usr/x86_64-pc-linux-gnu/bin/bzip2 
/usr/x86_64-pc-linux-gnu/bin/bzip2: error while loading shared libraries: /usr/x86_64-pc-linux-gnu/bin/bzip2: wrong ELF class: ELFCLASS64

What do we need is a 64bit dynamic linker/loader and for this we must install a basic cross-compile environment.

Installing a cross-compiler

To have a cross-compile environment on gentoo you need to install the crossdev package. This is simple as:

emerge crossdev

Now you use the crossdev script to build the essential tools for the desired target. In this case we need 64bit target:

crossdev -t x86_64-pc-linux-gnu

This will build the essential tools so you can generate 64bit ELF binaries even being in a host 32bit system. After this we have a 64bit dynamic loader/linker that we use to run any 64bit ELF binary:

daltonmatos@jetta:~
> file /usr/x86_64-pc-linux-gnu/lib/ld-linux-x86-64.so.2
/usr/x86_64-pc-linux-gnu/lib/ld-linux-x86-64.so.2: symbolic link to `ld-2.17.so'

daltonmatos@jetta:~
> file /usr/x86_64-pc-linux-gnu/lib/ld-2.17.so 
/usr/x86_64-pc-linux-gnu/lib/ld-2.17.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, stripped

daltonmatos@jetta:~
> /usr/x86_64-pc-linux-gnu/lib/ld-2.17.so 
Usage: ld.so [OPTION]... EXECUTABLE-FILE [ARGS-FOR-PROGRAM...]
You have invoked `ld.so', the helper program for shared library executables.
This program usually lives in the file `/lib/ld.so', and special directives
in executable files using ELF shared libraries tell the system's program
loader to load the helper program from this file. This helper program loads
the shared libraries needed by the program executable, prepares the program
to run, and runs it. You may invoke this helper program directly from the
command line to load and run an ELF executable file; this is like executing
that file itself, but always uses this helper program from the file you
specified, instead of the helper program file specified in the executable
file you run. This is mostly of use for maintainers to test new versions
of this helper program; chances are you did not intend to run this program.
--list list all dependencies and how they are resolved
 --verify verify that given object really is a dynamically linked
 object we can handle
 --inhibit-cache Do not use /etc/ld.so.cache
 --library-path PATH use given PATH instead of content of the environment
 variable LD_LIBRARY_PATH
 --inhibit-rpath LIST ignore RUNPATH and RPATH information in object names
 in LIST
 --audit LIST use objects named in LIST as auditors
daltonmatos@jetta:~
>

This means that now we have a way to run an arbitrary 64bit binary. Let’s test it with the first example we saw earlier.

daltonmatos@jetta:~
> file /usr/x86_64-pc-linux-gnu/bin/bzip2 
/usr/x86_64-pc-linux-gnu/bin/bzip2: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, stripped

daltonmatos@jetta:~
> /usr/x86_64-pc-linux-gnu/lib/ld-2.17.so /usr/x86_64-pc-linux-gnu/bin/bzip2 
/usr/x86_64-pc-linux-gnu/bin/bzip2: error while loading shared libraries: libbz2.so.1: wrong ELF class: ELFCLASS32
daltonmatos@jetta:~
>

Not so easy! This error means that the wrong shared library was chosen by the dynamic loader/linker. Now we have a new problem, we cannot run an 64bit binary using 32bit libraries. Fortunately there is an easy fix for this.

Solving shared libraries problems

What we need is to change the directories where the dynamic loader/linker will search for shared libraries when resolving the dependencies. This can be done using the LD_LIBRARY_PATH environment variable.

daltonmatos@jetta:~
> LD_LIBRARY_PATH=/usr/x86_64-pc-linux-gnu/lib /usr/x86_64-pc-linux-gnu/lib/ld-2.17.so /usr/x86_64-pc-linux-gnu/bin/bzip2 
bzip2: I won't write compressed data to a terminal.
bzip2: For help, type: `bzip2 --help'.
daltonmatos@jetta:~
>

That’s much better! The crossdev script puts the target shared libraries inside lib/ and usr/lib directories, so it’s advisable to add both to the LD_LIBRARY_PATH variable.

Putting all the pieces together

Now that we know how to run a 64bit binary being in a 32bit userland we need to run the main point of all this, the chroot binary.

daltonmatos@jetta:~
> LD_LIBRARY_PATH=/usr/x86_64-pc-linux-gnu/lib:/usr/x86_64-pc-linux-gnu/usr/lib/ /usr/x86_64-pc-linux-gnu/lib/ld-linux-x86-64.so.2 /usr/x86_64-pc-linux-gnu/bin/chroot 
/usr/x86_64-pc-linux-gnu/bin/chroot: missing operand
Try '/usr/x86_64-pc-linux-gnu/bin/chroot --help' for more information.

Awesome! Now that we can run the chroot binary, let’s do the chroot.

daltonmatos@jetta:~
> sudo LD_LIBRARY_PATH=/usr/x86_64-pc-linux-gnu/lib:/usr/x86_64-pc-linux-gnu/usr/lib/ /usr/x86_64-pc-linux-gnu/lib/ld-linux-x86-64.so.2 /usr/x86_64-pc-linux-gnu/bin/chroot /mnt/gentoo64 /bin/bash
jetta / #

Here we are already inside the 64bit jail, let’s check.

jetta / # which bzip2
/bin/bzip2
jetta / # file /bin/bzip2 
/bin/bzip2: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.9, stripped
jetta / # /bin/bzip2 
bzip2: I won't write compressed data to a terminal.
bzip2: For help, type: `bzip2 --help'.
jetta / #

And that’s it! We are now in a full 64bit jail, ready to install a new gentoo system while running an hybrid (32bit userland, 64bit kernel) system, that is, a brand new gentoo with no downtime!

Thanks for reading!

  1. Deixe um comentário

Deixe uma resposta

Preencha os seus dados abaixo ou clique em um ícone para log in:

Logotipo do WordPress.com

Você está comentando utilizando sua conta WordPress.com. Sair / Alterar )

Imagem do Twitter

Você está comentando utilizando sua conta Twitter. Sair / Alterar )

Foto do Facebook

Você está comentando utilizando sua conta Facebook. Sair / Alterar )

Foto do Google+

Você está comentando utilizando sua conta Google+. Sair / Alterar )

Conectando a %s

%d blogueiros gostam disto: