Building the LLVM based binary translator.
Lars.Rasmusson@sics.se 2010-10-21
1. Create clean build root file system
To make sure that we know what software we are using to build, we first create an entire root file system in a catalog. It containts its own build tools, independent of what the host OS itself uses.
Use debootstrap to install a bare bones root file system from a local repository:
sudo -s BUILDROOT=$(pwd)/xenbuild mkdir $BUILDROOT debootstrap --variant=buildd --arch i386 lucid $BUILDROOT http://se.archive.ubuntu.com/ubuntu
2. chroot into it, configure and install build tools and git
Set time zone, and use apt to install the build tools
chroot $BUILDROOT export LC_ALL=C hostname builder dpkg-reconfigure tzdata adduser --disabled-password larsr
Use a local repository and install a lot of packets
echo > /etc/apt/sources.list deb http://se.archive.ubuntu.com/ubuntu lucid main restricted universe apt-get update apt-get install -y wget nano mercurial gawk openssh-client libz-dev \ gettext libssl-dev libx11-dev bin86 bcc libncurses5-dev python-dev \ texinfo bridge-utils uuid-dev git-core man iasl m4 flex bison cmake
3. check out xen from git on mac (edit on mac?)
Initialize my git user name
su larsr cd $HOME cat >> .gitconfig <<EOF [user] name = Lars Rasmusson email = Lars.Rasmusson@sics.se EOF git clone ssh://lra@abakus.sics6.se/home/lra/src/xen-4.0-testing.git
4. Building and installing the Xen kernel
NOTE! These steps are only necessary if you are not already running xen on your testing machine.
4.1 Build
Prepare a new branch llvm-domain and check out a specific version of Xen. In the buildroot tree, do
cd ~/xen-4.0-testing git checkout -b llvm-domain RELEASE-4.0.1 git clean -fd make -j8 xen make -j8 tools
4.2 Install
In the real root tree, copy the new xen-image to the /boot directory, and configure the bootloader (grub/grub2/whatever) to load the newly compiled xen.
First pack the xen files into a tar file and then into a deb. It makes it easier to remove or update later.
tar -czf xen.tgz -C dist/install/ . fakeroot alien --to-deb xen.tgz
Then install the files in the real root file system. The update-rc.d commands makes xend and xendomains start when the system boots.
dpkg -i /home/larsr/xenbuild/home/larsr/xen-4.0-testing/xen_1-2_all.deb update-rc.d xend defaults update-rc.d xendomains defaults
I'm running Ubuntu 10.04 Lucid Lynx which uses the grub2 boot loader. I installed Xen in grub by putting the following into /etc/grub.d/35_xen
#!/bin/sh exec tail -n +3 $0 # This file is an example on how to add custom entries menuentry "Xen 4.0 / Ubuntu 10.04 kernel 2.6.31.13 pvops" { insmod ext2 set root=(hd0,1) multiboot (hd0,1)/xen-4.0.gz com1=115200,8n1 console=com1,vga module (hd0,1)/vmlinuz-2.6.31.13 console=tty0 console=hvc0 root=/dev/mapper/cluster5-root ro module (hd0,1)/initrd.img-2.6.31.13 savedefault }
and then run
update-grub
A linux running in Xen gets its console on hvc0, and for Ubuntu to start a getty on hvc0, put hvc0.conf in /etc/init/hvc0.conf
# hvc0 - getty # # This service maintains a getty on Xen's ttyhvc0 from the point the system is # started until it is shut down again. start on stopped rc RUNLEVEL=[2345] stop on runlevel [!2345] respawn exec /sbin/getty -8 115200 hvc0
Now you should be able to reboot and select the xen kernel from grub's boot menu. When you have booted, verify with
xm info
that Xen and xend are running.
6. build stub domain
This is done in the chrooted xenbuild directory.
First we build a stub domain, which makes the Xen Makefiles pull in several packages such as lwip, newlib, etc. and then compile mini-os which uses the packages.
D=~/xen-4.0-testing/stubdom cd $D make c-stubdom
Now the mini-os object files are compiled. We put them for convenience into a library
MO=$(pwd)/mini-os-x86_32-c ar cr $D/../extras/minios.a $MO/{blkfront,events,fbfront,fs-front,gntmap,gnttab,hypervisor,kernel,lock,main,mm,netfront,pcifront,sched}.o \ $MO/lib/{ctype,math,printf,stack_chk_fail,string,sys,xmalloc,xs}.o $MO/xenbus/xenbus.o $MO/console/{console,xencons_ring}.o
Now we are ready to build the cross g++ for minios
9. Building a cross compiler
You can compile some c++ programs for minios using the system's normal gcc, but it can't use iostreams etc, because the header files for g++ depend on the linux header files, and will break when compiling for minios.
We need a special c++ compiler that uses the right system include files, and newlib instead of glibc, or else
include iostream will not work. A g++ compiled for linux relies on some definitions
in the ctype.h file. Those definitions do not exist in minios, and therefore the iostream header files
generated for a linux g++ will not work on a minios system.
To build a g++ for minios, the sysroot of mini os must exist.
It got created (above) when we did make c-stubdom in the stubdom directory in the xen repository.
The minios sysroot is placed in $HOME/xen-4.0-testing/stubdom/cross-root-i686
# build tool chain TARGET=i686-xen-elf INSTALL=$HOME/sw GCC_VER=4.5.0 ### export PATH=$INSTALL/bin:$PATH mkdir -p $INSTALL/build cd $INSTALL/build ( wget http://ftp.sunet.se/pub/gnu/gmp/gmp-5.0.1.tar.bz2 & wget http://ftp.sunet.se/pub/gnu/mpfr/mpfr-2.4.2.tar.bz2 & wget http://ftp.sunet.se/pub/gnu/gcc/infrastructure/mpc-0.8.1.tar.gz & wait ) tar -xjf gmp-5.0.1.tar.bz2 tar -xjf mpfr-2.4.2.tar.bz2 tar -xzf mpc-0.8.1.tar.gz ( wget http://ftp.sunet.se/pub/gnu/binutils/binutils-2.20.tar.bz2 & wget http://ftp.sunet.se/pub/gnu/gcc/releases/gcc-$GCC_VER/gcc-core-$GCC_VER.tar.bz2 & wget http://ftp.sunet.se/pub/gnu/gcc/releases/gcc-$GCC_VER/gcc-g++-$GCC_VER.tar.bz2 & wget http://ftp.sunet.se/pub/gnu/gdb/gdb-7.1.tar.bz2 & wait ) tar -xjf binutils-2.20.tar.bz2 tar -xjf gcc-core-$GCC_VER.tar.bz2 tar -xjf gcc-g++-$GCC_VER.tar.bz2 tar -xjf gdb-7.1.tar.bz2 wget ftp://sources.redhat.com/pub/newlib/newlib-1.18.0.tar.gz tar -xzf newlib-1.18.0.tar.gz mkdir gmp-5.0.1/build; cd gmp-5.0.1/build ../configure --prefix=$INSTALL/build/gmp --disable-shared --enable-static make -j8 && make install cd ../../ mkdir mpfr-2.4.2/build; cd mpfr-2.4.2/build ../configure --prefix=$INSTALL/build/mpfr --with-gmp=$INSTALL/build/gmp \ --disable-shared --enable-static make -j8 && make install cd ../../ mkdir mpc-0.8.1/build; cd mpc-0.8.1/build ../configure --prefix=$INSTALL/build/mpc --with-gmp=$INSTALL/build/gmp \ --with-mpfr=$INSTALL/build/mpfr --disable-shared --enable-static make -j8 && make install cd ../../ patch -l -p0 <<EOF --- binutils-2.20/gas/as.h 2010-06-15 18:49:19.771279712 +0000 +++ binutils-2.20/gas/as.h 2010-06-15 18:50:10.400278790 +0000 @@ -238,7 +238,7 @@ #define know(p) gas_assert(p) /* Verify our assumptions! */ #endif /* not yet defined */ #else -#define know(p) /* know() checks are no-op.ed */ +#define know(p) do {} while(0) /* know() checks are no-op.ed */ #endif /* input_scrub.c */ EOF cd $INSTALL/build mkdir binutils-2.20/build; cd binutils-2.20/build ../configure \ --prefix=$INSTALL --target=$TARGET --enable-interwork \ --enable-multilib --disable-nls --disable-shared --disable-threads \ --with-sysroot=$HOME/xen-4.0-testing/stubdom/cross-root-i686/i686-xen-elf \ --with-gcc --with-gnu-as --with-gnu-ld make -j8 && make install cd ../../ mkdir gcc-$GCC_VER/build; cd gcc-$GCC_VER/build ../configure --prefix=$INSTALL --target=$TARGET --enable-interwork \ --disable-multilib --enable-languages=c --with-newlib --disable-nls \ --disable-shared --disable-threads --with-gnu-as --with-gnu-ld \ --with-gmp=$INSTALL/build/gmp --with-mpfr=$INSTALL/build/mpfr \ --with-mpc=$INSTALL/build/mpc \ --without-headers make -j8 all-gcc && make install-gcc cd ../../ mkdir newlib-1.18.0/build; cd newlib-1.18.0/build ../configure --prefix=$INSTALL --target=$TARGET --enable-interwork \ --enable-multilib --with-gnu-as --with-gnu-ld --disable-nls make -j8 && make install cd ../../ cd gcc-$GCC_VER/build rm -rf * ../configure \ --prefix=$INSTALL --target=$TARGET --enable-interwork \ --disable-multilib --enable-languages=c,c++ --with-newlib --disable-nls \ --disable-shared --disable-threads --with-gnu-as --with-gnu-ld \ --with-gmp=$INSTALL/build/gmp --with-mpfr=$INSTALL/build/mpfr \ --with-mpc=$INSTALL/build/mpc \ --with-sysroot=$HOME/xen-4.0-testing/stubdom/cross-root-i686/i686-xen-elf make -j8 && make install cd ../../
Now the cross compiler and build tools for mini-os are installed in ~/sw.
9.1 Building a stub domain with the new g++ compiler
Here we assume that there are two c++ files in stubdom/c called main.cpp and mi.cpp.
We use the g++ and ld compiled for mini os.
GPP=i686-xen-elf-g++ LD=i686-xen-elf-ld
Compile the source files
cd $D $GPP c/main.cpp -c -o c/main.o $GPP c/mi.cpp -c -o c/mi.o
Link with the mini os object files and our own libstdc++.
$LD -r -d \ -m elf_i386 \ $D/mini-os-x86_32-c/arch/x86/x86_32.o \ -\( \ $D/c/main.o \ $D/c/mi.o \ $D/../extras/mini-os/app.lds \ --undefined main \ -\) \ -whole-archive \ $D/../extras/minios.a \ -no-whole-archive \ $D/mini-os-x86_32-c/lwip.a \ $D/libxc-x86_32/libxenguest.a \ $D/libxc-x86_32/libxenctrl.a \ $D/mini-os-x86_32-c/arch/x86/libx86_32.a \ -L$D/cross-root-i686/i686-xen-elf/lib \ -lstdc++ \ -lc -lm -lz -lpci \ -L$D/cross-root-i686/lib/gcc/i686-xen-elf/4.5.0 \ -lgcc \ -o $D/mini-os-x86_32-c/mini-os.o
Create the final executable by placing the code segments where they should be and pack the file
$LD \ -m elf_i386 \ $D/mini-os-x86_32-c/mini-os.o \ -T $D/../extras/mini-os/arch/x86/minios-x86_32.lds \ -o $D/mini-os-x86_32-c/mini-os gzip -f -5 -c $D/mini-os-x86_32-c/mini-os >$D/mini-os-x86_32-c/mini-os.gz
Voila! It should run on xen (start it from dom0, not from inside the buildroot).
xm create /dev/null name=minios disk=file:/home/larsr/disk2.img,/dev/sda,r vif="" kernel=/home/larsr/xenbuild/home/larsr/xen-4.0-testing/stubdom/mini-os-x86_32-c/mini-os.gz -c
10. Build LLVM for minios.
You have to use a g++ and gcc that is compiled for minios. To use LLVM inside the minios, we must have the c++ header files in the compilation paths, and the correct libraries included.
cd ~ git clone ssh://lra@abakus.sics6.se/home/lra/src/llvm.git llvm.git wget http://www.sics.se/~lra/lars-llvm.patch wget http://www.sics.se/~lra/include-llvm-Config-config.h.patch cd llvm.git/ git checkout 27ff7ebce9a8d5dd2631156d910fd4898a028dbb patch -p1 <../lars-llvm.patch mkdir build cd build ../configure --prefix=$INSTALL --enable-targets=x86,x86_64 --disable-threads --target=i686-xen-elf make BUILD_DIRS_ONLY=1 -j8 sed -i -e"s@int\t_EXFUN(getpagesize@size_t\t_EXFUN(getpagesize@" \ $INSTALL/i686-xen-elf/include/sys/unistd.h patch include/llvm/Config/config.h ../../include-llvm-Config-config.h.patch # minios has MAP_ANON but not MAP_ANONYMOUS and no TEMP file support sed -i -e"s@#define HAVE_MMAP_ANONYMOUS 1@//#define HAVE_MMAP_ANONYMOUS 1@" include/llvm/Config/config.h sed -i -e"s@#define HAVE_MKDTEMP 1@//#define HAVE_MKDTEMP 1@" include/llvm/Config/config.h sed -i -e"s@#define HAVE_MKSTEMP 1@//#define HAVE_MKSTEMP 1@" include/llvm/Config/config.h sed -i -e"s@#define HAVE_MKTEMP 1@//#define HAVE_MKTEMP 1@" include/llvm/Config/config.h make VERBOSE=1 CXX="i686-xen-elf-g++" CC="i686-xen-elf-gcc" -j8 -C lib -k # use `-k` because the dynamic linking example 'lib/Transforms/Hello' will fail because -ldl is not supported rm Debug/lib/libLLVM{Support,System}.a cd lib/Support rm -rf Debug make VERBOSE=1 CXX="i686-xen-elf-g++" CC="i686-xen-elf-gcc" -j8 cd ../.. cd lib/System rm -rf Debug make VERBOSE=1 CXX="i686-xen-elf-g++ -isystem ${D:?}/../extras/mini-os/include/posix" CC="i686-xen-elf-gcc" -j8 cd ../..
We should also build the llvm tools to get llvm-config
make -C tools -j8
This breaks when it compiles llvm-mc, but llvm-config is compiled ok.
Now llvm-config is in build/Debug/bin/llvm-config
11. Building an LLVM program for minios
The following has to be defined in some c source file, because newlib doesn't provide
lseek and fstat correctly after the xen patches (which renames them).
The domain crashes if linked with crt{begin,end}.o so instead
declare the missing variable __dso_handle.
The code below could be placed in main.cpp, for instance.
extern off_t lseek64(int a, off_t b, int c); off_t lseek(int a, off_t b, int c) { return lseek64 (a, b, c); } int fstat64(int a, struct stat* b); int fstat(int a, struct stat* b) { return fstat64(a,b); } void *__dso_handle = 0;
12. To compile it
Suppose the program is in $D/c/jit-test.o and that llvm-config is in the path.
cd $D i686-xen-elf-g++ $(~/llvm.git/build/Debug/bin/llvm-config --cflags) -c c/jit-test.cpp -o c/jit-test.o i686-xen-elf-ld -r -d -m elf_i386 $D/mini-os-x86_32-c/arch/x86/x86_32.o -\( \ $D/c/jit-test.o \ $D/../extras/mini-os/app.lds --undefined main -\) \ -whole-archive $D/../extras/minios.a -no-whole-archive \ $D/mini-os-x86_32-c/lwip.a $D/libxc-x86_32/libxenguest.a $D/libxc-x86_32/libxenctrl.a \ $D/mini-os-x86_32-c/arch/x86/libx86_32.a \ -L/home/larsr/llvm.git/build/Debug/lib \ -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86AsmPrinter -lLLVMX86CodeGen -lLLVMSelectionDAG \ -lLLVMAsmPrinter -lLLVMMCParser -lLLVMX86Info -lLLVMJIT -lLLVMExecutionEngine -lLLVMCodeGen -lLLVMScalarOpts \ -lLLVMInstCombine -lLLVMTransformUtils -lLLVMipa -lLLVMAnalysis -lLLVMTarget -lLLVMMC -lLLVMCore \ -lLLVMSupport -lLLVMSystem \ -L$INSTALL/i686-xen-elf/lib -lstdc++ \ -L$D/cross-root-i686/i686-xen-elf/lib -lc -lm -lz -lpci \ -L$INSTALL/lib/gcc/i686-xen-elf/4.5.0 -lgcc \ -o $D/mini-os-x86_32-c/mini-os.o i686-xen-elf-ld \ -m elf_i386 \ $D/mini-os-x86_32-c/mini-os.o \ -T /home/larsr/xen-4.0-testing/extras/mini-os/arch/x86/minios-x86_32.lds \ -o $D/mini-os-x86_32-c/mini-os gzip -f -5 -c $D/mini-os-x86_32-c/mini-os >$D/mini-os-x86_32-c/mini-os.gz
13. Run the mini-os instance
From the dom0, as root, type
sudo xm create /dev/null kernel=/home/larsr/xenbuild/home/larsr/xen-4.0-testing/stubdom/mini-os-x86_32-c/mini-os.gz name=minios disk=file:/home/larsr/disk2.img,/dev/sda,r vif="" -c
