Building the LLVM based binary translator.

Lars.Rasmusson@sics.se 2010-10-21

1. Create clean build root file system

To make sure that we know what software we are using to build, we first create an entire root file system in a catalog. It containts its own build tools, independent of what the host OS itself uses.
Use debootstrap to install a bare bones root file system from a local repository:

sudo -s
BUILDROOT=$(pwd)/xenbuild
mkdir $BUILDROOT
debootstrap --variant=buildd --arch i386 lucid $BUILDROOT http://se.archive.ubuntu.com/ubuntu

2. chroot into it, configure and install build tools and git

Set time zone, and use apt to install the build tools

chroot $BUILDROOT
export LC_ALL=C
hostname builder
dpkg-reconfigure tzdata
adduser --disabled-password larsr

Use a local repository and install a lot of packets

echo > /etc/apt/sources.list deb http://se.archive.ubuntu.com/ubuntu lucid main restricted universe
apt-get update
apt-get install -y wget nano mercurial gawk openssh-client libz-dev \
  gettext libssl-dev libx11-dev bin86 bcc libncurses5-dev python-dev \
  texinfo bridge-utils uuid-dev git-core man iasl m4 flex bison cmake

3. check out xen from git on mac (edit on mac?)

Initialize my git user name

su larsr
cd $HOME
cat >> .gitconfig <<EOF
[user]
        name = Lars Rasmusson
        email = Lars.Rasmusson@sics.se
EOF
 
git clone ssh://lra@abakus.sics6.se/home/lra/src/xen-4.0-testing.git

4. Building and installing the Xen kernel

NOTE! These steps are only necessary if you are not already running xen on your testing machine.

4.1 Build

Prepare a new branch llvm-domain and check out a specific version of Xen. In the buildroot tree, do

cd ~/xen-4.0-testing
git checkout -b llvm-domain RELEASE-4.0.1
git clean -fd
make -j8 xen
make -j8 tools

4.2 Install

In the real root tree, copy the new xen-image to the /boot directory, and configure the bootloader (grub/grub2/whatever) to load the newly compiled xen.

First pack the xen files into a tar file and then into a deb. It makes it easier to remove or update later.

tar -czf xen.tgz -C dist/install/ .
fakeroot alien --to-deb xen.tgz

Then install the files in the real root file system. The update-rc.d commands makes xend and xendomains start when the system boots.

dpkg -i /home/larsr/xenbuild/home/larsr/xen-4.0-testing/xen_1-2_all.deb 
update-rc.d xend defaults
update-rc.d xendomains defaults

I'm running Ubuntu 10.04 Lucid Lynx which uses the grub2 boot loader. I installed Xen in grub by putting the following into /etc/grub.d/35_xen

#!/bin/sh
exec tail -n +3 $0
# This file is an example on how to add custom entries
 
menuentry "Xen 4.0 / Ubuntu 10.04 kernel 2.6.31.13 pvops" {
    insmod ext2
    set root=(hd0,1)
    multiboot (hd0,1)/xen-4.0.gz com1=115200,8n1 console=com1,vga
    module (hd0,1)/vmlinuz-2.6.31.13 console=tty0 console=hvc0 root=/dev/mapper/cluster5-root ro
    module (hd0,1)/initrd.img-2.6.31.13
    savedefault
}

and then run

update-grub

A linux running in Xen gets its console on hvc0, and for Ubuntu to start a getty on hvc0, put hvc0.conf in /etc/init/hvc0.conf

# hvc0 - getty
#
# This service maintains a getty on Xen's ttyhvc0 from the point the system is
# started until it is shut down again.
 
start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]
 
respawn
exec /sbin/getty -8 115200 hvc0

Now you should be able to reboot and select the xen kernel from grub's boot menu. When you have booted, verify with

xm info

that Xen and xend are running.

6. build stub domain

This is done in the chrooted xenbuild directory.
First we build a stub domain, which makes the Xen Makefiles pull in several packages such as lwip, newlib, etc. and then compile mini-os which uses the packages.

D=~/xen-4.0-testing/stubdom
 
cd $D
make c-stubdom

Now the mini-os object files are compiled. We put them for convenience into a library

MO=$(pwd)/mini-os-x86_32-c
ar cr $D/../extras/minios.a  $MO/{blkfront,events,fbfront,fs-front,gntmap,gnttab,hypervisor,kernel,lock,main,mm,netfront,pcifront,sched}.o \
$MO/lib/{ctype,math,printf,stack_chk_fail,string,sys,xmalloc,xs}.o $MO/xenbus/xenbus.o $MO/console/{console,xencons_ring}.o

Now we are ready to build the cross g++ for minios

9. Building a cross compiler

You can compile some c++ programs for minios using the system's normal gcc, but it can't use iostreams etc, because the header files for g++ depend on the linux header files, and will break when compiling for minios.

We need a special c++ compiler that uses the right system include files, and newlib instead of glibc, or else
include iostream will not work. A g++ compiled for linux relies on some definitions
in the ctype.h file. Those definitions do not exist in minios, and therefore the iostream header files
generated for a linux g++ will not work on a minios system.

To build a g++ for minios, the sysroot of mini os must exist.

It got created (above) when we did make c-stubdom in the stubdom directory in the xen repository.
The minios sysroot is placed in $HOME/xen-4.0-testing/stubdom/cross-root-i686

# build tool chain
TARGET=i686-xen-elf
INSTALL=$HOME/sw
GCC_VER=4.5.0
###
 
export PATH=$INSTALL/bin:$PATH
 
mkdir -p $INSTALL/build
cd $INSTALL/build
 
(
wget http://ftp.sunet.se/pub/gnu/gmp/gmp-5.0.1.tar.bz2 &
wget http://ftp.sunet.se/pub/gnu/mpfr/mpfr-2.4.2.tar.bz2 &
wget http://ftp.sunet.se/pub/gnu/gcc/infrastructure/mpc-0.8.1.tar.gz &
wait
)
 
tar -xjf gmp-5.0.1.tar.bz2
tar -xjf mpfr-2.4.2.tar.bz2
tar -xzf mpc-0.8.1.tar.gz
 
(
wget http://ftp.sunet.se/pub/gnu/binutils/binutils-2.20.tar.bz2 & 
wget http://ftp.sunet.se/pub/gnu/gcc/releases/gcc-$GCC_VER/gcc-core-$GCC_VER.tar.bz2 &
wget http://ftp.sunet.se/pub/gnu/gcc/releases/gcc-$GCC_VER/gcc-g++-$GCC_VER.tar.bz2 &
wget http://ftp.sunet.se/pub/gnu/gdb/gdb-7.1.tar.bz2 &
wait
)
tar -xjf binutils-2.20.tar.bz2
tar -xjf gcc-core-$GCC_VER.tar.bz2
tar -xjf gcc-g++-$GCC_VER.tar.bz2
tar -xjf gdb-7.1.tar.bz2
 
wget ftp://sources.redhat.com/pub/newlib/newlib-1.18.0.tar.gz
tar -xzf newlib-1.18.0.tar.gz
 
mkdir gmp-5.0.1/build; cd gmp-5.0.1/build
../configure --prefix=$INSTALL/build/gmp --disable-shared --enable-static
make -j8 && make install
cd ../../
mkdir mpfr-2.4.2/build; cd mpfr-2.4.2/build
../configure --prefix=$INSTALL/build/mpfr --with-gmp=$INSTALL/build/gmp \
   --disable-shared --enable-static
make -j8 && make install
cd ../../
mkdir mpc-0.8.1/build; cd mpc-0.8.1/build
../configure --prefix=$INSTALL/build/mpc --with-gmp=$INSTALL/build/gmp \
   --with-mpfr=$INSTALL/build/mpfr --disable-shared --enable-static
make -j8 && make install
cd ../../
 
patch -l -p0 <<EOF
--- binutils-2.20/gas/as.h      2010-06-15 18:49:19.771279712 +0000
+++ binutils-2.20/gas/as.h      2010-06-15 18:50:10.400278790 +0000
@@ -238,7 +238,7 @@
 #define know(p) gas_assert(p)  /* Verify our assumptions!  */
 #endif /* not yet defined */
 #else
-#define know(p)                        /* know() checks are no-op.ed  */
+#define know(p)        do {} while(0)  /* know() checks are no-op.ed  */
 #endif
 
 /* input_scrub.c */
EOF
 
cd $INSTALL/build
mkdir binutils-2.20/build; cd binutils-2.20/build
../configure \
   --prefix=$INSTALL --target=$TARGET --enable-interwork \
   --enable-multilib --disable-nls --disable-shared --disable-threads \
   --with-sysroot=$HOME/xen-4.0-testing/stubdom/cross-root-i686/i686-xen-elf \
   --with-gcc --with-gnu-as --with-gnu-ld
make -j8 && make install
cd ../../
 
mkdir gcc-$GCC_VER/build; cd gcc-$GCC_VER/build
../configure --prefix=$INSTALL --target=$TARGET --enable-interwork \
   --disable-multilib --enable-languages=c --with-newlib --disable-nls \
   --disable-shared --disable-threads --with-gnu-as --with-gnu-ld \
   --with-gmp=$INSTALL/build/gmp --with-mpfr=$INSTALL/build/mpfr \
   --with-mpc=$INSTALL/build/mpc \
   --without-headers
make -j8 all-gcc && make install-gcc
cd ../../
 
mkdir newlib-1.18.0/build; cd newlib-1.18.0/build
../configure --prefix=$INSTALL --target=$TARGET --enable-interwork \
   --enable-multilib --with-gnu-as --with-gnu-ld --disable-nls 
make -j8 && make install
cd ../../
 
cd gcc-$GCC_VER/build
rm -rf *
../configure \
   --prefix=$INSTALL --target=$TARGET --enable-interwork \
   --disable-multilib --enable-languages=c,c++ --with-newlib --disable-nls \
   --disable-shared --disable-threads --with-gnu-as --with-gnu-ld \
   --with-gmp=$INSTALL/build/gmp --with-mpfr=$INSTALL/build/mpfr \
   --with-mpc=$INSTALL/build/mpc \
   --with-sysroot=$HOME/xen-4.0-testing/stubdom/cross-root-i686/i686-xen-elf 
make -j8 && make install
cd ../../

Now the cross compiler and build tools for mini-os are installed in ~/sw.

9.1 Building a stub domain with the new g++ compiler

Here we assume that there are two c++ files in stubdom/c called main.cpp and mi.cpp.

We use the g++ and ld compiled for mini os.

GPP=i686-xen-elf-g++
LD=i686-xen-elf-ld

Compile the source files

cd $D
$GPP c/main.cpp -c -o c/main.o
$GPP c/mi.cpp -c -o c/mi.o

Link with the mini os object files and our own libstdc++.

$LD -r -d \
-m elf_i386 \
$D/mini-os-x86_32-c/arch/x86/x86_32.o \
-\( \
$D/c/main.o \
$D/c/mi.o \
$D/../extras/mini-os/app.lds \
--undefined main \
-\) \
-whole-archive \
$D/../extras/minios.a \
-no-whole-archive \
$D/mini-os-x86_32-c/lwip.a \
$D/libxc-x86_32/libxenguest.a \
$D/libxc-x86_32/libxenctrl.a \
$D/mini-os-x86_32-c/arch/x86/libx86_32.a \
-L$D/cross-root-i686/i686-xen-elf/lib \
-lstdc++ \
-lc -lm -lz -lpci \
-L$D/cross-root-i686/lib/gcc/i686-xen-elf/4.5.0 \
-lgcc \
-o $D/mini-os-x86_32-c/mini-os.o

Create the final executable by placing the code segments where they should be and pack the file

$LD \
-m elf_i386 \
$D/mini-os-x86_32-c/mini-os.o  \
-T $D/../extras/mini-os/arch/x86/minios-x86_32.lds \
-o $D/mini-os-x86_32-c/mini-os
 
gzip -f -5 -c $D/mini-os-x86_32-c/mini-os >$D/mini-os-x86_32-c/mini-os.gz

Voila! It should run on xen (start it from dom0, not from inside the buildroot).

xm create /dev/null  name=minios disk=file:/home/larsr/disk2.img,/dev/sda,r vif="" kernel=/home/larsr/xenbuild/home/larsr/xen-4.0-testing/stubdom/mini-os-x86_32-c/mini-os.gz -c

10. Build LLVM for minios.

You have to use a g++ and gcc that is compiled for minios. To use LLVM inside the minios, we must have the c++ header files in the compilation paths, and the correct libraries included.

cd ~
git clone ssh://lra@abakus.sics6.se/home/lra/src/llvm.git llvm.git
wget http://www.sics.se/~lra/lars-llvm.patch
wget http://www.sics.se/~lra/include-llvm-Config-config.h.patch
cd llvm.git/
git checkout 27ff7ebce9a8d5dd2631156d910fd4898a028dbb
patch -p1 <../lars-llvm.patch 
 
mkdir build
cd build
../configure --prefix=$INSTALL --enable-targets=x86,x86_64 --disable-threads --target=i686-xen-elf
 
make BUILD_DIRS_ONLY=1 -j8
 
sed -i -e"s@int\t_EXFUN(getpagesize@size_t\t_EXFUN(getpagesize@" \
  $INSTALL/i686-xen-elf/include/sys/unistd.h
 
patch include/llvm/Config/config.h ../../include-llvm-Config-config.h.patch
# minios has MAP_ANON but not MAP_ANONYMOUS and no TEMP file support
sed -i -e"s@#define HAVE_MMAP_ANONYMOUS 1@//#define HAVE_MMAP_ANONYMOUS 1@" include/llvm/Config/config.h
sed -i -e"s@#define HAVE_MKDTEMP 1@//#define HAVE_MKDTEMP 1@" include/llvm/Config/config.h
sed -i -e"s@#define HAVE_MKSTEMP 1@//#define HAVE_MKSTEMP 1@" include/llvm/Config/config.h
sed -i -e"s@#define HAVE_MKTEMP 1@//#define HAVE_MKTEMP 1@" include/llvm/Config/config.h
 
make VERBOSE=1 CXX="i686-xen-elf-g++" CC="i686-xen-elf-gcc"  -j8 -C lib -k
# use `-k` because the dynamic linking example 'lib/Transforms/Hello' will fail because -ldl is not supported
 
rm Debug/lib/libLLVM{Support,System}.a
cd lib/Support
rm -rf Debug
make VERBOSE=1 CXX="i686-xen-elf-g++" CC="i686-xen-elf-gcc" -j8
cd ../..
 
cd lib/System
rm -rf Debug
make VERBOSE=1 CXX="i686-xen-elf-g++ -isystem ${D:?}/../extras/mini-os/include/posix" CC="i686-xen-elf-gcc"  -j8
cd ../..

We should also build the llvm tools to get llvm-config

make -C tools -j8

This breaks when it compiles llvm-mc, but llvm-config is compiled ok.

Now llvm-config is in build/Debug/bin/llvm-config

11. Building an LLVM program for minios

The following has to be defined in some c source file, because newlib doesn't provide
lseek and fstat correctly after the xen patches (which renames them).
The domain crashes if linked with crt{begin,end}.o so instead
declare the missing variable __dso_handle.
The code below could be placed in main.cpp, for instance.

extern off_t lseek64(int a, off_t b, int c);
off_t lseek(int a, off_t b, int c) {
    return lseek64 (a, b, c);
}
 
int fstat64(int a, struct stat* b);
int fstat(int a, struct stat* b) {
    return fstat64(a,b);
}
 
void *__dso_handle = 0;

12. To compile it

Suppose the program is in $D/c/jit-test.o and that llvm-config is in the path.

cd $D
 
i686-xen-elf-g++ $(~/llvm.git/build/Debug/bin/llvm-config --cflags) -c c/jit-test.cpp -o c/jit-test.o
 
i686-xen-elf-ld -r -d -m elf_i386 $D/mini-os-x86_32-c/arch/x86/x86_32.o -\( \
 $D/c/jit-test.o \
 $D/../extras/mini-os/app.lds --undefined main -\) \
 -whole-archive $D/../extras/minios.a -no-whole-archive \
 $D/mini-os-x86_32-c/lwip.a $D/libxc-x86_32/libxenguest.a $D/libxc-x86_32/libxenctrl.a \
 $D/mini-os-x86_32-c/arch/x86/libx86_32.a \
 -L/home/larsr/llvm.git/build/Debug/lib \
 -lLLVMX86Disassembler -lLLVMX86AsmParser -lLLVMX86AsmPrinter -lLLVMX86CodeGen -lLLVMSelectionDAG \
 -lLLVMAsmPrinter -lLLVMMCParser -lLLVMX86Info -lLLVMJIT -lLLVMExecutionEngine -lLLVMCodeGen -lLLVMScalarOpts \
 -lLLVMInstCombine -lLLVMTransformUtils -lLLVMipa -lLLVMAnalysis -lLLVMTarget -lLLVMMC -lLLVMCore \
 -lLLVMSupport -lLLVMSystem \
 -L$INSTALL/i686-xen-elf/lib  -lstdc++ \
 -L$D/cross-root-i686/i686-xen-elf/lib -lc -lm -lz -lpci  \
 -L$INSTALL/lib/gcc/i686-xen-elf/4.5.0  -lgcc \
 -o $D/mini-os-x86_32-c/mini-os.o
 
 
i686-xen-elf-ld \
-m elf_i386 \
$D/mini-os-x86_32-c/mini-os.o  \
-T /home/larsr/xen-4.0-testing/extras/mini-os/arch/x86/minios-x86_32.lds \
-o $D/mini-os-x86_32-c/mini-os
 
gzip -f -5 -c $D/mini-os-x86_32-c/mini-os >$D/mini-os-x86_32-c/mini-os.gz

13. Run the mini-os instance

From the dom0, as root, type

sudo xm create /dev/null kernel=/home/larsr/xenbuild/home/larsr/xen-4.0-testing/stubdom/mini-os-x86_32-c/mini-os.gz name=minios disk=file:/home/larsr/disk2.img,/dev/sda,r vif="" -c

Notes to self

torsdag 21 oktober 2010