CSE 522S: Studio 1

Welcome To Linux

"A crash reduces
your expensive computer
to a simple stone."

—Anonymous, GNU error message haiku page

Welcome to the first meeting of the Advanced Operating Systems course! Today we're going to configure and compile a Linux kernel from source code. Even if you've done this kind of thing before, this studio will introduce the source code and some new resources that this class depends on, along with giving you additional experience working with the Raspberry Pi platform.

In this studio, and throughout the semester, you will use a two-stage cross-compilation approach:

Having the code and the build process on other machines has some key advantages besides probably building faster than on your Pi. First, if you get locked out of your Pi, or at some point a (modified) kernel crashes (see the error message haiku above :-) most of the work you have done is safe in another location which allows you to start over efficiently by re-imaging your Pi to the default setup, fixing whatever bugs may have caused the crash, recompiling the kernel, installing it, and proceeding from there. Second, the CEC servers (such as shell.cec.wustl.edu) give you a place to create backup copies of any code before you modify it - or even to run version control software such as svn or git so that if a change you make goes awry you can quickly recover to a previous (working) version of your code.

Please work in a group of 2-3 people to complete the required exercises below, as well as any optional enrichment exercises that you wish to complete.

As you work through these exercises, please record your answers, and when finished please send just one email for the whole group to eng-cse522@email.wustl.edu with the phrase Welcome To Linux Studio in the subject line.

Make sure that the name of each person who worked on these exercises is listed in the first answer, and make sure you number each of your responses so it is easy to match your responses with each exercise.

Required Exercises

  1. As the answer to the first exercise, list the names of the people who worked together on this studio.

    To save time, and to avoid potential freezing of your Raspberry Pi during long-running local compiles, we will be cross-compiling the kernel using a university-managed server (shell.cec.wustl.edu) and dedicated Linux machines.

    First, ssh into shell.cec.wustl.edu using your WUSTL Key id and password. The following directions will let you do that from one of the Windows lab machines - on a Mac or Linux laptop, you can just run ssh directly from within a terminal window.

    1. Log into the Windows lab machine (as needed use the KVM switch button on the monitor's stalk to toggle between that machine and your Pi).
    2. Click on the start icon that appears on the lower left corner of your desktop.
    3. In the search field that appears on the lower left of your screen, type "Secure" and then click on Secure Shell Client when it appears (another useful application that also should appear is Secure File Transfer Client, and in fact either of those can be launched via a button found in the other).
    4. Click on Quick Connect on the upper left of the window that appears, and enter shell.cec.wustl.edu and your WUSTL Key id in the fields for the host and user id (the port number field already should be set to 22, but set it to that value if not).
    5. Click on Connect, and when prompted enter your WUSTL Key password.

  2. Because kernel compiles are computationally intensive, you should not run them on the shell.cec.wustl.edu server, and instead you should run:

    qlogin -q all.q

    which will let you log into a dedicated Linux host using your WUSTL Key id and password.

    You must remember the hostname of the machine you are now running on. To determine the hostname, execute the hostname command. The result should look somthing like:


    You must remember this hostname. It will store your linux kernel as well as other files you implement as part of this and future studios and labs.

    As the answer to this exercise, write down the hostname of the linuxlab machine you re using. Note: this can and likely will be different for all students in your group

  3. Now, exit out of the machine you were issued via qlogin by typing exit. You are now running on the shell.cec.wustl.edu machine. Now, use ssh to log back into the linuxlab machine you wrote down by issuing the ssh command:

    ssh linuxlab{id}.seas.wustl.edu

    replacing {id} with the specific ID found in your hostname.

  4. Now, you are just about ready to start downloading and compiling the Linux kernel. However, in order to ensure that your compilation does not interfere with others using the machine, you must first create a private directory on the local filesystem that is isolated from the traffic created by other users.

    To do so, issue the following commands:
    cd /tmp/compile
    mkdir "your username"
    chmod 700 "your username"
    cd "your username"

    replacing "your username" with your actual username.

    These commands created a directory that you should use to store your linux kernel. Whenever you log into the machine via ssh, you must first change your directory to this location by typing:

    cd /tmp/compile/"your username"

    Note that this directory only exists on the linuxlab machine that you are currently using. You must always ssh to this specific machine, as you did at the beggining of this step.

  5. Now it's time to download the Linux kernel source code. For a general-purpose project, you would go to kernel.org and download the Linux source from there. However, since we're using the Raspberry Pi we'll be starting with a version that is designed for our platform. The Raspberry Pi project maintains it's own distribution on GitHub, at https://github.com/raspberrypi. We'll be modifying the kernel based on the building guide that you can find here.

    From your specific /tmp/compile/user-name/ directory, create a folder called linux_source in which to keep all of your source code and build files organized. Use the cd command to move into your new folder and issue the following commands, each of which may take a while to finish:

    wget https://github.com/raspberrypi/linux/archive/raspberrypi-kernel_1.20160506-1.tar.gz

    tar -xzf raspberrypi-kernel_1.20160506-1.tar.gz

    This has the effect of downloading a specific version of the Raspberry Pi 3 Linux distribution, which is the version that this course was developed with. Newer versions exist, which we won't use for this class but you can pursue on your own time if you'd like. Once the files finish unpacking you'll have a new directory, which we suggest renaming as something simpler with the mv command. We renamed ours linux, which is what the rest of these instructions will assume you've done as well.

    To save space, please delete the .tar.gz file once it is unpacked.

    Move into your new linux directory, and issue the command make kernelversion, which will tell you which Linux kernel version you've just checked out. As the answer to this exercise, give the output from that command.

  6. Now it's time to configure the Linux kernel. There are thousands of possible configuration options- many quite interesting and many quite dull. Rather than setting all these options manually, we'll we'll create the default Raspberry Pi 3 configuration and then tweak it. Issue the commands:

    module add raspberry (this adds the cross-compiler to your PATH variable)

    KERNEL=kernel7 (this is used by some build scripts)

    Note that in some terminal environments (depending on how your account is set up and what shell is running in it on shell.cec.wustl.edu) you may need to do this using the set command instead: set KERNEL=kernel7

    echo $KERNEL (this outputs the current value stored in the KERNEL environment variable, which should be kernel7 if the previous command was successful)

    make -j8 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig

    To reiterate, if you were building a general purpose kernel, you wouldn't use the previous commands, which set up a default configuration for the Raspberry Pi .

    Next, we want to set custom configuration options. Issue the command:

    make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- menuconfig

    After a moment, you'll get a kernel configuration menu. As you can see, there are a _lot_ of options. For now, we're just going to do two things. First, we will choose the kernel's preemption model. Go into "Kernel Features", select "Preemption Model", and then select the "Preemptible Kernel (Low-Latency Desktop)" option.

    Next you'll add your own unique identifier to the kernel you build. Exit "Kernel Features" and navigate to "Local version" under "General setup". The local version string you specify here will be appended to the output of the uname command. If you applied the default Raspberry Pi configuration correctly, this currently should be set to "-v7". Go ahead and append your own unique identifier to that string, though recall that uname already gives you the kernel version number, the day and time the kernel was compiled, as well as other info.Warning: do not include any spaces in the local version string - this will break the build script when you run sudo make modules_install in the future.

    Choose an interesting sounding option and use the "H" key to bring up a short description. As the answer to this exercise, give the option's name, a short summary, and the option's symbol. Once you're done, exit the configurator and be sure to answer "Yes" when asked to save your changes.

  7. Finally, we're ready to start compiling the kernel. This could take even longer if we were building the kernel locally on your Raspberry Pi instead of cross-compiling.

    To track how long this step takes, which you will need for the answer to this exercise, make note of (and write down) the current time. Then, issue the following commands to cross-compile the kernel:

    module add raspberry

    KERNEL=kernel7 (or if your shell requires it set KERNEL=kernel7)

    echo $KERNEL (confirm the KERNEL variable was set to kernel7)

    make -j8 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs

    The compile may take a while (possibly an hour or more, depending on the machine you're compiling on). Once it's done, issue the following command:

    mkdir ../modules

    This will create a directory that the cross-compiler will use to store the kernel modules that it creates. You will later transfer these files to your Raspberry Pi. Then issue the command:

    make -j8 ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=../modules modules_install

    Note the time again, and as the answer to this exercise please indicate approximately how long it took to complete it.

    Congratulations! You've just compiled an operating system from source!