CSE 422S: Studio 5

Linux Pipes, FIFOs, and Sockets


Regular pipes are the method used to "pipe" the output of one program into the input of another; they are created in memory via a system call and do not exist on any filesystem. Named pipes act like regular pipes but are accessed via a file, called a FIFO special file. Unrelated processes can access this file and communicate.

—Robert Love, Linux System Programming, 2nd Edition, Chapter 1, pp. 14-15.

Pipes, FIFOs, and sockets provide efficient ways to do file-like input and output between processes. Their biggest benefit is the ability to treat the endpoints as formatted input and output streams, just like you would read and write from files, or from streams like standard input and output.

In this studio, you will:

  1. Create and pass data through pipes with the pipe() system call
  2. Create and pass data through FIFOs with the mkfifo() function
  3. Implement a rudimentary active object pattern with processes and FIFOs
  4. Use sockets to send data between processes locally within a machine
  5. Use sockets to send data between processes running on different machines

Especially if you've not done so already, in addition to reading LPI pp. 515-522 and 563-571 and LKD pp. 29-33 and 38-40, please run the following commands on the Linux command line (on your Raspberry Pi or on one of the Linux Lab machines) for more documentation on what the fork and exec library functions do:
man 2 fork
man 3 exec

Later in the semester we will discuss Linux processes in greater depth, though if you'd like to do so now you can skim the slides and studio exercises to learn more about parent and child processes.


Please complete the required exercises below, as well as any optional enrichment exercises that you wish to complete. We encourage you to please work in groups of 2 or 3 people on each studio (and the groups are allowed to change from studio to studio) though if you would prefer to complete any studio by yourself that is allowed.

As you work through these exercises, please record your answers, and when you finish them each and every person who worked on them should please log into Canvas, select this course in this semester, and then upload a file containing them and also upload any other files the assignment asks for, and submit those files for this studio assignment (there should be a separate submission from each person who worked together on them).

Make sure that the name of each person who worked on these exercises is listed in the first answer, and make sure you number each of your responses so it is easy to match your responses with each exercise.


Required Exercises

  1. As the answer to the first exercise, list the names of the people who worked together on this studio.

  2. First we will use the pipe() system call to create a program with communicating child and parent processes. Once created, pipes cannot be shared between unrelated processes. Thus, the standard way to use this system call is first to call pipe(), which will create separate pipe endpoints for reading and writing, and then to call fork(), which will create a child process. Thus, the child will inherit the pipe endpoints from the parent, and both processes now will have access to the pipe.

    On your Raspberry Pi, create a C program that:

    1. Calls pipe() to create a pipe with read and write file descriptors (see man 2 pipe for details)
    2. Calls fork() to create a child process
    3. After the fork, the child process should close its copy of the read file descriptor with close(), and the parent process should close its copy of the write file descriptor similarly
    4. After the fork, the child process should write several test messages to the pipe
    5. After the fork, the parent process should read several messages from the pipe and print them to the standard output stream (stdout)

    Pipe I/O, like file I/O in C, is comparatively low level. There are two approaches you can use to read from and write to a pipe. You could use the integer file descriptors returned by pipe() directly with the read() and write() system calls, which are documented at man 2 read and man 2 write. However you might find it easier to instantiate a FILE* stream variable, which allows you to work with your pipe like it was any other file object (i.e. you write to the pipe with fprintf() and read from the pipe with input operations such as fgets() or fscanf(). You can open a file stream from a file descriptor with the fdopen() function. You can Read more about the difference between file descriptors and file streams here. Of course, all of the above functions have informative man pages as well.

    Hint: Blocking functions that read the pipe will block indefinitely until they are able to return successfully or the write-end of the pipe is closed by the writer. The writer should write all of it's data and then call close() on its file descriptor when it is done. This allows you to write code such as while( fgets() != NULL ) to do open-ended reads, on the pipe, which will eventually fail and return once the write-end is closed.

    Build and run your program on your Raspberry Pi. As the answer to this exercise, copy and paste the terminal output that is produced by running your program.

  3. Now we are going to implement a program that provides a rudimentary active object. An active object is an executable context that performs services for other contexts upon request. In this case our active object will be a process that listens to one end of a FIFO, does some trivial computation when data is recieved, and prints the result of that computation to standard output. Any number of other processes can open the FIFO's read side and submit data, and the order that requests are scheduled depends on a race for the FIFO (recall that writes and reads are atomic for less than PIPE_BUF sized data).

    To begin, create a new file for your program code. This program should:

    1. Create a new FIFO with the function mkfifo(), giving it both read and write permissions (S_IRUSR | S_IWUSR per the table on page 295 of the LPI textbook). This is documented at man 3 mkfifo. Be careful not to confuse this function with the program called mkfifo that is documented at man 1 mkfifo (which is also the man page you get if you say just man mkfifo). Note: if the FIFO already exists, the call to mkfifo() will fail and errno (declared in errno.h, which you can find in the /usr/include/ directory on the machine on which you are compiling in case you would like to browse through it and the other files it includes, to see the symbols and functions it declares) will be set to EEXIST - in that case your program should simply continue to the next step.
    2. Open the FIFO for reading (as though it were a file) using fopen() (run the command man 3 fopen to see return values, parameters, etc. for that call).
    3. Enter a while loop that continually attempts to read from the FIFO until EOF is read. Whenever something is read, print it to the standard output.

    Build and run your program: it will create a FIFO that will appear in the filesystem. In a separate terminal, write some data into the FIFO and validate that it is read by your program correctly. For example, if your FIFO is named my_ao_fifo, the command echo "my_message" > my_ao_fifo will insert the string "my_message" into the FIFO. Test your program with larger amounts of input as well, for example: cat my_file.c > my_ao_fifo.

    Now, we will make two modifications to your active object program. After opening a read stream to the FIFO, also open a write stream (i.e. fopen( fifo_name, "w" ). Your active object will not ever write to this stream, but the fact that it is held open will prevent your program from automatically quitting.

    Next, modify the active object to read from the FIFO with the function fscanf(). This allows you to perform formatted input. Your program should read integer inputs from the pipe, double the input, and then print out the original and the new values. Your input stream should not have any non-integer characters and you do not need to handle this case: test your program by passing integers into your FIFO.

    Finally for this exercise, we want to write a set of programs that concurrently use the FIFO to request that the active object perform work. Write two programs that open the write end of the FIFO with fopen(fifo_name, "w") and insert integer values with fprintf(). One of these programs should only insert even numbers, and the other should only insert odd numbers. Run these programs long enough to verify that they can both write concurrently to the FIFO without problem. Examine the output of your active object to verify correctness.

    As the answer to this exercise, please explain briefly (and show output fragments as evidence) why your program is working correctly based on that output.

  4. You will now use sockets to establish a local connection between two processes on your Raspberry Pi. We'll write two programs, a server and a client. First we will write the server, whose role is to create the communications channel and to listen for incoming connections. A good example of how to construct a basic server can be found in man 2 bind under the section labeled EXAMPLE.

    Your server should perform the following actions:

    1. First, create a socket with the socket() system call. To create a local connection use the domain AF_LOCAL (or equivalently AF_UNIX), the connection type SOCK_STREAM, and protocol zero.
    2. Second, create the communications channel with the bind() system call. This associates your socket from step one with a machine-visible address. In the case of AF_LOCAL, this address is a path in the file system, and your local server program should use the first argument passed to it as a string containing that path. Directions on how to specify a local address in this way (and which header files you'll need in order to make that work) can be found at man 7 unix, and there is a good example showing how to do this on page 1166 of the LPI (Kerrisk) text book.
    3. Now, having created the channel, your program needs to declare that it is going to listen for incoming connections with the listen() system call. Sockets only allow one process to connect at a time, so the second parameter determines how many connection requests can wait (queue up) before subsequent ones are rejected.
    4. Use the accept() system call to accept a connection over the socket interface, passing in NULL and NULL for the second and third arguments. If no connection is immediately available then this system call will, by default, cause your program to block until another program attempts to connect. When your program successfully returns from the call to accept() the new communication socket connection has been established, and the return value from the call is a file descriptor for the new socket.
    5. Your server program then should repeatedly read integers from the new socket, by instantiating a stream interface for it via the fdopen function and then using fscanf() as your active object code did to read integers from a FIFO. When creating the stream, be sure to use the socket descriptor that was returned by the call to accept() (the communication endpoint), not the one created by the initial call to socket() (the connection establishment endpoint). As long as the communication endpoint socket remains open, the server should print out all integers that it receives, to its standard output stream.
    6. Once the client disconnects, use the unlink() system call in order to destroy the original local socket (connection establishment endpoint) file before the program returns.

    Note: Many things can go wrong with socket-based communication. As usual, you should always check function return codes to detect errors. Recall that most functions return error codes that allow you to print a descriptive error statement with a line of code such as: printf("Program error, reason: %s\n", strerror(errno)). This will require including the headers string.h and errno.h.

    Now you will write a client program that should do the following:

    1. Create a communications endpoint with a call to the socket() system call.
    2. Establish a connection with the connect() system call. This requires using the same local socket address as was used previously in your local server program's call to bind().
    3. At this point, barring any errors, the communcations channel is ready to be used from the client side. Your client program should send some integers to the server program and print them to its standard output stream in order to validate the functionality; the client program should then close the socket and end.

    Compile your client and server programs, and run them in separate terminal windows on your Raspberry Pi. Copy and paste the client and server programs' output as the answer to this exercise.

  5. Refactor your server so that (in addition to the loop it uses to read repeatedly from the same socket as long as it's open) it uses an outer event loop, so that it will stay alive through multiple connection attempts. This requires that your server make multiple calls to accept() (but not to listen() or bind()). Your server program should also print a message announcing each time a new connection is established and should then print out the integer values it reads from the socket each time, to the standard output stream.

    Rather than having your server stay alive forever, now modify your server program so that it terminates when it recieves a special integer value (e.g., the http status code 418). Modify your client so that it can accept zero or one command line arguments, and if it is given the command line argument "quit" it sends that special code to the server after it has sent its other integers (thus terminating the server).

    Compile your refactored server code and run it in its own terminal window. In another window, run your client program several times along with that server to validate that it can stay alive and keep accepting new connections until it is told to quit. Copy and paste the server output as the answer to this exercise.

  6. The primary use of sockets is to manage networked connections between remote computers. This exercise will have your client program connect from your Raspberry Pi, to your server program that is running on the shell.cec.wustl.edu host.

    Note that for this exercise, your server program will run directly on shell.cec.wustl.edu. You should not qlogin into a Linux Lab node.

    Make copies of your client and server programs, and modify the new copies to use an internet connection rather than a local connection. Choose a port number between 30000 and 40000 that both of your programs will use wherever a port number is needed in the instructions below. Run the command

    ping shell.cec.wustl.edu

    (on the Raspberry Pi you may need to preface that command with sudo)

    from a terminal window to obtain the numeric IP address of the shell.cec.wustl.edu machine, which your client program should use.

    1. Include the files netinet/ip.h and arpa/inet.h in both programs.
    2. In both programs, modify your calls to socket() (and bind() and connect(), respectively) to specify an internet connection type with AF_INET (instead of AF_LOCAL or AF_UNIX). Also remove any artifacts of the local domain socket from both programs, including removing the call to unlink() from the server code.
    3. In both programs, declare a struct of type sockaddr_in (instead of sockaddr_un) to specify the internet address. This struct is documented at man 7 ip. In both programs, set the sin_family field to AF_INET, and the sin_port field to htons(port_num), where htons() is a function that translates between the numeric representations used on your computer and in the network, and port_num is the port number that both your server and client programs are using. In the server program assign the sin_addr.s_addr field the value INADDR_ANY, and in the client program use the function inet_aton() to translate a c-style string containing the target IP address and store it in the sin_addr field of the sockaddr_in struct.

    Compile and run your server on shell.cec.wustl.edu. Compile your client on your Raspberry Pi and run it there twice, once without telling the server to quit and the second time telling the server to quit. As the answer to this question, copy and paste the output from your client and server terminal windows. If you cannot connect for whatever reason, give a descriptive error message using strerror() and errno.

    Please Note: If your Raspberry Pi is not on the school's network, you will need to first establish a remote connection, either with a VPN connection or through an SSH tunnel.

    Installing the VPN Client

    To install the VPN client on your Raspberry Pi, first click the raspberry icon on the top left corner of your desktop, select Preferences, and then select Add/Remove Software. In the menu that pops up, search for the keyword "openconnect". Then click on the following 3 items:

    After you have selected all 3 options, click OK to install the packages (you will be prompted to enter the new password you created on your Pi above).

    These instructions have been updated to use the new WashU 2FA (two-factor authentication) based VPN. If you have any problems with enrolling in or using WashU 2FA please call the WashU IT ServiceDesk at (314) 933-3333, or email support@seas.wustl.edu.

    Once you have installed the VPN, and have enrolled in WashU 2FA and have chosen which authentication method to use, please open up a terminal window on your Raspberry Pi, and type:

    sudo openconnect danforthvpn.wustl.edu

    At the Username: prompt please enter you WUSTL key ID. After that you should get a Password: prompt, where you should enter your WUSTL key password. You will then get a second prompt that also says Password: which is where you should enter one of the following keywords (depending on which authentication method(s) you set up when you enrolled in WashU 2FA):

    push -- enter this if you have set up DUO mobile notification (be aware that if you have more than one device enrolled, you may receive the push on only one device)

    phone -- enter this if you want to authenticate via a phone call to the number you have registered with the WashU 2FA system (this may only work if you only have registered a single number there).

    (note that other methods like entering sms also may work but involve other details that we have not tested out - please contact the WashU IT ServiceDesk at (314) 933-3333 or ithelp@wustl.edu, or contact Engineering IT at support@seas.wustl.edu, for help using those other methods.)

    Note: you will need to go through this process to start the VPN every time you reboot your Raspberry Pi and want to access the linuxlab machines directly from it.

    Establishing an SSH Tunnel

    An SSH tunnel allows you to forward traffic on a specific port to a remote server via SSH.

    To establish a tunnel, simply run the following command on your Raspberry Pi:

    ssh -f -N your-username@shell.cec.wustl.edu -L port-number:shell.cec.wustl.edu:port-number

    You must replace your-username with your WUSTL Key username, and port-number with the port you selected to establish the socket connection.

    For example, your command might look like: ssh -f -N msudvarg@shell.cec.wustl.edu -L 31522:shell.cec.wustl.edu:31522

    After issuing the command, you will be prompted for your WUSTL Key password. Alternatively, if you are already set up to log in with an SSH key, you may replace your-username@shell.cec.wustl.edu with the server nickname, as detailed on this page.

    If you use the SSH Tunnel approach, your socket code should be configured to instead connect to the local loopback IP address (i.e. 127.0.0.1) instead of the IP address you obtained for shell.cec.wustl.edu.


Things to turn in:


Page updated Wednesday, September 23, 2020, by Marion Sudvarg and Chris Gill.