CSE 532 Spring 2008

Lab 2: "Threads and Synchronization"

Points: 15% of final grade

Teams Declared/Assigned: by 11:59pm Sunday March 30, 2008

Due: by 11:59pm Sunday April 13, 2008, by electronic submission to cse532@cec.wustl.edu

Purpose

In this assignment you will again use the basic wrapper facades for networked OO programming provided by the ACE framework, that you used in the previous labs. In this lab you will refactor your Lab 1 solution to make extensive use of algorithms and iterators from the STL, which will require you to think carefully about how to integrate the synchronously invoked interfaces like the ++, *, and = operators provided by STL iterators, with asynchronously invoked interfaces like handle_input and handle_output. Multi-threading can help with this issue significantly, and careful thought up front about where to incorporate threads into your design and how to synchronize those threads to avoid concurrency hazards is strongly encouraged.

Two key ideas are worth considering as possible ways to add multi-threading to your Lab 1 solution - you are free to apply some, all, or none of them to your Lab 2 solution as is appropriate to obtain a good design.

1. Making an acceptor or connector into an active object (that is, giving it a separate thread) may be useful decouple it from the rest of the program, but then may require that additional synchronization be used to prevent race conditions between its thread and the other threads in the program.

2. Using active objects selectively to manage client connections may allow simpler relationships among service handlers, iterators, algorithms, and other parts of the program than would required if you used only reactive concurrency to implement asynchronous handling of multiple games at once.

You will also extend the capabilities of your client and server programs so that different kinds of poker games (5 card draw as in Lab 1, and in this lab also 5 card stud) can be played by the clients and servers, with the type and name of each game being selected dynamically on demand, and so that a clean termination protocol across a set of interconnected clients and servers can be triggered from the command line. We will use these ideas and the design and implementation skills you'll develop using them in Lab 3 as well, to extend your Lab 2 solution to make use of the concurrency architecture and service access and configuration patterns that will be covered in the remainder of this semester.

Assignment

The connection semantics of your client and server programs will be similar to those in Labs 0 and 1. Servers should again be started first and should listen passively for connections from the clients. Each client should initiate a separate connection for each line of a provided definition file, and keep each connection alive until it is closed by the server on the other end or until the client itself is shut down. As in Lab 1, both servers and clients should be able to manage multiple connections at once reactively.

A new key feature of both servers and clients for this lab, is that the service handlers in them should make extensive use of STL algorithms and iterators to perform their work. In particular, streaming of data across a socket should be implemented using iterators for STL containers, files and socket streams, and algorithms such as copy or transform, to move data among STL containers, files, sockets, and standard I/O streams as needed.

To do this, please study the provided code in the lab2.zip file. The provided client and server programs use a Lab 0 style of communication, but do so with socket iterators and the copy algorithm. If you use the provided code, your design and implementation will need to replace the simple blocking calls used in the provided code with an asynchronous (reactive and/or multi-threaded) implementation as noted in the comments in the provided iterator code files.

You will also add the notion of a game type, and will implement an additional game protocol for this assignment: 5 card draw. Last, you will implement a protocol to ensure timely and consistent shutdown of clients and servers using a reactor signal handler and asynchronous completion tokens.

To do this please perform the following tasks:

Servers

• Use the STL copy, transform, or other suitable algorithms in the appropriate places, to perform all transfer of data among sockets, data structures, files, etc., via iterators for those different container abstractions. You may want to base your approach on the approach in the provided lab2.zip file, but if so you should extend the synchronous blocking send and receive calls to be asynchronous (again, using reactive and/or multi-threading techniques) according to the comments in the iterator header files provided.

• In this lab, the new game (5 card draw) will replace some, all, or none of the cards in the hand, while in the other game (5 card stud) the cards will remain the same once they have been dealt to the hand.

  An important consideration in maintaining the containers for these data, especially when adding new data to or removing data from those containers, is whether adding/removing data may invalidate iterators already positioned within the container's data range. Austern pp. 73-75 has a good discussion of the issues involved with this: for example, an STL list does not invalidate iterators on insertion but an STL vector does.

• At startup, the server should register a signal handler for the SIGINT and SIGTERM signals with the reactor so that it can be told to terminate from the command line, for example using the UNIX kill utility or simply hitting Ctrl-C. Please see pp. 74 of C++NPv2 and the Signal_Test.cpp file provided with ACE for more details. In general this signal handler should merely set a termination flag to true in some shared data structure, and take no other action, since the handler will be called in signal context, and thus can only make a restricted set of system calls safely.

  The server should be initialized with its termination flag set to false. When the termination flag is set to true, the server should print out a ACT indicating that it is in the process of shutting down. The server should then send at least one asynchronous completion token (ACT) indicating termination is in progress, at least to each client with which it is associated: depending on how you structure the termination protocol you may want to go as far as to have the server send each player on each client a separate ACT for each game of which it is a member.

  One subtlety of this is that if a game is in progress and the server is still waiting on replies from clients (enough players have joined and cards are being dealt, but all players in the game have not yet sent their scores) when the server detects that its termination flag is set to true, the server should go ahead and distributed the termination ACTs to the clients in that game. However, if the server has received scores from all players it should wait to distribute the termination token(s) to the players in that game until after it has declared a winner.

  To ensure your server program checks the termination flag reasonably frequently, you probably should add a timeout to your calls to the reactor's run_reactor_event_loop and/or handle_events methods. If you use multiple threads along the end-to-end data path, you may need to add mechanisms for passing termination ACTs between threads as well.

• The server should also be able to receive termination ACTs from players, and if a player disconnects or sends a termination ACT after receiving cards but before sending back its score, the server should take any cards that had been dealt to the player, and put them back into the remaining deck and re-shuffle it, and also take the player out of the list of players from which the server is expecting to receive scores. If there are too few players after that player's departure, the server should declare a misdeal and notify the players, and then start the game over with a freshly shuffled new deck (and again waiting until enough players have joined before proceeding).

  Thus, termination (and also disconnection due to network problems or other causes) of clients and servers is decoupled, with a server continuing to run whether or not any players remain associated with it, until it gets its own direct termination signal.

• The same kind of simple main function as in Lab 1 should be used to initialize each server from the command line, using switches with the following syntax:
  server -n <dealer name> [-p <server port>]
  where:
  • <dealer name> is a string with no intervening spaces (for example, -n Dealer1), and <server port> is an alternative port number at which the server should listen for connections (if specified).

  The -p switch should be optional, and if one is not specified the server should default to listening on some port number specified by the Dealer class: please remember ports through 1024 are reserved for protocols like telnet, ftp, or http, so you should only use port numbers greater than 1024. Also, you may want to try a different port (and/or run ps -u <uid> to see what processes you already have running) if the server fails to bind on a given port.

  If the user fails to specify either the -n switch, or gives another switch not recognized by your sensor program (you are free to add any additional switches you think useful), your program should print out a useful syntax message before exiting, like the following:

usage: server -n <dealer name> [-p <server port>]

• When the server is first initialized (after the command line parameters have been processed) it should print out its name to the standard output stream.

• The server program should again create or otherwise obtain a reactor that it can use to demultiplex connection requests onto the acceptor, and to demultiplex service requests onto service handlers the acceptor creates and registers with the reactor, according to the reactive version of the Acceptor/Connector pattern.

• The server program's acceptor should again listen on the given (or default if none is given) port on the host machine where it is running, and should do so either (1) reactively, as described in Section 7.3 of the C++NPv2 book, or (2) by making the acceptor into an active object. To bootstrap this in either case, the server program should first invoke the acceptor's open call with the appropriate parameters. When the open call succeeds, the program should then again obtain the server's listening address and port number as a string from the address on which it was asked to listen, and print them out to the standard output stream or other location where you can give it to the client. After the acceptor has been opened, the server program's main thread of execution should then enter the reactor's event loop, by calling the reactor's handle_events or run_reactor_event_loop method.

• The acceptor should again:
  1. accept a socket connection request from a client,
  2. create a new instance of a socket service handler for that connection and associate the connection with the handler,
  3. ensure the service handler is registered with the reactor for reading/writing as appropriate, and
  4. allow the thread of execution to return to the reactor's event loop.

• Whenever a server service handler is opened, it should perform the following activities in sequence in a way that (1) makes use of appropriate STL algorithms and iterators, and (2) asynchronously (i.e., reactively or using multi-threading) reads and/or writes information from/to the socket connection:
  1. read the player's name from the newly created connection;
  2. send its dealer name back to the client on the newly created connection;
  3. read the complete series of game types and names;
  4. add the newly connected player's name and socket and list of game types and names to the server's data structure(s);
  5. in each game deal 5 cards to the player; for the 5 card draw game also perform the following steps:
     1. read back the number of cards the client wants to draw,
     2. send that many cards to the client as well,
     3. read back that same number of cards which the client is discarding, and
     4. add them back to the deck and reshuffle the deck;
  6. read back the score from the player, and declare a winner as in Lab 1 by printing a ranked ordering of the players names and scores, from highest score to lowest score.

  Note: A key design question is which method will call the STL algorithms for copying data. Each service handler will naturally need to call those algorithms to move data into or out of sockets, output streams, and/or file streams.

• If a client drops its end of the connection, the appropriate service handler should be removed from the reactor, release its resources gracefully, and be destroyed. If a client sends the server a termination ACT (see below) the server should behave as though the client had simply dropped the connection, and in both cases should maintain all other game and player information and connections, either continuing each game if enough players remain, or declaring a misdeal and restarting the game if it had started and there are no longer enough players, as described above.

• To demonstrate your Server class, you should please start at least two separate copies of the server program (using a different port and/or host machine for each), as in:
  server -n Dealer1 -p 10070
server -n Dealer2 -p 10071
  

Clients

• In this lab you will make several modifications to your Lab 1 client implementation, similar to what you did for the server above.

• Similar to the server program, each client should again create or obtain a reactor.

• Like the server, the client should register a signal handler for SIGINT and SIGTERM with the reactor. Please see pp. 74 of C++NPv2 and the Signal_Test.cpp file provided with ACE for more details. In general this signal handler should merely set a termination flag to true in some shared data structure, and take no other action, since the handler will be called in signal context, and thus can only make a restricted set of system calls safely.

  Again like the server, the client should be initialized with its termination flag set to false. When it detects that the termination flag has been set to true, the client should print out a message indicating that it is in the process of shutting down. The client should then send at least one asynchronous completion token (ACT) indicating termination is in progress to each server with which it is associated: depending on how you structure the termination protocol you may want to go as far as to have the client send each dealer on each server a separate ACT for each game of which one of its players is a member.

  If the client is in the process of sending/receiving a particular piece of data to/from a server (e.g., the number of cards it wants to draw) it should finish that activity before sending a termination ACT to that server, but should send the ACT before it sends/receives any other data (e.g., reading the new cards the server sent) and proceed to shut down gracefully. To ensure your client program checks its termination flag reasonably frequently, you may want to add a timeout to its calls to its reactor's run_reactor_event_loop and/or handle_events methods, as you likely did in the server.

• The client should be able to receive termination ACTs from a server, and when a server is disconnecting its handlers should be removed from the client's reactor. When all servers with which the client is associated have terminated, or if the client itself received a signal through its termination handler and has notified all its servers, the client should shut down gracefully by closing its reactor, releasing all other resources, and terminating.

  You should add the ability to handle a -q command line switch which says that the client should automatically send termination ACTs and then shut down gracefully as soon as all of its games are finished. If that switch is not give the client should simply remain quiescent until it is shut down via an explicit Ctrl-C.

• A main function should again initialize the client from the command line, through switches with the following syntax:
  client -n <client name> -f <definition file> [-q]
  where:
  • <client name> is a string with no intervening spaces (for example, -n Player1), and
  • <definition file> is a string with no intervening spaces that gives the name (and possibly also the directory path) for the client's definition file: (for example, -f Player1Definition).

  If the user fails to specify either the -n or -f switches, or gives another switch not recognized by your program (you are free to add any additional switches you think useful), your client program should print out a useful syntax message before ending gracefully, like the following:

  usage: client -n <client name> -f <definition file>

• Using the file name given with the -f switch, your client program should open that file and parse its contents in preparation to making connection requests and communicating with servers once connections are established. If the file given by the -f switch cannot be opened or read, the client program should print out an error message indicating the kind of problem it encountered, send termination ACTs to its associated servers, then release its resources and shut down gracefully.

  The format of the client's definition file is again up to you to choose, though the following text file example in Player1Definition is a reasonably good (and simple) approach. In general each line of the file should specify:

  • a game type,
  • a game name, and
  • optionally the port and/or the host name or IP address for the server on which the client will join the game.

  If a line in the definition file leaves off the game type or the game name, your client program will not be able to use it, and should print out an error message for that line, but should then continue to parse later lines and try to set up and use those connections.

  If a line in the definition file leaves off the server port or IP address, your program should use a default port and/or a default IP address (the local host) as appropriate.

• For each well-formed line in the definition file, the client should:

  1. initialize a connector to request a connection to that server, and open the connector with the appropriate parameters.

  2. initiate a socket connection to the server at the address for that server, but allow the completion of that request to be handled reactively and continue to the next server in the list rather than blocking on the connection establishment request.

  One way to do this is to construct each connector within a sequence container, and then use the for_each STL algorithm to apply a unary function object to each connector in the sequence, which initiates a connection request and does either (1) a reactor registration (reactive) or (2) a thread activation (multi-threaded) for each of the connectors -- it's not mandatory to do this, but it's another good way to use the existing capabilities of the STL in your programs. When the client program has finished creating and registering connectors and initiating connection requests on them, it should call the reactor's handle_events or run_reactor_event_loop method to start the event loop.

• When each connection request completes, the connector should create a service handler for that connection and register that service handler with the reactor.

• Each service handler should perform the following steps (1) using appropriate STL algorithms and iterators, and (2) implementing the synchronous iterator methods (like operator++ and operator=) that need to move data onto or off of the socket asynchronously (using only one read or write per up-call from the reactor, or using a separate thread which can then make multiple calls until done or use send_n, etc.) and in sequence:

  1. send the player name to the server over the newly created connection,

  2. read the dealer name from the connection and print out a message to the standard output stream indicating the name of the dealer to which it has connected,

  3. send the server a list of the game types and names it wants to join on that server: again, how you structure the protocol for sending the list (in particular, how to detect the end of the list) is up to you, but watch out for cases where the server and client could have different expectations which can lead to deadlock,

  4. in both 5 card draw and 5 card stud games, read 5 cards from the server and store them in an STL container,

  5. in 5 card draw games only, determine which cards to replace as follows, and send that number of cards (between 0 and 5) and the cards themselves, to the server, then read back that many new cards from the server:

     • straight flush, full house, flush or straight: do not replace any cards ("stand pat").

     • four of a kind: if and only if there are more ranks (not counting the rank of the four cards) above the unmatched card than below it, replace that one card with a new card from the deck.

     • three of a kind: keep the three cards of the same rank, and replace the other two.

     • two pair: keep both pairs, and replace the other card.

     • one pair: if the pair is jacks or higher or if none of the other three cards is an ace, keep the pair and replace all three of the other cards; otherwise, keep the pair and the ace and replace the other two cards.

     • no rank: (1) if there are four cards of the same suit or that have consecutive ranks (not including an ace) replace the one other card; otherwise if (2) one of the cards is an ace, replace the other four cards; otherwise replace all five cards.

     If any cards are replaced in any player's hand, this method should print out a line with the name of the player, the cards that were replaced, and the new cards that replaced them.

     To ensure that the data from multiple streams is printed coherently, in this lab you may want to copy all the data that is destined for output into a separate synchronized data structure, and then print out the data from that data structure. Note that this kind of approach also may be needed on the server as well.

  6. If the server shuts down the connection or sends a termination ACT through it, the service handler(s) for that server (and only that server) should release its (their) resources gracefully (e.g., calling close on the socket once), be removed from the reactor, and be destroyed.

• To demonstrate your client class, you should first please start separate copies of the server program (using a different port and/or host machine for each), and then start separate copies of the client, as in:
  
  client -n Player1 -f Player1Definition
client -n Player2 -f Player2Definition
client -n Player3 -f Player3Definition
client -n Player4 -f Player4Definition
client -n Player5 -f Player5Definition
  

  You should also test different sequences of client and/or server startup and shut-down to verify correct behavior of the different games and of your termination protocols.

Resources

Also, there are several more mundane (but as in any software project, important) areas where you can exploit tools and examples to greatly speed your progress on this project and the ones later in the semester. In particular, code fragments and Makefile examples are provided as exemplars to follow or discard, as you choose.

Wrapper Facades

• ACE_Acceptor -- a generic abstraction for accepting socket connection requests
• ACE_Connector -- a generic abstraction for requesting network connections
• ACE_Get_Opt -- an option-string parameterized abstraction for parsing command line arguments
• ACE_INET_Addr -- an internet domain socket abstraction
• ACE_Reactor -- a generic abstraction for reactive event demultiplexing
• ACE_SOCK_Acceptor -- an acceptor abstraction for sockets
• ACE_SOCK_Connector -- a connector abstraction for sockets
• ACE_SOCK_Stream -- an internet domain stream socket abstraction
• ACE_Svc_Handler -- a full featured service handler abstraction
• ACE_Thread_Manager -- a class used for managing threads directly
• ACE_Thread_Manager -- a class used for managing threads directly
• ACE_TSS -- an implementation of thread specific storage
• ACE_Guard -- an implementation of a guard for scoped locking
• ACE_Thread_Manager -- a class used for managing threads directly
• ACE_Null_Mutex -- implements a do-nothing mutex (for strategized locking)
• ACE_Thread_Mutex -- implements a thread mutex synchronization object
• ACE_Recursive_Thread_Mutex -- wraps or emulates a recursive thread mutex
• ACE_RW_Thread_Mutex -- implements a readers/writer thread mutex
• ACE_Null_Semaphore -- implements a do-nothing semaphore (for strategized locking)
• ACE_Thread_Semaphore -- implements a thread semaphore synchronization object
• ACE_Condition_Thread_Mutex -- implements a thread mutex condition variable
• ACE_Message_Queue -- implements a queuing layer used with multi-threading
• ACE_Task -- active object that can interoperate with other threads, reactors, etc.

The Reactor and Acceptor/Connector pattern implementations in ACE are discussed in detail in chapters 3, 4, and 7 of the C++NPv2 book:

• Schmidt and Huston C++ Network Programming, Volume 2, Addison-Wesley, 2003.

The concurrency and synchronization pattern implementations and wrapper facades in ACE are discussed in detail in chapters 5, 6, 9, and 10 of the C++NPv1 book:

• Schmidt and Huston C++ Network Programming, Volume 1, Addison-Wesley, 2003.

Implementation of the Active Object and Half-Sync/Half-Async patterns and their associated wrapper facades is discussed in detail in chapter 6 of the C++NPv2 book.

STL Iterators and Algorithms

• input iterators -- Austern 7.2, pp. 94-96
• output iterators -- Austern 7.3, pp. 96-100
• istream_iterator -- Austern 14.2.1, pp., 354-357
• ostream_iterator -- Austern 14.2.2, pp., 357-359
• for_each algorithm - Austern 11.4, pp. 218-220
• copy algorithm - Austern 12.1.1, pp. 233-235
• transform algorithm -- Austern 12.3, pp. 240-243
• unary function objects -- Austern 8.1.2, pp. 111-112
• sequence containers -- Austern 5.3.1, pp. 73-75
• list container class -- Austern 16.1.2, pp. 441-448

EXTRA CREDIT: A bonus of up to 5 points will be given to any team submitting a lab that in my judgement makes highly effective use of the ACE wrapper facades and STL abstractions, and whose writeup makes and provides suitable justification for that claim. In general, simpler and shorter programs will be preferable to longer and more complicated ones. Clearly written programs will be preferred over ones that are harder to follow. Programs with better documentation in the README will be preferred over those with weaker documentation (please also comment your code well). Of course, correct programs will be preferred over those with bugs in them. The solutions that achieve this goal will be awared up to an additional 5 extra credit points out of a total of 100 points for this assignment (scores above 100 are possible), and the team(s) receiving such an award will be invited to present a brief summary of their approach in class.

Makefile & Environment

You will need a Makefile that links appropriately with the ACE libraries and include files. Please feel free to adapt or use the following example Makefile provided in the lab2.zip file.

What to Submit

README

The first section of your README file should include:

1. the number of the lab (e.g., "CSE 532 Spring 2008 Lab 2")
2. the names and e-mail addresses of all team members
3. an overview of how your programs are designed,
4. the Wrapper Facades you used or extended and how they helped in your design and code,
5. any insights and questions you encountered while completing the assignment, and
6. if you are applying for extra credit please also make your case for why your use of the STL and ACE is exemplary.

The second section of your README file should provide detailed instructions for how to:

1. unpack your files,
2. build your programs, and
3. run your programs.

Electronic Submission

1. your source code files,
2. your Makefile,
3. any example files you think useful (for example, a copy of the Player1Definition file showing the format you chose), and
4. your README file.

Please send a separate copy of your README file, with clear instructions on how your submission file was produced and how to unpack it using tools available on the CEC Linux machines.

Grading

Correct Compilation and Operation

Design Quality

Implementation Quality

Coding Style

Documentation

Missing Files

Late Submission

Grading Issues from Previous Labs

• using hard coded constants
• using dynamic allocation when a class member would do: if it's dynamically allocated in the constructor and dynamically deallocated in the destructor, then there must be some reasonable design force that motivates this approach rather than using an object directly as a member of the class (which your readme file should of course please explain)
• using the same error message for different failures
• using a pointer/size arithmetic formula without a comment explaining it: in general any non-obvious code should have a comment giving an intuition of what it does in general terms (and even better explaining why or how it does it that way)
• putting non-inline function definitions in a header file
• initializing class members in the constructor body when it is possible to intialize them in the base/member initialization list
• initializing class members in the base member initialization list when that initialization could fail, e.g., when a function whose return value matters or that could throw an exception is called
• unnecessary format restrictions, i.e., not tolerating blank lines and extra whitespace in a file
• leaving "commented out" code fragments left in code - remove these or use conditional compilation to bring them in/out
• adding comments for things that are already fairly obvious from the code (e.g., begin and end of a method)
• using exit to end the program in the middle of a class method or any method that could be called from another method
• using fixed length buffers when the lengths of inputs to them are variable/unknown
• unnecessarily cramping symbols or lines together, which impacts readability - use whitespace generously: it's free!

• use precompilation include guards in header files to avoid duplicate inclusion:
  
#if ! defined (MY_FILE_H)
#define MY_FILE_H

// body of the header file

#endif /* defined MY_FILE_H */


• check the return value from any call that could return an error code after a failure (e.g., new (nothrow) )
• use try/catch blocks around any call that could throw an exception after a failure (e.g., new)
• use the constructor's base/member initializer list in preference to the body of the constructor for member intialization, except were it is necessary to use the constructor body (please see above)
• using typed constant variables with descriptive names in preference to precompiler #define macros
• use consistent and appropriate comments
• use inlining for efficiency, and use the technique in ACE that allows inlining to be turned on or off by a precompilation macro
• discuss design decisions in your README file
• ask remaining questions you have in your readme (!)
• test your code for its handling of error cases as well as of the "happy path"
• perform "tiger team" testing - take turns trying to find cases that break your program, and then fixing them
• using the string class for variable length character buffers
• using whitespace generously: it's free and it improves readability!