Description
Objectives
In this assignment, you will implement a server that communicates with clients using an HTTP-like protocol. After completing this assignment, you should (1) be able to implement a simple TCP-based server, and (2) have a good understanding of the mechanisms of HTTP.
Group Work
Grading
We will test and grade your program on the sunfire server. Please make sure that your program run properly on sunfire. Moreover, you are allowed to use libraries installed in public folders of sunfire (e.g. /usr/lib) only.
We accept submission of Python 3 (in particular, 3.7), Java, or C/C++ program, and we recommend that you use Python 3 for your assignments. Programming languages other than Python 3, Java, and C/C++ are not allowed. For Python 3, we use the python3 program installed in folder /usr/local/Python-3.7/bin on sunfire for grading. If you use Java or C/C++, we will compile and run your program for grading using the default compilers on sunfire (java 9.0.4 installed in /usr/local/java/jdk/bin, or gcc 4.8.4 installed in /usr/local/gcc-4.8/bin). The grading script infers your programming language from the file extension name (.py, .java, .c). Therefore, please ensure your files have the correct extension names. For a Java program, the class name should be consistent with the source file name, and please implement the static main() method so that the program can be executed as a standalone process after compilation. We will deduct 1 mark for every type of failure to follow instructions (e.g. wrong program name).
Note that for Java programs, name your main class as WebServer and hence source file name WebServer.java during development. When you want to test your program using the provided script on Sunfire, or to make submission, rename your file name according to program submission and group submission section while keeping the class name as WebServer. Grading script will rename your file accordingly during compilation.
We will grade your program based on its correctness only. A grading script will be used to test your program and no manual grading will be provided.
Testing Your Program
To test your program, please use your SoC UNIX ID and password to log on to sunfire as instructed on Assignment 0 paper. To make the python3 alias permanently and thus avoid typing the alias command every time you login, you can run the following command once to store the shortcut into the configuration file: echo alias python3=/usr/local/Python-3.7/bin/python3 >> ~/.bash_profile
Your server program should receive one command-line argument which is the port number to listen(), as the following command shows: python3 WebServer-A0165432X.py <port>
If you test your server manually, please select a port number greater than 1024, because the OS usually restricts usage of ports lower than 1024. If you get a BindException: Address already in use (or similar errors for other languages), please try a different port number. Note that your program should not read from stdin. Your program can print anything to stdout or stderr, and our test script will silently ignore them.
To use the grading script, please upload your program along with the test folder given in the package to sunfire. Make sure that your program and the test folder are in the same directory. Then, you can run the following command to test your server program: bash test/WebServer.sh
By default, the script runs through all test cases. You can also choose to run a certain test case by specifying the case number in the command:
bash test/WebServer.sh 3
To stop a test, press ctrl-c. If pressing the key combination once does not work, hold the keys until the script exits.
Program Submission
For individual submission, please name your single source file as WebServer-<Matric number>.py and submit it to the Assignment_1_student_submission folder of LumiNUS Files. Here, <Matric number> is your matriculation number which starts with letter A. An example file name would be WebServer-A0165432X.py. If you use Java, C, or C++ to implement the web server, please use .java, .c, or .cpp respectively as the extension name. Note that file names are case-sensitive on sunfire.
You are not allowed to post your solutions to any publicly accessible site on the
Internet.
Special Instructions for Group Submission
For group submission, please include matriculation numbers of both students in the file name, i.e. WebServer-<Matric number 1>-<Matric number 2>.py. Submit it to the same Assignment_1_student_submission folder. An example file name would be WebServer-A0165432X-A0123456Y.py. For each group, there should be one designated member who submits the file, to avoid problems caused by multiple branches within a group. Do not change the designated submitter! If the group needs to upload a new version, it should be done by the same designated submitter as well.
You are free to discuss this assignment with your friends. However, you should refrain from sharing your program, program fragments, or detailed algorithms with others. If you want to solve this assignment in a group, please do so and declare it as group work.
Question & Answer
If you have any doubts on this assignment, please post your questions on LumiNUS forum before consulting the teaching team. However, the teaching team will NOT debug programs for students and we provide support for language-specific questions as a best-effort service. The intention of Q&A is to help clarify misconceptions or give you necessary directions.
The Server
In this assignment, you are required to implement an HTTP-like server that provides key-value store services and a frequency counter service. The server consists of two components. The first is a custom protocol simplified from HTTP 1.1. It retains the basic request-response mechanism and the persistent connection feature of HTTP 1.1, but uses a different header format. On top of this protocol, an interactive in-memory keyvalue store and a frequency counter are to be implemented under the path /key/ and /counter/ respectively. Figure 1 shows the overall architecture of the web server.
Figure 1: The layered architecture of the server. You need to implement the top two layers which are highlighted with bold fonts.
The TCP Socket
The lowest-level layer of this server is the TCP socket interface. Basically, the server should
1. bind() to a port
2. listen() to the socket and accept() a new connection
3. Parse and respond to client request(s) sequentially, by using recv() and send() for socket I/O.
4. Going back to 2 after the client disconnects
As discussed in the “Testing Your Program” section, port number is passed to your server as a command line argument. The grading script ensures that only one client connects to your server at any time so that no connection multiplexing or polling is required; your server can simply process connections sequentially.
It is necessary to detect disconnection events reliably, because your server is required to support the persistent connection feature originated from HTTP 1.1. If the bytes object returned by recv() is of zero length, then no more data could be recv()’ed from the connection. Since this only indicates half close (from client side only) of a TCP connection, your server should finish processing the final request(s) before closing the connection on server side. The client is guaranteed to keep the connection half open until either the server finishes sending all responses or a timeout occurs. Therefore, there will be no send() failures on the server side, unless it times out during this process.
The HTTP-like Protocol
After establishing a new TCP connection with a client, your server should read a customized HTTP request from the socket, which consists of request headers and an optional request body. Your server parses the request and sends the corresponding response as defined below. After such a request-response cycle, the connection is persisted (i.e. not closing immediately), and the server waits for another request from the same client until a notification of socket disconnection.
Since a real-world TCP connection could be intermittent, your server is also required to work properly when requests are delivered in chunks of random sizes with delays between chunks. Note that a TCP connection only guarantees ordering of data, but not segment sizes or delays. The test script simulates this type of intermittent connection in some test cases, while keeping the delays between two consecutive transfers below one second. To handle chunks that contain incomplete requests, it is sufficient for your server to handle data buffering well.
In addition, your server should be responsive in the sense that upon receiving a complete request, the server processes it and sends back the response immediately. The test script sets a one-second timeout after each request is fully sent to the server. If a single chunk contains multiple requests, your server is also expected to respond quickly to all complete requests inside.
To grasp the core ideas of HTTP, we omit other tedious aspects of the protocol. For this assignment, the server only needs to handle basic request and response information, as well as the Content-Length header field. The header format of this HTTP-like protocol is simplified as follows:
no more or colons).
3. For a request, the first two substrings (compulsory) are HTTP method and path, respectively. The HTTP method is case-insensitive, while path is case-sensitive.
(a) GET␣/key/CS2105␣␣
(b) POST␣/key/ModuleCode␣Content-Length␣6␣␣CS2105
(c) GET␣/key/ModuleCode␣␣
(d) GET␣/counter/CS2105␣␣
(e) POST␣/counter/StudentNumber␣␣
(f) GET␣/counter/StudentNumber␣␣
(Here, space is illustrated by ␣ and content body is underlined.)
4. For a response, the first two substrings (compulsory) are HTTP status code and description of status. The description has no real effects and is kept as a convention as in HTTP, and it should not have any spaces. If the response has a content body, a Content-Length header should similarly be included, if there’s content body. The corresponding responses for the above three requests could be:
(a) 404␣NotFound␣␣
(b) 200␣OK␣␣
(c) 200␣OK␣content-length␣6␣␣CS2105
(d) 200␣OK␣content-length␣1␣␣0
(e) 200␣OK␣␣
(f) 200␣OK␣content-length␣1␣␣1
The Key-value Store
The key-value store should be accessible to the client through paths with prefix /key/. To operate on a specific key, the complete HTTP path is /key/ appended with the actual key string. For example, a request to operate on key ModuleCode specifies /key/ModuleCode as the HTTP path. The server should support the following key-value operations over the HTTP-like protocol:
1. Insertion and update
(a) The HTTP request method should be POST and the value to be inserted (or updated) constitutes the content body. There should also be a ContentLength header indicating the number of bytes in the content body.
(b) The server should respond with a 200 OK status code after inserting or updating the value. The client always expects success statuses for insertions and updates.
2. Retrieval
(a) The HTTP request method should be GET, and there is no content body.
(b) If the key does not exist in the key-value store, the server returns a 404 NotFound status code. Otherwise, the server should return a 200 OK code, the correct Content-Length header and the value data in the content body.
3. Deletion
(a) The HTTP request method is DELETE, and there is no content body.
(b) If the key does not exist, the server returns a 404 NotFound code. Otherwise, it should delete the key-value pair from the store and respond with a 200 OK code and the deleted value string as the content body. The Content-Length header should also be sent accordingly.
The Counter Store
A counter store is very similar to key-value store, except that it stores non-negative frequency count, instead of bytes value, of a particular counter key. For simplicity, it should support update, retrieval of counter values, with counter keys functioning as pointers to the frequency values. Counter keys are case-sensitive ASCII strings. Unlike key-value store, no content-length header is present in a POST request, while it is needed in the response to a GET request. The server should keep all data in memory only and avoid accessing the disk.
The counter store should be accessible to the client through paths with prefix /counter/. To operate on a specific counter key, the complete HTTP path is /counter/ appended with the actual counter key string. For example, a request to operate on counter key
1. Update
(b) The server should respond with a 200 OK status code after updating the counter. The client always expects success status for updates. 2. Retrieval
(a) The HTTP request method should be GET, and there is no content body.
(b) If the counter key was never updated before, corresponding frequency should be 0. The server should return a 200 OK code, the correct Content-Length header and the frequency value in the content body without leading 0s.
Grading Rubric
1. The server program is free from syntax errors and can be successfully executed
(or compiled, for Java/C/C++). (1 mark)
(a) The server parses valid HTTP requests and sends correctly formatted responses.
(b) The server supports persistent connection and correctly processes sequential requests as specified.
(c) The server works properly and responsively over intermittent connections simulated by the test script.
(a) The server supports insertion/update of key-value pairs through the POST method according to the spec.
(b) The server supports retrieval of value by key through the GET method according to the spec.
(c) The server supports removal of key-value pairs through the DELETE method according to the spec.
4. The counter store (1 mark)
(a) The server supports update/retrieval of counter store according to the spec.
