Handy tools for testing programming assignments and maintaining class directories

Index

Thanks to the many TA's (particularly David Newman) who have used these tools and provided valuable input for improving them. And thanks to Jack Applin who provided several scripts and suggested the method for handling the symbolic link CurrentSemester.

Fritz Sieker -

Getting Everything

To get a local copy of these tools do the following:
  1. Create a location for them. I suggest the directory ~/bin/tools.
  2. cd to the directory of step 1.
  3. make -f ~cs000/public_html/tools/Makefile
  4. Set your $PATH variable to include the directory from step 1.

Top Level Tools

The following are various tools and other files I wrote for maintaining a class directory structure, generating Java assignments, and for testing the students' code (Java/C/LC-3 assembler to date). There is a lot of detail following, but you might be able to tell from the name whether it would be of interest.
  1. Maintaining Directories/Checkin
  2. Structure of java Assignments
  3. Testing Programming Assignments

File/Script index

This is an index of all directories, files, and tools mentioned in this document.

Maintaining Course Directories and Checkin

Motivation

When I first began to teach, I had a difficult time finding materials from previous semesters. Often I could only determine what material went with what semester by looking at file dates. To make my life easier, I started to organize the directories of the courses according to what would be convenient for me (Your mileage may vary). I have also built a series of tools to make creating and maintaining things easier and more consistent.

In talking with various faculty members, I found that it would be convenient for the faculty be able to easily browse various semesters of a course to see how it was evolving. It was also suggested that we might want to publish our historical versions to promote our department. The directory structure incorporates these suggestions. The "openness" is controlled by Linux permissions.

This directory structure is currently in place for cs160, cs161, cs200, cs270. It was in place for cs370, but has been disabled. To see the practical result of this directory structure, visit this

Course Directory Lifecycle

In this structure, the directories follow a lifecycle:
  1. Creation - directory structure is created (tool newSemester) and populated with content.
  2. Goes Live - directory becomes the default accessed by the course web page (tool makeCurrent).
  3. Retirement - directory is no longer active. It should have all write permissions removed and perhaps be password protected. Ideally the web pages woulg get a watermark background indicating "retirement". No tools for this yet.

Directory Structure

The course materials are stored under the directory of the actual class, rather than in the instructor's directories or on RamCT. There is one top level directory per semester, located in ~csXXX/public_html. This has several advantages: As an example, consider the (partial) listing of ~cs370 
     drwxr-xr-x  3 cs370 class 4096 Jul 19 13:09 bin/
     lrwxrwxrwx  1 cs370 class   35 Jul 19 13:15 Checkin -> public_html/CurrentSemester/Checkin/
     drwxr-xr-x 17 cs370 class 4096 Jul 19 14:47 public_html/
     -rw-r--r--  1 cs370 class 1307 Jul 19 14:04 README

and the (partial) listing of ~cs370/public_html 
     lrwxrwxrwx  1 cs370 class    8 Jul 19 14:46 CurrentSemester -> .Spring11/
     drwxr-x---  7 cs370 class 4096 Jul 19 13:49 .Fall09/
     drwxr-x---  7 cs370 class 4096 Jul 19 13:50 .Fall10/
     drwxr-x--x  9 cs370 class 4096 Jul 19 13:57 .Fall11/
     -rw-r-----  1 cs370 class 1281 Jul 19 14:19 .history.html
     -r--r--r--  1 cs370 class   62 Jul 19 13:17 index.php
     -rw-r--r--  1 cs370 class 1307 Jul 19 14:03 README
     drwx--x--x 10 cs370 class 4096 Nov 19  2010 .Shared/
     drwxr-x---  7 cs370 class 4096 Aug 17  2010 .Spring09/
     drwxr-x---  8 cs370 class 4096 May  3  2010 .Spring10/
     drwxr-x--x  3 cs370 class 4096 Jul 19 13:25 .Spring11/
     drwx------  3 cs370 class 4096 Feb 12  2009 .Spring09/

There are several things to note:

Creating a new Semester

The script newSemester facilitates the creation of a new semester. It creates a directory named for its first parameter. The second parameter is an option of what to create. The options are:

Getting the Checkin directory set up

The checkin program require a users file that defines who is allowed to check things in. It can be opened up to general submissions, but I prefer to actually specify the users enrolled in the class. To do this, first use AriesWeb or RamCT to access the class list(s). Save the info as a comma separated list of values. If there is more than one section, save the class list for each section. I save these files in the Checkin directory. Then use the script genUsers. This script parses the file(s), looks up the login of each student and produces the file ~/Checkin/users. Any errors are students that do not currently have a cs login. This may be a result of a new student or perhaps a name change (e.g. someone gets married). I always add myself and the TA's to the file so that we can run checkin just like the students do.The script also produces a number of other files that are used to upload grades to RamCT and used to split grades into multiple sections.

The script also produces the files allLogins logins.001 ... . Theses files are used by the grading tools. The files allLogins is a sorted list of all the students in the course. This is used to produce the users file. The logins.00X file(s) correspond to the logins for the students in the section(s) of the course.

The script also produces a start for the file ~/Checkin/assignments. This file defines the assignments and when they can be checked it. See the comments at the beginning of the file for details.

The script also produces a number of mapping files (*.map, mapCStoRamCT) that are used to upload grades to RamCT.

NOTE: Since these tools access the file via ~/Checkin/ they should only be run after the makeCurrent script resets the public_html/currentSemester link.

Creating assignment directories

I have adopted a standard directory layout for programming assignments. All assignments live in the ~/public_html/CurrentSemester/assignments directory. There is one sub-directory per assignment. My convention is to use the names PAx for programming assignments and HWx for written assignments. The names of the directories are then the key in the ~/Checkin/assignments file that defines when assignments may be checked in. Although any key may be used, I keep them short because they are used as file suffixes by many of the testing tools.

For every project, the assignment sub-directory contains the following: 

    -rw-r--r-- 1 cs370 class 5510 2010-04-29 15:40 PAx.html
    drwx------ 2 cs370 class 4096 2010-04-22 16:15 class
    drwxr-xr-x 3 cs370 class 4096 2010-04-22 16:16 doc
    drwx------ 2 cs370 class 4096 2010-05-09 16:32 soln
    drwxr-xr-x 2 cs370 class 4096 2010-05-09 16:00 src
    drwx------ 8 cs370 class 4096 2010-08-26 11:21 test
Things to note: The script newAssignment creates this directory structure.

Once the source files have been generated, the script genDoc creates javadoc for the file. This script expects to be executed from the src directory. The script produces javadoc including all private fields/methods so that students can view everything. There is a tool gendoxy that generates documentation for non java files using doxygen.

Force a checkin AFTER drop dead date

Used to place an assignment in the Checkin directory after the drop dead date of the assignment. See .

Back to index

Structure of Java Assignments

Sample assignment here.

Motivation

When I taught cs161 in Spring 2010, I reworked some Java code I had developed for another class. This became the basis for the assignments. The purpose of this freshman class is to introduce object oriented programming as well as some discrete math. The assignments were independent. That is, no assignment built on a previous assignment. However, the same code has been used successfully in a course in which assignments built on one another. I also wanted something that would encourage the students to do incremental development and testing. And, I wanted something that would allow easy testing of discrete pieces of each assignment. The basis of each assignment was a command interpreter which is described in the next section. Every assignment consisted of multiple Java files. Some were provided. Others were written/modified by the students. To keep things more obvious, all files that the students wrote or modified were named with the pattern MyClassName using the My to emphasize what the students should work on.

Most of the documentation for assignments came from javadoc of the provided file(s). This forces the students read javadoc to get necessary information and to encourage them to use the online javadoc to learn about the many Java classes that are part of the java environment.

The Command Interpreter

Every assignment contained the Shell. This class provides some simple generic commands and some helper methods for collecting parameters. It also provides a simple debugging/tracing interface that can be toggled at runtime. In writing this code I had a number of objectives: This code, by itself, does not do much. However, it provides a convenient base that can be extended for individual assignments. One handy command that it implements is input which takes commands from a file and processes them just as if they were typed at the console. Multiple input may be nested to any depth and provides a nice example of recursion.It Also has an exit which terminates the current input stream and resumes processing at the next outer layer. If there is no outer layer, the program is terminated.

The Assignment Interpreter

Every assignment also had an AssignX.java class. This file extends the class Shell, thus illustrating inheritance. It adds the commands that test the functionality of this assignment. This happens in the processOneCommand() method. This nicely illustrates method overwriting and how the super keyword is used. We provided this code for many assignments, but later just provided a skeleton of the class and had the students complete it. This structure separates the testing of commands from the implementation of the assignment details.

If one had a series of assignments that built on one another, this could still be used by simply having each assignment extend the previous one and add some new functionality. On could even override commands in later assignments to provide additional functionality to something done earlier.

The example code is the first assignment of that semester. Its purpose was to get the students writing code and to test their ability to write loops, nested loops, to read from files, and do simple logic. This example will be used in the section on testing to illustrate test cases.

The MyAssignment Code

To complete an assignment, one needs to implement methods that provide the required functionality. There are several ways of doing this. The first is to make the class AssignX abstract by adding the methods (and javadoc) that the students must implement. This introduces a 3rd file MyAssignX.java to the assignment. This class extends AssignX. Using an IDE such as Eclipse, empty implementations can be produced automatically. Now one has a compilable, runnable program, but the methods will not function correctly. With the command interpreter (often one command per method the student needs to implement), the student may code and test incrementally.

Another alternative is to define the methods via an interface that the students code must implement. This decouples the students code form the interpreter hierarchy. Unfortunately, nether solution works when you want to introduce a static method into the assignment.

Back to index

Testing Programming Assignments

Motivation

To aid in grading programming assignments, I wrote a series of tools and have include contributions from other instructors and students. This framework has been used to grade Java/C/LC3 assembly programs, but is not limited to those. The tools are tightly coupled to the checkin program in that they rely on the key used to check in an assignment and the location where the checkin program stores things. The tools have been used to grade programs submitted via RamCT. This only requires grabbing the submissions from RanCT and putting them in the correct stucture. Wim Boehm has written a program to do this. Some of the goals of the work included: In the end, the tools could easily grade 200+ students per hour on a very old linux box. Thus, we were able to return results to the students only a few hours after the submission deadline.

Checkin directory structure (for RamCT users)

This is only of interest if you do not want to use checkin but still utilize the testing framework. The tools assume that they can look in ~/Checkin and its subdirectories for information. The organization is as follows:

Strategy

The basic idea of the tools is to capture the output of a run and diff it against a master. Recognizing that diff may be too "harsh", the grader may supply additional tools to either grade the output directly or "cook" it and then diff the result against the "cooked" master output. And beyond this, the grader may manually review the results and give credit as needed. The idea was to keep things as simple as possible and only do additional work as needed.

The tests are also somewhat like unit tests in that each test case tests one small thing. Typically there were 20+ test cases per assignment. Often the individual test cases tested the same feature with different values to check completeness of implementation. The test cases often supplied "bad" values (e.g. null) to see if students handled erroneous input.

Directory Structure For Testing

Testing is performed in the test directory. This directory may be anywhere, but my convention is to make it a subdirectory of the assignment. The following is a listing of a typical test directory: 
    -rw------- 1 cs161 class  107 2010-05-10 14:03 grade.ini
    drwx------ 2 cs161 class 4096 2010-08-26 11:21 grades.PAx
    -rw------- 1 cs161 class 6788 2010-05-10 14:51 gradeThese
    drwx------ 2 cs161 class 4096 2010-05-10 14:54 input
    drwx------ 2 cs161 class 4096 2010-05-10 14:54 master
    drwx------ 2 cs161 class 4096 2010-05-15 13:27 output
    drwx------ 2 cs161 class 4096 2010-05-11 15:21 provided
    drwx------ 2 cs161 class 4096 2010-05-11 15:21 regrade
    -rw------- 1 cs161 class  290 2010-05-10 14:02 testCases.PAx
    drwx------ 2 cs161 class 4096 2010-05-15 13:27 work

Several directories/files have suffixes which correspond to the assignment tag. This was done so that we could grade the original assignment (e.g. PA5) and the resubmitted assignment (e.g. PA5-R) in the same directory. The only real difference was the test cases we ran.

Test subdirectories/files

Using The Testing Scripts

The following tools are used in the grading process. They are presented in the order they are normally used. Each script takes one or more command line arguments. The first argument is always the name of the assignment (e.g. PAx). This must correspond exactly to the name used to checkin in the assignment. To find the other parameters, execute the script with no parameters and it will produce a usage message.
  1. Generate a solution to the assignment and check it in using the TA's or instructors name. This will be used to generate the master output.
  2. If the student's submission does not include a makefile, create one appropriate to the language of the programs being graded and store it in the provided directory. The build is performed in the test directory by executing

    make -C work -s.

    The makefile I used for java is here .

  3. Customize the latePolicy script to take off points for late submissions. See details.
  4. Customize the stdFilter script. See details.
  5. Develop the various test cases (the contents of the input directory. Assign points to each test case and generate the testCases.PAx file. See details.
  6. Run cleanup See details.
  7. Run genMaster Review the directory to make sure they are what was expected. See details.
  8. Run gradeAll. See details.
  9. The next step is an optional manual step. We would eyeball all the grade files looking for grading errors. If an error is detected, the master code is fixed and the process is restarted from the beginning.
  10. Run genGrades to generate the final grade file for each student. See details.
  11. Run genStats to get an overview of the results of the grading. See details.
  12. Run releaseGrades to make the grades visible to the students. See details.

Hints on "fixing" problems - and manual grading

There are times when you may want to override something that the grading tools have done. Here are several things I ran into and how I handled them:

Back to index

Take the test tools for a ride

If you would like to try out the test tools, do the following.
  1. WARNING:Do not do this as a user such as cs161 or any other user that has a real Checkin directory.
  2. Save this tar file in your home directory. It must be your home directory because the Checkin directory is referenced by ~/Checkin.
  3. In your home directory, tar -xvf Checkin.tar This will create a Checkin directory and additional files/directories below it.
  4. cd Checkin
  5. mkdir tools.
  6. cd tools
  7. make -f ~fsieker/public_html/teaching/Makefile
    You now have all the tools. Make sure the tools directory is in your $PATH.
  8. cd ~/Checkin/test. Peruse the various files and directories you see there and refer to the documentation for details.
  9. genMaster PAx perfect. This produces the master output files in the master subdirectory.
  10. gradeAll PAx. This will grade the submissions found in the ~/Checkin/allLogins file. This is a superset of the actual submissions found in ~/Checkin/PAx to illustrate a student who did not submit anything.
  11. You may see the results of each student in the grades.PAx subdirectory. There is also a file called ERRORS there.
  12. genGrades PAx. This will generate the final grade files in the grades.PAx subdirectory. Note the files now contain a summary section at the top.
  13. genStats PAx. This will generate statistics about the grading in the file grades.PAx/PAx.stats.
  14. Experiment with creating an adjust.PAx file and regenerate the grades.
  15. Experiment by adding/changing test cases.
  16. Create a different version of the MyAssignX.java and use forceCheckin to add it to the Checkin directory.
  17. cleanup PAx, then gradeAll PAx ...
  18. Experiment with the gradeThese file.
  19. Remove the directory ~/Checkin to clean everything up. Reset your $PATH.

Back to index

Grade Scripts and Files - Details

The grading tools may be copied to the test directory or stored in a central location (I suggest ~/bin/tools). Your $PATH shell variable must include either . or the directory where the tools are stored. If you invoke any of the tools without parameters, they will print a usage message with an example.

Back to index

The adjust.PAx File/Directory

This optional file/directory exists in the test directory. Its purpose is to allow manual modifications to the results of the automated grading. It is separate file so that grading may be rerun as many times as necessary without losing the information stored here. The information is incorporated into the students grade file by the script genGrade.

The file adjust.PAx consists of multiple sections, one per student. There need only be sections for students whose grade requires adjustment.

If there are adjustments for every student (e.g. something which MUST be done manually), then the adjustments may be stored in individual files in the directory adjust.PAx. The name of the file is the login name. For example, in one assignment, the students were expected to include a README file containing answers to several questions. By creating the simple script genAdjust and including the line in the testCases.PAx file of the form: 

 adjust % 5 % ../genAdjust $assignName $login

a file was created for each student that contained the students README as well as a place for the grader to assign a grade. Here is the result for a sample user.

The format of a section/file is: 


  <BEGIN loginName1>
  adjust Score: -10 // Commented out unused code that used an undeclared variable called "shortestPath"  -Maggie
  <END>

  <BEGIN loginName2>
  adjust Score: -10 // Removed '\r' from the Cities toString method -Maggie
  <END>
All the info for a particular user is added to the grade file when it is generated. Lines containing the word Score: cause the total to be changed. All other lines are simply copied into the grade file. Multiple score adjustments for a single login are OK.

Back to index

The checkComments Script

This script was written by David Newman to find if there are comments for each method in the students submission. This was used as part of assignment grading in cs200.

Back to index

The cleanup Script

This script removes the grades.PAx directory. It is normally used before rerunning gradeAll script to make sure the results are fresh.

Back to index

The forceCheckin Script

This script may be used to place an assignment in the Checkin directory after the drop dead date of the assignment. For example, a student may have an official excuse (e.g. traveling for team sport), and this allows the TA to "submit" the assignment without it being marked as late.

Back to index

The genAdjust Script

This script is useful when manual grading is required for some portion of the assignment. For example, in a recent assignment, we provided a file PA2.QandA that contained several questions about the code the students wrote for the assignment. The students were expected to answer the questions in the file (simple text) and turn it in as part of the submission. This script was used to produce an adjustment file for each student. The graders then edit the file, add a score for each questiopn and perhaps some comments. This is then inserted into the students grade file. By providing a file per student, multiple graders could work simultaneously. In general, this script will be customized for each assignment.

A multi-step process was then used for grading. The example assumes that genAdjust is in the test directory.

  1. The genAdjust script was customized for this assignment. In this case cat P2.QandA was all that was needed
  2. A testCases.PAx file was produced containing the single line 
    
    QandA % 10 % ../genAdjust $testName $login %
    This has the effect of assignming 10 points to the questions. See testCases for more detail.
  3. run gradeAll to produce one file per student. The files may now be manually graded.
  4. Modify the testCases.PAx file to add the remaining test cases and modify the command line portion of QandA test to be echo ""
  5. Run genMaster. The master output for the QandA test case will be an empty file.
  6. Modify the testCases.PAx file and modify the command line portion of the QandA test to be echo "see attached". This will cause the automated grading to assign a score of 0 to this test. The points will be added back in via the adjustment file.
  7. If for some reasom the master needs to be updated, repeat steps 4 to 6.

Back to index

The gendoxy Script

Here is an example assignment generated with gendoxy. Most of the documentation is in the .h files as normal doxygen comments. I put the TODO in the .c files as this is what the students were expected to complete. The tools assumes the file ~/bin/tools/doyxgen.config exists. This is the default file generated with doxygen -g with the following changes: 

   > REPEAT_BRIEF           = NO
   > OPTIMIZE_OUTPUT_FOR_C  = YES
   > EXAMPLE_PATH           = ..
   > HTML_OUTPUT            = ../doc
   > SEARCHENGINE           = NO
   > GENERATE_LATEX         = NO
The script assumes a directory structure as create by newAssignment. All source code lives in the src directory. The description of the assignment (PAx.html) is in the PAx directory. In the html file links to the provided files are coded with the path src/fileName. Finally in one of the source files (I normally choose the one containing main()), the following lines are added: 

  /** @mainpage cs270 Fall 2012 Programming Assignment 1 - Bit Fields in C
   *  \htmlinclude "PA1.html"
   */
To generate the documentation, in the src directory execute gendoxy PAx The final documentation is accessed as http://.../doc/index.html

Back to index

The genGrade Script

This script generates the final grade file for a single student. The info for each test case is generated by the gradeAssignment script. This script produces a summary, assesses late penalties and includes information (if any) from the adjust.PAx file. The script is rarely used directly. Rather, it is invoked by the genGrades script.

Back to index

The genGrades Script

This script invokes genGrade. It is driven by a list of login names (one per line). It uses the file gradeThese.PAX or gradeThese or ~/Checkin/allLogins, whichever it finds first. The script extracts the scores for the individual test cases and generates an assignment total. It then regenerates the grade file with the total as the first line of the file. The script is also responsible for deducting late points and adding info from the adjust.PAx file. As part of the run, the script produces the file grades.PAx/SUMMARY. This file contains the login and assignment grade for each student. There are tools for uploading this information to RamCT.

Back to index

The genMaster Script

This script is used to generate master output against which all other submissions are compared. It puts empty results in the master directory. It then uses gradeAssignment to test the submission. The output then becomes the master for all subsequent runs. If a students submission is used to generate the master, that student should be regraded, since in the run of this program produces a grade file with all 0's.

The script is invoked as follows

     genMaster assignName login

The parameter assignName is the name used in the checkin of the assignment. It is a subdirectory of the directory ~/Checkin.

The parameter login is the login of the person whose code will be used to generate the master output. This code goes through exactly the same process as any student submission.

Back to index

The genStats Script

This script is used to generate statistics about the overall assignment. It looks at each grade file and produces statistics for each test case. This is useful to understand what test cases where hard (or where the master might be wrong). The file is test/grades.PAx/PAx.stats. A sample file is here.

Back to index

The gradeAll Script

This script invokes gradeAssignment. It is driven by a list of login names (one per line). It uses the file gradeThese.PAX or gradeThese or ~/Checkin/allLogins, whichever it finds first. As it works it echoes the logins to the screen so the tester can monitor progress. If a test hangs (typically infinite loop) just kill the process(s). It will likely take multiple ctrl-C's to stop everything. Then create a file gradeThese containing the logins of the remaining submissions and restart the script. As each submission is graded, it produces a file grades.PAx/login.grade.

The script also produces the file grades.PAx/ERRORS. This contains the list of all logins for which there was no submission or the compile failed. It provides a quick check on the overall status of the class.

Each grade file contains multiple sections corresponding to each test case. Each section contains the name of the test case, the contents of the test case file, the output produced by the submitted program, the master output, and the grade for this test. The output of the tested program is limited to the maximum of the length of the master output or 60 lines. This was to prevent huge stack traces (infinite recursion) from filling up the file. An example grade file with the side-by-side output (the default) is here. An example grade file with linear output (configurable) is here. An example grade file containing a runtime error is here.

Back to index

The gradeAssignment Script

This script is only used directly for debugging purposes. It is normally executed by the gradeAll script. It is the "meat" of the process, doing all the work for grading the assignment for a single student. The script contains many embedded tools that do portions of the work.

The script is invoked as follows

     gradeAssignment assignName login

The parameter assignName is the name used in the checkin of the assignment. It is a subdirectory of the directory ~/Checkin.

The parameter login is the login of the person whose code will be graded. The gradeAll simply invokes this script once for each user to be graded.

As a program executes, stdout and stderr are captured in $outputFile. See testCases.PAx for details.

The flow of the script is:

  1. extract the command line parameters and set default values
  2. source in the grade.ini file if it can be found
  3. remove, then create work directory
  4. initialize the grade file for the student (grades.PAx/login.grade)
  5. run embedded script getSubmission()
    1. get the file with the latest date
    2. check for prefix LATE_ and rename as needed (see embedded script chekLate())
    3. unpack it, if necessary (see embedded script extractFiles())
    4. flatten any directory structure that might exist (see embedded script flatten())
    5. remove package statements that might exist (see embedded script fixPackage())
    6. copy files to Moss, if option turned on
    7. copy files from provided to work (see embedded script addProvided())
  6. build the assignment using make
  7. run embedded script checkMakeAndRunTests()
    1. if make was successful, run run embedded script gradeSubmission(). Otherwise, assign grade of 0
    2. Process the lines (test cases) of the testCases.PAx file one at a time. Determine the name of the test and other parameters and actually run the test. Output is captured in a temporary file.
    3. pass the output thru the script stdFilter if it exists
    4. diff the output and master. The result is appended to the grade file in a side-by-side format so the user can easily compare their results to the master. When viewing results, maximize the window to accommodate very long lines.
    5. If there are no differences, assign full credit. Otherwise assign 0.
    6. If the score was 0, optionally regrade the results (if a regrade is specified in the test case) by
      • See this for details on how this is done.

Back to index

The grade.ini File

The file grade.ini is an optional file that overrides some of the behavior of the grading script gradeAssignment. That script first looks in the current directory and then in the home directory for the file. If this file is present, it is sourced in. An example of the file is here.

Back to index

The gradeThese File

This optional file in the test directory is used to restrict the number of users processed by gradeAll and genGrades. The file consists of logins, one per line. It is commonly used when a run of gradeAll "breaks" because a submission enters an infinite loop. The grader kills the grading process, and produces a file of the remaining students who have not yet been graded, then restarts the process. If the file is not present, the list of students is taken from ~/Checkin/allLogins. It is also useful during resubmissions to grade only those students who actually resubmit. The file can be produced by a simple ls of the resubmit directory.

Back to index

The latePolicy Script

This script is designed to be modified for each course. It is responsible for determining the number of points to deduct for a late assignment. It is passed two parameters, the number of hours the assignment is late and the total number of points in the assignment. The sample implementation deducts 10 percent per day. An even simpler one might simply return a constant, disregarding the actual number of days late. The script may be implemented in any scripting language you want.

The script is used by the genGrades script and splits out the late policy from the rest of the work for convenient modification.

The reason that hours late is passed was to provide some "wiggle" room in the calculation of late points. For example, if the assignment is 1 hour late, perhaps only a small penalty (1 point) might be deducted. Anything beyond this might be calculated as 10 percent per day.

Back to index

The login.Grade File

As gradeAssignment executes, its output is collected and placed in the file grades.PAx/login.grade.

Each grade file contains multiple sections corresponding to each test case. Each section contains the name of the test case, the contents of the test case file, the output produced by the submitted program, the master output, and the grade for this test. The output of the tested program is limited to the maximum of the length of the master output or 60 lines. This was to prevent huge stack traces (infinite recursion) from filling up the file. An example grade file with the side-by-side output (the default) is here. An example grade file with linear output (configurable) is here. An example grade file containing a runtime error is here.

Back to index

The regrade Scripts

These optional executable script(s) provide a means of doing additional grading beyond what is done by the original diff. The script can function in one of two ways:
  • additional filtering - in this case the script refilters the output and that result is diff'ed against the results obtained by performing the same filtering on the master output.
  • direct grading - in this case the script does its work and echoes the score to stdout. The format of the line is:

    testName Score: value / max

The script is stored in the regrade directory and is executed as: 


    regrade/scriptName testName regradePoints originalFileName cookedFileName
Consider a simple example from a beginning Java class. The students were instructed to write a program that prompts for several inputs, then calculates the income tax owed. The following is a sample student output and the master output from the .grade file: 

   Your output                       Master output
   Number of Exceptions: 2	     |	Number of Exemptions: 2
   Gross Salary: 10000		        Gross Salary: 10000
   Interest Income: 500		        Interest Income: 500
   Capital Gains:250	             |	Capital Gains: 250
   Charitable Contributions: 30	        Charitable Contributions: 30
   Total Income: $10750.00		Total Income: $10750.00
   Adjusted Income: $7450.00	        Adjusted Income: $7450.00
   Total Tax: 0.00		     |	Total Tax: $0.00
Notice the "minor" errors. The diff awards a score of 0 as there are differences. However, with a simple regrade filter, one can easily give back full/partial credit. The most important result was the computed tax. Thus, the following script was used. It simply grabbed the last line and isolated the number at the end (the tax). It is up to the grader to decide how much the final value was worth. 

    #!/bin/sh
    # Fritz sieker
    #set -x

    testName=$1
    points=$2
    testFile=$3
    cookedFile=$4

    # get the last line (should contain "total Tax: XXX.YY
    # remove everthing up to :
    # remove everything up to $
    # remove leading/trailing whitespace
    # only thing remaining should be computed number

    tail -n1 $testFile | sed -e 's/^.*\://g; s/^.*\$//g; s/^[ \t]*//; s/[ \t]*$//' > $cookedFile
In another case, one might sort the output and give partial/full credit if the sorted output matches.

One of our assignments involved a brute force traveling salesman problem. If there are N cities, there are 2N equivalent circuits (start at any city, go either clockwise or counter-clockwise). For this case we wrote a script that preserved their solution, but started at a particular city and traversed in a particular direction.

If one is using the additional filtering method, the cooked results must be written to the file specified in the fourth parameter. Here is a another example of a regrade script. The script to reorder the TSP data is here. For that assignment the original output is here while the "cooked" output (produced by the regrade script) is here.

A handy source of sed scripts is here.

Here is a another example regrade script. The script splits the file into words (one per line) gets rid of all blanks and sorts the output. In this case, the grader might choose to give partial credit on a match. 

   #!/bin/sh
   # Fritz Sieker
   #set -x

   testName=$1
   points=$2
   testFile=$3
   cookedFile=$4

   sed -s 's/ /\n/g' $testFile | sed -s 's/ //g' | sort > $cookedFile

Back to index

The noLoop Script

This script provided by Ryan Stern aids in running programs that may have infinite loops. This is common in lower lever courses, particularly assignments involving recursion. The script is used by prefacing the command line to run the program by noLoop. For example 

    noLoop java P2 < test
If the program P2 executes more that 10 seconds (configurable in script), the program is killed and an error printed. This is usefull in the automated testing so that one students problems do not prevent completion of testing all of the students programs. The runner script has this and other capabilities.

Back to index

The releaseGrades Script

This script simply moves the individual grade files from the directory test/grades.PAx to the directory ~/Checkin/PAx. Once this has been done, the grades become visible to the students using the ~/bin/grade program. This script may be used multiple times if regrading is performed after initial grades are released.

Back to index

The runner Script

This script provided by Jack Applin aids in grading programs that do console I/O. Suppose you are grading a simple program that writes promps and collects input from the user. It then does something and typically prints some results. This can be tested by invoking the program and and redirecting input from a file. Here is and example of that for a simple cs160 program: 

    java P2 < test1

    Number of Exemptions: Gross Salary: Interest Income: Capital Gains: Charitable Contributions: Total Income: $10750.00
    Adjusted Income: $7450.00
    Total Tax: $0.00
If the runner is used, the output is as follows: 

    runner java P2 < test1

    Number of Exemptions: 2
    Gross Salary: 10000
    Interest Income: 500
    Capital Gains: 250
    Charitable Contributions: 300
    Total Income: $10750.00
    Adjusted Income: $7450.00
    Total Tax: $0.00

runner has other capabilites including limiting the total CPU time (detect infinite loops) of the tested program, limiting the output, ... Talk to Jack for more info. or read the script.

Back to index

The stdFilter Script

This optional script is used to to process the output of each test run to remove common things that make diff "break". For example, if the case of the output is not important, one could use tr to convert everything to a single case. Transformations that go here will not need to be processed by any optional regrade tools. The example script removes all trailing white space on lines and removes all null characters. Filters are expected to read from stdin and write to stdout. No filtering is performed if the command which stdFilter results in an error.

A handy source of sed scripts is here.

Back to index

The sumCol.pl Script

This script simply sums of a column of values in a test file. It is used by genGrade to sum up individual test case scores (possibly real numbers).

Back to index

The testCases.PAx File

This is a required file of the test cases to run. The file is line oriented and each line defines one test case. Fields in a line are separated by the character '%'. The hash character ('#') denotes the beginning of a to-end-of-line comment. Blank lines are ignored.

Each line of this file contains seven parameters. Parameters may be blank, and parameters after the last actual parameter may be left off. The parameters are:

  1. testName - this names the test case.
  2. points - the number of points this test case is worth.
  3. cmd line - the command line used to execute the test case. All test cases execute in the work directory. There may be IO redirection and pipes in the command line as needed. To make this field easier to generate, there are several macros defined that one may use. They are:
    • $testName - the name of this test case.
    • $points - the number of points for this test case.
    • $inputDir - the path to the input directory.
    • $inputFile - the value $inputDir/$testName
    • $outputDir - the path to the output directory. ALL output of the program must be stored in this directory, because this directory becomes the master directory upon running genMaster..
    • $outputFile - the value $outputDir/$testName. Anything written to stdout or stderr is captured in this file.
    • $workDir - the path to the work directory.
    • $login - the login of the user whose assignment is being graded.

    The command line may be blank, in which case nothing is executed. This is useful in the case where a previous test case produces multiple files, each of which is to be graded individually. The first test case executes the program and grades one of the result files. Subsequent test cases leave the command line blank, and simply specify the remaining files, one per test case.

    Programs prompting for console input: If you have a program prompting for input, consider wrapping the command in the runner script.

    Programs using only command line parameters: While the tools allow you run a program that takes all of its input from command line parameters, the output will not show those parameters unless the program explicity print them. Thus, it is hard for students to understand their errors. However, this is easily fixed without modifying the tested program. Simply put the actual command line in a file. Then in the testCases file the line for that test case becomes. 

    
  testName % 4 % . $inputFile % $inputFile % $outputFile
    where the file testName contains the actual command line. This file will be printed in the resulting output, so the students can see how the program was run.

  4. input file(s) - any file(s) listed here will be copied into the students grade file. If the parameter is blank, no files are copied. My convention is to name the input file the same as the test name. This is not required, but makes developing the test case file easier, because the name is directly available with the macro $inputFile

    An interesting use of this field is to set its value to $outputFile and leave the output field blank. Suppose the command line specified executed a script which checks for comments in the students code and generates a grade based on the comments. Since there is really no master output to diff against, the script simply writes its results (including a score) to stdout. The output is captured and copied into the grade file.

  5. output file - the file listed here will be passed thru stdFilter (if it exists), and then diff'ed against files with the same name in the master directory. If no file is specified, no diff'ing occurs.
  6. regrade script - the name of the regrade script for this test case
  7. regrade points - the points for the regrade (defaults to points).

Here is the file used to test the sample assignment referenced in another section. Note that three of the test cases use the regrade scripts nullToEmptyArray, and that three of the tests have multiple input files.

Back to index