Task properties

There are many task properties. Most of them have default values, and when you make grade.properties file only a few properties need explicit values. The basic syntax of that file involves writing lines in the following format: <property>=<value>. Some lines can be comments by adding # in the start. We will describe the properties and how they should be added in the grade.properties file.

Time limits

1. time - execution time of the program on each test case. This is the standard time limit for the tasks. Default value: 1 second. Example: time=1.5 for 1.5 seconds time limit

2. io_time - Time limit of the operations not counted in the regular execution time. Sometimes in interactive tasks, the time for reading the input and writing the output is removed from the execution time. In this case, the normal time limit applies only to the execution time of the contestant's program and the other time (for the other operations) also needs a time limit. For more details on setting up a task in such a way, see pseudo interactive). Default value: 0 seconds. Example: io_time=1.5 for 1.5 seconds limit (the name comes from input/output time).

3. compile_time - compile time limit. This is the standard time limit for the time needed to compile the programs. By default, this limit is used for all allowed languages! Default value: 10 seconds. Example: compile_time=1.5 for 1.5 seconds compile time limit

4. java_compile_time - Java compile time limit. Compilation time limit only for Java programs. Default value: 300 seconds. Example: java_compile_time=1.5 for 1.5 seconds java compile time limit

Memory limits

1. memory - execution memory of the program on each test case. This is the standard memory limit for the tasks. Default value: 256 MB (megabytes). Example: memory=1024 for 1 GB (gigabyte = 1024 MB)

2. compile_memory - compile memory limit. This is the standard memory limit for the memory needed to compile the programs. By default, this limit is used for all allowed languages! Default value: 512 MB. Example: compile_memory=1024 for 1 GB compile memory limit

3. java_compile_memory - Java compile memory limit. Compilation memory limit only for Java programs. Default value: 1536 MB. Example: java_compile_memory=1024 for 1 GB java compile memory limit

Test cases names

The system expects the tests to be numbered consecutively, starting from 1. Each test case must have input file and output files. Only in case there are no output files, then during testing the output files are considered to be empty files. The system has a complicated algorithm to find the test cases and recognize their numbers. This algorithm works in most cases. There are also some properties that help the system find and number the tests in case of failure.

1. patterns - gives multiple names to use for the paths of the tests. It is expected that the input and output file for each test still have approximately the same name except for marking what is the type of the file (extensions .in and .out, input and output in the names, etc.). The test cases have to be in folder tests or test (only when this property is used). Additionally, the patterns are considered to describe subtasks (only property weights needs to be added, see subsection 1 from subtasks and groups) in the same order. To set up the tests to be evaluated individually, the scoring property (see subsection 1 from scoring) needs to be used. Default value: none - this is an extra property. Example: patterns=small,medium,large for recognizing test cases with the following names: inp.small.01, inp.medium.01, inp.large.01, inp.large.02, out.small.01, out.medium.01, out.large.02, out.large.02

2. input and output - gives patterns for the input and output files paths of the tests Regular expression is given for the paths of the input files and another one for the output files. The test cases are numbered according to the lexicographical order of the paths. It is important that the regular expressions match exactly the paths (relative to the ZIP archive) of the tests. Default value: none - this is an extra property. Example: input=t/first.* output=t/second.* (you can consider these two to be in separate lines) for recognizing test cases in folder t with the following names: first01, first05, first11, second01, second05, second11 (the first 3 files will be considered input files for the first 3 tests and the latter 3 files will be considered the corresponding output files; the tests will be numbered from 1 to 3)

Subtasks and groups

The two are closely connected. Subtasks are used in IOI for tasks - tests are grouped into groups, called subtasks, each subtask worth a certain number of points if all tests pass. If groups are used, they usually contain the same number of tests and are worth the same number of points. For more details on scoring, refer to scoring section. As described in the previous section, the system numbers the tests consecutively, starting from 1. This also applies to subtasks. It is necessary that the first set of tests belongs to the first subtask, the next set - to the second, and so on.

1. groups - describes the intervals with tests for the subtasks. Default value: none - this is an extra property. Example: groups=1-5,6-6,7-25,26-100 for a task with 4 subtasks, where the first subtask covers the tests with numbers from 1 to 5, the second covers only test 6, the third covers tests 7 to 25 and the fourth covers tests 26 to 100.

2. weights - describes the ratio for the points of the subtasks. Default value: none - this is an extra property and expected if groups is present! If no value is provided, then the subtasks are considered to have equal 'weights'. Example: weights=10,20,30,40 for a task with 4 subtasks worth 10, 20, 30, and 40 points if the task is worth 100 points (if, for example, the task was worth 50 points, then the points for the subtasks would be 5, 10, 15, and 20).

3. dependencies - describes the dependencies between the subtasks. Usually, all subtasks are evaluated in order from the first to the last. The evaluation of some subtasks may require previous subtasks to be evaluated and passed successfully. If, for a subtask, one of the required subtasks does not pass, then that subtask is not evaluated and its status is skipped. Default value: none - this is an extra property and expected if groups is present! Example: dependencies=,,1;2,3 for a task with 4 subtasks where the first and second subtasks have no required subtasks, the third subtask requires subtasks 1 and 2 to be passed, and the fourth subtask requires only subtask 3 to be passed.

If the property groups is present and weights is absent, the system considers the task to have groups, and the points for the groups will be equal.

Points

points - the number of points the task is worth. Default value: 100 points. Example: points=1 for making the task worth 1 point (useful for ICPC contests).

Scoring

This section describes properties for indicating how the score of a submission is calculated and how the score for a task is calculated when there are multiple submissions. We consider each test to have a score which is a decimal number from 0 to 1. For simplicity, we will use the term "groups" when the task has subtasks or groups. In that case, each group will have a weight (given by the property weights). If the task does not have groups, then each test will have a weight, which is the same for all tests. Finally, we consider the task to be worth p points (given by the property points).

1. scoring - describes how the score of a submission is calculated. There are only three possible values:

   • sum: if s is the sum of (score of a test) x (weight of a test) for all tests, then the score of a submissions is s x p; all tests are evaluated individually.

   • min: the score of each group is the minimum of the scores of each test; if s is the sum of (score of a group) x (weight of that group) for all groups, then the score of a submissions is s x p; if a subtask is evaluated, then all tests in it are evaluated.

   • min_fast: the score of a submissions is calculated in the same way, and the only difference is that if a test fails in a subtask (equivalently it has a score of 0), then the other tests in that subtask are with status skipped and not evaluated.

2. scoring_type - describes how the score for a task is calculated. There are only two possible values:

   • best: the score for a task is the highest score among all submissions;

   • aggregated: here, it is expected that the task has groups; for each group, the maximal score of that group is calculated among all submissions, then the score for a task is the sum of (maximal score of a group) x (weight of that group) for all groups.

The following table shows the default values for the aforementioned properties depending on the type of task.

Type of task	Default value of scoring	Default value of scoring_type
Individual tests	sum	best
Groups	min_fast	best
Subtasks	min_fast	aggregated

Other properties

1. feedback - specifies the feedback, showed to the contestants, for the submissions. Usually, the contestant can see some feedback for the submissions - the verdicts, sometimes scores of all the tests, time and memory of the execution on the tests. This can be restricted by setting the feedback property and adding the numbers of the tests or subtasks/groups that are included in the feedback. Contestants won't see any feedback for those tests/subtasks/groups. Default value: none - this is an extra property (by default, all tests are included in the feedback). Examples:

   • feedback=1,3,5 for providing feedback only for tests with numbers 1, 3 and 5 or subtasks/groups with numbers 1, 3 and 5 (depending on the type of task).

   • feedback=42: for providing no feedback for submissions if a task has fewer than 42 tests.

2. sample - making some tests to be samples. This property is used only for tasks with individual tests. When tests become samples, they give no points and are displayed appropriately. Default value: none - this is an extra property. Example: sample=1,2 for making tests 1 and 2 to be sample tests.

3. precision - decimal precision for the scores of a task. This property is also used to tell the system that a task can have partial scores for tests - scores strictly between 0 and 1 (if there will be no decimal scores, it is enough to set the precision to 0). Default value: -1, which indicates that the tests will have a score of either 0 or 1. Example: precision=2 for making the scores for a task rounded to the second digit.

4. extensions - allowed extensions for the submitted files (programs in most cases). Supported extensions are .cpp, .c, .h (for special type of tasks, see limiting global memory), .java, .go, .cs, .zip (for output-only tasks, see multiple files), .txt (also for output-only tasks), and .pdf (for task translation purposes). Other extensions shouldn't be used.

   Default value: cpp for C++ programs. Example: extensions=txt,java to allow submissions that are Java programs and also .txt files.

5. processes - allowed number of processes that a program can have. Usually, this is used in multiple-run tasks where the contestant's program is run several times and the property is the number of runs that should be done. For more details on setting up a task in such a way, see multiple-run. Default value: 1 (only one process is allowed). Example: processes=2 to allow 2 processes for the contestant's program (and if this is a correctly setup 2-run task, the contestant won't be able to create new processes in the program because they will already be exhausted by the system).

6. open_files - allowed number of open files per process. This property sets the limit on the number of files that can be opened by each process and more precisely the number of opened file descriptors (which counts standard input, output and error). There is also a system limit. Default value: 64. Example: open_files=256 to allow up to 256 open files per process.

7. rejudge - number of times submissions should be rejudged on a test. When a submission exceeds the time limit for a test, the system can rejudge the submission on that test several times until the time is small enough. This is only done if the time has not exceeded the extra time limit - a more generous time limit. Default value: 1, which indicates that submissions are run at most once on every test and there is no rejudging. Example: rejudge=3 for making at most two additional executions of a submissions on a test that exceeds the time limit but it still within the extra time limit.

8. blacklist - blacklist strings in the submitted files. This property allows restricting programs from using some strings. If a submitted file contains any of those words, a compilation error is given describing the specific reason. Default value: empty list. Example: blacklist=pragma,Pragma to disallow pragma and Pragma strings in the submissions.

9. info - adding some information for the task in the grade.properties file. Default value: empty string. Example: info=dp,segment trees.