Quick Reference
This page is the command and workflow reference you are most likely to revisit while developing and evaluating your project. Run all commands from the repository root unless noted otherwise.
Project selection and repository creation
Project Selection (before Checkpoint 1): Follow Getting Started to form your team (or work individually with approval), read the problem definitions, rank three preferences, and submit project.json on Gradescope. There is no repository to create in this stage.
Beginning of Checkpoint 1 (after assignments): Follow Checkpoint 1: Foundation to create or join the team repository through Classroom 50, clone it, transfer project.json, and fill in assigned_problem.
One-time setup after your problem is assigned
Once your team has received its assigned problem, confirm that
project.json contains the correct assigned_problem.
Then run:
python tools/setup_project.py
This validates project.json and installs any external benchmark data configured for your assigned problem. It is safe to run the command again later; already-installed benchmark data is reused unless you explicitly force a reinstall.
To verify that course-owned infrastructure has not been accidentally changed:
python tools/check_course_files.py
Run one algorithm on one instance
The common solver interface is:
python src/solve.py INPUT_FILE --problem PROBLEM --algorithm ALGORITHM
Problem identifiers may be written using the short forms:
mvc tsp mgc lp mc
Algorithms are:
exhaustive improved heuristic1 heuristic2
For example:
python src/solve.py tests/public/minimum_vertex_cover/instances/cycle5.txt \
--problem mvc \
--algorithm heuristic1 \
--seed 17
--seed is available for randomized algorithms. Use the same seed when you want a run to be reproducible.
Run the provided public tests against your code
Checkpoint 1 / Foundation
Run the public Foundation tests for your assigned problem:
python tools/run_cp1_tests.py
Useful options include:
python tools/run_cp1_tests.py --algorithm exhaustive
python tools/run_cp1_tests.py --test star
python tools/run_cp1_tests.py --jobs 4
--test TEXT runs only public tests whose names contain TEXT.
Checkpoint 2 / Improved Algorithms
Run the public tests for the improved exact algorithm, polynomial-time bound, and first heuristic:
python tools/run_cp2_tests.py
To run independent public test cases concurrently:
python tools/run_cp2_tests.py --jobs 4
Are the test runners parallel?
By default, no: --jobs defaults to 1.
When you supply --jobs N, independent solver/bound test cases are run concurrently. Each test executes student code in a separate worker process. Interface/verifier setup and the major Checkpoint 2 test groups are still coordinated sequentially.
Run the benchmark/experiment program
First see which suites exist for your assigned problem:
python tools/run_experiments.py --list-suites
Then run one suite, for example:
python tools/run_experiments.py --suite readiness
python tools/run_experiments.py --suite exact_frontier
python tools/run_experiments.py --suite quality_known
python tools/run_experiments.py --suite heuristic_scale
python tools/run_experiments.py --suite structure
python tools/run_experiments.py --suite challenge_open
You can run every suite in the manifest with:
python tools/run_experiments.py --all
Be careful: --all can include intentionally expensive experiments.
Results are written under experiments/results/, with one stable JSON/CSV
pair per suite. For example, --suite exact_frontier writes:
experiments/results/exact_frontier.json
experiments/results/exact_frontier.csv
Timestamped copies are retained in the same directory. Running another suite does not overwrite the results from earlier suites.
Are benchmark runs parallel?
No. The benchmark runner executes one solver/bound worker at a time. Each individual run is isolated in a child process, but the suite itself is sequential. This makes timing comparisons easier to interpret and avoids multiple benchmark jobs competing for the same CPU at once.
Where do benchmark defaults come from?
For each problem, the course benchmark manifest is:
benchmarks/<problem>/manifest.json
The manifest controls course experiment settings such as:
- which instances belong to each suite;
- which algorithms run on each instance;
- timeouts;
- fixed random seeds;
- repeats;
- known optimum / best-known / published-bound metadata; and
- structural metadata used in the final analysis.
You normally read but do not edit the course manifest.
You can override benchmark timeouts for a local exploratory run:
python tools/run_experiments.py --suite exact_frontier --timeout 30
python tools/run_experiments.py --suite exact_frontier --timeout-scale 0.5
The benchmark timeout is an external safety limit enforced by the runner. It is not an internal heuristic stopping rule.
Where do heuristic/anytime parameters go?
There is currently no project-wide --attempts, --iterations, or --anytime command-line option passed through the benchmark runner.
Algorithm-specific choices such as number of randomized restarts, local-search iterations, or an internal anytime budget belong in your student heuristic implementation, for example:
src/student/problems/<problem>/heuristic1.py
src/student/problems/<problem>/heuristic2.py
If your heuristic uses a tunable budget, make that choice explicit in your code and report it in the returned statistics dictionary so that your experiments are reproducible. The course runner supplies the fixed random seed; your algorithm decides how to use its computation budget.
Do not confuse an internal heuristic budget with the manifest timeout: the timeout terminates a run that takes too long; it is not intended to be the algorithm’s search strategy.
Can I run only some benchmarks?
The standard runner selects benchmarks by suite:
python tools/run_experiments.py --suite quality_known
There is not currently a command-line option that selects arbitrary individual instance IDs within a suite.
For a single benchmark instance, run the solver directly with src/solve.py.
For an instructor-designed or exploratory subset, the runner can use an alternate manifest:
python tools/run_experiments.py --manifest path/to/my_manifest.json --suite my_suite
This is mainly useful for custom experiments; the required project experiments should use the course-provided manifest.
9. A useful workflow while developing
A typical cycle is:
edit algorithm
↓
run one small instance with src/solve.py
↓
run the relevant public test suite
↓
run a small benchmark suite such as readiness or quality_known
↓
inspect the corresponding files under experiments/results/
↓
make one algorithmic change and repeat
For the final experimental analysis, use the course-provided fixed seeds and benchmark suites so comparisons remain reproducible.