Quick Reference

This page is the command and workflow reference you are most likely to revisit while developing and evaluating your project. Run all commands from the repository root (the directory containing project.json) unless noted otherwise.

Develop and test incrementally. The workflow below is not the same as the formal checkpoint sequence. Run small examples and correctness tests throughout development, rather than waiting until an entire checkpoint is implemented.

Project selection and repository creation

Project Selection (before Checkpoint 1): Follow Getting Started to form your team (or work individually with approval), read the problem definitions, rank three preferences, and submit project.json on Gradescope. You do not create the team repository at this stage.

Beginning of Checkpoint 1 (after assignments): Follow Checkpoint 1: Foundation to create or join your team’s repository through Classroom 50, clone it, transfer your team’s project.json, and fill in assigned_problem with the identifier announced by the instructor. Do not substitute a clone of the general starter repository for your assigned team repository.

One-time setup after your problem is assigned

Prerequisites: Python 3.10 or newer, Git, and an Internet connection for downloading external benchmarks. These commands use python3; if your computer names the same Python installation python, use that instead. Run them from the root of your team repository.

Once your team repository exists, confirm that project.json contains your team’s name and member names, its three project preferences, and the correct, full assigned_problem identifier (for example, minimum_vertex_cover, not the CLI abbreviation mvc). Preserve your original team information and preferences.

Check your Python version, then run the three setup commands in order:

python3 --version
python3 tools/check_setup.py --phase assigned
python3 tools/setup_project.py
python3 tools/check_course_files.py
  • check_setup.py --phase assigned validates the team configuration, repository structure, and course-owned files. Look for Project setup check: PASSED.
  • tools/setup_project.py is the setup script; there is no separate top-level setup.sh. It downloads any external benchmark collections required for your assigned problem. Look for Benchmark setup complete. External instances available/configured: .... The downloads may be large.
  • check_course_files.py verifies the protected course infrastructure. Look for Course infrastructure check: PASSED (version 17 in the current release).

The downloaded benchmarks are stored under benchmarks/<problem>/external/ and are ignored by Git: do not commit them. You can rerun python3 tools/setup_project.py safely; already-installed instances are reused. Use python3 tools/setup_project.py --force only when you intentionally need to redownload them.

If setup fails, first inspect the error message and verify project.json, your current directory, and Internet access. If a benchmark download fails, report the failing URL and error to the instructor; do not edit the course-owned installer or manifest to work around it.

Note: A successful setup does not mean the supplied student algorithms are implemented. Public solver/verifier tests are expected to fail on the untouched starter until you write the required code.

Run one algorithm on one instance

The common solver interface is:

python3 src/solve.py INPUT_FILE --problem PROBLEM --algorithm ALGORITHM

Problem identifiers may be written using the short forms:

mvc   tsp   mgc   lp   mc

Algorithms are:

exhaustive   improved   heuristic1   heuristic2

For example:

python3 src/solve.py tests/public/minimum_vertex_cover/instances/cycle5.txt \
    --problem mvc \
    --algorithm heuristic1 \
    --seed 17

--seed is available for randomized algorithms. Use the same seed when you want a run to be reproducible.

Run the provided public tests against your code

Checkpoint 1 / Foundation

Run the public Foundation tests for your assigned problem:

python3 tools/run_cp1_tests.py

Useful options include:

python3 tools/run_cp1_tests.py --algorithm exhaustive
python3 tools/run_cp1_tests.py --test star
python3 tools/run_cp1_tests.py --jobs 4

--test TEXT runs only public tests whose names contain TEXT.

Checkpoint 2 / Improved Algorithms

Run the public tests for the improved exact algorithm, polynomial-time bound, and first heuristic:

python3 tools/run_cp2_tests.py

To run independent public test cases concurrently:

python3 tools/run_cp2_tests.py --jobs 4

Are the test runners parallel?

By default, no: --jobs defaults to 1.

When you supply --jobs N, independent solver/bound test cases are run concurrently. Each test executes student code in a separate worker process. Interface/verifier setup and the major Checkpoint 2 test groups are still coordinated sequentially.

Run the benchmark/experiment program

First see which suites exist for your assigned problem:

python3 tools/run_experiments.py --list-suites

Then run one suite, for example:

python3 tools/run_experiments.py --suite readiness
python3 tools/run_experiments.py --suite exact_frontier
python3 tools/run_experiments.py --suite quality_known
python3 tools/run_experiments.py --suite heuristic_scale
python3 tools/run_experiments.py --suite structure
python3 tools/run_experiments.py --suite challenge_open

You can run every suite in the manifest with:

python3 tools/run_experiments.py --all

Be careful: --all can include intentionally expensive experiments.

Results are written under experiments/results/, with one stable JSON/CSV pair per suite. For example, --suite exact_frontier writes:

experiments/results/exact_frontier.json
experiments/results/exact_frontier.csv

Timestamped copies are retained in the same directory. Running another suite does not overwrite the results from earlier suites.

Are benchmark runs parallel?

No. The benchmark runner executes one solver/bound worker at a time. Each individual run is isolated in a child process, but the suite itself is sequential. This makes timing comparisons easier to interpret and avoids multiple benchmark jobs competing for the same CPU at once.

Where do benchmark defaults come from?

For each problem, the course benchmark manifest is:

benchmarks/<problem>/manifest.json

The manifest controls course experiment settings such as:

  • which instances belong to each suite;
  • which algorithms run on each instance;
  • timeouts;
  • fixed random seeds;
  • repeats;
  • known optimum / best-known / published-bound metadata; and
  • structural metadata used in the final analysis.

You normally read but do not edit the course manifest.

You can override benchmark timeouts for a local exploratory run:

python3 tools/run_experiments.py --suite exact_frontier --timeout 30
python3 tools/run_experiments.py --suite exact_frontier --timeout-scale 0.5

The benchmark timeout is an external safety limit enforced by the runner. It is not an internal heuristic stopping rule.

Where do heuristic/anytime parameters go?

There is currently no project-wide --attempts, --iterations, or --anytime command-line option passed through the benchmark runner.

Algorithm-specific choices such as number of randomized restarts, local-search iterations, or an internal anytime budget belong in your student heuristic implementation, for example:

src/student/problems/<problem>/heuristic1.py
src/student/problems/<problem>/heuristic2.py

If your heuristic uses a tunable budget, make that choice explicit in your code and report it in the returned statistics dictionary so that your experiments are reproducible. The course runner supplies the fixed random seed; your algorithm decides how to use its computation budget.

Do not confuse an internal heuristic budget with the manifest timeout: the timeout terminates a run that takes too long; it is not intended to be the algorithm’s search strategy.

Can I run only some benchmarks?

The standard runner selects benchmarks by suite:

python3 tools/run_experiments.py --suite quality_known

There is not currently a command-line option that selects arbitrary individual instance IDs within a suite.

For a single benchmark instance, run the solver directly with src/solve.py.

For an instructor-designed or exploratory subset, the runner can use an alternate manifest:

python3 tools/run_experiments.py --manifest path/to/my_manifest.json --suite my_suite

This is mainly useful for custom experiments; the required project experiments should use the course-provided manifest.

A useful workflow while developing

Use this cycle as you implement and improve your algorithms. The commands below use Minimum Vertex Cover as an example; substitute the appropriate instance, problem identifier, and algorithm for your assigned problem. These are development activities, not additional checkpoint submissions.

Step 1 — Make a small change to your algorithm

Edit the appropriate student-owned file, for example:

src/student/problems/minimum_vertex_cover/exhaustive.py

Keep course-owned files under src/course/ and tools/ unchanged. Begin with a small case whose answer you can check independently.

Step 2 — Run one small instance

From the repository root, test the exhaustive solver on a five-vertex cycle:

python3 src/solve.py \
  tests/public/minimum_vertex_cover/instances/cycle5.txt \
  --problem mvc \
  --algorithm exhaustive

Inspect the reported solution and statistics. An untouched starter will fail here because its student solver is not implemented yet. After implementing your code, use additional small instances and randomized seeds as appropriate.

Step 3 — Run the public correctness tests

For Checkpoint 1 / Foundation work:

python3 tools/run_cp1_tests.py

For Checkpoint 2 / Improved Algorithms work:

python3 tools/run_cp2_tests.py

Run the tests relevant to what you have implemented. Later-checkpoint tests will not pass while the corresponding student algorithms remain stubs. You can narrow Checkpoint 1 tests (for example, --algorithm exhaustive), as described earlier on this page.

Step 4 — Run a benchmark suite

Once the relevant correctness tests pass, start with the small readiness suite:

python3 tools/run_experiments.py --suite readiness

As you develop exact and heuristic algorithms, select additional suitable suites:

python3 tools/run_experiments.py --suite exact_frontier
python3 tools/run_experiments.py --suite quality_known

Run only the suite relevant to your current investigation; these are alternative commands, not a required sequence to execute all at once. Some experiments are intentionally expensive.

Step 5 — Inspect the results

For example, after running readiness:

less experiments/results/readiness.csv

Or inspect the corresponding structured output:

python3 -m json.tool experiments/results/readiness.json | less

Look for failed or invalid solutions, known-optimum mismatches, timeouts, runtime variation, and—when evaluating heuristics—solution quality. A completed experiment does not necessarily indicate that every solution was correct or optimal. The other suites write corresponding files under experiments/results/.

Step 6 — Improve one thing and repeat

Make a specific change, rerun the individual instance and public tests, and then repeat the relevant benchmark suite. For meaningful comparisons, keep the input instances, seeds, and benchmark settings consistent.

For your final experimental analysis, use the course-provided fixed seeds and benchmark suites so results remain reproducible.

Last modified November 6, 2026: BBF updates (ac3f9c6)