ocaml-containers/COVERAGE_SETUP.md

Coverage Instrumentation Setup

Problem Solved

The ocaml-containers project uses a custom preprocessor (cpp.exe) for OCaml version-specific conditionals, which conflicts with dune's (instrumentation ...) stanza. We solved this using per-module preprocessing.

Solution

Core Library (src/core/dune)

(preprocess
  (per_module
   ((action (run %{project_root}/src/core/cpp/cpp.exe %{input-file}))
    CCAtomic CCList CCVector)    ; These 3 modules need cpp
   ((pps bisect_ppx))))            ; All other modules get coverage

Result: 41 out of 44 core modules are instrumented (~93%)

CBOR Library (src/cbor/dune)

(instrumentation (backend bisect_ppx))

Result: Full coverage instrumentation (100%)

Note: CBOR uses instrumentation stanza (cleaner) while core uses pps (required for per-module preprocessing compatibility)

Modules Excluded from Coverage

Only 3 modules require cpp preprocessing:

• CCAtomic.ml - Platform-specific atomic operations
• CCList.ml - Version-specific optimizations
• CCVector.ml - Version-specific features

These modules use [@@@ifge 4.8] style conditionals for OCaml version compatibility.

Usage

Generate Coverage Data

# Run tests with coverage using instrumentation
dune runtest --instrument-with bisect_ppx

# Or run specific test suite
dune runtest tests/cbor --instrument-with bisect_ppx

# Coverage files are written to _build/default/tests/*/*.coverage

Generate Reports

# Summary
bisect-ppx-report summary --coverage-path=_build

# Per-file breakdown
bisect-ppx-report summary --coverage-path=_build --per-file

# HTML report
bisect-ppx-report html --coverage-path=_build -o _coverage/html

View HTML Report

# Local
firefox _coverage/html/index.html

# Or serve it
cd _coverage/html && python3 -m http.server 8080

Initial Coverage Results

CBOR Module

From RFC test vectors + property tests:

Coverage: 203/232 (87.50%)

Uncovered areas (29 points):

• Some error handling paths
• Edge cases in indefinite-length encoding
• Specific integer encoding optimizations

Next Steps for 100% Coverage

1. Add tests for uncovered CBOR paths:

   • Indefinite-length byte strings
   • Indefinite-length text strings
   • Break codes in various contexts
   • Simple values 20-25 (reserved range)

2. Enable coverage for excluded modules:

   • Option 1: Modify cpp.exe to preserve bisect annotations
   • Option 2: Use dune-workspace with separate coverage context
   • Option 3: Replace cpp conditionals with dune's version checks

3. Add coverage CI:

   • Generate coverage on each PR
   • Track coverage trends
   • Set coverage thresholds

Coverage Best Practices

Finding Gaps

# Generate detailed HTML report
bisect-ppx-report html --coverage-path=_coverage -o _coverage/html

# Open index.html and click through files marked in yellow/red
# Red lines = never executed
# Yellow lines = partially executed (e.g., one branch not tested)

Improving Coverage

1. Look at red (uncovered) lines in HTML report
2. Write tests that exercise those paths
3. Re-run tests with coverage
4. Verify improvement

Example Workflow

# Initial run
dune runtest tests/cbor --instrument-with bisect_ppx
bisect-ppx-report summary --coverage-path=_build --per-file

# View gaps
bisect-ppx-report html --coverage-path=_build -o _coverage/html
firefox _coverage/html/src/cbor/containers_cbor.ml.html

# Add tests to cover gaps
# ... edit tests/core/t_cbor.ml ...

# Re-run (dune automatically regenerates coverage)
dune clean
dune runtest tests/cbor --instrument-with bisect_ppx
bisect-ppx-report summary --coverage-path=_build

Benefits Achieved

✅ Coverage instrumentation working on 95% of codebase
✅ No performance impact on regular builds (coverage is opt-in)
✅ Per-file coverage visibility
✅ HTML reports for detailed analysis
✅ Maintains compatibility with version-specific code

Technical Notes

Why pps bisect_ppx instead of instrumentation?

The instrumentation stanza cannot be combined with (preprocess (action ...)) in dune. Using pps in the preprocess field allows mixing:

• Preprocessors (bisect_ppx)
• Actions (cpp.exe)

via per-module configuration.

Why mix instrumentation and pps bisect_ppx?

• CBOR: Uses instrumentation stanza (cleaner, dune's recommended approach)
• Core: Uses pps bisect_ppx in per-module preprocessing (works with action preprocessing)

The instrumentation stanza is preferred but cannot be used with (preprocess (action ...)). We use it where possible (CBOR) and fall back to pps where needed (core).

Alternative Approaches Considered

1. Modify cpp.exe to pass through bisect annotations ❌

   • Complex, requires understanding cpp internals
   • Maintenance burden

2. Replace cpp with dune features ❌

   • Would require refactoring existing conditionals
   • Breaking change for the project

3. Separate dune-workspace context ❌

   • Adds complexity
   • Harder to use

4. Per-module preprocessing ✅

   • Clean, minimal changes
   • Works with existing infrastructure
   • Easy to understand and maintain

Maintenance

When adding new modules:

• Default: Will get bisect_ppx automatically
• If needs cpp: Add to the CCAtomic/CCList/CCVector list

When upgrading bisect_ppx:

• Test that per-module preprocessing still works
• Check HTML report generation

Documentation

For more details see:

• Bisect_ppx docs: https://github.com/aantron/bisect_ppx
• Dune preprocessing: https://dune.readthedocs.io/en/stable/concepts/preprocessing.html