bisect_ppx

Code coverage for OCaml and Reason

Downloads in past

Stats

StarsIssuesVersionUpdatedCreatedSize
bisect_ppx
297172.8.2a year ago4 years agoMinified + gzip package size for bisect_ppx in KB

Readme

Bisectppxself is a code coverage tool for OCaml and Reason. It helps you test thoroughly by showing what's not tested.
!Bisectppx usage examplesamplegh-pages-report
You can browse the report seen above online heregh-pages-report.

Table of contents

- Dune   (starter repodune-repo, reportdune-report) - esy   (starter repoesy-repo, reportesy-report) - ReScript   (starter reporescript-repo, reportrescript-report) - Jsofocaml   (starter repojsoo-repo, reportjsoo-report) - Ocamlfind, Ocamlbuild, and OASIS


Usage

Dune

Refer to aantron/bisect-starter-dunedune-repo, which produces this reportdune-report.
  1. Depend on Bisectppx
in your `opam` file:

 ```
 depends: [
   "bisect_ppx" {dev & >= "2.5.0"}
   "dune" {>= "2.7.0"}
 ]
 ```
  1. Mark the code under test for instrumentation by
`bisect_ppx`](https://github.com/aantron/bisect-starter-dune/blob/03cb827553d1264559eab19fdaa8c0056c9b2019/dune#L4) in your `dune` file:

 ```ocaml
 (library
  (public_name my_lib)
  (instrumentation (backend bisect_ppx)))
 ```
  1. Build and run your test binary. In addition to testing your code, when
exiting, it will write one or more files with names like
`bisect0123456789.coverage`:

 ```
 find . -name '*.coverage' | xargs rm -f
 dune runtest --instrument-with bisect_ppx --force
 ```

 The `--force` flag forces all your tests to run, which is needed for an
 accurate coverage report.

 To run tests without coverage, do

 ```
 dune runtest
 ```
  1. Generate the coverage reportdune-report in _coverage/index.html:
```
bisect-ppx-report html
```

You can also generate a short summary in the terminal:

```
bisect-ppx-report summary
```


esy

Refer to aantron/bisect-starter-esyesy-repo, which produces this reportesy-report.
The instructions are the same as for regular Dune usage, but...
  1. Depend on Bisectppx in package.json,
instead of in an opam file:
```json
"devDependencies": {
  "@opam/bisect_ppx": "^2.5.0"
},
"dependencies": {
  "@opam/dune": "^2.7.0"
}
```
  1. Use the esy command for the build and for running binaries:
```
esy install
esy dune runtest --instrument-with bisect_ppx --force
esy bisect-ppx-report html
```


ReScript

Refer to aantron/bisect-starter-rescriptrescript-repo, which produces this reportrescript-report.
  1. Depend on Bisectppx in package.json,
and install it:
```json
"devDependencies": {
  "bisect_ppx": "^2.0.0"
},
"dependencies": {
  "rescript": "*"
}
```

```
npm install
```

If pre-built binaries aren't available for your system, the build will
automatically fall back to building Bisect_ppx from source using
[esy](https://esy.sh), which will take a few minutes the first time. If this
happens, you may need to install esy, if it is not already installed:

```
npm install -g esy
npm install
```
  1. Add Bisectppx to your bsconfig.json:
```json
"bs-dependencies": [
  "bisect_ppx"
],
"ppx-flags": [
  "bisect_ppx/ppx"
]
```
  1. If you are using Jest, add this to your package.json:
```json
"jest": {
  "setupFilesAfterEnv": [
    "bisect_ppx/lib/js/src/runtime/js/jest.js"
  ]
}
```

Or, if you have enabled the `package-specs.in-source` flag in
`bsconfig.json`, replace the path by

```json
"bisect_ppx/src/runtime/js/jest.js"
```

You can exclude your test cases from the coverage report by adding this to
`bsconfig.json`:

```json
"ppx-flags": [
  ["bisect_ppx/ppx", "--exclude-files", ".*_test\\.res$$"]
]
```

Usage with Jest requires Bisect_ppx version 2.4.0 or higher. See the
[**aantron/bisect-starter-jest**][jest-repo] for a complete minimal example
project. That repo produces [this report][jest-report].

If the tests will be running in the browser, at the end of testing, call

```reason
Bisect.Runtime.get_coverage_data();
```

This returns binary coverage data in a `string option`, which you should
upload or otherwise get out of the browser, and write into a `.coverage`
file.
  1. Build in development with BISECT_ENABLE=yes, run tests, and generate the
coverage reportrescript-report in _coverage/index.html:
```
BISECT_ENABLE=yes npm run build
npm run test
npx bisect-ppx-report html
```

To exclude your test files from the report, change your PPX flags like so:

```json
"ppx-flags": [
  ["bisect_ppx/ppx", "--exclude-files", ".*test\\.re"]
]
```

The last argument is a regular expression in the syntax of OCaml's [`Str`
module](https://caml.inria.fr/pub/docs/manual-ocaml/libref/Str.html). Note
that backslashes need to be escaped both inside the regular expression, and
again because they are inside a JSON string.

Multiple `--exclude-files` option can be specified if you want to provide
multiple patterns.
  1. If your project uses both ReScript and native Dune, native Dune will start
picking up OCaml files that are part of the ReScript `bisect_ppx` package.
To prevent this, add a `dune` file with the following contents to the root of
your project:

```
(data_only_dirs node_modules)
```


Jsofocaml

Refer to aantron/bisect-starter-jsoojsoo-repo, which produces this reportjsoo-report.
  1. Follow the Dune instructions above, except that the final test
script must be linked with bisect_ppx.runtime (but not instrumented):
```scheme
(executable
 (name my_tester)
 (modes js)
 (libraries bisect_ppx.runtime))
```
  1. If the tests will run on Node, call this function
at the end of testing to write bisect0123456789.coverage:
```ocaml
Bisect.Runtime.write_coverage_data ()
```

If the tests will run in the browser, call

```ocaml
Bisect.Runtime.get_coverage_data ()
```

to get binary coverage data in a string option. Upload this string or
otherwise extract it from the browser to create a `.coverage` file.
  1. Build the usual Jsofocaml target, including the instrumented code under
test, then run the reporter to generate the coverage reportjsoo-report in _coverage/index.html:
```
dune build my_tester.bc.js --instrument-with bisect_ppx
node _build/default/my_tester.bc.js   # or in the browser
bisect-ppx-report html
```


Ocamlfind, Ocamlbuild, and OASIS

and OASIS instructions can be found at aantron/bisectppx-ocamlbuild.
  • With Ocamlfind, you must have your build script issue the right commands, to
instrument the code under test, but not the tester:
```
ocamlfind opt -package bisect_ppx -c src/source.ml
ocamlfind opt -c test/test.ml
ocamlfind opt -linkpkg -package bisect_ppx src/source.cmx test/test.cmx
```

Running the tester will then produce `bisect0123456789.coverage` files,
which you can process with `bisect-ppx-report`.


Sending to Coveralls and Codecov

bisect-ppx-report can send reports to Coveralls and Codecov directly from Travis, CircleCI, and GitHub Actions. To do this, run
bisect-ppx-report send-to Coveralls

or
bisect-ppx-report send-to Codecov

When sending specifically from GitHub Actions to Coveralls, use
- run: bisect-ppx-report send-to Coveralls
  env:
    COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    PULL_REQUEST_NUMBER: ${{ github.event.number }}

Put these commands in your CI script in the same place you would run bisect-ppx-report html locally. See bisect-ci-integration-megatest for example CI scripts and current status of these integrations.
If you'd like Bisectppx to support other CI and/or coverage services, please send a pull request!
As a workaround for missing CI/coverage integrations, and for development, bisect-ppx-report can also generate a JSON report in Coveralls format, which can be uploaded to a service of your choice using a separate command. For example, to send manually from Travis to Coveralls:
bisect-ppx-report \
  coveralls coverage.json \
  --service-name travis-ci \
  --service-job-id $TRAVIS_JOB_ID
curl -L -F json_file=@./coverage.json https://coveralls.io/api/v1/jobs

For other CI services, replace --service-name and --service-job-id as follows:
| CI service | --service-name | --service-job-id | | ---------- | ---------------- | -------------------- | | Travis | travis-ci | $TRAVIS_JOB_ID | | CircleCI | circleci | $CIRCLE_BUILD_NUM | | Semaphore | semaphore | $REVISION | | Jenkins | jenkins | $BUILD_ID | | Codeship | codeship | $CI_BUILD_NUMBER | | GitHub Actions | github | $GITHUB_RUN_NUMBER |
Note that Coveralls-style reports are less precise than the HTML reports generated by Bisect
ppx, because Coveralls considers entire lines as visited or not visited. There can be many expressions on a single line, and the HTML report separately considers each expression as visited or not visited.


Controlling coverage with [@coverage off]

You can tag expressions with [@coverage off], and neither they, nor their subexpressions, will be instrumented by Bisectppx.
Likewise, you can tag module-level let-declarations with [@@coverage off], and they won't be instrumented.
You can also turn off instrumentation for blocks of declarations inside a module with [@@@coverage off] and [@@@coverage on].
Finally, you can exclude an entire file by putting [@@@coverage exclude_file] into its top-level module. However, whenever possible, it is recommended to exclude files by not instrumenting with Bisect
ppx to begin with.


Other topics

See advanced usageadvanced for:
  • Exhaustiveness checking.
  • Excluding generated files from coverage.
  • SIGTERM handling.
  • Environment variables.

Cornell CS3110 offers a Bisectppx tutorialcs3110, featuring a video.


Bisectppx users

A small sample of projects using Bisectppx:

  • Core tools
- Dunedune   (report) - Lwtlwt   (report) - Odocodoc - ocamlformatocamlformat - OCamlocaml - Reasonreason - ctypesctypes
  • Libraries
- Irmin   (report) - Markup.mlmarkupml   (reportmarkupml-coveralls) - Lambda Soupsoup   (report) - Trie   (report) - ocaml-ooxml   (report) - routes   (report)
  • Applications

- Tezos - XAPI   (1, 2, 3, 4, 5) - Scilla   (report) - Coda - snarky - comby   (report)


Contributing

Bug reports and pull requests are warmly welcome. Bisectppx is developed on GitHub, so please open an issueissues.
After cloning the repo, try these Makefile targets:
  • make test for unit tests.
  • make usage for build system integration tests, except ReScript.
  • make -C test/js full-test for ReScript. This requires npm and esy.

If you'd like to build an npm package, run npm pack. You can install the resulting .tgz file in another project with npm install. This requires esy, as the Bisect binaries will not be pre-built. The npm package will use esy to build them automatically.