CLI¶

Bowtie is a versatile tool which you can use to investigate any or all of the implementations it supports. Below are a few sample command lines you might be interested in.

Running Commands Against All Implementations

Many of Bowtie’s subcommands take a -i / --implementation option which specifies which implementations you wish to run against. In general, these same subcommands allow repeating this argument multiple times to run across multiple implementations. In many or even most cases, you may be interested in running against all implementations Bowtie supports. In the future, Bowtie’s CLI will default to running against all implementations in a number of additional cases where it makes sense to do so.

For now, if you wish to run against all implementations you can make use of the filter-implementations command to simply output the full list, along with some simple shell vomit to insert the needed -i options. Specifically, running $(bowtie filter-implementations | sed 's/^/-i /') will expand out to something you can use to run against all implementations. See below for a concrete example.

Examples¶

Validating a Specific Instance Against One or More Implementations¶

The bowtie validate subcommand can be used to test arbitrary schemas and instances against any implementation Bowtie supports.

Given some collection of implementations to check – here perhaps two Javascript implementations – it takes a single schema and one or more instances to check against it:

$ bowtie validate -i js-ajv -i js-hyperjump <(printf '{"type": "integer"}') <(printf 37) <(printf '"foo"')

Note that the schema and instance arguments are expected to be files, and that therefore the above makes use of normal shell process substitution to pass some examples on the command line.

Piping this output to bowtie summary is often the intended outcome (though not always, as you also may upload the output it gives to https://bowtie.report/ as a local report). For summarizing the results in the terminal however, the above command when summarized produces:

$ bowtie validate -i js-ajv -i js-hyperjump <(printf '{"type": "integer"}') <(printf 37) <(printf '"foo"') | bowtie summary

                                            Bowtie
┏━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Schema              ┃                                                                      ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│                     │                                                                      │
│ {                   │   Instance   hyperjump-json-schema (javascript)   ajv (javascript)   │
│   "type": "integer" │  ──────────────────────────────────────────────────────────────────  │
│ }                   │   37         valid                                valid              │
│                     │   "foo"      invalid                              invalid            │
│                     │                                                                      │
└─────────────────────┴──────────────────────────────────────────────────────────────────────┘
                                        2 tests ran

Running the Official Suite Across Implementations¶

The following will run all Draft 7 tests from the official test suite (which it will automatically retrieve) across all implementations supporting Draft 7, showing a summary of any test failures.

$ bowtie suite $(bowtie filter-implementations | sed 's/^/-i /') draft7 | bowtie summary --show failures

Running a Single Test Suite File¶

To run the draft 7 type-keyword tests on the Lua jsonschema implementation, run:

$ bowtie suite -i lua-jsonschema https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft7/type.json | bowtie summary --show failures

Running Test Suite Tests From Local Checkouts¶

Providing a local path to the test suite can be used as well, which is useful if you have additional tests in development locally:

$ bowtie suite $(bowtie filter-implementations | sed 's/^/-i /') ~/path/to/json-schema-org/suite/tests/draft2020-12/ | bowtie summary --show failures

Checking An Implementation Functions On Basic Input¶

If you wish to verify that a particular implementation works on your machine (e.g. if you suspect a problem with the container image, or otherwise aren’t seeing results), you can run bowtie smoke. E.g., to verify the Golang jsonschema implementation is functioning, you can run:

$ bowtie smoke -i go-jsonschema

Connectables¶

In all of the examples presented above, we passed our desired implementations to Bowtie’s -i / --implementation option.

In truth, this option is more flexible than indicated above, though generally this extra flexibility is useful for more advanced use cases, which this section elaborates on.

The full syntax of the -i option is known as a connectable.

Connectables implement a mini-language for connecting to supported harnesses.

They allow connecting to implementations supported by Bowtie without making assumptions about the specific mechanism used to run or execute them.

In most simple use cases, users likely will not interact directly with the connectable syntax. Passing -i some-supported-implementation will generally do The Right Thing™. But for more advanced usage, the full collection of supported connectables will now be described.

The general syntax for connectables looks like:

[<kind>:]<id>[:<arguments>*]

The kind of connectable is optional, and when unspecified defaults to the happy connectable, making an example of the simplest connectable look like lua-jsonschema (which expands to happy:lua-jsonschema). A slightly longer description of what the happy connectable means is below, but in brief, the happy connectable will speak directly to any implementation which can be imported from Python. For any other implementation, an OCI container will be created (and later destroyed) using the image connectable.

More generally, the id is an identifier whose exact form depends on each kind of connectable. Arguments customize the actual connection to the implementation, and which arguments are supported again depends on the kind of connectable.

Connectables are loosely inspired by Twisted's strports.

`image`¶

A container image which Bowtie will start, stop and delete when finished.

The image id should be the image name. Providing a repository is optional, and if unprovided, will default to pulling images from Bowtie’s own public repository of images.

The image must be an image whose entrypoint speaks Bowtie’s harness protocol (which of course all of Bowtie’s own published harnesses images will do).

Examples:

image:example: an image named example, retrieved from Bowtie’s repository

example: with no explicit image, referring to the same image as previous

image:foo/bar:latest: an image with fully specified OCI container repository which will be pulled if not already present

`container`¶

An externally running container which Bowtie will connect to.

The container must be listening on standard input for input valid under Bowtie’s harness protocol.

Bowtie will not attempt to manage the container, so this connectable is suitable for cases where you wish to spin up a container externally, leave it running and potentially have Bowtie connect to it multiple times.

The id is a connector-specific identifier and should indicate the specific intended implementation. For example, for container images, it must be the name of a container image which will be pulled if needed. It need not be fully qualified (i.e. include the repository), and will default to pulling from Bowtie’s own image repository.

Examples:

container:deadbeef: an OCI container with ID deadbeef which is assumed to be running (and will be attached to)

`direct`¶

An implementation which is directly importable from Python

This connectable is useful for implementations Bowtie can speak directly to without inter-process communication. The list of such implementations is small relative to those reachable using containers.

Examples:

direct:python-jsonschema: a direct connection to the Python implementation known as jsonschema

`happy`¶

A connectable which intelligently picks the best possible way to connect to an implementation

The happy connectable is a sort of “higher-level” connectable. Given the name of an implementation, the happy connectable will pick the best possible way to connect to the implementation. Best here is defined to be “most reliable” and/or “fastest”.

Generally this means “connect directly when possible, falling back to spinning up a container”.

This connectable is used when no connector is specified, effectively giving the best behavior when no specific connectable is requested.

Examples:

happy:python-jsonschema: a direct connection to the Python implementation known as jsonschema, equivalent to direct:python-jsonschema

happy:some-not-directly-connectable-implementation: for any non-directly connectable implementation, equivalent to image:some-not-directly-connectable-implementation

Enabling Shell Tab Completion¶

The Bowtie CLI supports tab completion using the click module's built-in support. Below are short instructions for your shell using the default configuration paths.

Add this to ~/.bashrc:

$ eval "$(_BOWTIE_COMPLETE=bash_source bowtie)"

Add this to ~/.zshrc:

$ eval "$(_BOWTIE_COMPLETE=zsh_source bowtie)"

Add this to ~/.config/fish/completions/bowtie.fish:

$ _BOWTIE_COMPLETE=fish_source bowtie | source

This is the same file used for the activation script method below. For Fish it’s probably always easier to use that method.

Using eval means that the command is invoked and evaluated every time a shell is started, which can delay shell responsiveness. To speed it up, write the generated script to a file, then source that.

Save the script somewhere.

$ _BOWTIE_COMPLETE=bash_source bowtie > ~/.bowtie-complete.bash

Source the file in ~/.bashrc.

$ . ~/.bowtie-complete.bash

Save the script somewhere.

$ _BOWTIE_COMPLETE=zsh_source bowtie > ~/.bowtie-complete.zsh

Source the file in ~/.zshrc.

$ . ~/.bowtie-complete.zsh

Save the script to ~/.config/fish/completions/bowtie.fish:

$ _BOWTIE_COMPLETE=fish_source bowtie > ~/.config/fish/completions/bowtie.fish

After modifying your shell configuration, you may need to start a new shell in order for the changes to be loaded.

Reference¶

bowtie¶

A meta-validator for the JSON Schema specifications.

Bowtie gives you access to the JSON Schema ecosystem across every programming language and implementation.

It lets you compare implementations either to each other or to known correct results from the official JSON Schema test suite.

bowtie [OPTIONS] COMMAND [ARGS]...

Options

--version¶: Show the version and exit.

-L, --log-level <log_level>¶

How verbose should Bowtie be?

Default:: 'warning'
Options:: debug | info | warning | error | critical

If you don’t know where to begin, bowtie validate --help or bowtie suite --help are likely good places to start.

Full documentation can also be found at https://docs.bowtie.report/

badges¶

Generate Bowtie badges for implementations using a previous Bowtie run.

Will generate badges for any existing dialects, and ignore any for which a report was not generated.

bowtie badges [OPTIONS]

Options

--site <site>¶

The path to a previously generated collection of reports, used to generate the badges.

Default:: PosixPath('site')

filter-benchmarks¶

Output benchmarks matching the specified criteria.

bowtie filter-benchmarks [OPTIONS]

Options

-D, --dialect <URI_OR_NAME>¶

A URI or shortname identifying the dialect of each test. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, and draft7.

Default:: 'Draft 2020-12'

-t, --benchmark-type <benchmark_type>¶

Specify the type of benchmark to filter.

Options:: default | keyword

-n, --name <benchmark_names>¶: Filter the benchmarks with given name. Use the option multiple times to filter multiple benchmarks.

filter-dialects¶

Output dialect URIs matching a given criteria.

If any implementations are provided, filter dialects supported by all the given implementations.

bowtie filter-dialects [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>¶: A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-d, --dialect <URI_OR_NAME>¶: Filter from the given list of dialects only.

-l, --latest¶: Show only the latest dialect.

-b, --boolean-schemas, -B, --no-boolean-schemas¶: If provided, show only dialects which do (or do not) support boolean schemas. Otherwise show either kind.

filter-implementations¶

Output implementations which match the given criteria.

Useful for piping or otherwise using the resulting output for further Bowtie commands.

bowtie filter-implementations [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>¶: A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-f, --format <format>¶

What format to use for the output

Default:: 'plain'
Options:: plain | json

--schema¶: Show the JSON Schema for this command’s JSON output.

-d, --supports-dialect <URI_OR_NAME>¶: Only include implementations supporting the given dialect URI or dialect shortname. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, and draft7.

-l, --language <LANGUAGE>¶

Only include implementations in the given programming language.

Options:: c++ | clojure | cpp | dotnet | elixir | go | java | javascript | js | kotlin | lua | php | python | ruby | rust | scala | swift | ts | typescript

--direct¶: Only include implementations with direct connectable functionality (i.e. which can run without the presence of a container runtime).

info¶

Show information about a supported implementation.

bowtie info [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>¶: A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-f, --format <format>¶

What format to use for the output

Default:: 'pretty if stdout is a tty, otherwise JSON'
Options:: json | pretty | markdown

--schema¶: Show the JSON Schema for this command’s JSON output.

--versions, --no-versions¶: Also show all implementation versions which Bowtie supports.

latest-report¶

Output the latest published report from Bowtie’s website.

bowtie latest-report [OPTIONS]

Options

-D, --dialect <URI_OR_NAME>¶

Default:: 'Draft 2020-12'

perf¶

Perform performance measurements across supported implementations.

bowtie perf [OPTIONS] [BENCHMARK]

Options

-i, --implementation <IMPLEMENTATION>¶: Required A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-D, --dialect <URI_OR_NAME>¶

Default:: 'Draft 2020-12'

-f, --format <format>¶

What format to use for the output

Default:: 'pretty if stdout is a tty, otherwise JSON'
Options:: json | pretty | markdown

--schema¶: Show the JSON Schema for this command’s JSON output.

-r, --runs <runs>¶

Number of runs used to run benchmarks.

Default:: 3

-v, --values <values>¶

Number of values per run.

Default:: 2

-w, --warmups <warmups>¶

Number of skipped values per run used to warmup the benchmark.

Default:: 1

-l, --loops <loops>¶

Number of loops per value.

Default:: 1

-q, --quiet¶

Enable quiet mode (Only output the final result).

Default:: False

-k, --keywords¶

Run keyword specific benchmarks to learn about how various implementations implement the keyword.

Default:: False

-b, --benchmark-file <benchmark_files>¶: Allows running benchmark from a file. Specify the path of the benchmark file

-t, --test-suite <test_suite>¶: Run Benchmarks over the official JSON Schema Test Suite.

Arguments

BENCHMARK¶: Optional argument

run¶

Run test cases written directly in Bowtie’s testing format.

This is generally useful if you wish to hand-author which schemas to include in the schema registry, or otherwise exactly control the contents of a test case.

bowtie run [OPTIONS] [INPUT]

Options

-D, --dialect <URI_OR_NAME>¶

Default:: 'Draft 2020-12'

-i, --implementation <IMPLEMENTATION>¶: Required A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-k, --filter <GLOB>¶: Only run cases whose description match the given glob pattern.

-x, --fail-fast¶: Stop running immediately after the first failure or error.

--max-fail <COUNT>¶: Stop running once COUNT tests fail in total across implementations.

--max-error <COUNT>¶: Stop running once COUNT tests error in total across implementations.

-S, --set-schema¶

Explicitly set $schema in all (non-boolean) case schemas sent to implementations. Note this of course means what is passed to implementations will differ from what is provided in the input.

Default:: False

-V, --validate-implementations¶: When speaking to implementations (provided via -i), validate the requests and responses sent to them under Bowtie’s JSON Schema specification. Generally, this option protects against broken Bowtie implementations and can be left at its default (of off) unless you are developing a new implementation container.

Arguments

INPUT¶: Optional argument

smoke¶

Smoke test implementations for basic correctness against Bowtie’s protocol.

bowtie smoke [OPTIONS]

Options

-i, --implementation <IMPLEMENTATION>¶: A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-q, --quiet¶: Don’t print any output, just exit with nonzero status on failure.

-f, --format <format>¶

What format to use for the output

Default:: 'pretty if stdout is a tty, otherwise JSON'
Options:: json | pretty | markdown

--schema¶: Show the JSON Schema for this command’s JSON output.

statistics¶

Show summary statistics for a Bowtie generated report.

If stdin is a TTY, the most recent public report for the latest JSON Schema dialect is downloaded. Otherwise, if it is not a TTY (e.g. if it is a pipe) then it should contain report data.

Piping input via:

$ bowtie latest-report –dialect <some-dialect> | bowtie statistics

can be useful to retrieve the latest report for a specific dialect.

bowtie statistics [OPTIONS] [REPORT]

Options

-f, --format <format>¶

What format to use for the output

Default:: 'pretty if stdout is a tty, otherwise JSON'
Options:: json | pretty | markdown

--schema¶: Show the JSON Schema for this command’s JSON output.

--quantiles <n>¶: How many quantiles should be emitted for the compliance numbers? Computing quantiles only is sensical if this number is more than the number of implementations reported on. By default, we compute quartiles.

Arguments

REPORT¶: Optional argument

suite¶

Run the official JSON Schema test suite against any implementation.

Supports a number of possible inputs:

file paths found on the local file system containing tests, e.g.:

{PATH}/tests/draft7 to run the draft 7 version’s tests out of a local checkout of the test suite

{PATH}/tests/draft7/foo.json to run just one file from a checkout

URLs to the test suite repository hosted on GitHub, e.g.:

https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft7/ to run a version directly from any branch which exists in GitHub

https://github.com/json-schema-org/JSON-Schema-Test-Suite/blob/main/tests/draft7/foo.json to run a single file directly from a branch which exists in GitHub

short name versions of the previous URLs (similar to those providable to bowtie validate --dialect, e.g.:

7, to run the draft 7 tests directly from GitHub (as in the URL example above)

bowtie suite [OPTIONS] DIALECT

Options

-i, --implementation <IMPLEMENTATION>¶: Required A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-k, --filter <GLOB>¶: Only run cases whose description match the given glob pattern.

-x, --fail-fast¶: Stop running immediately after the first failure or error.

--max-fail <COUNT>¶: Stop running once COUNT tests fail in total across implementations.

--max-error <COUNT>¶: Stop running once COUNT tests error in total across implementations.

-S, --set-schema¶

Explicitly set $schema in all (non-boolean) case schemas sent to implementations. Note this of course means what is passed to implementations will differ from what is provided in the input.

Default:: False

-V, --validate-implementations¶: When speaking to implementations (provided via -i), validate the requests and responses sent to them under Bowtie’s JSON Schema specification. Generally, this option protects against broken Bowtie implementations and can be left at its default (of off) unless you are developing a new implementation container.

Arguments

DIALECT¶: Required argument

summary¶

Generate an (in-terminal) summary of a Bowtie run.

bowtie summary [OPTIONS] [REPORT]

Options

-f, --format <format>¶

What format to use for the output

Default:: 'pretty if stdout is a tty, otherwise JSON'
Options:: json | pretty | markdown

--schema¶: Show the JSON Schema for this command’s JSON output.

-s, --show <show>¶

Configure whether to display validation results (whether instances are valid or not) or test failure results (whether the validation results match expected validation results)

Default:: 'validation'
Options:: failures | validation

Arguments

REPORT¶: Optional argument

trend¶

Show trend data of an implementation across its multiple versions.

bowtie trend [OPTIONS] [VERSIONED_REPORTS_TAR]

Options

-i, --implementation <IMPLEMENTATION>¶: Required A connectable ID for a JSON Schema implementation supported by Bowtie. Run bowtie filter-implementations for the full list of supported implementations.

-D, --dialect <URI_OR_NAME>¶: A URI or shortname identifying the dialect of the test suite you wish to see the trend data for. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, and draft7. If none specified then defaults to all known dialects.

-f, --format <format>¶

What format to use for the output

Default:: 'pretty if stdout is a tty, otherwise JSON'
Options:: json | pretty | markdown

--schema¶: Show the JSON Schema for this command’s JSON output.

Arguments

VERSIONED_REPORTS_TAR¶: Optional argument

validate¶

Validate instances under a schema across any supported implementation.

bowtie validate [OPTIONS] SCHEMA [INSTANCES]...

Options

-D, --dialect <URI_OR_NAME>¶: A URI or shortname identifying the dialect of each test. Possible shortnames include: 2019, 2019-09, 201909, 2020, 2020-12, 202012, 3, 4, 6, 7, draft2019-09, draft201909, draft2020-12, draft202012, draft3, draft4, draft6, and draft7.

-i, --implementation <IMPLEMENTATION>¶: Required A connectable ID for a JSON Schema implementation supported by Bowtie. May be repeated multiple times to select multiple implementations to run. Run bowtie filter-implementations for the full list of supported implementations.

-S, --set-schema¶

Explicitly set $schema in all (non-boolean) case schemas sent to implementations. Note this of course means what is passed to implementations will differ from what is provided in the input.

Default:: False

-V, --validate-implementations¶: When speaking to implementations (provided via -i), validate the requests and responses sent to them under Bowtie’s JSON Schema specification. Generally, this option protects against broken Bowtie implementations and can be left at its default (of off) unless you are developing a new implementation container.

-d, --description <description>¶: A (human-readable) description for this test case.

--expect <expect>¶

Expect the given input to be considered valid or invalid, or else (with ‘any’) to allow either result.

Default:: 'any'
Options:: valid | invalid | any

Arguments

SCHEMA¶: Required argument

INSTANCES¶: Optional argument(s)

CLI¶

Examples¶

Validating a Specific Instance Against One or More Implementations¶

Running the Official Suite Across Implementations¶

Running a Single Test Suite File¶

Running Test Suite Tests From Local Checkouts¶

Checking An Implementation Functions On Basic Input¶

Connectables¶

image¶

container¶

direct¶

happy¶

Enabling Shell Tab Completion¶

Reference¶

bowtie¶

badges¶

filter-benchmarks¶

filter-dialects¶

filter-implementations¶

info¶

latest-report¶

perf¶

run¶

smoke¶

statistics¶

suite¶

summary¶

trend¶

validate¶

`image`¶

`container`¶

`direct`¶

`happy`¶