Skip to main content

Quickstart

Testiny integrates with test automation tools and frameworks to collect and manage automated test results in one place. Whether tests are run via a CI/CD pipeline or manually, Testiny helps debug, troubleshoot, and analyze failures efficiently. Learn how to submit your first automated test run to Testiny and set up reporting with your automation framework and CI/CD pipeline.

Submit your first automation results​

  1. Download or install the Testiny CLI and create an API key in Testiny.

  2. Run your automated tests & generate a report. Testiny CLI supports JUnit-style XML files, Playwright JSON report, Cypress JSON reports and more. See the CLI reference for all supported report formats.

  3. Import your automated results to Testiny. Specify the Testiny API key, project ID or key, a source name, the report type and the report file(s):

export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --junit report.xml
See command output
API key: user '[email protected]' in organization 'testiny'
Using project: Demo (DE)
Import source: integration-tests
Environment: gitlab
Processing nunit file: report.xml
  Submitted 168/168 results
Completed 1 automation test run(s)
Imported all 1 file(s) successfully
Submitted Testiny automated run ids: 264

The CLI will parse the report file, create an automation run in Testiny and add all results, errors and attachments to the run. The run is added to the specified project. The source name is used to group your test runs together (so choose a name that fits your test suite, e.g. frontend-tests, backend-tests, e2e-tests, api-tests, ...). In the console output, you can see how many test files have been processed and how many results and attachments have been uploaded.

Run Fields​

Run Fields (and the specified source) are used to identify test runs and to group together automated results. For example, in CI environments, the Pipeline ID, Build ID, Job ID or any combination can be used for Run Fields.

If you don't provide any Run Fields with --run-fields, an automatically generated run_id is used (or if in a known CI environment, the CLI sets some default Run Fields). If you would like to specify your own run key you can override the default Run Fields. You can use any field(s) from your automated results as a Run Field and pass them as a comma-separated list. To add a field value, use --field-values and specify it as a Run Field with --run-fields:

export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --field-values my_run_key=1234 --run-fields my_run_key --junit report.xml

After submitting results to Testiny, the test run is automatically marked as completed and no new results can be added to this run. If you would like to submit more results to the same run, add the --incomplete flag to every command submitting results to the run. See the next section Distributed Results for an example.

Distributed Results​

Run Fields are really powerful in CI environments when importing results from split tests (e.g. parallel testing, test sharding, different environments, ...), as you can use the available environment variables to add the distributed results to the same test run.

If you would like to manually group together reports and import them to the same run in Testiny, you can specify your own run key and mark them as --incomplete and finally, mark the run as complete:

export TESTINY_API_KEY=****
  
# report 1
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --incomplete --field-values my_run_key=1234 --run-fields my_run_key --junit report1.xml

# report 2
npx @testiny/cli automation --project ID_OR_KEY --source "unit-tests" --incomplete --field-values my_run_key=1234 --run-fields my_run_key --junit report2.xml

# complete run
npx @testiny/cli automation  --project ID_OR_KEY --source "unit-tests" --complete-runs --field-values my_run_key=1234 --run-fields my_run_key

If the CLI is executed in a CI environment, it auto-detects various environment variables like commit hashes and tags, pipeline and job IDs, and adds these fields to your automation run. See our examples for CI/CD integration or learn how import results from parallel/shared tests.

Importing multiple files​

Some automation frameworks like Cypress or Mochawesome output multiple result files. You simply import multiple report files at once by specifying a directory or a pattern to select multiple files for import:

export TESTINY_API_KEY=****
npx @testiny/cli automation --project ID_OR_KEY --source unit-tests --junit results/*.xml

Test Automation Guide​

Learn more about automation concepts, adding fields to your run and parallel testing in our detailed Automation Guide.

Reporters & examples​

For some automation frameworks, we offer custom reporters where you don't need to install and handle the CLI yourself. See all available reporters and usage examples.

Next steps​