Reporting Test Automation with consistent IDs

Ben_Gruber · September 12, 2025, 1:08pm

I’ve finally gotten a chance to start trying out the Test Automation feature with your reporter, and so far it’s looking great. I do have some questions about the linking automated tests to specific Test IDs though.

I saw in the testinyapp/junit-automation-example project on github (I can’t link to it directly it seems) that you should be able to set one of two properties (testid and testkey) to tell Testiny it should link it to a consistent ATC ID. However, in practice I haven’t seemed to get it to work.

Here’s the sample junit file I’m working with:

junit.xml

<?xml version="1.0" encoding="utf-8"?>
<testsuites name="pytest tests">
    <testsuite name="pytest" errors="0" failures="1" skipped="2" tests="13" time="3.055"
               timestamp="2025-09-12T13:00:03.976481+00:00" hostname="hostname">
        <testcase classname="tests.test_assertion" name="test_assertion" time="0.000">
            <properties>
                <property name="requirement" value="RS-1"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_two" time="0.000">
            <properties>
                <property name="requirement" value="RS-2"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_three" time="3.001">
            <properties>
                <property name="requirement" value="RS-3"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_four[1]" time="0.001">
            <properties>
                <property name="requirement" value="RS-4"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_four[2]" time="0.001">
            <properties>
                <property name="requirement" value="RS-4"/>
            </properties>
            <failure message="Failed: Intentional failure for case 2">variation = 2, record_property = &lt;function
                record_property.&lt;locals&gt;.append_property at 0x790f933231a0&gt;

                @mark.parametrize("variation", [1, 2, 3])
                def test_four(variation, record_property):
                print(variation)
                record_property("requirement", "RS-4")
                if variation == 2:
                &gt; pytest.fail(f"Intentional failure for case {variation}")
                E Failed: Intentional failure for case 2

                tests/test_assertion.py:25: Failed
            </failure>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_four[3]" time="0.001">
            <properties>
                <property name="requirement" value="RS-4"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_not_validation" time="0.001"/>
        <testcase classname="tests.test_assertion" name="test_skipped_test" time="0.000">
            <skipped type="pytest.skip" message="Intentional skip for prototype">
                /home/ben/project/qaet/testiny-automation-reporter/tests/test_assertion.py:30: Intentional skip for
                prototype
            </skipped>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_xfail_test" time="0.001">
            <skipped type="pytest.xfail" message=""/>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_with_test_id_modified" time="0.001">
            <properties>
                <property name="testid" value="a34422f1-86f5-4fc0-b2d1-746b96fd61f8"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_with_test_key" time="0.001">
            <properties>
                <property name="testkey" value="prototype_A123_01"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_with_test_id_and_key" time="0.001">
            <properties>
                <property name="testid" value="a34422f1-86f5-4fc0-b2d1-746b96fd61f8"/>
                <property name="testkey" value="prototype_A123_02"/>
            </properties>
        </testcase>
        <testcase classname="tests.test_assertion" name="test_with_explicit_test_id" time="0.001">
            <properties>
                <property name="testid" value="ATC-24"/>
            </properties>
        </testcase>
    </testsuite>
</testsuites>

And the invocation I’m using to import the results:

./testiny-importer-linux automation --project 1 --source "pytest-prototype-cicd" "--custom-result-fields=requirement,testid,testkey" --junit junit.xml

I’ve been modifying just the names of the tests, and despite having the testid or testkey field set, it links the results to a new ATC ID. Changing the name of the test back links the result to the previous ATC ID.

I’ve noticed that testid doesn’t appear as a custom field in the test results, but testkey does. Strangely, for test_with_test_id_and_key the testkey field has a value, but for the test_with_test_key test the testkey field has no value.

Any help or guidance would be great.

mgi · September 15, 2025, 9:26am

Hello Ben,

We investigated this and found that our JUnit-loader processes test keys incorrectly.
Your usage of the test keys is correct - they are intended to be used for pinning to a specific automation test case with the given key value instead of inferring the test case by title/folder

We will try to fix this in one of the upcoming patch releases.

Regards,
Michael

Ben_Gruber · September 15, 2025, 9:48am

Good to know I’m not going crazy or missing something obvious.

Relatedly, do you mind explaining how the importer will handle each of testid and testkey, when each should be used and what would happen if you supplied both? I would like to be able to start writing our internal documentation on how to use the properties with test cases so we’re ready when it’s working again.

Alex_Hoisl · September 15, 2025, 10:24am

Only the first testid or testkey property will be used, any other properties with such name on the same test will be ignored.
So both “testkey” and “testid” are interchangeable.

Best regards,
Alex

Ben_Gruber · September 15, 2025, 11:04am

Two follow up questions:

If I set the testid to an ATC-# value, would it then map to that existing ATC ID? (If not, that would be helpful)
Are you going to expose the testid value? In my testing so far that wasn’t, but that might be due to the bug.

Alex_Hoisl · September 16, 2025, 11:14am

The testid is actually meant to be independent of Testiny. You can use any identifier in your automated tests and when you import these tests the first time, the testkey-ATC-link will thus be set in Testiny.
If you would use Testiny’s ATC ID as the testkey, then you would need a second pass to achieve this link (with the first matching being done by title only, which is undesirable):
first import without key → note down the ATC ID in Testiny → add testkey to your code → re-import the tests (matching will be done by title this time).

We will show the current key for ATCs in the automated test run results view with one of the next updates.

Best regards,
Alex