Testiny API doesn’t seem to take into account our folder structure when using JUnit or CSV

Support
, ,
1

Hi there,
We are running into some limitations with the Testiny CLI when uploaded test results from Bitrise to Testiny.
Since the CLI doesn’t support directly uploading the native xcresult format for our iOS tests, we need to first convert xcresult to junit. As a result, the uploaded results don’t seem to include attachments.
Additionally, the test results uploaded in the junit format also lack the relevant folder structure which is essential for our QA team.

As a result of these shortcomings, we will continue to use the Testiny API but wanted to see if you plan on adding support for the xcresult files or if you plan on fixing the CLI such that the folder structures from JUNIT files are respected. Or should we just continue to use the API going forward?

Any tips and guidance is greatly appreciated.

2

Hi,

Thanks for your feedback!

We plan to extend the CLI automation command to support more result formats, but it might be a while until we get to that. I’ve added the importance of the xcresult format to our backlog.

Concerning the attachments: Are these lost during the conversion from xcresult to junit? The testiny-importer automation command is able to upload attachments to results if the attachment path is in the junitfile. Ar the attachments paths or file names in your junit file?

The CLI is also able to import the folder structure. You can also specify the option --junit-classname-as-folder if you would like to use the classname as the test case folder.
How are the folders represented in your junit file?

It would be great if you could share a sample junit file with us (can be anonymized, and also just be a snippet of the whole file). Then we can try to reproduce the problems on our side (and if necessary, also deploy a fix for the CLI :slight_smile: ).

Best regards,
Hanna

Just a side note: The testiny-importer testrun command (not the automation command) does not support importing attachments, and we don’t plan to extend the testrun command.

3

Thank you for your message. After converting from xcresult to junit format, we observed that all attachments were lost during the conversion and file paths were not preserved. For reference, we’ve attached a sample junit file converted from an xcresult file:

<?xml version='1.0' encoding='UTF-8'?>
<testsuites tests='1' failures='1' skipped='0' time='50.0'>
  <testsuite name='Compa****PushNotificationUITests' time='50.0' tests='1' failures='1' skipped='0'>
    <testsuite name='SavedSearchPushNotificationTests' time='50.0' tests='1' failures='1' skipped='0'>
      <testcase name='testAgentSavedSearchListing()' classname='SavedSearchPushNotificationTests' time='50.0'>
        <failure message='SavedSearchPushNotificationTests.swift:40: failed - Error: Compa****AppError(type: UITestDSLCore.DSL.Compa****AppError.ErrorType.other(GRPC.ConnectionPoolError.deadlineExceeded(connectionError: nil)), file: &quot;Sources/ios/iOS/Tests/Compa****PushNotificationUITests/SavedSearchPushNotificationTests.swift&quot;, line: 40)'/>
      </testcase>
    </testsuite>
    <properties>
      <property name='Configuration' value='Test Scheme Action'/>
      <property name='device' value='iPhone 16 Pro (18.3.1)'/>
    </properties>
  </testsuite>
</testsuites>
4

Thank you for the sample file!

Concerning the attachments: those would need to be preserved during the conversion.

For the folder/name issue: The import of the folder name works on my end (it’s an extra column in the table). But I now understand that you’re probably refering to the “folder tree”, that is available in the test cases repository but not yet in the automation view.
This feature is already very high on our roadmap for the automation feature. So thanks for your feedback and helping to improve Testiny!

Best regards,
Hanna