Pfthb

I am running a pytest-based suite of tests during my Azure DevOps build process. I have two jobs arranged to run these tests against two different environments.

In each job, I run the pytest tests using a script task and generate a junit-style xml output file, then have a PublishTestResults task publish that xml file. This is working great, and I'm able to peruse my test results in the azure build tests report UI -- but only if all the tests pass. If any tests fail, the publish task is skipped, and the tests aren't reported in the UI.

YML extract:

- job: 'RunTestsQA'
 continueOnError: True
 steps:
 - task: UsePythonVersion@0
 inputs:
 versionSpec: '3.6'
 architecture: 'x64'
 - task: DownloadSecureFile@1
 inputs:
 secureFile: 'ConfigFile'
 - script: pip install -r requirements.txt
 displayName: 'Install Requirements'
 - script: |
 pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
 displayName: 'Test with pytest'
 # PUBLISH JUNIT RESULTS
 - task: PublishTestResults@2
 inputs:
 condition: succeededOrFailed()
 testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit
 testResultsFiles: '**/TEST-*.xml' 
 #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional
 mergeTestResults: false # Optional
 testRunTitle: 'API_CHECK QA'
 #buildPlatform: # Optional
 #buildConfiguration: # Optional
 publishRunAttachments: true # Optional

Through some experimentation, I've been able to confirm the XML file is always created. What do I need to fix here? A test report isn't super helpful if it only shows up when the tests pass.

I'm using Ruby and Minitest, but I have found that the following setting allows the PublishTestResults task to run:

- script: |
 pytest -m smoke --ENV=qa --log-file $SYSTEM_ARTIFACTSDIRECTORY/smoke-qa.log --junitxml="TEST-qa-smoke.xml"
 displayName: 'Test with pytest'
 continueOnError: true

The only issue that I have found with this setting is that if the build fails, it reports as "Partially Succeeded" and not "Failed".

edit:

Of course, if your build process has any deploy tasks after the test task, you may not want to use this setting.