How to exclude execution of certain specflow tests marked with @ignore keyword in Azure Pipeline

Question

I have many specflow feature files in my solution and there are multiple UI test cases. At the page level, I have defined a tag eg @Feature1 for the first file and @feature2 for the second file. They are passed onto as the parameter in the yaml file I pass the tag to my pipeline yml. Now I am in a situation wherein I have few test cases marked as @ignore as well So then the pipeline runs, these test cases are not exculded but eventually fail. I want to skip the test cases marked with the @ignore attribute/tag. Here is a snippet from my pipeline

  parameters:
  - name: 'featuresToRun'
    type: object
    default:
    - Performance
    - AutoComplete
    - Benches
    - CATMI
    - Export
    - GemIntegration
    - Keyboard
    - MainMenu
    - NewVoyages
    - ReferenceData
    - Settings
    - SimilarVoyages
    - Validation
    - Views
    - VolumeConversion
    - Voyages
    - LaycanRanges

trigger: none

jobs:
- job: startVM
  timeoutInMinutes: 10
  pool:
    vmImage: 'windows-latest'

  steps:
  - checkout: none

  

- job: runTests
  timeoutInMinutes: 1800
  dependsOn: startVM
  condition: not(canceled())

  pool:
    name: 'UI Automation'

  steps:

  - task: ScreenResolutionUtility@1
    inputs:
      displaySettings: 'optimal'

  - task: VisualStudioTestPlatformInstaller@1
    inputs:
      packageFeedSelector: 'nugetOrg'
      versionSelector: 'latestStable'

  - task: NuGetCommand@2
    inputs:
      command: 'restore'
      restoreSolution: '**/*.sln'
      feedsToUse: 'config'

  - task: MSBuild@1
    inputs:
      solution: 'UIAutomation.sln'
      msbuildArchitecture: 'x64'
      clean: true

  - ${{each feature in parameters.featuresToRun}}:
    - task: VSTest@2
      displayName: ${{feature}} Tests
      inputs:
        testSelector: 'testAssemblies'
        testAssemblyVer2: |
          UIAutomation.Specs\bin\Debug\UIAutomation.Specs.dll
          !**\*TestAdapter.dll
          !**\obj\**
        searchFolder: '$(System.DefaultWorkingDirectory)'
        uiTests: true
        testRunTitle: '${{feature}}'
        testFiltercriteria: 'Category=${{feature}}'        
        rerunFailedTests: true
        rerunMaxAttempts: 2
        rerunFailedThreshold: 80
        codeCoverageEnabled: true
      continueOnError: true

Greg Burghardt · Accepted Answer · 2022-12-01 22:06:23Z

1

Modify the testFiltercriteria to exclude that category:

testFiltercriteria: 'Category=${{feature}}&Category!=ignore'
                                          ^^^^^^^^^^^^^^^^^

it appears the testFiltercriteria property translates to the --filter argument to dotnet test or the --testcasefilter:<Expression> argument to vstest.console.exe.

The @ignore tag in Gherkin gets translates to a [TestCategory(...)] attribute above the test methods (when using MS Test). Other unit test providers have a similar conversion.

More info:

edited Dec 1, 2022 at 22:06

answered Dec 1, 2022 at 21:28

Greg Burghardt

19.2k10 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to exclude execution of certain specflow tests marked with @ignore keyword in Azure Pipeline

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related