A critical vulnerability was discovered in React Server Components (Next.js). Our systems remain protected but we advise to update packages to newest version. Learn More. 

AI OnAI Off
Johan Antila
Johan Antila
Sep 16, 2022
  68
(0 votes)

Azure DevOps and failing VSTest task

Sometime during the summer, our Test task for a customers Azure Devops CI/CD environment stopped working for no apparent reason and as it had been unchanged for more than a year, the problem looked like it would be related to the Azure Devops environment itself. Checking build logs, we could conclude that Microsoft did an update to the default build runner used in Azure DevOps sometime in August or perhaps late July, 2022 (We lack builds from just before so we don't know the exact date) and it included a later version of the VSTest task that had a different behaviour than the previous one. It turned out that our test task included a lot of dlls targeted for different frameworks that were not actual tests that should not have been included but this selection of dlls hadn't changed, it was always the same and it had always picked up a lot more test dlls than it should have done - all with different targets - but since that was never a problem with the previous runner, we didn't pay attention to this. Where the old version would ignore this situation, the newer version of the VSTask returned a fail when it picked up dlls from different framework targets, something it does not support, thus failing the build even though the tests it actually managed to run all passed. The solution to this is to specify a less exclusive search pattern than the default one that you get if you define the test task in your yaml file like this:

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
(Not specifying wildcard pattern defaults to this search pattern: **\*test*.dll,!**\*TestAdapter.dll,!**\obj\**)

In my build, this would include all these files:

vstest.console.exe "D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Core.dll"
"D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Dal.dll"
"D:\a\1\s\Customer.Web\bin\EPiServer.Marketing.Testing.Web.dll"
"D:\a\1\s\Tests\Tests.Customer.Framework\bin\Release\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.Interface.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pl\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pl\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\pt\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\ru\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\tr\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hans\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.MSTest.TestAdapter.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\_common\zh-Hant\Microsoft.VisualStudio.TestPlatform.TestFramework.resources.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\netcoreapp1.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll"
"D:\a\1\s\packages\MSTest.TestAdapter.2.1.2\build\uap10.0\Microsoft.VisualStudio.TestPlatform.MSTestAdapter.PlatformServices.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\net45\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\netstandard1.0\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\netstandard1.0\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\uap10.0\Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions.dll"
"D:\a\1\s\packages\MSTest.TestFramework.2.1.2\lib\uap10.0\Microsoft.VisualStudio.TestPlatform.TestFramework.dll"


After changing the way test assemblies are found like this:

- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\tests.customer*.dll
    searchFolder: '$(System.DefaultWorkingDirectory)'

    
It now includes only the relevant assemblies:

vstest.console.exe "D:\a\1\s\Tests\Tests.Customer.Framework\bin\Release\Tests.Customer.Framework.dll"
"D:\a\1\s\Tests\Tests.Customer.Framework\obj\Release\Tests.Customer.Framework.dll"
"D:\a\1\s\Tests\Tests.Customer.Web\bin\Release\Tests.Customer.Web.dll"
"D:\a\1\s\Tests\Tests.Customer.Web\obj\Release\Tests.Customer.Web.dll"

And with this, the test now passes. Succes! In other words, make sure you have a coherent naming scheme for your tests that is easy to pick up with a wildcard pattern and make sure your test task only picks up these specific dlls. You can see the output of the VSTest task in the pipeline in Azure Devops to detrmine what dlls it's trying to probe for tests.

Sep 16, 2022

Comments

Please login to comment.
Latest blogs
A day in the life of an Optimizely OMVP: Learning Optimizely Just Got Easier: Introducing the Optimizely Learning Centre

On the back of my last post about the Opti Graph Learning Centre, I am now happy to announce a revamped interactive learning platform that makes...

Graham Carr | Jan 31, 2026

Scheduled job for deleting content types and all related content

In my previous blog post which was about getting an overview of your sites content https://world.optimizely.com/blogs/Per-Nergard/Dates/2026/1/sche...

Per Nergård (MVP) | Jan 30, 2026

Working With Applications in Optimizely CMS 13

💡 Note:  The following content has been written based on Optimizely CMS 13 Preview 2 and may not accurately reflect the final release version. As...

Mark Stott | Jan 30, 2026

Experimentation at Speed Using Optimizely Opal and Web Experimentation

If you are working in experimentation, you will know that speed matters. The quicker you can go from idea to implementation, the faster you can...

Minesh Shah (Netcel) | Jan 30, 2026

See all blogs