How to fix the Newman task for Team Foundation Server silently failing

The Problem

Running the NewmanPostman_VSTS_Task on our on-premise Microsoft Visual Studio Team Foundation Server 2017.3.1 finished successfully, but no tests were run:

##[section]Starting: Newman - Postman
==============================================================================
Task         : Newman the cli Companion for Postman
Description  : Using Newman, one can effortlessly run and test a Postman Collections directly from the 
Version      : 3.0.21
Author       : Carlo Wahlstedt
Help         : [More Information](https://github.com/carlowahlstedt/NewmanPostman_VSTS_Task/blob/master/README.md)
==============================================================================
found 1 files
No specific path to newman, using default of 'newman'
No custom report configured
Reporter list is : cli,json,junit
[command]C:\Users\[BUILDSERVER-USERNAME]\AppData\Roaming\npm\node_modules\newman\bin\newman.js run D:\TFS\Delivery\01\119\s\Delivery\ApiTests\BTW2019\BTW2019.postman_collection.json -r cli,json,junit -n 1 -e D:\TFS\Delivery\01\119\s\Delivery\ApiTests\BTW2019\BTW2019.Dev03.postman_environment.json
2019-01-09T09:49:11.3043192Z ##[section]Finishing: Newman - Postman

TL;DR

When the NewmanPostman_VSTS_Task tries to run AppData\Roaming\npm\node_modules\newman\bin\newman.js instead of AppData\Roaming\npm\newman.cmd ensure that the PATH environment variable for the user that the builds run on has C:\Users\[BUILDSERVER-USERNAME]\AppData\Roaming\npm in it, and reboot the server after that.

The Solution

I’ve already spoiled the ending of this blog post in the section above so the following is only interesting for people who like to know how I - despite starting with looking in the wrong place - managed to find a solution.

When I tried to run the same command directly on the build server …

C:\Windows\system32>C:\Users\[BUILDSERVER-USERNAME]\AppData\Roaming\npm\node_modules\newman\bin\newman.js run D:\TFS\Delivery\01\119\s\Delivery\ApiTests\BTW2019\BTW2019.postman_collection.json -
r cli,json,junit -n 1 -e D:\TFS\Delivery\01\119\s\Delivery\ApiTests\BTW2019\BTW2019.Dev03.postman_environment.json

… it opened the newman.js file in Notepad, instead of running the command!

This however did work as expected, so Newman was installed properly:

C:\>newman --help
Usage: newman [options] [command]

Options:
  -v, --version               output the version number
  -h, --help                  output usage information

Commands:
  run [options] <collection>  URL or path to a Postman Collection.

I disassociated .js files from Notepad by running the ASSOC command:

C:\Windows\system32>assoc .js=
The system cannot find the file specified.

“If ASSOC is invoked with just a file extension, it displays the current
file association for that file extension. Specify nothing for the file type and the command will delete the association for the file extension.”

This did not change anything, so I decided to reboot the build server. That did not change a thing too.

I then decided to search everywhere for “change the file type” and clicked “Change the file type associated with a file extension”. There I noticed that .js was still associated with Notepad! I decided to change the association to “Microsoft ® Windows Based Script Host”, because the .jse extension had that too.

Now when I excuted the newman.js run command I got an error instead of Notepad being opened:

There is no script engine for file extension “.js”.

So I tried another reboot, once more without success. I decided to have a look at our Development build server, where Newman was running correctly. There everything looked to be set up in the same way: no file association according to assoc .js, associated to “Microsoft ® Windows Based Script Host” according to “Set Associations”, and no mention of Newman in Environment PATH variables.

Looking at the logs of the “Newman - Postman” task of the build I noticed a difference. The successful one had this:

[command]C:\Windows\system32\cmd.exe /D /S /C “C:\Users[BUILDSERVER-USERNAME]\AppData\Roaming\npm\newman.cmd run …

The one that used to run the [NewmanPostman_VSTS_Task] but not executing any tests now failed with:

found 1 files
No specific path to newman, using default of ‘newman’
##[error]Unable to locate executable file: ‘newman’.
Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable.
Also verify the file has a valid extension for an executable file.

With that information I decided to have a look at the PATH variable in the Environment Variables of the Development build server: no mention of Newman there too.

I then realized that the builds run on a particular account. Here’s what I found in that User’s PATH variable (without the line-breaks):

C:\Users\[BUILDSERVER-USERNAME]\AppData\Local\Programs\Python\Python37-32\Scripts\;
C:\Users\[BUILDSERVER-USERNAME]\AppData\Local\Programs\Python\Python37-32\;
C:\Users\[BUILDSERVER-USERNAME]\AppData\Local\Programs\Python\Launcher\;
C:\Users\[BUILDSERVER-USERNAME]\.dotnet\tools;
C:\Program Files\Nodejs;
C:\Users\[BUILDSERVER-USERNAME]\AppData\Roaming\npm

And nothing related to npm or newman in the system’s Environment Variables.

I found only this for the User PATH variable on the ‘problem server’:

C:\Users\[BUILDSERVER-USERNAME]\.dotnet\tools

And nothing related to npm or newman in that system’s Environment Variables too.

Looking at those difference I decided to try to add C:\Users\[BUILDSERVER-USERNAME]\AppData\Roaming\npm to the PATH variable for the [BUILDSERVER-USERNAME] user.

Tried the build again but it still failed with Unable to locate executable file: 'newman'. Time for a restart. That finally did the trick!

Now the build ran Newman like this:

C:\Windows\system32\cmd.exe /D /S /C “C:\Users[BUILDSERVER-USERNAME]\AppData\Roaming\npm\newman.cmd run D:\TFS\Delivery\01\119\s\Delivery\ApiTests\AangifteStatusAPI\AangifteStatusAPI.postman_collection.json -r “cli,json,junit” -n 1 -e D:\TFS\Delivery\01\119\s\Delivery\ApiTests\AangifteStatusAPI\AangifteStatusAPI.Delivery.postman_environment.json”

Instead of this:

C:\Users[BUILDSERVER-USERNAME]\AppData\Roaming\npm\node_modules\newman\bin\newman.js run D:\TFS\Delivery\01\119\s\Delivery\ApiTests\BTW2019\BTW2019.postman_collection.json -r cli,json,junit -n 1 -e D:\TFS\Delivery\01\119\s\Delivery\ApiTests\BTW2019\BTW2019.Dev03.postman_environment.json

So I was looking in the wrong place all the time: it wasn’t a problem with running .js files, it was actually a problem with the wrong command being run.

Conclusion

When the NewmanPostman_VSTS_Task tries to run AppData\Roaming\npm\node_modules\newman\bin\newman.js instead of AppData\Roaming\npm\newman.cmd ensure that the PATH environment variable for the user that the builds run on has C:\Users\[BUILDSERVER-USERNAME]\AppData\Roaming\npm in it, and reboot the server after that.

Don’t be fooled by the task executing newman.js but not running any test, and don’t go on an expedition trying to fix that!