MSTest unit tests started randomly failing
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty height:90px;width:728px;box-sizing:border-box;
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
add a comment |
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
add a comment |
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
A few days ago (11/12) our CI builds for a .NET 4.7 Windows Forms app started "partially succeeding" unexpectedly. I traced the problem back to the Visual Studio Test steps. After breaking out the unit tests into separate steps for each DLL I've been further able to isolate it to the UI test DLL.
It is a plain vanilla MSTest project where the classes test portions of the UI that can be manipulated behind the scenes. Mostly manipulating a control's data model and then inspecting details of controls to ensure that data model changes have the expected effect. UI elements are instantiated in code, either through new declarations within a class or TestClasses that are children of the controls being tested. No message boxes are presented nor UI elements actually rendered for humans or external automation to review.
These tests are running in an Azure "Hosted VS2017" environment. The test runs that fail all have exactly the same error message:
The active test run was aborted. Reason: Unhandled Exception: System.AppDomainUnloadedException: The application domain in which the thread was running has been unloaded.
I have not been able to discern a pattern with the failures. The exception can occur after different individual tests. Running the tests in isolation, in parallel, or not parallel doesn't appear to impact the failures. The failures do appear to be increasing in frequency however. All attempts to recreate this locally fail; any Visual Studio 2017 installation we've used to run the unit tests has passed them all without issue.
All I can really find on this particular topic via Google is either entries to long-past versions of Visual Studio where this was a bug that's been corrected, or to other unanswered questions. Has anyone ran into this kind of trouble before?
azure-devops mstest azure-pipelines azure-devops mstest azure-pipelines asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
asked Nov 15 '18 at 15:53
Robb BromleyRobb Bromley
112
112
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
Self-followup to document how I resolved this for my situation.
The cause: instantiating objects that derive from System.Windows.Forms.Control or System.Windows.Forms.Form without properly disposing of them.
My fix: wrapped all instantiations inside a using() block.
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
add a comment |
When a background thread is created in any test and throws an exception while other tests are still running this will crash the test run.
It's important that any code that starts threads correctly waits for these threads to finish and to handle any exceptions that may occur. Or to set up your test so that it can perform this error handling and/or clean-up.
Having background threads running while other tests execute can make your whole test suite flakey :). As you may have found out. And probably can cause problems with your production code as well.
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
add a comment |
Your Answer
StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");
StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);
else
createEditor();
);
function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);
);
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53323190%2fmstest-unit-tests-started-randomly-failing%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
Self-followup to document how I resolved this for my situation.
The cause: instantiating objects that derive from System.Windows.Forms.Control or System.Windows.Forms.Form without properly disposing of them.
My fix: wrapped all instantiations inside a using() block.
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
add a comment |
Self-followup to document how I resolved this for my situation.
The cause: instantiating objects that derive from System.Windows.Forms.Control or System.Windows.Forms.Form without properly disposing of them.
My fix: wrapped all instantiations inside a using() block.
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
add a comment |
Self-followup to document how I resolved this for my situation.
The cause: instantiating objects that derive from System.Windows.Forms.Control or System.Windows.Forms.Form without properly disposing of them.
My fix: wrapped all instantiations inside a using() block.
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
Self-followup to document how I resolved this for my situation.
The cause: instantiating objects that derive from System.Windows.Forms.Control or System.Windows.Forms.Form without properly disposing of them.
My fix: wrapped all instantiations inside a using() block.
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
answered Feb 21 at 13:35
Robb BromleyRobb Bromley
112
112
add a comment |
add a comment |
When a background thread is created in any test and throws an exception while other tests are still running this will crash the test run.
It's important that any code that starts threads correctly waits for these threads to finish and to handle any exceptions that may occur. Or to set up your test so that it can perform this error handling and/or clean-up.
Having background threads running while other tests execute can make your whole test suite flakey :). As you may have found out. And probably can cause problems with your production code as well.
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
add a comment |
When a background thread is created in any test and throws an exception while other tests are still running this will crash the test run.
It's important that any code that starts threads correctly waits for these threads to finish and to handle any exceptions that may occur. Or to set up your test so that it can perform this error handling and/or clean-up.
Having background threads running while other tests execute can make your whole test suite flakey :). As you may have found out. And probably can cause problems with your production code as well.
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
add a comment |
When a background thread is created in any test and throws an exception while other tests are still running this will crash the test run.
It's important that any code that starts threads correctly waits for these threads to finish and to handle any exceptions that may occur. Or to set up your test so that it can perform this error handling and/or clean-up.
Having background threads running while other tests execute can make your whole test suite flakey :). As you may have found out. And probably can cause problems with your production code as well.
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
When a background thread is created in any test and throws an exception while other tests are still running this will crash the test run.
It's important that any code that starts threads correctly waits for these threads to finish and to handle any exceptions that may occur. Or to set up your test so that it can perform this error handling and/or clean-up.
Having background threads running while other tests execute can make your whole test suite flakey :). As you may have found out. And probably can cause problems with your production code as well.
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
answered Feb 21 at 13:51
jessehouwingjessehouwing
70.1k11165245
70.1k11165245
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53323190%2fmstest-unit-tests-started-randomly-failing%23new-answer', 'question_page');
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function ()
StackExchange.helpers.onClickDraftSave('#login-link');
);
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
