Testing.Platform, the new way to run .NET tests

Рет қаралды 11,437

Күн бұрын

Пікірлер: 12

@IAmFeO2x 12 күн бұрын

These are great innovations! I‘m especially thrilled about Native AOT support! Will upgrade some of my OSS projects the upcoming weekend.

@fieryscorpion 13 күн бұрын

21:35 That is pretty neat!

@ilh86 6 күн бұрын

Does hot reload run the tests that cover the changed code or does it run all tests?

@nithinb9671 12 күн бұрын

You left me with a few technical questions regarding this new framework. Probably I am asking for too much technical details. It would be great if you can answer them. 1. URL provided is pointing to MS Test and not the Test platform. I guess both are in different libraries because Test Platform is independent supporting different test frameworks. 2. What is the execution model now. VSTest supports python, typescript and other programming languages if test runners and adapters are implemented for them. This is needed for IDEs like Visual Studio Test Explorer because we can do python, typescript programming in them. Do we still have this adapter model for extensibility? Also, each runner runs independently with some socket connection or something right? Is that same here or there is no network-based execution happening 3. No reflection lookup. Does that mean that discovery of test cases in done without reflection and probably like source code parsing? I guess still you need folder scanning for this. Or do you mean about lookup for adaptors from NuGet package directories, I wonder why that was being done because I expect dotnet to pick/load DLL based on Test framework NuGet version provide in csproj file. Isn't it? 4. Static extension registration, so here it will avoid scanning for extensions like coverage and output logging and load based on extensions provided in csproj file. Now we should provide extensions from command prompt which is moved to csproj file. But what do you mean by static registration. I think you still need to scan for the extensions using reflection to initialize them, right? 5. Custom assembly resolver. Not sure what do you mean by that. Some background would have been great.

@nohwnd5914 12 күн бұрын

> Probably I am asking for too much technical details. It would be great if you can answer them. Not at all, the talk was only skimming the surface :) 1. URL provided is pointing to MS Test and not the Test platform. I guess both are in different libraries because Test Platform is independent supporting different test frameworks. Yes, the url leads to microsoft/testfx repository, where both the new platform and MSTest test framework live. The platform does support multiple test frameworks, MSTest among them. 2. What is the execution model now. VSTest supports python, typescript and other programming languages if test runners and adapters are implemented for them. This is needed for IDEs like Visual Studio Test Explorer because we can do python, typescript programming in them. The execution model is that the tests are some kind of executable, either one that runs by itself, or one that interprets a file. Right now we are focusing on implementing this for .NET, but we do have a quite simple json protocol above, that might eventually allow for example JS tests to run, when executed e.g. via Node, or maybe via browser. But this is not our main focus right now. In VSTest there are actually two levels of extensibility, test runtime provider and test adapter. Most test executors opt into the second one, because it is easier to implement. But it also means that a .NET Testhost is started first, loading a dll with the adapater, and then the test execution is delegated to the actual test runner e.g. Node, that will communicate back over some one off protocol to the test host. The new model cuts out this middle man, and would allow Node to talk directly back to VS. Rather than VS to vstest.console to testhost to node. > Do we still have this adapter model for extensibility? In theory yes, for backward compatibility and for ease of onboarding nunit, mstest and xunit, we are still hosting the frameworks in an adapter layer, but this is not the desired state, because it requires us to carry over dependencies and behaviors of VSTest (for example the ObjectModel.dll). So in reality no. > Also, each runner runs independently with some socket connection or something right? Is that same here or there is no network-based execution happening That depends on the execution, when running the exe there is no connection, when running under VS we talk over tcpip via local host, when running in dotnet test we talk over pipe. These transports might change, but this is how it is now. The difference from vstest is that we don't have vstest.console in the middle, but rather we talk with VS directly, so there is one less serializatio and deserialization od data. 3. No reflection lookup. > Does that mean that discovery of test cases in done without reflection and probably like source code parsing? I guess still you need folder scanning for this. No, we do still use reflection to lookup tests in the assembly. It is probably faster that you might expect. Generating the same code via source gen takes a lot more time to do in compilation. We do not scan folders for this, we only ask the assembly for all its types (and methods etc.) via reflection. > Or do you mean about lookup for adaptors from NuGet package directories, I wonder why that was being done because I expect dotnet to pick/load DLL based on Test framework NuGet version provide in csproj file. Isn't it? I did mean this in the talk. When nuget with extension, e.g. MSTest test framework is installed, it will add msbuild item to the build, that says which type in this assembly is holding the extension "hook". The hook is simply a type that has a well known method "AddExtension". This item is picked up by a target in the Extensions.MSBuild package we ship, and eventually will generate the Program.cs and add it to compilation. Think about it as poor mans source generator (because we need to support also VB and F#). (If you build with binlog you can see all the details of what happens and what code we write.) For MSTest we will call this AddExtension, that wil call something like testApp.AddMSTest(() => Assembly.GetCurrentAssembly()), to register MSTest into the test app, and to point it at the current assembly so it knows where to look for tests. Similarly for trx we would write test testApp.AddTrx(), and that would add TRX extension to the run. Notice that we only register dlls that are installed via nuget, we have them referenced because they are installed via nuget, and we refer directly to them because we have the exact type. This avoids searching for all .TestAdapter.dll and finding all "IExtension" implementations in them. The folder scanning is not bad by itself, what is bad is that for VSTest the TestExplorer, AzureDevops and other runners include huge amounts of dlls into the run throgh --additional-test-adapter. e.g. by adding / of the repo as a place to search for extension dlls, this leads to test adapters of different versions being used in incorrect places, version mismatches etc. All of this is avoided by the approach above, and we always know what exactly we are referencing. 4. Static extension registration, so here it will avoid scanning for extensions like coverage and output logging and load based on extensions provided in csproj file. Now we should provide extensions from command prompt which is moved to csproj file. But what do you mean by static registration. I think you still need to scan for the extensions using reflection to initialize them, right? I am not sure what is being asked, apart from the last sentence, which I hopefully answered above. Please clarify the question. 5. Custom assembly resolver. Not sure what do you mean by that. Some background would have been great. My bad for not going deeper here. This is also related to the dynamic extension registration that VSTest does, because we can add additional paths to extensions, we need to be able to resolve their dependencies e.g. C:/MSTest.TestAdapter would hold MSTest.TestAdapter, and MSTest.TestAdapter.Utilities.dll. But my tests would be in C:/MyTests/MyTests.dll. I somehow need to tell to .NET to also look for dlls in C:/MyTests where my test dll resides but also in C:/MSTest.TestAdapter. This is done by a custom implementation of assembly resolver (actually there are 2, one for VStest, and another for MStest). Now multiply this by 100 extension paths and add in the mix that many of the dlls will have the same dependencies but with different versions etc. And you will have a typical VSTest test run. This part is one of the biggest sources of test crashes in VStest. In the new test platform we don't add a custom assembly resolver, we just rely on the fact that all extensions should come via nugets, and so their conflicts will be resolved by nuget (following nuget rules), copied to bin via msbuild (following their rules, and providing great tooling like binlog) and loaded by runtime (following runtime rules). Making it way more stable, and way more deterministic how tests will run.

@nithinb9671 11 күн бұрын

@@nohwnd5914 Thanks you for clarifying the questions by taking time.