Remove semver promise from testing API

While working on improving the text, I realized that https://www.drupal.org/core/d8-bc-policy doesn't link to the definition of semv.

TBH I just don't understand why we would do this. WebTestBase, KernelTestBase, etc. are APIs and should be treated with the same care we treat other APIs. Contrib tests aren't the only things that depend on them.

The Testing framework requires development that won't be the focus during main product development therefore aligning it to the product semver promise effectively hamstrings innovation in this idea and results in major disadvantage.

I think that's really hyperbolic. How does it hamstring innovation when we can innovate right now in the minor development branch and it is never frozen? I feel like every time one of these issues are coming up, people are forgetting that the production branch isn't supposed to innovate; it's supposed to scoot sites along with bugfixes until the next innovation-rich minor is released in six months, but there is never any barrier or freeze to innovating in the development branch. We can commit awesome new APIs to 8.2.x every day.

I can see the case for saying BrowserTestBase was supposed to be considered experimental/under development/whatever.

I really feel like this issue feeds misconceptions. 8.0.x was inconsistently restricted since summer 2013 and very restricted during the year-long beta because of the incredible technical debt, and we all have PTSD about it. But there will never be a situation like that in D8 again, ever. The development branch is never frozen, never restricted, always open for additions.

Can we please separate the breaking discussion from the feature additional discussion? IMHO we are discussion about both things at the same time ...

Just imagine a feature addition as in some helpful new assertion or what not. People would profit from it ...

people are forgetting that the production branch

IMHO the term production simply doesn't make any sense for testing. The endusers can update without any issues.

Here is my opinion. Testing is one of the places where we can still actually make people's life easier and better and speed up contrib/custom development time, and especially in the case where this discussion started, we talk about a fundamentally change. I challenge you to debug a BrowserTestBase test.
This improvement would last for 6 months, which noone can deny is a long time.

Well, I think I just don't get things. It was the case for a long time that the testing system could be improved at any point, and actually was.

Yeah, I we should at least support the main base classes IMHO.

With one exception I guess, which is kernel test base which is marked as deprecated and to be removed in 8.2.x. I think that would be nice to remove then. We did a ton of work recently to prepare for that in core.

Even changes in the production branch can be problematic for tests.

For example, we removed the field.module kernel test base class (added again) and the migrate kernel test base class (gone for real I guess). That's annoying for contrib because d.o runs tests already against 8.2.x and until recently against 8.1.x. Which means modules had no way to have tests passing for 8.0.x and 8.1.x with the same code base, without stopping to use those base classes.

Whatever we do, due to how testing works atm on d.o, I would recommend to have a BC period for testing changes in minor branches that matches how we run tests. Meaning, as long as 8.0.x is supported, it should be a possible for a module to have its test running against 8.0.x, 8.1.x (and currently also 8.2.x). Once 8.0.x is officially unsupported, we could make breaking changes there.

SO berdir is arguing for BC changes, while other people are arguing for features, nice!

For me there are good reasons to treat the testing framework differently from the rest of core.

With the rest of core, we specifically do not want to encourage people to use new APIs that we might introduce in patch releases, regardless of semver or not:

- people might be running an older patch release of core with a newer contrib release that doesn't have the method, then we'd break their site
- we don't want to commit refactoring core patches to patch releases due to risk of unintentional breakage

However for tests there are things we do want to do:

- increase test coverage of core - i.e. via writing tests with real browsers that execute javascript that run against 8.1.x - this will allow us to fix bugs in 8.1.x with more confidence that they're tested
- write tests for contributed modules, which currently have to support a single set of tests which run against two to three core branches at once
- in 99.9% of cases, we always run tests against the HEAD of core branches, so don't need to worry about incompatibilities with older patch releases if contrib starts using the newer APIs, only divergence between minor releases

The text of the policy, Automated test classes, was updated in 2018. As I read it, it meets the proposal here.

I am closing this as outdated. If that is incorrect, re-open and add a comment explaining what remains to be completed.