[meta] Make sure Drupal 8 looks good and works right on browsers (both mobile+desktop)

Related

Actually if this is critical, it blocks a release candidate anyway. Untagging.

Just found out after testing extensively that CKEditor is incompatible with Android browsers. Here is some more info https://dev.ckeditor.com/ticket/10380
Quoting from that ticket:
"As explained on our compatibility page, Android support is, sadly, not available yet. It's not that we do not want to provide it, the problem is with Google not implementing some vital features of contenteditable support :( Since this is the case, we had to blacklist Android in our env.js file. However, if you want to try it out, you can uncomment the blacklisting code and see for yourself if the level of support that Android browser provides is OK for you, because some features work quite all right already.

You can see the list of open Android bugs by using the ​Android keyword."

http://ckeditor.com/support/faq/features#question9

Actually, let's switch this around, since there's nothing specific to patch here and bugs discovered as part community testing will be critical, major, normal, etc. in their own right, and this tracking issue does not itself block release.

#4 sounds like an explicit issue should be filed for that if we confirm it.

What is the list of mobile browsers and devices to be supported in relation to 8.0.0-rc1? This issue currently sounds (I assume) we are considering all core themes and modules providing visibility and usability must be mobile ready.

@stevesmename there's no official list, #2390621: [policy, no patch] Update Drupal's browser support policy would be the issue to decide that, but it hasn't had a post for six months. However for this issue we don't necessarily need a full list - whatever we support, the list of what's going to be pro-actively tested is going to be a lot shorter.

(I assume) we are considering all core themes and modules providing visibility and usability must be mobile ready.

This issue #2289619: Add a new framework base theme to Drupal core would make this task way easier :) Don't reinvent the wheel!

#11: That is an entirely separate issue. We are definitely not adding a new framework prior to 8.0.x, and that would not affect either Bartik or Seven, both of which we need to test now, prior to RC1.

While I agree that #2289619 is not related to this issue specifically and definitely won't happen in 8.0.x, #11 it is related conceptually and still a valid point.

I think @giorgio79 was just trying to bring to people's attention to the fact that we could be doing things much better than the manual "front end testing" process we have now. That's all. I encourage people to read my blog post (and its comments) found in that issue.

edit: just trying to "soften the blow" of that issue being mentioned. I don't think the intent was meant to derail this issue but rather meant more as an "FYI".

Before we test anything, we need to decide:

1) What are we testing?

2) Where are we testing it?

3) What are our criteria for something that needs fixing?

What are we testing

So, first things first, I would suggest that we focus on testing Bartik and Seven. Those are the only core themes with user interfaces. We'll need to narrow that down further though.

- Do we test every possible admin interface? Or a select few? Do we need to test UI for every enabled core module in the standard profile? What about modules that are not enabled by default?
- Do we also need to test every possible type of content that can appear based on the site using core's enabled modules? Core's optional modules?
- Depending on how many things we need to test, can we set up a stable build for testing use? With sample content? That would make it much easier for multiple people to test without having to set things up themselves and possibly get inconsistent results. Maybe simplytest.me or something similar could be used for this?

There are a lot of potential things to test! Lots of UI elements, lots of possible combinations of configurations that could cause configuration. We would need a detailed list of what needs testing. My guess is that this will be a very long list.

Where are we testing it?

Since Drupal is used worldwide, we need to test it based on worldwide browser usage data.

The two sources I found with the most detailed info on current worldwide mobile and tablet browser data are the following:
- http://gs.statcounter.com/#mobile+tablet-browser-ww-monthly-201505-20150...
- http://www.netmarketshare.com/browser-market-share.aspx?qprid=2&qpcustom...

I would suggest focusing primarily on browsers with greater than 1% usage. Below 1%, you quickly get into the weeds. Those people are important, but usages below 1% are likely to be obscure browser with extremely challenging fixes.

Stat Counter identifies the following usage statistics on mobile/tablet from May 2015 to July 2015:
- Chrome (31%)
- Safari (25%)
- Android (16%)
- UC Browser (13%... I had never heard of this, but apparently popular in China and India)
- Opera (10%)
- IE Mobile (1.8%)

Below that you get Blackberry (0.8%), Nokia (0.75%), Firefox (0.4%), Silk (0.36%... Amazon Fire/Kindles)

Net Market Share has some more helpful numbers on particular browser versions, which is key.
- Safari 8 (24.8%)
- Safari 600 (7.7%... what the heck is this?)
- Safari 7 (5.1%)
- Safari 6 (0.81%)
- Safari 5.1 (0.81%)

- Chrome 43 (14.4%)
- Chrome 30 (3.5%)
- Chrome 28 (3.3%)
- Chrome 34 (3%)
- Chrome 38 (2.2%)
- Chrome 39 (1.25%)
- Chrome 33 (1%)
- Chrome 42 (1%)

- Android Browser 4.0 (12.5%)

- IE11 Mobile (1.8%)
- IE10 Mobile (0.5%)

- Opera Mini 7.5 (1.6%)
- Opera Mini 7.6 (0.85%)
- Opera Mini 4.5 (0.8%)
- Opera Mini 8 (0.62%)

- UC Browser (1%)

So... that's terrifying.

Here are some browsers we should probably test in:
- Safari 8
- Safari 7
- latest Chrome
- Android 4... my memory is 4.4 is the most popular version of Android 4, but I could be wrong
- Opera Mini (maybe the latest version?)
- Maybe IE11

Ideally we would do these tests on actual devices, not simulators, which reveals things that might not be apparent from a simulator. Particularly for Safari and Chrome, we should test on both phones and tablets, as the different dimensions could uncover problems.

What are our criteria for something that needs fixing?

Surprise! Drupal is not going to look the same in all of these browsers. There will very likely be loads of small differences. The question is which do we absolutely need to fix. In descending order:

- Something absolutely does not work in a certain browser. Ever.
- Something doesn't work very well in a particular browser and is annoying.
- Something looks significantly visually off in a particular browser, but still works okay.
- Something looks a tiny bit off and works just fine.

The first item is definitely a problem. The second would be good to fix. The third would be nice to fix. The fourth, ain't nobody got time for that.

From my experience, there are quite a number of weird things that may crop up with mobile Safari. Particularly if we are animating anything, as that will freeze while scrolling which can lead to strange results.

Opera Mini is popular in areas with low bandwidth because it does loads of server proxy caching, which means that lots of dynamic JS things don't work in Opera Mini. I would expect a lot of our behavioral stuff will break.

Ideally, we'd also be testing in low bandwidth situations, because that's very important worldwide on mobile. We've done a lot of backend caching work in Drupal 8, but front-end performance is an entirely different matter and usually has an order of magnitude more difference on site speed (if the backend is optimized).

Ideally, we'd also test to see how Drupal works with JS turned off. Not because very many people actively turn off JS, but because sometimes there will be a glitch, either in the network or in something that is loaded arbitrarily from an external source, and suddenly some or all of the JS fails. A recent Pew study pegged this at 15% of traffic, which seems really high to me, but even something as modest as 2-5% of traffic would be huge.

And ideally, we'd also test for a wide variety of accessibility situations, which probably has an even bigger impact than some of these small traffic mobile browsers. There may well be accessibility testing that's being done, in which case, kudos.

And as much as we care about mobile browsers (which is important!), ideally we should also be testing our UI in desktop/laptop browsers as well. That's still probably around 50% of usage and is also important.

Here's the part where I'm very doubtful that we can come up with a list of 1) things to test, 2) places to test it, and 3) actually fix the things we've found in a way that doesn't severely delay Drupal's release.

Particularly because our markup and front-end assets aren't frozen until RC1, but we can't make changes to those items after RC1. So how do we test, make changes, without knowing that other changes might cause regressions?

All of this points to the fact that for Drupal 9, one of our key goals should be building a visual regression and behavioral regression testing infrastructure into d.o. Even if we had that, though, that likely wouldn't ensure no regressions in the wide variety of popular browsers, across devices.

The real challenge is that we're almost at the point where front-end assets are going to be locked down... for five years or so. My understanding is that Drupal's HTML, CSS and JS is unable to be changed once we hit RC1. At the very minimum, we won't hit Drupal 8.3 until 2018, and while the talk is that it won't take as long to do Drupal 9, I'd put good money we won't see it until 2020. That's a long time in the front-end world, a very long time. So... this is really our only chance to fix problems. And there are a whole slew of core developers who will likely be irate if the release drags out longer due to browser bugs.

Wish us all well!

Thanks for the detailed notes, webchick!

I think the only thing I'd add to that is that we should probably test on recent Safari desktop as well. Definitely significant enough that I wouldn't leave it off the list.

When I am more conscious tomorrow, I'll update the issue summary.

+1 on testing on opera mini
+1 on testing on slow/high latency connections

+1 to proxy browsers #2119299: Make sure Drupal 10 works with mobile proxy browsers

This seems to be growing beyond test mobile browsers to test everything? Marc's browser usage numbers are just mobile. We'd obviously see a different landscape if desktop was included.

As far as what to test, I'd focus on anonymous display of various content, (making sure fields, views, etc, come out right,) test breakpoint functionality, RTL text, content editing experience all around, and things a site visitor might do outside of just viewing, like commenting. I don't think we need a huge amount of testing for most administrative functions. I'm not convinced anyone will spend significant time editing content types or views on their phone. Yes, do some first level testing, but our limited hours are best put elsewhere. We aren't going to lose sleep if there is bug in editing taxonomy filters on your phone, but if you can't insert an image into a node mobile editing might be useless.

I sent an email to a few people about organizing a New Jersey sprint to do some of this testing before Barcelona. I'll how that goes.

In the process of updating the issue summary. More later.

Initial testing plan finished. We still need to define key testing tasks and feedback is welcome.

Definitely include anything with an autocomplete. I have no idea how well that does or does not work on mobile.

Updated the issue summary with a set of testing tasks.

Printed off testing instructions so I can test while on a road trip (as passenger not driver of course!). We'll see how Drupal 8 does living on the EDGE!

@mrdrummond: do you mind sharing those instructions? I'm trying to get some of my QA guys to run through Drupal 8 on all of the mobile devices we have but they have very little knowledge of the inner workings of Drupal so any help we can give them in terms of numbered steps/instructions would be great.

@rtraveller: Testing instructions are in the issue summary. Thanks for helping!