Add more telemetry to Drupal core

It'd be wonderful if as part of this, we'd get usage statistics per core module (IOW: a breakdown of https://www.drupal.org/project/usage/drupal). It'd allow us to see, for example:

• how many sites are using experimental modules, and hence estimate how well they have been tested
• which modules probably belong in contrib rather than in core

AFAIK this data is already gathered, just not exposed! So perhaps this can be a first quick win? :)

I think #1627676: Display stats on enabled components (e.g. modules included in a project) is the issue for #3.

Thanks, @effulgentsia!

I love this idea. More telemetry helps a lot.

One thing I've seen people doing in my $dayjob, which is extremely valueable, is catching exceptions and logging them over time. This seems like a small thing, but it allows us to catch bugs, before actual users potentially start to see them.

Another nice thing would be data around overriding PHP classes, usage of APIs etc. This way potential minor BC breaks could be much better evaluated and justified (aka. We know it will just break 0.5% of all sites potentially)

Making all of this telemetry sending opt-outable.

Why do we think this is the right idea? Do we value the project as more important as the right/opinions of the actual users of the project?

Making sure that headless sites can be identified as Drupal sites underneath by crawlers, and by the d.o metrics system

I think the only way which is remotely achievable is: Enable a way for sites to identify themselves as Drupal site.

"One thing I've seen people doing in my $dayjob, which is extremely valueable, is catching exceptions and logging them over time"

This is part of my day job too....Reviewing high crash rate events for a multi million install app.

Seeing and auto reporting Drupal core / module bugs, eg collecting data how many sites reported that bug would be an immense quality tool. Eg seeing that a php exception occurred 4566 times the past day for 2502 Drupal sites... Error reporting could be completely automated this way. Big +++++

Why do we think this is the right idea? Do we value the project as more important as the right/opinions of the actual users of the project?

I dont think it's as clear cut as saying 'opt-out means we dont value users rights and opinions, and opt-in does'.

On one end of the spectrum, we are going to have a percentage of users for whom any data sharing at all is either non-compliant (HPAA), or seen as an invasion of privacy, a potential security risk, and they will not want to share data. We absolutely want to support these users wishes, we don't want to use any dark patterns to trick them into sending data or anything of that sort.

On the other end of the spectrum we have a percentage of people who recognize the value of sending us data, who work on the project as part of their $dayjob, and understand how much value sending us this data will be, how it can help guide product decisions, priorities, and help drupal as a whole.

Finally, in between, we are going to have a third group of people, who don't particularly care if they send us data, but aren't going to go out of their way to do it either. If I were to speculate, I would say this is the *largest* group of people by far. Probably 90% of our users would fall into this category.

So, deciding how something like this is enabled, or disabled, and presented to the user definitely needs to take all those groups into consideration.

I personally think we should err on the side of opt-in by default, with an obvious, prominent way to opt out. This seems like it is pretty much a solved pattern.. I just reinstalled my laptop today, and I was asked by OSX, by PHPStorm, by Chrome, and by Iterm all as part of the installation process if I wanted to send telemetry data. I let phpstorm and iterm have my data, because I like them, and trust them, and didnt want to give it to apple and google. I appreciated having the ability to choose.

If this ends up being some setting buried in the admin interface that somebody has to turn on, I think we're going to end up with poor, invalid data in that case, which would defeat the purpose of creating this, we'll end up with a self selected sample of just the people who care enough to turn it on, and then end up making bad decisions based on incomplete data.

This seems like it is pretty much a solved pattern.. I just reinstalled my laptop today, and I was asked by OSX, by PHPStorm, by Chrome, and by Iterm all as part of the installation process if I wanted to send telemetry data.

+1

Hi there! Very interested in this initiative, I think this is an essential way of knowing how Drupal is used.

Just a few questions:

Making sure that headless sites can be identified as Drupal sites underneath by crawlers, and by the d.o metrics system

In what way are headless Drupal sites different then traditional installations in terms of sending telemetry? And why do we need crawlers for sending telemetry data to Drupal?

Also, this post mainly talks about core usage and modules, but I would like it very much to have more in-depth data about other modules too. So, is this initiative just about core modules or is data about contrib modules also being sent? It would be great for module maintainers to see more data about what types of users are using their modules.

Ultimately this initiative is about providing a framework for gathering data from installed sites. *what, exactly* that data is, is likely to change over time, but this is just the first step - being able to gather different data points as our needs dicate.

In what way are headless Drupal sites different then traditional installations in terms of sending telemetry? And why do we need crawlers for sending telemetry data to Drupal?

We dont need crawlers for telemetry data, thats a reference to the way that much of that data is gathered now. Tools like BuiltWith cannot know that drupal is under the hood on sites like weather.com because it's headless, so drupal gets underreported in those sorts of scenarios.

if they are sending telemetry then we dont have to worry about whether they are headless or not.

Opt-out seems like the perfect balance between the needs of the project and the needs of the minority for whom telemetry is a problem.

The data from this would be invaluable when helping teams architect their Drupal sites. For example, I've worked with teams who insist on running ancient versions of PHP and never upgrade. Being able to show that "90% of Drupal 8 sites run PHP 7.0+" or similar would help highlight the risk of running less-well-tested and potentially insecure PHP versions.

The experimental module use case is a particularly good one - a core experimental module with tens of thousands of sites probably has more real-world testing behind it than a contrib module with 5000 installs that we don't worry about enabling.

I do wonder about CI builds polluting stats. For example, if a site runs Behat tests on every commit, wouldn't those show up as unique "sites". There are probably some good technical reasons (beyond security or privacy perceptions) to make it easy to opt-out.

Because I like to keep this initiative going, let me do a proposal.

Architecture

I think the following things are needed to get valuable insights for the Drupal project:

1. Drupal Telemetry client(s) - A Drupal module that sends data to the telemetry server
2. Drupal Telementry server - A place where all telemetry data is stored
3. Drupal Telemetry front-end - Makes all data visually available

Telemetry client

The telemetry client should be a Drupal module (for at least Drupal 8 but maybe also for Drupal 7) that is able to send data at set times to the telemetry server. This could start as a contrib module, that gets ported to core once the module is tested well and advanced to the point that everyone can use it without problems. The core module should be enabled by default (with an update hook) but users can opt out by disabling the module. Of course, developers should be notified that the module is turned on so no data is sent without the developers knowing. Also information should be shown on what data is sent to the telemetry server, and what is going to be sent the next ping. This way, there is full transparency on what data is collected.

Telemetry server

The telemetry server can be a Drupal 8 installation that receives data from all clients, and saves it in a database. Data should be stored structured, so maybe webforms or just custom entities should be used for that. There should be an API endpoint for posting this data to the server. Also, there should be APIs for requesting data from the server.

Telemetry front-end (stats)

To actually see the telemetry data there should be a place where people can view the data gathered on the server. Of course this could be on the telemetry server itself, but this is also a perfect opportunity to show that Drupal can be used for decoupled applications. So, we can create a React application that requests data from the telemetry server and displays them in an intuitive and visually appealing way.

Communication between clients and server

To send data from the client to the server, a REST endpoint on the server should be used. At a set interval (for example, every week) the client should gather all the information and should try to send it to the server. If the server isn't available at the moment or the client has trouble to connect to the internet somehow, the data should be stored on the client. Then, after set time, the client should retry to send the data to the server. Because the original data is sent (with the timestamp of when the data was gathered) even when the server is unavailable for a day no data will be lost. The client will just insert the data retroactively.

Every ping should get a random uuid, so that the server can reject / deduplicate data when the client sends the same ping twice. The client should keep retrying until it receives a successful response code from the server. The server should only sent a successful response code when it successfully received all data.

Server read API / data that will be public

I think that as much data as possible should be available trough the API of the server. This way it can be used to generate a very specific set of data that can be used for specific insights. Also it's very transparent if all data is available to everyone, instead of only a specific amount of people.

What data will be sent to the server

I think the general rule here is to never send data that could be traced back to an individual site. So never the url of the website, for example. To get a better idea where Drupal is used for, we can send this data for example:
- What core modules are enabled
- What contrib modules are enabled
- How many custom modules are enabled
- How many nodes are created
- How many users the site has
- PHP version / Composer version / Database system & version
- Type of hosting (SaaS, containerized, managed)
- Way of installation (manually by downloading the .zip / installation with composer)
- Number of visitors
- Ratio API requests / 'traditional' requests (to detect headless websites)
- Is this a multisite?
- ...?

I hope that in the future, contrib modules can send their own metrics. For example, that we can see how many webforms are created in the webform module.

Unanswered questions

There should be a way to know which pings are from the same site. We could always send the site UUID with every ping, but I don't know if the site UUID is secret or if there is a better way to differentiate sites. Also, we should be able to automatically disable sending data to the server on acceptance and development sites to not pollute the data. So we should make a reliable way to differentiate between environments.

Steps to proceed

MVP

1. Discuss above points and agree on MVP
2A. Start working on a contrib module that collects data and sends it to a server. The data that is sent doesn't have to be extendable on this point.
2B. Start working on a Drupal 8 installation that collects this data.
3. If both work, we can start on porting the contrib module to core.
4. At the same time, we can start working on creating a nice React front-end that can visualize the data and give handy insights.

After MVP

It would be very nice if the data can be extended trough plugins, so contrib modules can send their own data. This also means that they should be able to login to the server and create their own metric fields that they will send. This way, the server knows what fields to expect.

Also, we should be able to use the Read API's from the server to show more data on the project pages. Think of things like 'This project is mostly used in combination with these projects'.

And maybe far in the future, we could start asking users little questions like 'Do you like the content editor experience in Drupal?' where users can give a rating. By combining that data with all other data from these sites, we can correlate this data and see which modules really improve the experience for people and what use-cases need the most improvements.

Discuss!

This is just my take on this initiative, and I would like to hear what you think of it. Let's discuss and get this initiative started!

I'm at Drupal Europe this week, so if you want to discuss in 'real life', you can contact me on Slack :-)

Meanwhile there's ongoing work happens in moving .inc files to classes makes sense to organize classes to read Requirements/Php/Memory in consistent way

Also, we should be able to automatically disable sending data to the server on acceptance and development sites to not pollute the data. So we should make a reliable way to differentiate between environments.

IMHO, that data (from non-prod) sites should be reported too, in order to have e.g. info about what modules are used for development/testing, etc.

The initiative was mentioned by dick olson as part of his drupalcon Amsterdam presentation:
https://youtu.be/IygX5-34__M

I don't agree that opt-out is a good way to go. Companies often go that way with the rationalization that "it's anonymous so it's okay", but often the data can be re-associated to users and they can still be tracked. Especially when those companies also start selling the data, which gets bought up and combined with other purchased data to further correlate data to users. Then the data inevitably gets leaked somehow.

I don't even think telemetry is necessary at all for Drupal to effectively serve its users.

The 90% of users who "don't care" are probably so because they are in the "don't know" category. Those who don't know they are being tracked are still dehumanized by the treatment of them as points of data rather than as humans with their own lives, thoughts, and feelings. People, Drupal included, need to stop assuming that tracking users without their consent and collecting data is okay just because "there's a way to do so and you can improve your service somehow". Just because 90% haven't told you otherwise, doesn't give you a good reason to go ahead collect the data anyway.

Make it opt-in, or no telemetry.

I don't even think telemetry is necessary at all for Drupal to effectively serve its users.

As a product manager, I'll respectfully disagree. :)

I definitely DO agree that telemetry/activity reporting needs to be handled with care... but the absence of access to usage data causes all manner of problems, including:

• Lack of clarity over the end user impact of deprecating dependencies, or requiring new versions of PHP, MySQL, etc. We might be screwing 70% of our user base in Drupal 10 if we were to require PHP 8, or we might be screwing over a tiny handful, and the benefits would far outweigh the negatives. We have absolutely no idea, so we can't make a decision.
• Endless deadlocks over whether anyone actually uses [insert module/feature here] when it comes to proposing those things for addition/deletion to core (e.g. Color module, Forum module, Book module, etc.). While it's possible now I think if the DA does some manual work to determine if a particular core module is turned on or not, we don't know if they're using, say, Forum Topic Shadowing or whether that's something we could cleave off to further simplify core.
• An utter impossibility to measure whether steps we took to mitigate a product defect actually worked. We added Layout Builder module to core. Cool. Does anyone actually use it? In what ways are they using it? Are they succeeding or failing at it? Where are their pain points? etc. Determining this currently requires the random whim of some major benefactor (company, university) to invest a crap-ton of time and effort in a round of usability testing, and as a result we've only done that a handful of times over the course of Drupal's 20+ year lifecycle. Other comparable products in our space are answering questions like this within minutes, in some cases (e.g. SaaS products such as wordpress.com or SquareSpace).
• We also aren't able to perform common best practices such as A/B testing to validate development/UI approaches. It's possible to ship an experimental module that offers a couple of different approaches, and compare metrics between them to see which fares best, and ship that one. In absence of this capability, we instead guess, and build and build and build and finally ship, and then blindly (literally; since we can't measure anything) hope it works out.
• And finally, prioritization. This again tends to happen ad-hoc, either because someone randomly chose to scratch an itch, or because Dries did a once-a-release-cycle community-wide survey to figure out what people wanted. If we had better (or any ;)) metrics, this would make it much easier to ensure end user pain points were being addressed outside of these feedback cycles.

Without any actual data to fall back on, many product discussions basically end up being an endless round of "who can argue for their position the loudest" among a tiny handful of people with Drupal.org accounts who happen to know about issue # 12345. Not only does this burn out core contributors, it also is not at all serving our actual users, the vast majority of whom do not have Drupal.org accounts, nor even know these discussions are happening at all. :\

So while there are valid privacy concerns, I think there are also valid inclusion concerns, and making sure our product decisions are representative of our actual user base and not just those who build Drupal. As well as competitive concerns, since we're not currently able to follow any product management best practices with respect to measuring how our improvements do, and learning from those insights to drive better improvements going forward, and others in this space definitely are.

Here's an idea I thought of that Drupal could perhaps use. This could still be a module, but it would prompt the admin to submit the data. So even if the module is in core and enabled by default, it requires a user action to send the telemetry data.

Something similar I've seen is how Steam (the gaming platform/store) sends a monthly Steam Hardware & Software Survey to users. When agreeing to submit data, Steam will gather all of the data it wants, and then it will show the user the file/data it will send (so that the user can once again see with complete transparency what is sent, and verify that no personal/private information is included), and then they can agree and send the survey data.

I think the way Steam does this is an acceptable strategy, because the user gets to see exactly what is sent first.

Agreed that something like that could be a nice balance between the concerns; thanks for being open to discussion!

Copied from slack:

@xjm re "update stats is effd" is there anything we can do to help with that? As in, perhaps we can modify what the update module does so that it is more acceptable to leave it enabled permanently?

I know I had switched it off on my D8 site for a couple of years because I didn't want unhelpful messages about updates that I was managing in my own way.

Perhaps update could be modified thus:

1. As many services do now, ask for permission about passing on improvement data to drupal org;
2. Be clear about exactly what info is exposed and what is stored;
3. Provide a way to 'hide' modules from update, so it will never mention them in any update. Perhaps by default it could be set to only ask about modules in a core/ or modules/contrib/ folder?
4. Finally, provide better fine-grained control about what it does: is it just checking in with d.o, is it sending emails, is it posting on status page, etc.

@Hestenet replied:
Yeah - those comments re update stats improvements would be welcome. We have some architectural thoughts about it as well. @mixologic (out sick) has worked out some of those ideas - but as xjm said, it hasn't bubbled to the top of the list yet.

@xjm replied:
I think the initiative scope should include a lot of what you mentioned. The first step would be to move telemetry outside update.module. There'd be config allowing you to opt out, but default would be opt-in.

As I see some legal stuff was not mentioned before:
1. There needs to be a page on d.o about "Data usage policy for data collected from Drupal websites"
2. Drupal core has to provide an option to see what is exactly sent to d.o (telemetry server)

copied from #3199472: Improved data sharing with d.o

Drupal's Cookbook

If we are going to do this, then I am wondering if we could do just a little bit more. A lot of this information can also be used to create a Drupal cookbook for (beginner) sitebuilders, themers and/or developers. To be really useful a bit more information is necessary. When we add an opt-in form in the administrator section of a site, where the sitebuilder add information about the site, how it is build and what is special about it (free text). In combination with the already collected data about the site, can be put on a page in lets say drupal.org/cookbook or cookbook.drupal.org. Learning how to build a great Drupal site or learning to create great modules and/or themes is not easy, and seeing how it is done on professional build sites is a huge help. Not all (beginner) sitebuilder, themers and/or developers have the luxury of having a senior Drupal person to ask questions about Drupal. It is great for the ones that have such a person, only that luxury is not for everybody.

Creating all that information might be a lot of work for a sitebuilder, only I think that for most professional build sites this information will already be documented somewhere for clients, support and/or management. Therefor I think we will get a lot of cookbook entries without being it a lot of work for sitebuilders.

In the form we can also add who has build the site (one or more Drupal users accounts) and organizations (one or more Drupal organizations accounts). When a (beginner) sitebuilder, themer and/or developer likes a cookbook entry, they can give the entry a star, same as Github. The persons and organizations get on their profile the number of cookbook stars they received. When you have a lot of cookbook star you must be doing something right and that might/will attract more work, better paid work, more interesting work or just a bit of fame in the Drupal world.

An added benefit might be that a lot more persons will get a Drupal user/organization account.

Cookbooks could be wrapped into help topics or tours so exportable

Just asking... re: https://www.drupal.org/project/ideas/issues/2940737#comment-14170618 is there a project for a Drupal Cookbook?

We have a local user group working on collecting Drupal Recipes which could also benefit from information collected with this initiative. I am also following this initiative as I am working with a site-builder committee working on the project browser initiative.

I was looking precisely for such an initiative! 🙋‍♂️Actually started a discussion a few days ago to see if there was some interests in gathering information about Drupal, users, usage...etc.

I am interested in contributing to Telemetry initiative. IMHO I think we all agree that more data about usage will help everybody to build better modules, better UX. It seems telemetry is a viable solution to gather data AND to respect privacy, performance and all legitimate concerns in-between.

Now the question is: How should/can we proceed to move things forward?

I'm afraid that there are too many areas of investigation detailed here.

Shall we/Can we start building something more specific?

For instance, limit the telemetry to one area such as :

• Usage of core modules
• Technical info (env, server...)

I would +1 an implementation like a telemetry module in core with plugins to enable/disable tracking usage per configuration.

In my understanding, the primary purpose of the telemetry phenomenon is to answer questions. The nature of these questions could be more focused if you are an individual IT company developing one software product (e.g. a browser maker like Mozilla measures Firefox's usage metrics).

However, the situation is absolutely different for the Drupal Community. As consisting of such a diverse mix of interest groups it can be challenging to agree on the set of questions we want to answer.

Here are some examples to better get my point:

• Core maintainers would want to know:
  
  

  “Which one is the least underutilized core module we should move to move out from core in the future?”

• Dev-oriented module (e.g. CTools) maintainers would want to know:
  
  

  “How many times is my module is installed as a dependency?”

• End-user facing module (e.g. Webform) maintainers would want to know:
  
  

  “How many times is a public page rendered by my module?”

• Localization translators would want to know:
  
  

  “Which languages are enabled on a Production site?”

Obviously, we cannot answer every need. So in my opinion this initiative could start off by channeling its efforts towards a common agreement of the audience (or a prioritization between them, if several) which we plan to serve with meaningful insights.

I landed here based off a comment from XJM from 3 years ago :) seem like things are still in the discussion phase?

I do think this is of great value. I landed here because of "weighted" credits which are calculated based on module installs, which we all know currently is not a very accurate representation.

I think the list in #17 is a great start, although I don't think everything is of as much value as other items. Below my comments on some of the items.

- How many custom modules are enabled -> What value does it add to know how many custom modules there are?
- PHP version / Composer version / Database system & version -> Composer version is irrelevant I think, because it will probably not be installed on many *modern* environments. (Especially when you deploy artefacts). I also think with only 2 big versions available and V1 being EOL for almost 2 years it doesn't add a lot of value.
- Type of hosting (SaaS, containerized, managed) -> How do we plan on detecting that?
- Way of installation (manually by downloading the .zip / installation with composer) -> Is this still relevant option?
- Number of visitors -> Difficult to detect if the statistics module is not enabled I guess?
- Ratio API requests / 'traditional' requests (to detect headless websites) -> Is this currently being logged in some way? I think it would be great if there is some way of detecting headless sites!
- Is this a multisite?

What contrib modules are enabled

This one would be very important to get a more accurate view for module maintainers on the usage of their module. I think it would be great for module maintainers to see a combination of the data. E.g version usage & PHP versions combined for example. Some advanced charts that could be filtered on would be awesome, but I guess I'm running a bit fast here ;-) those changes could only be added anyway after phase 1 where Telemetry is a thing and it has been released and data has been gathered.

Personally I'm fine with it being an opt-out setting somewhere in the settings screen. If it really get's too much negative feedback or something I guess we can also add a step for it in the install process of a new Drupal site.

So for me an updated list of things to send would become (removed inline comments and added some extra):
- Drupal Core Version
- What distribution is used (if any)
- What core modules are enabled
- What contrib modules are installed/enabled
- Which themes are installed/enabled
- How many nodes are created
- How many users the site has
- PHP version
- Database system + version + maybe size?
- Ratio API requests / 'traditional' requests (to detect headless websites)
- Is this a multisite?
- What country? (Based on domain name/hosting location/...)
- Amount of installed/enabled languages + the default one
- Hosting architecture (ARM/AMD, Linux flavour,...)
- ...

Feel free to re-add, remove, discuss,... :)

Also happy to join this initiative to actually put some elbow grease into coding when we come to that stage!

It might also be relevant to know which PHP extensions are installed, though I'm not sure if that's beneficial to anyone or not.

How about Apache vs Nginx vs other?

@solideogloria I had PHP extensions written down but removed it because the list would maybe get way too long? Also not really sure to what extend it would be meaningful to know such thing. I think it would add more value if we would know the php settings like memory limits and stuff.

+1 for Apache & Nginx + version

This might land under contrib, but what about which search engine is used? Solr/Elastic + version for example? I guess there are not that many sites anymore that don't use search :)

This could probably be its own issue later, but just wondering, some of this information could reveal sites with security vulnerabilities or available updates for PHP, database, web server, OS, etc. Should there be an entry added to the Status Report if, for example, checking Apache version reveals that the website could be vulnerable?

Not sure I would consider that as in-scope for the Telemetry initiative.

It's a great idea though, but I personally think it would make more sense to cover that part under the Drupal Status Report and add it there as a thing of its own. Telemetry is more about pushing general data to d.o and not so much about checking if the site is secure and reporting it back to that site.

I agree. I think that once the service file for getting such information is in core, that will make it much easier to add to the status report.

Some of that information can be obtained from PHP info. I wrote this function just now as proof of concept:

/**
 * Get PHP info as an array.
 *
 * @param int $flags
 *   Flags to pass to the phpinfo() function.
 *
 * @return array
 *   An array of PHP info.
 */
function phpinfo_to_array(int $flags = INFO_GENERAL | INFO_CONFIGURATION | INFO_ENVIRONMENT): array {
  ob_start();
  phpinfo($flags);
  $phpinfo = ob_get_clean();
  //echo $phpinfo;

  $lines = explode("\n", $phpinfo);
  $arr = [];
  foreach ($lines as $line) {
    $parts = explode(' => ', $line);
    if (count($parts) < 2) {
      continue;
    }
    $arr[$parts[0]] = $parts[1];
  }

  return $arr;
}

$phpinfo = phpinfo_to_array();
print_r($phpinfo);

Run it online here.

Note that this will only work when on CLI (see php_sapi_name), as phpinfo() returns HTML otherwise.

phpinfo() outputs plain text instead of HTML when using the CLI mode.

A comment on the documentation for phpinfo makes an attempt at parsing the HTML.

Great stuff! Output looks very usable :)

There are many great ideas on this issue for telemetry data points that we could collect. To make this more feasible, we should being by experimenting with collecting telemetry with as small scope as possible. For instance, we could create some leading indicators for the current lagging indicators. One of the indicators we're following is the number of Drupal sites reported by the update status module, which is a lagging indicator. However, there are leading indicators such as how many Drupal sites are being installed or how many Drupal composer projects are being initialized using composer create-project drupal/recommended-project drupal.

This data would be valuable when assessing the overall state of Drupal. If we could expand this to getting data on module installations too, we would have leading indicators for experimental modules as well, which would be extremely valuable for working on Drupal Core initiatives.

While there is a lot of other useful data that we would want to collect, starting with something small like this would make the scope more achievable. Once we have figured out this use case, we can move on to the next use case and add additional data points for the other proposed ideas.

I was just chatting with @rkoller about how nice it would be to actually know how people are using the admin breadcrumbs. Would be so useful to know if folks are going to our help.

If there is a way to enable this so that we can learn more about how authors and admins use their sites it would be so valuable. So often we just have to make assertions based on what we think would be useful to these users. Sadly, extrapolations from developers is rarely as good as watching those trying to use their Drupal site.

I agree with lauriii in #44 that the scope of this needs to be reigned in. If we shoot for the moon, this initiative will never take off (that's not to knock any of the very good ideas I've read through above!!)

I'm here because I maintain a Drupal platform of 1,000+ websites and I don't have the update module enabled on any of them, but want to contribute our module usage telemetry back to Drupal.org.

I don't have the update module enabled on any of these sites because the vast majority of the module's code (and it's a lot of code) is unrelated to sending the telemetry to Drupal.org, and it's for functionality I don't want or need. It's mostly around facilitating module updates/installs through the UI and reporting on modules that are out of date or have security issues. As a platform maintainer, I don't need that capability. I have other ways to monitor those concerns. As a platform maintainer, I try to keep unnecessary code off the platform as much as possible.

I know other platform maintainers (there's a lot of us in Higher Ed) that also don't have the update module enabled. We're collectively contributing to a massive under-reporting of project (module) usage on Drupal.org, all because the current code for collecting module telemetry is bundled with something we don't want.

So I feel the first step of launching this initiative is to separate the project usage reporting out of the update module and put it in core. Basically don't change any of the existing functionality of project usage reporting that Drupal.org already provides yet (I guess via the Project module?) We do that at a later time.

I see another user already created an issue about this #3199472: Improved data sharing with d.o. I think that's a good place to start focusing the effort.

Actually that last issue has a broader scope and I didn't feel comfortable hijacking it entirely, so I created a new issue #3448401: Move project telemetry reporting from the update module into core as a child of this one.

I develop the OpenTelemetry contrib module for Drupal. It is still in Alpha, but I hope it can help with generating interesting traces and metrics for receiving by Drupal Community as telemetry data.

To not insert into Drupal Core some telemetry-specific things, we can apply auto-instrumentation https://github.com/open-telemetry/opentelemetry-php-instrumentation and hook the needed functions in the wrapper.

And maybe eventually we will bring the opentelemetry as a Drupal Core module, that can be installed by default if the user opts in.

https://duanestorey.com/posts/down-the-rabbit-hole-a-deep-look-at-the-wo... is likely worth a read as it relates to Wordpress and the recent concern that Wordpress.org may have used telemetry data for less than ideal purposes (the Wordpress Engine Tracker site) raising questions about data collection in general by a CMS.

There is a lot of talk in this issue about collecting random data on a whim of someone had the idea it would be useful.

I would suggest that if this initiative ever does go forward that the first question asked regarding any data point should be "what problem will this solve in the next 3-6 months?" if you can not identify a specific concrete problem that can not be solved without the data point the collection is questionable.

I note this in other issues, even the collection of a site specific ID is enough to cause Update Manager to be disabled in many environments I work with. In general, the more data collection you add the less likely users are to be willing to share it.

@cmlara Very interesting article. I had no idea WordPress did that. I definitely agree that Drupal should only collect what it needs. I think that if users are given more granular control over which data is sent, they are more likely to allow it.

Moving this to Drupal core because the ideas queue is deprecated, however I think we need concrete suggestions for changes to be laid out a bit more. The only active issue I can think of is #3489066: Add tracking for when recipes from Drupal.org are applied.

edit: no at least two - #3448401: Move project telemetry reporting from the update module into core.

Might be a good fit first for Drupal CMS? Think we could be starting to ask more questions there? It is perhaps more useful to start asking questions and gathering information from those new to the community, than those already entrenched into D.o.

Both could be useful. I'm not arguing against involvement in Core. I see lots of ways that we could use Drupal itself to start promoting involvment in the Drupal community.

ISTR we have booted companies from d.o. for sending phoning back even such anonymous data (long ago). Couldn't it be all opt-in with an incitation to enable, like the update requirements, for example ?

I think all telemetry, including domain, should be opt-in, personally.

Goin back to comment #50:

Since some of us were not there. What is the use case for this data? What problem will be solved by collecting this data?

back even such anonymous data

This far from anonymous data. The domain can identify an individual and so can email address. This would PII under multiple privacy laws I’m aware of.

I don't think it would qualify as PII under GDPR for instance: we're talking server public IP there, aren't we ? which is public from DNS already and not tied to an individual directly, and only indirectly via their registry.

I don't think it would qualify as PII under GDPR for instance: we're talking server public IP there, aren't we ? which is public from DNS already and not tied to an individual directly, and only indirectly via their registry.

Some of this depends on is this a mega corp or an individual that is hosting the Drupal Site and are they using personally controlled resources (dedicated IP) or shared hosting.

Article 4 of GDPR:

‘personal data’ means any information relating to an identified or identifiable natural person (‘data subject’); an identifiable natural person is one who can be identified, directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier or to one or more factors specific to the physical, physiological, genetic, mental, economic, cultural or social identity of that natural person;

IP is specifically called out in Recital 30 as PII.

Email address are generally considered personal data (unless we’re talking something like webmaster@drupal.org which goes to a team/company).

Domains can be PII too:
Drupal.org, unlikely a single person
Cmlara.example.org (personal blog) identifies it as me personally and it becomes PII.

Note: I am not covered under GDPR, so I can’t assert a breach under it. CCPA is my primary regulation however the D.A. is likely legally exempt from it currently so no claim from me there either. Others may be able to assert a claim under their applicable privacy laws that I can not.

Side note: if this data also goes with the Drupal site key that would also potentially move that identifier under PII too (same as a customer ID is PII).

I think we can agree to disagree on GDPR / CCPA implications anyway since the point is even that should be opt-out, are we in agreement about that ?

I agree that having more information about the purpose of the collected data may help moving this forward. Especially the statement below is not clear to me, what the data will be used for and how these data (for example the domain) can help the DA (other than marketing, which I hope will be excluded):

Can we send your domain back to Drupal.org to better understand our users? (checked by default, can opt-out)