Meaningful metrics?

I'm not moving the rest of the original discussion. Though there are some reference links worth reviewing there.

• Cornell study on tracking corporate diversity in the US
• Minnesota (US) government case-study in increasing diversity of job applicants
• An introduction to the social identifiers that mark diversity
• NTEN's digitial inclusion toolkit

I started looking at the initial work here and doing some thinking, and I wanted to capture some ideas about how we might measure DD&I outside of issue attribution and other factors. These are rough notes, open to discussion.

We might be best served by breaking our metrics into a few separate categories. Fundamentally, I think we want to look at two group metrics:

• Individuals and their experience
• Groups and leadership

The focus on contribution metrics are an individual metric that tries to understand who is contributing to Drupal and how. It is a quantitative metric. There is an argument to be made that we also need some qualitative metrics that capture the nature of personal and group experiences in the community.

If we take it that way, that gives us four sets of metrics we might care about:

1. Individual, qualitative
2. Individual, quantitative
3. Group, qualitative
4. Group, quantitative

These metrics should, I think, also consider both our accomplishments and our efforts. Good metrics around efforts to increase our diversity -- which would show both successes and failures of particular approaches -- should be a goal.

I'm thinking in particular of how we can incentivize the behavior and outcomes that we are looking for. In the US, one approach that suggests a good metric is what's known as the Rooney Rule, which a sports league put in place to increase minority representation in head coach positions. This kind of proactive check -- hey, organizers, did you invite people from under-represented groups to be part of your planning -- can be a very valid way to make improvements.

The Cornell study listed above starts by asking three questions that are relevant here:

1. What do you want diversity to help you achieve?
2. What do you need to implement?
3. How can you measure it?

That study also has a good recommendation for starting with Individual, qualitative measurements:

Employee climate surveys help capture the perceived benefits of programs that many qualitative
metrics miss. They allow for employee input with regard to effectiveness of diversity initiatives, and
provides invaluable insight into which diversity programs are most critical to staff and might yield the
greatest efficiencies.

The most effective employee climate surveys ask questions pertaining to perceptions of diversity,
welcomeness, trust, transparency, and fairness, etc. Note it is important to establish a baseline for these
questions and track progress over that baseline.

So as a baseline, I'd suggest that we start working along these four lines, with a focus on qualitative measures for individuals, and quantitative measures for groups.

Individual, qualitative

Periodic (and post-event) surveys around experience, rated on a scale. For instance:

• Did you feel welcome at the event?
• Did you feel safe?
• Did you feel like you were treated with respect?
• And so on....

And map the results along major social identifiers (Age, Ethnicity, Gender, Race, Sexual Orientation) and community-specific ones (Drupal Experience, Events attended, Profession).

Individual, quantitative

These metrics are around individual contributions and participation. The work that we need here (which @mtift noted in his original post) is about how better to measure a contribution to Drupal. Some ideas:

• Code contribution
• Documentation contribution
• Issue triage
• Event organization
• Speaker selection
• Mentorship
• Speaking / workshopping
• Event volunteering

Group, qualitative

These items would be about the experience of a leadership team with regard to diversity initiatives, outreach, and outcomes. Rated questions such as:

• Do you feel like your ideas were heard and acknowledged?
• Do you feel that your group accurately represented your community / attendees?
• Were you satisfied with the diversity of the leadership team?
• Were you satisfied with the diversity of the community / attendees / speakers?

Group, quantitative

Here, I think we dive into active, measurable steps that leadership groups can take to promote D&I.

• What was the makeup of the leadership team?
• Was the [baseline steps like the Rooney rule] followed in composing the leadership team?
• Was the [baseline steps like the Rooney rule] followed in composing the speaker list?
• What was the makeup of the attendees?
• What was the makeup of the speakers?
• What outreach programs were supported at the event?
• Were D&I resources available before and during the event?

There's some additional work here to decide what is or is not an appropriate measure to ask leadership teams to take, etc.

Next Steps

As a next step, I would propose that we ratify this -- or another -- approach to metrics and then divide into teams to tackle each set.

See also https://www.linuxfoundation.org/blog/chaoss-project-creates-tools-to-ana... and https://chaoss.community/

See also https://discourse.mozilla.org/t/diversity-inclusion-in-open-source-surve...

It's fine. I was discussing with @sugaroverflow this morning that this issue was originally framed as about the entire community. That may not be appropriate.

I still think the 4-quadrant approach will help us focus if we narrow to metrics for DD&I only.