Test RDFa by parsing RDFa (add the easyrdf library) [#1866858]

Problem/Motivation

As discussed in #1778226: [META] Fix RDF module, core's RDF module provides unreliable RDFa. To ensure this isn't the case in Drupal 8, we need to have better testing.

So far in Drupal 7 we used xPath expressions to check the markup produced, but it leads to verbose and obscure expressions such as

//div[contains(@class, "comment") and contains(@typeof, "sioct:Comment")]//span[@rel="sioc:has_creator"]/a[contains(@class, "username") and @typeof="sioc:UserAccount" and @property="foaf:name" and @href="http://example.org/" and contains(@rel, "foaf:page")]

RDFa processing can't be handled accurately using xpath, but instead a compliant parser should be used to parse the page and verify that the results are what we expect. Writing a library for RDFa processing is way too complex to maintain ourselves and out of scope for the Drupal project. There are libraries which already do it well.

Proposed resolution

Use the EasyRdf PHP library:

98.4% conformant to W3C HTML5+RDFa 1.1 see RDFa processors conformance report
licensed under BSD-3-Clause (same as Twig), compatible with GPL2
PSR-0 and PEAR namespaces (like Twig)
Composer integration
actively maintained by Nicholas Humfrey, who is very supportive and even extended easyrdf API to make our lives easier in Drupal testing

It's important to note that this library is only loaded and used during testing only.

Issues which depend on this library:
- #1867096: Rewrite RdfaMarkupTest to parse RDFa
- #1869914: Refactor RdfMappingDefinitionTestCase and split it in dedicated test cases
- #1886108: Rewrite CommentAttributesTest to parse RDFa
- #1886102: Rewrite TrackerAttributesTest to parse RDFa

Comment	File	Size	Author
#15	1866858_easyrdf_15.patch	427.48 KB	scor
#15
#12	easy-rdf-12.patch	416.5 KB	scor
#12
#7	easy-rdf.patch	1.36 MB	linclark
#7

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #0.0

Anonymous (not verified) CreditAttribution: Anonymous commented 14 December 2012 at 16:49

Issue summary:

View changes

Updated EasyRdf compliance.

Comment #1

Anonymous (not verified) CreditAttribution: Anonymous commented 14 December 2012 at 17:15

These are the current fails for EasyRdf:

Test 0072: Relative URI in @about (with XHTML base in head)
Test 0073: Relative URI in @resource (with XHTML base in head)
Test 0074: Relative URI in @href (with XHTML base in head)
Test 0075: Reserved word 'license' in @rel with no explizit @about
Test 0117: Fragment identifiers stripped from BASE
Test 0177: Test @prefix
Test 0178: Test @prefix with multiple mappings
Test 0179: Test @prefix vs @xmlns priority
Test 0182: Test prefix locality
Test 0183: Test @xmlns redefines @prefix
Test 0186: @vocab after subject declaration
Test 0187: @vocab redefinition
Test 0188: @vocab only affects predicates
Test 0197: Test TERMorCURIEorAbsURI requires an absolute URI
Test 0217: @vocab causes rdfa:usesVocabulary triple to be added
Test 0224: @inlist hanging @rel
Test 0254: @datatype="" generates plain literal in presence of child nodes
Test 0255: lang="" clears language setting
Test 0303: For HTML+RDFa 1.1, remove term elements of @rel/@rev when on same element as @property

Comment #1.0

Anonymous (not verified) CreditAttribution: Anonymous commented 14 December 2012 at 17:15

Issue summary:

View changes

Updated compliance again.

Comment #2

scor CreditAttribution: scor commented 14 December 2012 at 17:39

yes, I was thinking that EasyPHP should be used, especially since it's PSR-0 + composer enabled, and could be part of core easily. Relying on a external service would be much slower to perform tests and prone to go down.

The current size of the library is 3.1MB. We have two options:
1. include full easyrdf (core would only use a small part of its features but contrib could build on the rest)
2. only include the RDFa parser code (with RDF output as PHP array). I've asked the maintainer if he would be willing to offer such minimal subset of easyrdf.

Comment #3

webchick

she/they

English

Vancouver 🇨🇦

CreditAttribution: webchick commented 14 December 2012 at 17:47

Category:

feature

» task

Re-classifying as a task. Better test coverage should never be blocked by thresholds. :D

Comment #4

Anonymous (not verified) CreditAttribution: Anonymous commented 14 December 2012 at 20:21

EasyRdf's PSR-0 handling seems a little odd to me, I'm not sure it is compatible with our autoloader.

Comment #5

scor CreditAttribution: scor commented 14 December 2012 at 21:13

add this to core/vendor/composer/autoload_namespaces.php

    'EasyRdf_' => $vendorDir . '/njh/easyrdf/lib/',

and then call the call like we do with Twig, with a \:

$foaf = new \EasyRdf_Graph("http://njh.me/foaf.rdf");
$foaf->load();
$me = $foaf->primaryTopic();
echo "My name is: ".$me->get('foaf:name')."\n";

Comment #6

Anonymous (not verified) CreditAttribution: Anonymous commented 14 December 2012 at 21:17

As mentioned in IRC, this should be nhj/easyrdf

Comment #7

Anonymous (not verified) CreditAttribution: Anonymous commented 19 December 2012 at 23:42

Status:

Active

» Needs review

File	Size
easy-rdf.patch	1.36 MB

I'm sure this is the wrong way to add something to composer, but hopefully someone will do it the correct way.

Currently, this library has whitespace errors. Do we correct those in external libraries?

Comment #8

20 December 2012 at 00:46

Status:

Needs review

» Needs work

The last submitted patch, easy-rdf.patch, failed testing.

Comment #9

scor CreditAttribution: scor commented 20 December 2012 at 01:09

Status:

Needs work

» Needs review

#7: easy-rdf.patch queued for re-testing.

Comment #10

scor CreditAttribution: scor commented 20 December 2012 at 01:32

Status:

Needs review

» Needs work

I ran composer update easyrdf and got an error saying it could not find 0.8.*. I think it would be fine for now to use "njh/easyrdf": "dev-master" since we want to keep up with future changes of easyrdf. Later we can switch to a more stable version of easyrdf as we get closer to a stable version of D8.

Currently, this library has whitespace errors. Do we correct those in external libraries?

we certainly don't want to fork external libs, since we want to manage them with composer. this could be fixed upstream, I'm sure it's something njh would be happy to incorporate.

Comment #11

Anonymous (not verified) CreditAttribution: Anonymous commented 20 December 2012 at 02:50

Right, I was expecting that we would fix it upstream if we were going to fix it. I was wondering whether we have to fix it before we can bring it in.

Comment #12

scor CreditAttribution: scor commented 31 December 2012 at 21:57

File	Size
easy-rdf-12.patch	416.5 KB

updating easyrdf patch using easyrdf-minimal which only includes the necessary lib folder. njh is working on packaging this for composer, so I'm not going to change anything composer related for now in this patch. But in the meantime we can move on in RDF testing issues such as #1867096: Rewrite RdfaMarkupTest to parse RDFa. The whitespace errors become moot now since these whitespaces were outside the lib folder (tests mostly).

Comment #13

Anonymous (not verified) CreditAttribution: Anonymous commented 10 January 2013 at 14:27

Status:

Needs work

» Needs review

Marking this as needs review just to get the green.

Comment #14

10 January 2013 at 14:33

Status:

Needs review

» Needs work

The last submitted patch, easy-rdf-12.patch, failed testing.

Comment #15

scor CreditAttribution: scor commented 10 January 2013 at 18:08

Status:

Needs work

» Needs review

File	Size
1866858_easyrdf_15.patch	427.48 KB

here is a reroll, which includes the latest version of easyrdf. We'll make use of a new feature which was added a few days ago for #1867096: Rewrite RdfaMarkupTest to parse RDFa.

Comment #16

Anonymous (not verified) CreditAttribution: Anonymous commented 10 January 2013 at 19:28

Status:

Needs review

» Reviewed & tested by the community

RTBC. We'll be able to improve the accuracy of RDF tests greatly once this is in.

Comment #17

webchick

she/they

English

Vancouver 🇨🇦

CreditAttribution: webchick commented 10 January 2013 at 20:32

Title:	Test RDFa by parsing RDFa	» Test RDFa by parsing RDFa (add the easyrdf library)
Assigned:	Unassigned	» Dries

This is one of those "add an external library to core" issues, which Dries typically likes to eyeball before they go in. Assigning.

An updated issue summary to provide a little bit more info on what this library is and does might help.

Comment #17.0

webchick

she/they

English

Vancouver 🇨🇦

CreditAttribution: webchick commented 10 January 2013 at 20:32

Issue summary:

View changes

add link to easyrdf lib

Comment #17.1

scor CreditAttribution: scor commented 10 January 2013 at 21:28

Issue summary:

View changes

more info on easyrdf

Comment #18

scor CreditAttribution: scor commented 10 January 2013 at 21:46

thanks webchick, I've updated the OP with more info on easyrdf, including a list of issues depending on the easyrdf library for more accurate testing.

Comment #18.0

scor CreditAttribution: scor commented 10 January 2013 at 21:46

Issue summary:

View changes

more on the maintainer and note on usage

Comment #19

Wim Leers

Ghent 🇧🇪🇪🇺

CreditAttribution: Wim Leers commented 11 January 2013 at 17:11

Correct RDFa is essential for Edit module to drop its own custom http://viejs.org service in favor of VIE's built-in RDFaService, so this will indirectly help with that.

Comment #20

webchick

she/they

English

Vancouver 🇨🇦

CreditAttribution: webchick commented 11 January 2013 at 17:44

Status:

Reviewed & tested by the community

» Fixed

Ok, Moshe, Alex, and I discussed this on a call. I'm going to go ahead and commit this, despite the lack of Dries's sign-off, because the earliest Dries will be available will be late next week (which blocks further RDF clean-up efforts until then) and this is being used only for testing, as opposed to a new developer-facing API. So I'm going to ask for forgiveness on this one. :) Hope that that's ok.

Committed and pushed to 8.x. Thanks!