[Spike] Research URL support for CCC

Quick glance and have some initial questions regarding "pulling context content from a URL"...

1. Is the intent on converting the entire html to md, ability to do a section of the html (eg. a div with an ID), or other specifics of an html looking to convert to md?
2. Nothing in the head would be relevant and context is within body tag

Maybe we could use the Guzzle approach in here?
https://brightdata.com/blog/web-data/php-web-scraping-libraries

It would be great to do something like we sync a copy of what's here as clean MD.
https://www.gov.uk/government/publications/government-functional-standar...

Just sharing a quick thought, what if the page is an index sort of page for a bunch of guidelines...
Since it's an issue still under discussion. I suggest instead of going for a simple grab and clean approach, which will later make us work on a crawl multiple layers and such appraoch, let's maybe use an agentic approach here? Maybe I'm overcomplicating it.
BUT imagine the time when you have given soemthing like Claude Code or such tools a page to extract something from and it realises that a specific sub-page is also important and grabs that as well...

These are options to achieve the single page criteria.

• https://github.com/thephpleague/html-to-markdown: This is already a required composer dependency for the the "AI" module. The thephpleague packages are highly regarded and dependable.
• Guzzle: This is a Drupal required dependency, obviously, so it can be used for fetching content from remote urls.
• Content: Various methods such as regex, DOMDocument, Drupal's HTML load to parse for the content (or other region).

To also note, https://www.drupal.org/project/markdownify is a module that uses thephpleague/html-to-markdown and can be reviewed for implementation ideas. I did not test out to see if an option to use.

Potential Gotchas

I did do a simple "drush scr" script but guzzle might run into issues such as when I was initially trying it with "https://www.drupal.org/drupalorg/style-guide/brand". I kept getting a 403 as it uses advanced security to prevent scraping or bots. Perhaps less of an issue if companies themselves are using CCC to get brand content and are able to whitelist internal IP ranges.

For content, realized may provide a lot of cruft in the markdown since it may have regions and actual content would be a specific div depending on how page is constructed. There may be a need to pass a selector to a function for grabbing specific content (eg. "div.content").

I've been working on the Document Loader (https://www.drupal.org/project/document_loader) which is mentioned in https://www.drupal.org/project/ai_context/issues/3547035

The Document loader had a bunch of submodules initially (see older commit), one of which was for loading content from web pages. It suits the need mentioned in this issue.
I took all the submodules out for now with the idea to make each of them standalone modules like document_loader_plugin_api

I have been working on the https://www.drupal.org/project/ai_simple_pdf_to_text module in the last few days in order to upgrade it to use the new Tools api and Document Loader - it's working, I only need to do some cleanup and push it for review.

What @unqunq said. The idea is that we would expose the Document Loader API to AI Context, so that you could feed it any document loader plugin, and it would try to transform what was given into text/markdown for AI Context to understand.

We will need to consider the UI around AI Context for all of these plugins though. Might want a cleaner Forms API component in Document Loader, or something to make it easier to use this.