Problem/Motivation
The HTML cleaning up brings in a bunch of stuff that's unrelated to the content you actually want to scrape.
Steps to reproduce
1. Visit the Explorer
2. Select that you'll be reading in a website
3. Select Webpage Docuemnt Loader Plugin
4. Enter in a blog post like https://www.kalamuna.com/blog/why-kalamuna-endorses-un-open-source-principles
5. Output to Markdown
6. Click Load
7. Expect to see just the content you care about, and nothing else
Proposed resolution
Readability.php is a PHP port of Mozilla's Readability to extract the readable version of HTML. It does a really good job on parsing and extracting, and bundling up.
Recommend bringing that in for parsing the HTML so that content is focused on what's actually desired.
| Comment | File | Size | Author |
|---|---|---|---|
| #4 | Screenshot from 2026-03-09 20-54-45.png | 247.23 KB | robloach |
Issue fork document_loader_webpage-3578143
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #3
robloachThe result actually cleans it up quite nicely...
Input:
There are also some Configuration options we could use to clean it up even further too.
Before
After
Comment #4
robloachComment #5
robloachComment #6
robloachUpdating summary
Comment #7
robloachComment #10
robloachComment #12
unqunq