Problem/Motivation

The HTML cleaning up brings in a bunch of stuff that's unrelated to the content you actually want to scrape.

Steps to reproduce

1. Visit the Explorer
2. Select that you'll be reading in a website
3. Select Webpage Docuemnt Loader Plugin
4. Enter in a blog post like https://www.kalamuna.com/blog/why-kalamuna-endorses-un-open-source-principles
5. Output to Markdown
6. Click Load
7. Expect to see just the content you care about, and nothing else

Proposed resolution

Readability.php is a PHP port of Mozilla's Readability to extract the readable version of HTML. It does a really good job on parsing and extracting, and bundling up.

Recommend bringing that in for parsing the HTML so that content is focused on what's actually desired.

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

robloach created an issue. See original summary.

robloach’s picture

Status: Active » Needs review
StatusFileSize
new336.53 KB

The result actually cleans it up quite nicely...

Input:

https://www.kalamuna.com/blog/why-kalamuna-endorses-un-open-source-principles

There are also some Configuration options we could use to clean it up even further too.

Before

After

robloach’s picture

StatusFileSize
new247.23 KB
robloach’s picture

Title: Readability.php? » Use Readability.php for HTML Parsing
robloach’s picture

Issue summary: View changes

Updating summary

robloach’s picture

Issue summary: View changes

unqunq made their first commit to this issue’s fork.

robloach’s picture

Status: Needs review » Fixed

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

unqunq’s picture

Status: Fixed » Closed (fixed)