xtractor

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 46 Open GitHub ↗

XTractor is an algorithmic text extractor from web pages written in Java. It builds upon the "commonly used web design practices" approach (from readability.js, goose and snacktory) to create a set of heuristics for fast article text extraction. It adds several features like paragraph preservation, better image detection heuristics, sibling score based enhancements to article detection

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# XTractor - heuristics based webpage text extraction demo
Demo: http://xtractor.herokuapp.com

Sample URL: Fox News review of the movie Oblivion : http://xtractor.herokuapp.com/xtractor/?url=http%3A%2F%2Fwww.foxnews.com%2Fentertainment%2F2013%2F04%2F18%2Freview-tom-cruise-tones-it-down-in-visually-stunning-oblivion%2F

Xtractor tries to guess the main body of the article and grab one significant image from the page. Then it strips off all formatting and presents the article with just the image as header, a short summary and paragraph preserved simple readable text.

I had written this close to 3 years back (https://news.ycombinator.com/item?id=5580719)

It uses snacktory for dom manipulation and tldrzr (http://github.com/mohaps/tldrzr) for summarization.

I was never happy with the code quality and always intended to get back to it. Received an email from someone who wished to use the code in an open source project. Opensourcing this under the BSD license. Feel free to hack on it and make it better. :)

While reading the code, please keep in mind that it was written within a couple of hours of hackathon style coding :) with the sole goal of getting something working end to end. If you have questions... well, it's been so long, I mightn't even remember why I did somethings the way I did. :) But feel free to drop me a line at mohaps AT gmail DOT com

Have fun! Keep hacking.

## Running the application locally

First build with:

$mvn clean install

Then run it with:

$java -cp target/classes:target/dependency/* com.mohaps.xtractor.Main

## Library usage example

### With Fetcher

Fetcher fetcher = new Fetcher();
    Extractor extractor = new Extractor(fetcher);
    ISummarizer summarizer = Factory.getSummarizer();

int summarySentenceNb = 1;
    long timeout = 1000; // 1s
    String url = "http://www.bbc.com/news/science-environment-34510869";
    FetchResult fResult = fetcher.fetch(url, timeout);
    ExtractorResult eResult = extractor.extract(fResult.getContent(), fResult.getCharset(), fResult.getActualUrl());
    String summary = summarizer.summarize(eResult.getText(), summarySentenceNb);

System.out.print("title:  \t"+ SHelper.replaceSmartQuotes(eResult.getTitle()) + "\n");
    System.out.print("summary:\t"+ SHelper.replaceSmartQuotes(summary) + "\n");
    System.out.print("image:  \t"+ eResult.getImage() + "\n");
    System.out.print("video:  \t"+ eResult.getVideo() + "\n");
    System.out.print("body:   \t"+ SHelper.replaceSmartQuotes(eResult.getText()) + "\n");

fetcher.shutdown(); // Have to be shutdown to finish the process

### Without Fetcher

Extractor extractor = new Extractor();
    ISummarizer summarizer = Factory.getSummarizer();

int summarySentenceNb = 1;
    String charset = "UTF-8"
    String url = "http://www.random.com";
    String html = "<html><body><div>Hello World</div></body></html>"
    ExtractorResult eResult = extractor.extract(content, charset, url);
    String summary = summarizer.summarize(eResult.getText(), summarySentenceNb);

System.out.print("title:  \t"+ SHelper.replaceSmartQuotes(eResult.getTitle()) + "\n");
    System.out.print("summary:\t"+ SHelper.replaceSmartQuotes(summary) + "\n");
    System.out.print("body:   \t"+ SHelper.replaceSmartQuotes(eResult.getText()) + "\n");

xtractor

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README