libcraigscrape

KO | EN

gitlite — search

#python #java #linux #github-actions #javascript #python3 #docker #android #react #hacktoberfest #covid-19 #flutter

★ 45 Open GitHub ↗

An easy library to do the heavy lifting between you and craigslist's posting database. Given a URL, libcraigscrape will follow links, scrape fields, and make ruby-sense out of the raw html from craigslist's servers.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# libcraigscrape - A craigslist URL-scraping support Library

An easy library to do the heavy lifting between you and Craigslist's posting database. Given a URL, libcraigscrape will
follow links, scrape fields, and make ruby-sense out of the raw html from craigslist's servers.

For more information, head to the {craiglist monitoring}[http://www.derosetechnologies.com/community/libcraigscrape] help section of our website.

## craigwatch
libcraigscrape was primarily developed to support the included craigwatch[link:files/bin/craigwatch.html] script. See the included craigwatch script for
examples of libcraigscape in action, and (hopefully) to serve an immediate craigscraping need.

## Installation

Install via RubyGems:

sudo gem install libcraigscrape

## Usage

### Scrape Craigslist Listings since Sep 10

On the 'miami.craigslist.org' site, using the query "search/sss?query=apple"

require 'rubygems'
    require 'libcraigscrape'
    require 'date'
    require 'pp'
    
    miami_cl = CraigScrape.new 'us/fl/miami'
    miami_cl.posts_since(Time.parse('Sep 10'), 'search/sss?query=apple').each do |post|
      pp post  
    end

### Scrape Last 225 Craigslist Listings

On the 'miami.craigslist.org'  under the 'apa' category

require 'rubygems'
    require 'libcraigscrape'
    require 'pp'
    
    i=1
    CraigScrape.new('us/fl/miami').each_post('apa') do |post|
      break if i > 225
       i+=1
       pp post
    end

### Multiple site with multiple section/search enumeration of posts

In Florida, with the exception of 'miami.craigslist.org' & 'keys.craigslist.org' sites, output each post in 
the 'crg' category and for the search 'artist needed'

require 'rubygems'
    require 'libcraigscrape'
    require 'pp'
    
    non_sfl_sites = CraigScrape.new('us/fl', '- us/fl/miami', '- us/fl/keys')
    non_sfl_sites.each_post('crg', 'search/sss?query=artist+needed') do |post|
       pp post
    end

### Scrape Single Craigslist Posting

This grabs the full details under the specific post http://miami.craigslist.org/mdc/sys/1140808860.html

require 'rubygems'
    require 'libcraigscrape'
    
    post = CraigScrape::Posting.new 'http://miami.craigslist.org/mdc/sys/1140808860.html'
    puts "(%s) %s:\n %s" % [ post.post_time.strftime('%b %d'), post.title, post.contents_as_plain ]

### Scrape Single Craigslist Listing

This grabs the post summaries of the single listings at http://miami.craigslist.org/search/sss?query=laptop

require 'rubygems'
    require 'libcraigscrape'
    
    listing = CraigScrape::Listings.new 'http://miami.craigslist.org/search/sss?query=laptop'
    puts 'Found %d posts for the search "laptop" on this page' % listing.posts.length

## Author
- Chris DeRose (cderose@derosetechnologies.com)
- DeRose Technologies, Inc. http://www.derosetechnologies.com

## License

See COPYING[link:files/COPYING.html]

libcraigscrape

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README