@elioway/spider

Scrape Schema.org objects into mongoose schema files the elioWay.

Downloads in past

Stats

StarsIssuesVersionUpdatedCreatedSize
@elioway/spider
1.2.54 years ago4 years agoMinified + gzip package size for @elioway/spider in KB

Readme

spider
Get your schemon! Tim Bushell

Scrape Schema.org objects into mongoose schema files the elioWay.
This is a requirement of bones but it can also be run as the boilerplate of a web spidering project with scheming intentions.

Install

npm install @elioWay/spider --save

Usage

// yourapp.js
const yourAppSpider = require('@elioWay/spider');
var today = new Date()
// Create schemon the spider.
let schemon = new yourAppSpider(
    version = today.getFullYear() + '.' + today.getMonth() + '.' + today.getDate(), // Do change.
    depth = 2, // The deeper you go, the more objects you get. Go crazy.
    thingsSelector = '#thing_tree', // Don't change - but there is a bigger tree on the page.
    useOjectFields = true // Instead of 1 to 1 relationships to other Things, force String type.
  )
// Let schemon do spider things.
schemon.spider(
  // Wrap what schemon scraped.
  data => Spider.optimize(
    data
  )
)

node yourapp

Seeing is believing

git clone https://gitlab.com/elioschemers/spider/
cd spider
node test_spider

Credits

License

MIT Tim Bushell
elioway