crawler-request

HTTP request module customized for crawlers.

Downloads in past

Stats

StarsIssuesVersionUpdatedCreatedSize
crawler-request
1.2.25 years ago5 years agoMinified + gzip package size for crawler-request in KB

Readme

crawler-request
HTTP request module customized for crawlers.
version downloads node status

Installation

npm install crawler-request

Usage

Simple Request

const crawler = require('crawler-request');

crawler("https://stackoverflow.com/").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

PDF Parse

const crawler = require('crawler-request');

crawler("http://careers.stackoverflow.com/stack_overflow_careers.pdf").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

Extend

const crawler = require('crawler-request');

function response_text_size(response){
    response["size"] = response.text.length;
    return response;
}

crawler("https://stackoverflow.com/",response_text_size).then(function(response){
    // handle response
	console.log(response.size);
});

Test

mocha or npm test
Check test folder and quickstart.js for extra usages.

Support

I use this package actively myself, so it has my top priority. You can chat on WhatsApp about any infos, ideas and suggestions.
WhatsApp

Submitting an Issue

If you find a bug or a mistake, you can help by submitting an issue to GitLab Repository

Creating a Merge Request

GitLab calls it merge request instead of pull request.

License

MIT licensed and all it's dependencies are MIT or BSD licensed.