Robust streaming parallel download manager with filesystem cache

  • cget

Downloads in past


0.2.16 years ago8 years agoMinified + gzip package size for cget in KB


build status dependency status npm version
cget is a robust streaming parallel download manager with a filesystem cache and a simple API.


  • Promise-based API, returns HTTP headers and a Node.js stream with contents.
  • Filesystem cache mirrors remote hosts and their directory structure.
- Easy to bypass cget and look at cached files.
  • Stores headers in separate .header.json files.
  • Caches HTTP errors to avoid repeating failing requests.
  • Limits concurrent downloads automatically using cwait.
  • Follows and caches redirect headers.
  • Built on top of request.
  • Optionally allow streaming from file:// URLs, bypassing the cache.
  • Add arbitrary files in the cache with any URI (URL or URN) as the key.
  • Written in TypeScript.

cget is perfect for downloading and caching various schema files, and is used in cxsd

Cached downloads

var Cache = require('cget').Cache;

// Store files in "cache" subdirectory next to this script.
var basePath = require('path').join(__dirname, 'cache');

// Initialize the download cache.
var cache = new Cache(basePath, {

  // Allow up to 2 parallel downloads.
  concurrency: 2


// Download a web page and print some info.

cache.fetch('').then(function(result) {

  console.log('Remote address:   ' + result.address.url);
  console.log('Local cache path: ' + result.address.path);
  console.log('HTTP status code: ' + result.status + ' ' + result.message);




Running it the first time prints and saves the downloaded file and its headers including any redirects in local files, for example:
  • cache/
  • cache/<COUNTRY>/<NONCE>
  • cache/<COUNTRY>/<NONCE>.header.json

The second time it prints the exact same output, but without needing a network connection.

Caching arbitrary files

The store method supports caching a string with any URI (URL or URN) as the key:
var cache = new (require('cget').Cache)();'urn:x-inspire:specification:gmlas:GeographicalNames:3.0', 'Some data');'', 'More data');
The MIT License
Copyright (c) 2015-2017 BusFaster Ltd