Streaming operations on S3 objects

Downloads in past


771.0.06 years ago6 years agoMinified + gzip package size for @mapbox/s3scan in KB


Build Status
Provides native Node.js streams for S3 operations such as
  • listing all the keys in a bucket/prefix
  • retrieving all the objects under a given bucket/prefix
  • deleting all the children of a given bucket/prefix

Also provides CLI tools:
s3keys: prints a line-delimited list of keys to stdout
$ s3keys s3://my-bucket/some-prefix

s3purge: deletes all objects beneath the provided prefix
$ s3purge s3://my-bucket/all-finished-with-these

s3scan: GET and print a prefix-worth of objects to stdout
$ s3scan s3://my-bucket/some-prefix
$ s3scan s3://my-bucket/some-prefix --gunzip
$ s3scan s3://my-bucket/some-prefix --gunzip --concurrency=1

Running tests

You can run tests against your own bucket/prefix by specifying them as environment variables:
TestBucket=my-bucket TestPrefix=my-prefix npm test

Otherwise these values default to buckets owned by Mapbox which require appropriate authentication.