loopback-ds-iterator-mixin

This module is designed for the Strongloop Loopback framework. It provides a mixin that makes it easy to iterate through large data sets without leaving a large memory footprint.
Data is fetched in batches, with each batch lazy-loading only when needed. The batch size can be configured by setting the batchSize config options.

npm install --save loopback-ds-iterator-mixin

Add the mixins property to your server/model-config.json like the following:

{
  "_meta": {
    "sources": [
      "loopback/common/models",
      "loopback/server/models",
      "../common/models",
      "./models"
    ],
    "mixins": [
      "loopback/common/mixins",
      "../node_modules/loopback-ds-iterator-mixin/lib",
      "../common/mixins"
    ]
  }
}

MODEL CONFIG
=============

To use with your Models add the `mixins` attribute to the definition object of your model config.

```json
  {
    "name": "Item",
    "properties": {
      "name": "String",
      "description": "String",
      "status": "String"
    },
    "mixins": {
      "Iterator": {}
    }
  }

The mixin provides a number of configurable values that control how and when results are fetched and processed.
The following options are available:
- batchSize (default: 100)

Number of items to fetch from datasource per batch.

- maxQueueLength (default: 50)

The maximum size to allow the queue length to grow to.

- queueWaitInterval (default: 100)

The number of milliseconds to wait until checking wether the queue has drained below *maxQueueLength*.

- concurrentItems (default: 50)

The number of concurrent items to process (used in *forEachAsync*).

To use different values for the default add the following parameters to the mixin options.

{
  "name": "Item",
  "properties": {
    "name": "String",
    "description": "String",
    "status": "String"
  },
  "mixins": {
    "Iterator": {
      "batchSize": "100",
      "maxQueueLength": "50",
      "queueWaitInterval": "100",
      "concurrentItems": "50"
    }
  }
}

An iterate method will be added to your model class that can be used to manually iterate over the results of a find query.

var iterator = this.Item.iterate(query)
iterator.next()
  .then(function (item) {
    expect(item.name).to.equal('Item1');
    return iterator.next();
  })
  .then(function (item) {
    expect(item.name).to.equal('Item2');
    return iterator.next();
  })

Additionally, a forEachAsync method will be added to your model class that can be used to iterate over the results of a find query. maxQueueLength, queueWaitInterval & concurrentItems options can be used to tune things.

var fn = function(task, callback) {
  process.nextTick(function() {
    callback();
  });
}
this.Item.forEachAsync({where: { ... }}, fn, {})
  .then(function () {
    done();
  })
  .catch(done);
});

You can override the default configuration options such as the number of items per page (as defined in the mixin configuration) by setting the optional settings parameter:

var iterator = this.Item.iterate(query, {batchSize: 1000})
iterator.next()
  .then(function (item) {
    ...

Run the tests in test.js

npm test

Run with debugging output on:

DEBUG='loopback:mixin:iterator' npm test