@bitblit/RatchetCommon utilities for Typescript and Node.
You may wish to read the changelog
This library is my first attempt to publish to NPM. I make no guarantees other than the standard one I make for all my libraries - that I will always rev the major version number if I do something that is not backwards compatible. As I improve this paragraph should go away.
yarn install @bitblit/ratchet
Barrel FilesA Note on barrel files - All of Ratchet's barrel files are one level down. This is because otherwise everything I said above about transitive dependencies gets thrown out the window if you put them all in one big barrel file
Utilities and Stuff
DaemonThe Daemon subpackage is to handle the case on AWS where you want to run a process asynchronously via Lambda (not waiting for the return on a API Gateway request) and check its results until it is finished. Daemon offers a thin wrapper around an S3 object that can be updated until it is finally replaced by the final results themselves. The end customer can be given the key to check on synchronously. Items are broken down by day (for easy flushing later) and by groups if desired.
Site UploaderThere is a tool in here called site uploader that is designed to help put completely static sites into S3 (basically a glorified aws s3 cp --recursive ...), while allowing you to set some of the more popular HTTP headers like Cache-Expires. If you use it you'll need to add this to your dev dependencies (since I didn't want it in the dependencies of Ratchet which is meant to also be used in the browser)
It will expect you to provide it a configuration file. I'll document it better later, but here is an example of such a configuration file (an Angular app I use, with a couple of custom HTML files with no extensions)
Direct DependenciesSince this is meant to be a very generic library, I don't want to pull in a bunch of transitive dependencies, I am keeping myself to very, very few upstream libraries. However, I am making the following exceptions, because I use these libraries in literally every project I have ever done:
Optional DependenciesCheck out the package.json file for libraries that are optional - basically, if you wish to use the code that uses those libraries, you need to also include those libraries. Otherwise, they can be safely ignored.
AWS Library NotesFor most of my non-super-heavy-load stuff I work in us-east-1. I do this both because I am lazy and because that is where AWS releases the new stuff first. Because of this, you will see that while my code allows you to override the region, I always set a biased default. If you don't like that... sorry?
AthenaAthenaRatchet is a special case because the datasets you use with Athena tend to be so large you'll often need to work with only a chunk of them at a time. The AthenaRatchet depends on a couple more libraries that you'll need to use a chunk of the functionality - csv parses output files from Athena locally (much faster than having them do it) and tmp creates local tmp files for storage. It also uses 'fs' so, in case it's not already abundantly clear, this only works in Node, not in the browser. Not that you'd do much Athena work in the browser anyway, but I may break this up later if I see a need for that.
To use the AthenaRatchet, in addition to aws-lib you will need csv and tmp
RXJSThe Observable ratchet is based on Observables through RXJS, you'll need to install it.
Handlebars and CrossFetchThe simplified mailer for SES (aws/ses/mailer) can be provided with a remote template renderer, which assumes the template is a Handlebars template. If you use it, you'll need Handlebars (and Handlebars-Layouts, which isn't required but is highly recommended if you are doing much Handlebars work needing templates) and CrossFetch installed.
I use CrossFetch to keep Ratchet usable on both the client and server side.
ModelValidator DependenciesRatchet includes a helper for validating objects that match swagger schemas in the model-validator package. If you use it, you'll need to include js-yaml and swagger-model-validator.
TestingHa! No, seriously - all testing is done using Jest. To run them:
Why not X? (Where X=Lodash, Underscore, Ramda, etc...)Originally, my answer would be because I just didn't know about them. I know about them now (2019) and I use them quite a lot myself. Any code has impedence mismatches (either with the problem domain, or just with how I think about the problem) and so Ratchet is how I tackle some of these. If you think like me, Ratchet is for you! If not, it's ok - go use X. We're still friends.
DeploymentI'll write notes-to-self here on how my deployment on Github actions is actually going to work.
Following the notes from here. Important points:
- Everything in the package that isn't in .gitignore or .npmignore gets uploaded. Review before post
- Set a release tag and push to Github, Github actions takes care of the rest:
git commit -m "New stuff" mytag release git push origin master --tags
ContributingPull requests are welcome, although I'm not sure why you'd be interested!
Notes on ECMAScript ModulesCAW 2022-03-22 : It is my full intention to convert Ratchet (and my other NodeJS libraries) to ECMAScript modules. However, I have lots of other things to do besides getting that mess working when Typescript doesn't support it cleanly yet. Support was pulled from Typescript 4.5 but as soon as it is supported it'll be on the top of my list of changes.
CAW 2022-09-05 : Took another multi-week run at making this an ECMAScript module. It is actually kinda amazing that a full 7 years after this spec came out support is STILL so weak across the ecosystem for it. Typescript has issues unless you use some hacks (e.g., default Typescript imports are extensionless, but ECMAScript REQUIRES an extension on the imports. Default ESLint hates this). Jest works, but only with some hacks ("cross-env NODEOPTIONS=--experimental-vm-modules jest").
Barrel files still a huge mess to try to make tree shaking working cleanly downstream. Barrelsby won't put extensions on the index files so that's a SED hack, and then unless you wanna completely destroy tree shaking you better list every folder separately in the "exports" section of the package.json file.
Worst of all, the Typescript infrastructure is such that if I make Ratchet ESM-only (as opposed to a dual-built ESM and CJS module) then everything downstream of it (notably Epsilon and ALL my various projects which use Ratchet, which is to say all of them) must also be ESM only. Given that Neon uses Ratchet, and that everything up until this point has been incredibly fragile, I'm going to commit everything I have into a branch (cw/feat/esm) and push it, and then put it back on the shelf for 6 months to age. We'll see how it looks then - until then, I'm not willing to take on that much risk just to get rid of a warning in my Angular builds.
If I was primarily a front-end, non-webpack guy the value would be higher since getting rid of sync imports is huge, but given that I'm a 92% Node / 8% web guy, this is still way too far out on the value chain for me.