
An opinionated tool for safely managing and deploying Serverless projects and their secrets
Contents
Why?
Problem: The Serverless framework currently offers no way to manage secrets, keys, etc. You could put them in with your environment variables, but what if you are working in a team? You could put it in the repo, but "secrets" in a git repo is bad practice. You could use other tools to simply encrypt with KMS, but you end up storing them in S3 or back in the repo again. Also, none of these solutions ensures that your secrets have actually been created before deployment...Solution: Serverless Secrets stores your secrets in a place designed for secrets. For AWS, this is the EC2 Parameter Store, which supports encryption, including custom KMS keys. In addition, Serverless Secrets offers automated validation of your secrets' presence, making your deployments that much closer to bulletproof (TM).
Requirements
Local requirements
Node.js 6.5 or greater for CLIClient requirements
The bundled client is for Node.js only. For other languages, see the following table.| Language | Link | |------------|------------------------------------------------------------------------------------------------------------------| | Python 2/3 | https://github.com/trek10inc/serverless-secrets-python |
If you develop a client, please send us a PR to link to your client. All clients should allow for multi-provider support. Clients need only implement the
getSecret
provider method.Framework requirements
Serverless Secrets 3.0.0 and greater is designed for use with Serverless >= 1.12.0. For Serverless 0.x, please see Serverless Secrets 2.xProvider requirements
Currently, Serverless Secrets only supports AWS. However, it has been designed with support for other providers in mind down the road. We welcome PRs for this too.AWS
The bundled client requires Node 6.10 (or greater in the future). Feel free to develop and contribute your own clients for other languages.Offline support
Serverless Secrets should work with Serverless Offline, but not in a fully offline (no Internet connection) setting. You will still need access to your provider to load the secrets.Configuration
Adding Serverless Secrets to your project
In the root of your Serverless project:npm install serverless-secrets --save-dev
or yarn add serverless-secrets --dev
Add the plugin to your
serverless.yml
:plugins:
- serverless-secrets
environmentSecrets
With a standard Serverless project, you can use theenvironment
property to add environment variables
to individual functions as well as to all of your functions via the provider
section. We augment this
concept by adding an environmentSecrets
section to the provider and any function. Just like
environment
, the properties under the environmentSecrets
property become environment variables,
with the keys becoming the environment variable names. However, the values of the properties under
environmentSecrets
are the names of the secrets in the secure store (e.g. Parameter Store for AWS).
Once you have set your secrets with the CLI (see below), just make sure they are all listed correctly in
environmentSecrets
. You should not duplicate any environmentSecrets
keys in environment
. This is
checked during the validation step. Here's an example:provider:
environment:
API_USER: asdf@asdf.com
environmentSecrets:
API_KEY: '/my-project/${opt:stage}/API_KEY'
If this project included a Node function, after loading the secrets,
process.env.API_KEY
would contain
the stored secret value.custom.serverlessSecrets section
There are a number of options avaliable to customize how Serverless Secrets operates. These should be set undercustom.serverlessSecrets
in your serverless.yml
. Here's an example showing that the
secrets are stored in us-west-2 and listing 2 KMS keys for use with the CLI:custom:
serverlessSecrets:
providerOptions:
region: us-west-2
keys:
default: "alias/myDefaultKey"
anotherKey: "alias/myOtherKey"
options
The following options apply to both the custom section and the client methods. The custom section values will be deployed to your functions and become the default values for the client methods.throwOnMissingSecret
- boolean: If set to true, an error will be thrown if any secret
false
.logOnMissingSecret
- boolean: If set to true, an message will be logged if any secret
true
.providerOptions
- object: The options object to be passed to the CLI/client provider. This will
apiVersion: '2014-11-06',
region: process.env.AWS_DEFAULT_REGION || 'us-east-1'
}
```The following options apply only to the custom section as they are only used in deploy/package CLI operations:
skipValidation
- boolean: If set to true, validation of the existence of your secrets in
false
.omitPermissions
- boolean: If set to true, permissions will not automatically be added
false
.
- AWS: This grants permission to the ssm:GetParameters
action.resourceForIamRole
- string | string: This is the string or array of strings that become the
Resource
in the IAM role that grants the ssm:GetParameters
action. This does nothing
if omitPermissions
is true. Default value: ''.The following options apply only to the custom section as they are only used in Serverless Secrets CLI operations:
keys
- object: This is an object that can be used create shortcuts for your KMS keys. You can
default
key, along with other named keys. Any operation that takes a key id will
search the shortcuts first. If no key id is provided, the default
key will be used, if set.
If no key id is provided and no default
exists, the provider default will be used.CLI
serverless secrets set parameters
Store secrets in your provider's secret storeParameters
--name / -n
: REQUIRED; name of the secret--text / -t
: Text to be stored; MUST supply this or --file
--file / -f
: File to be stored; MUST supply this or --text
--region / -r
: OPTIONAL; if not supplied, region will fallback to configured (or default) providerOptions
--key / -k
: OPTIONAL; KMS Key ID or the shortcut name from the keys
section of the config--description / -d
: OPTIONAL; Description of the secret--plaintext
: OPTIONAL; Store the secret in plaintextUsage samples
Simple encrypted storageserverless secrets set --name my-secret --text my-secret
serverless secrets set -n my-secret -t my-secret
Encrypted storage with description, custom KMS key, and region
serverless secrets set --name my-secret --text my-secret --key "alias/aws/ssm" --description "My secret" --region us-west-2
serverless secrets set -n my-secret -t my-secret -k alias/aws/ssm -d "My secret" -r us-west-2
Plaintext storage
serverless secrets set --name not-so-secret --plaintext --text my-secret --description "Not-so-secret"
File storage (pay attention to your provider's limits here!)
serverless secrets set --name my-secret-file --file file.txt
serverless secrets set -n my-secret-file -f file.txt
serverless secrets get name
Retrieve decrypted values of your secrets from the provider's secret storeParameters
--name / -n
: REQUIRED; name of the secret--region / -r
: OPTIONAL; if not supplied, region will fallback to configured (or default) providerOptions
Usage samples
serverless secrets get --name my-secret --region us-west-2
serverless secrets delete name
Delete your secrets from the provider's secret storeParameters
--name / -n
: REQUIRED; name of the secret--region / -r
: OPTIONAL; if not supplied, region will fallback to configured (or default) providerOptions
Usage samples
serverless secrets delete --name my-secret --region us-west-2
serverless secrets list-remote
List all of the secrets (name and description only) stored in a given regionParameters
--region / -r
: OPTIONAL; if not supplied, region will fallback to configured (or default) providerOptions
Usage samples
serverless secrets list-remote --region us-west-2
serverless secrets validate
2 step validation process:- No duplicates between
environment
andenvironmentSecrets
sections: This ensures that there are no conflicts
- All secrets exist in the provider's secret store: all values listed in all
environmentSecrets
sections are
If any issues are found, an error is thrown with a detailed log of all detected issues.
Parameters
--region / -r
: OPTIONAL; if not supplied, region will fallback to configured (or default) providerOptions
Usage samples
serverless secrets validate --region us-west-2
other serverless commands
These notes cover commands likeserverless deploy
and serverless package
Flags
You can provide--skipValidation
and/or --omitPermissions
to activate these options, even if you have not
turned them on in the custom.serverlessSecrets
section of your serverless.yml
. See options documentation for details.The
--region / -r
flag for things like serverless deploy
is completely ignored by the Serverless Secrets plugin during
packaging/deployment. You should have this configured in your custom.serverlessSecrets
section of your serverless.yml
.Processing during packaging
Serverless Secrets performs a good amount of its magic during any operation that include packaging of your project. Let's cover those steps:- All configuration data is written to a JSON file called
.serverless-secrets.json
.
.gitignore
(or other VCS exclusion config)..serverless-secrets.json
is added to your package- All of your
environmentSecrets
are converted intact to regular environment variables. This is strictly
- Permissions to access the secret store are injected into roles.
- Secret validation is performed. For details on this process, see the
serverless secrets validate
CLI command.
Client
The client can automatically load all of your secrets into environment variables, or you can choose to load them individually. Decryption is done automatically, meaning that the full plaintext will be loaded into the environment variable. You may still want to do post processing on it, particularly in the case of the files.Remember: do not
require
modules that need the environment variables loaded by the client before the
environment variables have actually been loaded.client.load(options)
Parameters:options
- object: The options object as described in thecustom.serverlessSecrets
section above.
custom.serverlessSecrets
configuration.Returns: Promise
Side effects: Uses generated configuration to determine the environment variables to be filled and the keys to request from the secret store to fill those variables. After the secret store responds, the environment variables are then set to the corresponding returned secrets.
Sample code:
// Given: a secret named '/my-project/dev/api-key' is stored in SSM with value 'mySecret'
// Given: an environmentSecret named 'API_KEY' exists with a value of '/my-project/dev/api-key'
const secretsPromise = require('serverless-secrets/client').load();
module.exports.handler = function (event, context, callback) {
secretsPromise.then(() => {
// process.env.API_KEY now contains 'mySecret'
// logic goes here
}).catch(err => {
// handle errors here
});
};
client.loadByName(environmentVariableName, parameterName, options)
Parameters:environmentVariableName
- string: name of the key to be added toprocess.env
that
parameterName
- string: name of the secret to be retrieved from the secret storeoptions
- object: The options object as described in thecustom.serverlessSecrets
section above.
custom.serverlessSecrets
configuration.Returns: Promise
Side effects: Retrieves
parameterName
from the secret store and loads it
into process.env[environmentVariableName]
Sample code:
// Given: a secret named '/my-project/dev/api-key' is stored in SSM with value 'mySecret'
const secretsClient = require('serverless-secrets/client');
const ssPromise = secretsClient.loadByName('API_KEY', '/my-project/dev/api-key');
module.exports.handler = function (event, context, callback) {
ssPromise.then(() => {
// process.env.API_KEY now contains 'mySecret'
// logic goes here
}).catch(err => {
// handle errors here
});
};
Use with serverless-webpack
Normally, the secrets client is initialized with a dynamic require. With webpack that does
not work very well, because any requires are evaluated at compile time to detect the
project's dependencies. If you use serverless-webpack
you have to configure your project
as follows.Include .serverless-secrets.json
Include the secrets configuration file by using the file-loader
, so that it is integrated
into the compiled sources and can be dereferences by webpack when required.// webpack.config.js
module.exports = {
...
module: {
rules: [
...
{
test: /\.serverless-secrets.json$/,
use: [
{
loader: 'file-loader'
}
],
},
...
]
},
};
Load and initialize the secrets configuration in your handler
const secretsClient = require('serverless-secrets/client');
// Initialize the client with the configuration file.
// Webpack will resolve this automatically. Adjust the relative path accordingly.
secretsClient.init(require('../../.serverless-secrets.json'));
const ssPromise = secretsClient.loadByName('API_KEY', '/my-project/dev/api-key');
module.exports.handler = function (event, context, callback) {
...
};
serverless-webpack
The plugin can be used with serverless-webpack. Just configure the service according to the following sections.The service
There are only a few requirements for the service definition. See the sample yaml snippet below to see how things have to be set.You have to add the webpack and the secrets plugin to your
serverless.yml
, so that the secrets plugin precedes the webpack plugin. This is important because webpack relies on the files generated by the secrets plugin.Define the
environmentSecrets
section as stated in the README.Currently
aws-sdk
is mentioned as production dependency in the secrets plugin. With webpack that would lead to a packaging of the SDK. We do not want that as it only bloats the package and AWS Lambda already has a working SDK installed. So we tell the webpack plugin to exclude the aws-sdk.It is important to use
serverless-webpack@^4.0.0
as the forced exclude is only available there.# serverless.yml
...
plugins:
- serverless-secrets
- serverless-webpack
...
provider:
...
environmentSecrets:
API_KEY: 'testsecret'
...
custom:
webpackIncludeModules:
forceExclude:
- aws-sdk
The Webpack configuration
Because serverless-secrets is a development dependency and the actual client, that is used in our production handlers is contained in there, the easiest way to get (only) the client in, is to have it just bundled. This can be achieved by tellingnode-externals
to include it, instead of marking it as external module.
Nothing more has to be done to the webpack configuration file.module.exports = {
entry: slsw.lib.entries,
target: 'node',
externals: [
nodeExternals({
whitelist: [
"serverless-secrets/client"
]
})
],
...
}
The Lambda handler
With the changes in the PR we now can initialize the secrets plugin client with a preloaded.serverless-secrets.json
. Because we bundle the client, we can just require the temporary config file in our handler. Then webpack will just bundle the client AND the config, so that the compiled handler now contains everything needed and does not have any external dependencies anymore.I commented the new behavior in the sample handler below.
// Given: a secret named 'testsecret' is stored in SSM with value 'myvalue'
// Given: an environmentSecret named 'API_KEY' exists with a value of 'testsecret'
// This require will be removed by webpack, because it bundles the client now.
const secrets = require('serverless-secrets/client');
// We use the new (PR) init function here, to initialize the secrets plugin client with the
// preloaded pregenerated JSON. With this require, webpack will automatically bundle the JSON
// and removes the needed dynamic require that would fail.
// This has to be a static path. The file is located in the root of the service. So you have to
// specify the proper backwards relative path from your handler directory.
secrets.init(require('./.serverless-secrets.json'));
module.exports.handler = function (event, context, callback) {
const secretsPromise = secrets.load();
secretsPromise.then(() => {
// process.env.API_KEY now contains 'myvalue'
console.log(process.env.API_KEY);
callback(null, { statusCode: 200 });
}).catch(err => {
// handle errors here
callback(null, { statusCode: 500, body: err.message });
});
};
The output from the very top is exactly from this handler.The package / deployment
You can check withserverless package
that it works. However there is a warning but it can be ignored (see below).$ node node_modules/serverless/bin/serverless deploy
Serverless: Generating Serverless Secrets Config
Serverless: Serverless Secrets beginning packaging process
Serverless: Writing .serverless-secrets.json
Serverless: Validating secrets
Serverless: Secrets validated
Serverless: Adding environment variable placeholders for Serverless Secrets
Serverless: Bundling with Webpack...
Time: 660ms
Asset Size Chunks Chunk Names
first.js 8.41 kB 0 [emitted] first
[0] C:/Projects/serverless/serverless-secrets/client/index.js 2.11 kB {0} [bu
ilt]
[1] ./first.js 725 bytes {0} [built]
[2] external "path" 42 bytes {0} [not cacheable]
[3] external "lodash" 42 bytes {0} [not cacheable]
[4] C:/Projects/serverless/serverless-secrets/lib/constants.js 68 bytes {0} [
built]
[5] C:/Projects/serverless/serverless-secrets/lib/providers/aws.js 1.85 kB {0
} [built]
[6] external "aws-sdk" 42 bytes {0} [not cacheable]
[7] C:/Projects/serverless/serverless-secrets/client 160 bytes {0} [built]
[8] ./.serverless-secrets.json 226 bytes {0} [built]
WARNING in C:/Projects/serverless/serverless-secrets/client/index.js
20:24-85 Critical dependency: the request of a dependency is an expression
The critical webpack warning emitted can be ignored, because with the PR the init function will use the argument given to init to initialize the configuration. Here is the code generated by webpack, where you see, that it works and is safe.
function init(config) {
if (!secrets) {
secrets = config || !(function webpackMissingModule() { var e = new Error("Cannot find module \".\""); e.code = 'MODULE_NOT_FOUND'; throw e; }());
}
}
Misc
AWS IAM
If you disable automatic permission injection, remember to grant your lambda functions access to get parameters from SSM in yourserverless.yml
. Example:provider:
iamRoleStatements:
- Effect: "Allow"
Action: "ssm:GetParameters"
Resource: "arn:aws:ssm:${region}:${awsAccountId}:parameter/*"
Future feature ideas
- Clone secrets from one region to another