aws: add missing entries
This commit is contained in:
thomasabishop
parent
86a79ef0a4
commit
7ebe6dcd5d
3 changed files with 449 additions and 0 deletions
24
zk/AWS_CLI.md
Normal file
24
zk/AWS_CLI.md
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
---
|
||||
tags: [AWS]
|
||||
---
|
||||
|
||||
# SAM frequent commands
|
||||
|
||||
### Retrieve current user
|
||||
|
||||
```
|
||||
aws-sts get-caller-identity
|
||||
```
|
||||
|
||||
### List users
|
||||
|
||||
```
|
||||
aws configure list
|
||||
aws configure list-profiles
|
||||
```
|
||||
|
||||
### View profile data
|
||||
|
||||
```
|
||||
vim ./aws/credentials
|
||||
```
|
||||
131
zk/Local_AWS_development_with_SAM.md
Normal file
131
zk/Local_AWS_development_with_SAM.md
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
---
|
||||
tags: [AWS, docker]
|
||||
---
|
||||
|
||||
# Local AWS development with SAM
|
||||
|
||||
We can run a local instance of our SAM stack for a given application without
|
||||
sending requests to the cloud. This is implemented through Docker.
|
||||
|
||||
The SAM CLI handles all the Docker-related tasks, such as pulling the required
|
||||
Lambda runtime images, creating containers, mounting your code and dependencies,
|
||||
and running your Lambda functions inside those containers.
|
||||
|
||||
## Basic set up
|
||||
|
||||
Enter your project directory.
|
||||
|
||||
First build your SAM application:
|
||||
|
||||
```sh
|
||||
sam build
|
||||
```
|
||||
|
||||
We then run:
|
||||
|
||||
```sh
|
||||
sam local start-api
|
||||
```
|
||||
|
||||
If you have an API Gateway endpoint that you want to call over the local server.
|
||||
You will be able to call it after executing the above.
|
||||
|
||||
This will be indicated by:
|
||||
|
||||
|
||||
|
||||
If we want to invoke the function directly we use:
|
||||
|
||||
```sh
|
||||
sam local invoke [FunctionName]
|
||||
```
|
||||
|
||||
## Using environment variables
|
||||
|
||||
If you have an API key or database credentials, you are going to typically want
|
||||
to use different values dependent on environment.
|
||||
|
||||
Even if the values are the same accross environments, it's a good idea to not
|
||||
call a secret when working locally since this request is billable.
|
||||
|
||||
In the example below I show how to set up environment variables for an API key
|
||||
locally and in production.
|
||||
|
||||
## Create secret
|
||||
|
||||
Go to AWS SecretsManager and add the API key as a secret. This will be sourced
|
||||
in production.
|
||||
|
||||
## Create local ENV file
|
||||
|
||||
> These must be in JSON to work with SAM:
|
||||
|
||||
### Local env
|
||||
|
||||
```json
|
||||
// local-env.json
|
||||
{
|
||||
"FunctionName": {
|
||||
"API_KEY": "xxx-yyy-xxx",
|
||||
"NODE_ENV": "development"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
We save these to the root of the given function's directory not at the global
|
||||
repo level.
|
||||
|
||||
> Be sure to add this to `.gitignore` so that it does not become public
|
||||
|
||||
### Update `template.yaml`
|
||||
|
||||
Every environment variable you intend to use, must exist in the `template.yaml`,
|
||||
otherwise it will not be sourced at runtime:
|
||||
|
||||
```yaml
|
||||
...
|
||||
Resources:
|
||||
Properties:
|
||||
Environment:
|
||||
Variables:
|
||||
SECRET_ARN: "arn:aws:secretsmanager:eu-west-2:885135949562:secret:wakatime-api-key-X9oF3v",
|
||||
NODE_ENV: production
|
||||
API_KEY:
|
||||
...
|
||||
```
|
||||
|
||||
> We go ahead and populate the values for production. But we leave the variable
|
||||
> we use in development blank, since we don't want it committed and we will
|
||||
> source it at the SAM invocation. It still needs to exist though.
|
||||
|
||||
### Pass in the environment variable at invocation:
|
||||
|
||||
```sh
|
||||
sam local start-api --env-vars /home/thomas/repos/lambdas/wakatime-api/get-coding-stats/local-env.json
|
||||
```
|
||||
|
||||
In production, the variables required will be automatically sourced from the
|
||||
`template.yaml`
|
||||
|
||||
### Create handler within the Lambda itself
|
||||
|
||||
You are obviously going to need to distinguish between the different deployments
|
||||
when the Lambda executes. Here is an example helper function:
|
||||
|
||||
```ts
|
||||
import * as AWS from "aws-sdk";
|
||||
|
||||
const secretsManager = new AWS.SecretsManager();
|
||||
|
||||
async function getApiKey(): Promise<string> {
|
||||
if (process.env.NODE_ENV === "production") {
|
||||
const response = await secretsManager
|
||||
.getSecretValue({ SecretId: process.env.SECRET_ARN as string })
|
||||
.promise();
|
||||
const secretValues = JSON.parse(response.SecretString as string);
|
||||
return secretValues.API_KEY;
|
||||
} else {
|
||||
return process.env.API_KEY as string;
|
||||
}
|
||||
}
|
||||
```
|
||||
294
zk/SAM.md
Normal file
294
zk/SAM.md
Normal file
|
|
@ -0,0 +1,294 @@
|
|||
---
|
||||
tags: [AWS]
|
||||
---
|
||||
|
||||
# AWS SAM
|
||||
|
||||
SAM stands for **serverless application model**. It is a framework developed by
|
||||
AWS to simplify the process of building, deploying and managing serverless
|
||||
applications. It provides a concise syntax for defining the components of a
|
||||
serverless application, such as
|
||||
[Lambda functions](zk/Lambda_programming_model.md),
|
||||
[API gateway](/zk/AWS_API_Gateway.md) and database tables.
|
||||
|
||||
The SAM infrastructure is defined in a YAML file which is then deployed to AWS.
|
||||
SAM syntax gets transformed into CloudFormation during the deployment process.
|
||||
(CloudFormation is a broader and more robust AWS tool for large, highly
|
||||
scaleable infrastructures).
|
||||
|
||||
## Key features of SAM
|
||||
|
||||
- Single deployment configuration
|
||||
- Integration with development tools
|
||||
- Local testing and debugging
|
||||
- Built on AWS CloudFormation
|
||||
|
||||
## Main technologies required
|
||||
|
||||
### Docker
|
||||
|
||||
Whilst SAM can be used to create a deployable file for AWS it can also be run as
|
||||
a container for local development with Docker.
|
||||
|
||||
### AWS CLI
|
||||
|
||||
This is installed using Python and allows you to interact directly with AWS via
|
||||
the command-line.
|
||||
|
||||
### AWS SAM CLI
|
||||
|
||||
See
|
||||
[https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html)
|
||||
|
||||
## Setting up credentials for the AWS CLI
|
||||
|
||||
You require an access key for the given
|
||||
[IAM user](zk/AWS_User_management_and_roles.md#iam). You should create an IAM
|
||||
account specific to the project with bounded permissions.
|
||||
|
||||
```
|
||||
aws configure
|
||||
AWS Access Key ID [None]: AK*******
|
||||
AWS Secret Access Key [None]: ukp******
|
||||
Default region name [None]:
|
||||
Default output format [None]:
|
||||
```
|
||||
|
||||
This information can be found in the Security Credentials section of the given
|
||||
[IAM](zk/AWS_User_management_and_roles.md#iam) user:
|
||||
|
||||
|
||||
|
||||
### Switching between credentials
|
||||
|
||||
You should set up a different IAM user for each project.
|
||||
|
||||
You can do this with:
|
||||
|
||||
```sh
|
||||
aws configure --profile <profile-name>
|
||||
```
|
||||
|
||||
This will then ask you to add the credentials for the user.
|
||||
|
||||
You can switch between different credentials for the user as follows:
|
||||
|
||||
```sh
|
||||
AWS_PROFILE=<profile-name> sam build
|
||||
|
||||
```
|
||||
|
||||
## Starting a SAM project
|
||||
|
||||
First create a directory for your project which will serve as the repository:
|
||||
|
||||
```sh
|
||||
mkdir aws-sam-learning
|
||||
cd aws-sam-learning
|
||||
```
|
||||
|
||||
Then we can use the `sam` cli to bootstrap the project:
|
||||
|
||||
```sh
|
||||
sam init --runtime nodejs16.x
|
||||
```
|
||||
|
||||
We can just click through and accept the basic HelloWorld Lambda.
|
||||
|
||||
This will create the Lambda as well as an API Gateway trigger URL.
|
||||
|
||||
### `template.yaml`
|
||||
|
||||
This is autogenerated and details the main constituents of the project. There
|
||||
are lots of fields but the most important are the following:
|
||||
|
||||
```yaml
|
||||
HelloWorldFunction:
|
||||
Type: AWS::Serverless::Function
|
||||
Properties:
|
||||
CodeUri: hello-world/
|
||||
Handler: app.lambdaHandler
|
||||
Runtime: nodejs16.x
|
||||
Architectures:
|
||||
- x86_64
|
||||
Events:
|
||||
HelloWorld:
|
||||
Type: Api
|
||||
Properties:
|
||||
Path: /hello
|
||||
Method: get
|
||||
```
|
||||
|
||||
This details the location of the [handler function](/Lambda_handler_function.md)
|
||||
which is contained at the path `hello-world/app.js`:
|
||||
|
||||
```js
|
||||
exports.lambdaHandler = async (event, context) => {
|
||||
try {
|
||||
// const ret = await axios(url);
|
||||
response = {
|
||||
statusCode: 200,
|
||||
body: JSON.stringify({
|
||||
message: "hello world",
|
||||
// location: ret.data.trim()
|
||||
}),
|
||||
};
|
||||
} catch (err) {
|
||||
console.log(err);
|
||||
return err;
|
||||
}
|
||||
|
||||
return response;
|
||||
};
|
||||
```
|
||||
|
||||
It also lists the `get` event that we can use to call API Gateway and trigger
|
||||
the Lambda.
|
||||
|
||||
The full template is below:
|
||||
|
||||
|
||||
|
||||
## Adding our own code
|
||||
|
||||
We will create our own function and API Gateway trigger.
|
||||
|
||||
We will place our function after the existing `HelloWorldFunction`
|
||||
|
||||
```yaml
|
||||
ClockFunction:
|
||||
Type: AWS::Serverless::Function
|
||||
Properties:
|
||||
CodeUri: clock/
|
||||
Handler: handler.clock
|
||||
Runtime: nodejs16.x
|
||||
Events:
|
||||
ClockApi:
|
||||
Type: Api
|
||||
Properties:
|
||||
Path: /clock
|
||||
Method: get
|
||||
```
|
||||
|
||||
We can test the syntax with:
|
||||
|
||||
```sh
|
||||
sam validate
|
||||
```
|
||||
|
||||
Just like with `HelloWorld`, we will create a directory for this function:
|
||||
`clock` and we will initialise it as an `npm` project.
|
||||
|
||||
```sh
|
||||
mkdir clock
|
||||
cd clock
|
||||
npm init
|
||||
```
|
||||
|
||||
We will use `handler.js` as our root, handler function.
|
||||
|
||||
We have said in the template file that our `Handler: handler.clock`, therefore
|
||||
the main function in the `handler` module should be `clock`:
|
||||
|
||||
```js
|
||||
const moment = require("moment");
|
||||
|
||||
exports.clock = async (event) => {
|
||||
console.log("Clock function run");
|
||||
const message = moment().format();
|
||||
const response = {
|
||||
statusCode: 200,
|
||||
body: JSON.stringify(message),
|
||||
};
|
||||
return response;
|
||||
};
|
||||
```
|
||||
|
||||
The directory structure is as follows:
|
||||
|
||||
|
||||
|
||||
When we call the API Gateway path `/clock` with `GET`, our function will be
|
||||
triggered.
|
||||
|
||||
## Deploying the project
|
||||
|
||||
We will now deploy our project to AWS from the local environment.
|
||||
|
||||
The process is as follows:
|
||||
|
||||
1. Build
|
||||
2. Package
|
||||
3. Deploy
|
||||
|
||||
### Build
|
||||
|
||||
We need to install the runtime dependencies for the function. We do this by
|
||||
running `sam build`. This ignores test files and development dependencies and
|
||||
installs the project dependencies and source files to a temporary subdirectory.
|
||||
|
||||
|
||||
|
||||
The build directory is `.aws-sam/build/`. There will be a subdirectory for each
|
||||
of our files.
|
||||
|
||||
### Package
|
||||
|
||||
As noted, CloudFront handles the deployment of the application. It can only
|
||||
receive one file as an input. The packaging process consists in creating that
|
||||
single file.
|
||||
|
||||
The packaging proces will first archive all of the project artefacts into a zip
|
||||
file and then upload that to [S3](zk/AWS_S3.md). A reference to this S3 entity
|
||||
is then provided to CloudFormation.
|
||||
|
||||
|
||||
|
||||
The command is as follows:
|
||||
|
||||
```sh
|
||||
sam package
|
||||
--template-file template.yaml
|
||||
--output-template-file pkg.yml
|
||||
--region eu-west-1
|
||||
```
|
||||
|
||||
This will automatically create a hashed bucket name for you in S3 (I have tried
|
||||
to add my own naming but it doesn't comply.)
|
||||
|
||||
### Local development with Docker
|
||||
|
||||
In order to work with your application locally without actually sending requests
|
||||
to AWS and using credit, you can run a local instance.
|
||||
|
||||
See [Local AWS Development with SAM](zk/Local_AWS_development_with_SAM.md).
|
||||
|
||||
### Deploy
|
||||
|
||||
Once you have packaged the app you can deploy with `sam deploy --guided`. This
|
||||
will talk you through the defaults and will deploy the package to
|
||||
CloudFormation. In CloudFormation each individual project is called a **stack**.
|
||||
|
||||
If we then go to Cloud Formation we will see the deployed application.
|
||||
|
||||
|
||||
|
||||
## Call the endpoint
|
||||
|
||||
If we now go to the Lambda console, we will see our function listed, and the API
|
||||
Gateway endpoint under `triggers`:
|
||||
|
||||
|
||||
|
||||
We can then call this from Postman to check everything is working as it should:
|
||||
|
||||
|
||||
|
||||
## Clean up and erase the stack
|
||||
|
||||
We can delete the stack and remove all the resources we have created with a
|
||||
single CLI method:
|
||||
|
||||
```sh
|
||||
aws cloudformation delete-stack --stack-name <name> --region <region>
|
||||
```
|
||||
Loading…
Add table
Reference in a new issue