Skip to content

Latest commit

 

History

History
512 lines (343 loc) · 10.7 KB

aws.md

File metadata and controls

512 lines (343 loc) · 10.7 KB

AWS Lambda

{id: intro}

What is Serverless?

{id: serverless}

  • Function as a Service - FaaS

  • 0 cost for deployment

  • $ for usage

  • No root access, just a simple function (or application)

  • No server maintenance costs.

What is it good for?

{id: what-is-it-good-for}

  • For short batch processes (5 min limit)
  • ETL
  • Real time data stream collection and manipulation
  • Scheduled jobs (cron-jobs)
  • REST API web endpoints
  • Bots

AWS Lambda Limits

{id: aws-lambda-limits}

  • AWS Lambda Limits

  • 5 minutes execution

  • disk capacity 500 MG

  • 1024 threads

  • 3Gb memory

  • 50 MB deployment zip

Development

{id: development}

  • In the Lambda console using Cloud9
  • Local environment
  • Local environment using SAM

How does FaaS work?

{id: how-faas-fork}

Trigger (Event Source) -> Function -> Resources (output)

Event Sources (Triggers)

{id: event-sources}

  • Supported Event Sources
  • Amazon API Gateway
  • Scheduled Events
  • Amazon S3
  • Amazon DynamoDB
  • Amazon Simple Email Service
  • ...

Function

{id: function}

  • Python 2.7 and 3.6
  • Node.JS 6.10 and 8.10
  • Java 8
  • C# .NET Core
  • Go 1.x

Resources (via IAM)

{id: resources-for-output}

  • See IAM - Identity and Access Management for more resources
  • S3
  • Amazon DynamoDB
  • SES - Simple Email Service
  • SNS - Simple Notification Service (Sending SMS)
  • Redshift (Data warehouse)
  • ElastiCache clusters
  • RDS - Relational Database Service
  • ...
  • External services

Create an AWS account

{id: create-account}

  • Create Free Account
  • You will have to supply a credit card, but AWS provides a Free tier.
  • You will need to supply a phone number and they will call you.

Start with AWS Lambda

{id: aws-lambda}

Task 1 - Hello World URL

{id: task-1}

  • Create script in Python that can be accessesed via curl.

Hello World in AWS Lambda

{id: hello-world}

  • Press "Create function"

  • Name: "hello"

  • Runtime: Python 3.6

  • Role: Create new role from template(s)

  • Role name: "myrole"

  • Policy templates: Basic Edge Lambda Permission

  • Press "Create function"

The default code will look like this:

  • Test it (click on "Test")
  • First it will make you create a new test-case.

API Gateway

{id: api-gateway}

ERROR 502 Bad Gateway

or

{"message": "Internal server error"}

Add header

{id: add-header}

  • The function needs to return a dictionary with the status code and the headers.
  • At least the Content-type.

  • curl ...

Send JSON

{id: send-json}

  • curl ...
{"message": "Hello World!"}

Event details

{id: event-details}

  • curl ...

Exercise 1

{id: exercise-1}

  • Create your own hello function.

Task 2 - Accept URL GET parameters

{id: task-2}

  • Accept parameter in the GET request and echo it back

Accept Parameters

{id: accept-parameters}

  • Save and Click "Test"
  • Observe the error
{
  "errorMessage": "'queryStringParameters'",
  "errorType": "KeyError",
  "stackTrace": [
    [
      "/var/task/lambda_function.py",
      4,
      "lambda_handler",
      "name = event['queryStringParameters']['name']"
    ]
  ]
}
  • Our Python code is not safe enough, we assume a field "name" to be passed in.

Error via the API

{id: error-via-the-api}

  • Before we fix the code, let's see what happens if we access the URL using curl ?

  • curl ...

{"message": "Internal server error"}

  • To see the error log, visit:

  • Monitoring

  • Invocation errors

  • Jump to logs

Test Event for API Gateway

{id: test-event-for-api-gateway}

  • Before we fix the code, we can fix the test:
{
    "queryStringParameters" : {
        "name": "Foo Bar"
    }
}
  • Try this using the "Test" button.

Also, try it from the console using curl or in your browser (use your own URL).

{"message": "Hello Foo Bar!"}

Exercise 2

{id: exercise-2}

  • Fix the Python code so even if the user does not supply the "name" field it still won't crash.

  • Instead make it return a JSON structure with status "400 Bad Request"

  • Use curl -I or curl -D err.txt to check the headers as well.

  • Create another function that will accept 2 numbers (parameters a and b) and add them together returning a JSON that looks like this:

{
   'a' : 23,
   'b' : 19,
   'op' : '+'
   'result' : 42
}

Solution 2 - echo

{id: solution-2-echo}

Solution 2 - add

{id: solution-2-add}

Task 3 - Multi file application

{id: task-3}

  • Create an application that has more than one files.

Multi-file application - json

{id: multi-file-application-json}

  • Create a file called a.json with some JSON content in it.

  • Change the code to read the file and return it

Multi-file application - python module

{id: multi-file-application-module}

Local development

{id: local-development}

mkdir project
cd project
vim lambda_function.py


zip ../project.zip *
  • Upload a .ZIP file.
  • Save.
  • Try it using curl.

Exercise 3

{id: exercise-3}

  • Create a 'calculator' application that accepts two numbers 'a' and 'b' and an 'operation' that can be either 'add' or 'multiply'.

  • Return the appropirate result.

  • Create it on you computer in two files. A main web serving file and a module with two functions 'add' and 'multiply'

  • On your local computer create a directory for a project.

  • In the directory create a file called 'lambda_function.py' this will hold the main function.

  • Create also a file called 'mymodule.py'.

  • Upload the whole thing using zip.

Solution 3

{id: solution-3-calc}

Task 4 - Use 3rd party Python modules.

{id: task-4}

Development machine

{id: development-machine}

mkdir app_pylev
cd app_pylev
pip install pylev -t pypi
zip -r ../project.zip *
  • curl 'https://qspmdah6oj.execute-api.us-east-1.amazonaws.com/v0/hello?a=abd&b=acx'

Error: must supply either home or prefix/exec-prefix - not both

{id: must-supply-either-home-or-prefix-exec-prefix-not-both}

On OSX you might get the above error. Create the 'setup.cfg' file.

Third party not pure-python

{id: third-party-not-pure-python}

  • editdistance is a Levenshtein distance module written in C++ and Cython
  • See

  • Needs a linux box either locally or on Amazon AWS.

Docker to build 3rd party modules

{id: docker-to-build}

amazonlinux

docker build -t aws .
  • In the project directory:
rm -rf pypi
docker run -v $(pwd):/opt  --rm aws pip install -r requirements.txt -t pypi

zip -r ../project.zip *
  • Upload the zip file.

Exercise 4

{id: exercise-4}

  • Web Client: A function that uses 'requests' to fetch a URL and return the text of the page.

  • A function that will accept the name of two cities.

  • Call the API of Open Weather Map and return the temprature difference in the two places.

Solution: Web client

{id: web-client}

pip install -r requirements.txt -t pypi
zip -r ../project.zip *

curl 'https://qspmdah6oj.execute-api.us-east-1.amazonaws.com/v0/hello?url=https://httpbin.org/get'

Task 5 - Using S3 - Simple Storage Service

{id: s3-storage}

  • Buckets - (all the users share a single name-space for buckets, so a lot of them are already taken)

  • Create a bucket in s3 (eg. your-name)

  • Add permission to the role Attach policy AmazonS3FullAccess

  • boto 3

S3 List bucket

{id: list-s3-bucket}

S3 write object from Lambda

{id: s3-write-object-from-lambda}

Read S3 object

{id: read-s3-object}

Trigger Lambda by S3

{id: trigger-lambda-by-s3}

Trigger by using:

aws s3 cp data.json s3://szabgab/new/data.json

Download the resulting file using

aws s3 cp s3://szabgab/out.json .

Handle S3 event in Lambda

{id: handl-s3-event-in-lambda}

Exercise 5

{id: exercise-5}

  • Create a function that can be triggered by a URL request passing in parameters like this: ?name="Foo Bar"

  • Create an object in S3 called something like "in/time.json" (with the current time)

  • Create another function that will be triggered by a newaobject with a perfix "in/"

  • Load that object and creare a new object called "out/time.json" using the same object name, but adding 3 new fields to it called

new_time:  the time portion of the name of the object
end_time:  the time when we read the object in the second function.
elapsed:   the difference.

AWS Resources

{id: aws-resources}

Resources

{id: resources}