Web Api Pass File Upload to Lambda

API Gateway supports a reasonable payload size limit of 10MB. One way to work within this limit, but even so offering a means of importing big datasets to your backend, is to allow uploads through S3. This article shows how to utilize AWS Lambda to betrayal an S3 signed URL in response to an API Gateway request. Finer, this allows you to expose a machinery assuasive users to securely upload data straight to S3, triggered by the API Gateway.

The basic menses of the import process is equally follows: the user makes an API, which is served past API Gateway and backed past a Lambda role. The Lambda part computes a signed URL granting upload access to an S3 bucket and returns that to API Gateway, and API Gateway forwards the signed URL back to the user. At this point, the user tin use the existing S3 API to upload files larger than 10MB.

Upload through S3 signed URL

In do, implementing this thought requires several interconnected parts: an S3 bucket, a Lambda function, and the API Gateway. Let'south walk through how to create a working upload system using these components.

Starting time, we need an S3 bucket for storing our data. All objects in S3 are private by default and merely the object owner has permission to admission these objects. Still, the object owner can optionally share objects with others by creating a pre-signed URL, using their own security credentials, to grant time-limited permission to upload or download the objects. Nosotros are going to take reward of this feature to allow users to upload objects to an otherwise private S3 bucket.

Second, we need a Lambda office that generates pre-signed URLs in response to user API requests. In this example, we volition use the Python AWS SDK to create our Lambda office.

Tertiary, nosotros need to expose our Lambda role through API Gateway. This requires creating a basic API that proxies requests to and from Lambda. We will ascertain this API using Swagger and import information technology to API Gateway to start serving requests.

The S3 Bucket

The S3 bucket can exist created via the AWS user interface, the AWS command line utility, or through CloudFormation. The only requirement is that the bucket be set to allow read/write permission only for the AWS user that created the bucket. This is the default prepare of permissions for any new bucket.

For example, we can easily create a new S3 bucket using AWS CLI past running the following command:

i	$ aws s3 mb s3://<my-unique-bucket>

The Lambda role

Given your bucket name, you can examination you take everything setup correctly by generating a pre-signed URL. In the following instance, we utilise the Python AWS SDK (boto) to generate the signed URL and then apply cURL to upload data using a PUT request.

1 2 3 iv 5 6 7 eight nine 10 xi 12 13 14 xv 16

import os import boto3 # Get the service client. s3 = boto3.customer('s3') temp_url = s3.generate_presigned_url( ClientMethod= 'put_object', Params={ 'Bucket': '<my-unique-bucket>', 'Central': '<my-key>' } ) # Upload a file named 'exam.txt' using scroll os.system('roll --request PUT --upload-file ./test.txt "' + temp_url + '"')

API Gateway expects responses to be returned equally JSON, which corresponds to a Python lexicon. We can convert this elementary test programme into a Lambda function usable by API Gateway by returning the signed url as a JSON payload.

i 2 three iv 5 6 7 8 9 10 xi 12 13 14 xv 16 17 eighteen 19 20 21 22 23 24 25

import uuid import boto3 def lambda_handler(outcome, context): # Get the service customer. s3 = boto3.client('s3') # Generate a random S3 primal name upload_key = uuid.uuid4().hex # Generate the presigned URL for put requests presigned_url = s3.generate_presigned_url( ClientMethod= 'put_object', Params={ 'Bucket': '<my-unique-bucket>', 'Key': upload_key } ) # Render the presigned URL return { "upload_url": presigned_url }

Finally, we cypher this function and upload it to AWS as a new Lambda office. Assuming your role is named url_signer.py,

            $ zip -r UrlSigner.zip url_signer.py                      

At present, nosotros tin create the lambda function. Caution must be taken here: the IAM part assigned to the Lambda function must have read/write access to S3 so that it can create the signed URLs, and information technology must be assumable by Lambda.

This implies the office must have the post-obit policy granting access to S3.

1 two 3 4 5 six 7 8 9 10

{ "Version": "2012-x-17", "Argument": [ { "Issue": "Let", "Activity": "s3:*", "Resource": "*" } ] }

And the post-obit trust human relationship, which makes the function assumable by Lambda functions.

            {   "Version": "2012-10-17",   "Statement": [     {       "Outcome": "Allow",       "Master": {         "Service": "lambda.amazonaws.com"       },       "Action": "sts:AssumeRole"     }   ] }                      

With this in place, you tin can create your Lambda role:

            aws lambda create-role \     --region the states-eastward-1 \     --office-name urlsigner \     --cypher-file fileb:///<path-to-zip>/UrlSigner.nix \     --handler url_signer.lambda_handler \     --runtime python2.7 \     --part <your-arn>                      

API Gateway

At this point, we accept an S3 bucket, and a Lambda part that creates signed URLs for uploading to that bucket. The final step is creating the API Gateway frontend that calls the Lambda part. For API Gateway to invoke a Lambda function, you lot must attach a office assumable by API Gateway that has permission to call Lambda's InvokeFunction activeness.

This means you lot must accept a role capable of being causeless by API Gateway with the post-obit trust relationship:

1 ii 3 4 5 6 vii eight 9 10 eleven 12 13

{ "Version": "2012-10-17", "Statement": [ { "Sid": "", "Effect": "Permit", "Principal": { "Service": "apigateway.amazonaws.com" }, "Activeness": "sts:AssumeRole" } ] }

And this role must be able to invoke Lambda functions using the following policy:

1 2 iii 4 5 vi 7 8 ix x

{ "Version": "2012-ten-17", "Argument": [ { "Effect": "Let", "Activeness": "lambda:InvokeFunction", "Resource": "*" } ] }

Now we can define the terminal API. It is fairly straightforward: we define a single API endpoint that integrates with the Lambda function nosotros previously created, and that has the right role for executing the Lambda function. This API returns the signed URL for uploading directly to S3.

one 2 3 four five 6 seven viii 9 ten eleven 12 13 xiv fifteen xvi 17 18 nineteen twenty 21 22 23 24 25 26 27 28 29 thirty 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45

--- swagger: 'ii.0' info: championship: Signed URL Import Service description: | An import service for uploading information to an S3 bucket through a signed URL. version: v1 consumes: - application/json produces: - application/json schemes: - https definitions: Job: type: object properties: upload_url: description: A self-expiring signed URL for uploading files. type: cord paths: /import/: post: summary: Create a new import job. responses: 200: description: Created schema: "$ref": "#/definitions/Job" x-amazon-apigateway-integration: type: "aws" uri: "arn:aws:apigateway:usa-e-i:lambda:path/2015-03-31/functions/arn:aws:lambda:us-e-1:820873945423:office:urlsigner/invocations" httpMethod: "POST" credentials: "arn:aws:iam::820873945423:role/ApiGatewayRole" responses: "default": statusCode: "200" responseTemplates: awarding/json: |- { "upload_url": $input.json('$.upload_url') }

Y'all tin can use the AWS user interface to create a new API using this definition, and deploying information technology as an API phase.

The Final Upshot

At present y'all should be able to make a POST request to your import endpoint and be returned a signed URL:

1 2 3 4 5 vi 7 eight ix 10

$ http Post https://<...>.execute-api.u.s.-east-one.amazonaws.com/v1/import HTTP/1.1 200 OK Connection: continue-alive Content-Length: 609 Content-Type: awarding/json Date: Tue, 23 May 2017 twenty:37:28 GMT { "upload_url": "https://s3.amazonaws.com/<your-saucepan-id>/..." }

You lot can use the signed URL to upload information directly to S3.

Encounter also

• The False Dichotomy of Design-First and Code-First API Development
• The Cathedral, The Bazaar, and the API Marketplace
• Marrying RESTful HTTP with Asynchronous and Event-Driven Services
• Overambitious API gateways
• Comparing Swagger with Thrift or gRPC

waddellwitho1976.blogspot.com

Source: https://sookocheff.com/post/api/uploading-large-payloads-through-api-gateway/