Microservices API’s with serverless framework, AWS Lambda, API Gateway and Auth0 – Part1

Over the last 3 years we have been building a product, and like most products, we have a suite of mobile applications and a web presence. When we started this journey, we created an “old school” tiered architecture and I made the decision to build the API layer using familiar technology, the old and trusted JavaEE JAX-RS stack.

We have since changed the architecture a couple of times, going from JavaEE to .NET CORE Web API’s containerised in docker, and deployed to AWS ECS and now to a completely serverless architecture, which is the base of this blog.

So why serverless?

Let me start by stating that there is nothing wrong with our current .NET CORE Web API containerised solution, the problem for us is purely a cost factor one at the moment. To get the true value from the containerised solution, we need to manage a cluster larger than what we might need, and that means paying for the underlying EC2 infrastructure whether used or not.

So back to the question, why serverless? Based on the above reason, hosting cost is a major factor in this move. Being able to start small, but still having the ability to scale as the load increases without having to manage clusters or change anything is the selling factor right now.

So with the intro done, let’s jump into the solution and the tech.

High-level solution overview

The above diagram details the high-level architecture I have in mind to achieve my micro-services API architecture. Clients will be authenticated, using oauth2 (powered by Auth0), and the bearer tokens will be passed to AWS API Gateway, which will validate the token against Auth0 (using a custom authoriser lambda function) before routing the traffic to the appropriate AWS Lambda function. All our information is stored in a relational database.

As part of the POC exercise, I also wanted to enable a CI/CD pipeline to enable automated build and deployment to AWS.

Technology overview

• Serverless framework – abstracts a lot of the AWS (and other cloud providers) specifics away. Allowing you to focus on writing the micro-service. It also handles infrastructure-as-code, so all AWS resources will be created or updated on deployment
• NodeJs – opted for NodeJs, as it is the widest adopted technology across the different cloud providers. Although moving the serverless functions from 1 cloud provider to the next will not be as simple as container-based solutions, the use of NodeJs and Serverless framework above will simplify the task if ever needed.
• Bitbucket – GIT repository and pipelines for CI/CD
• VSCode – Open source IDE for modern web development.
• Ohhh wait, and AWS 🙂

The POC

Now that you have an overview of the solution ( POC ), let’s go through the steps required to get a working hello-world example.

Setting up the serverless framework

At this point, I assume that you have already downloaded and setup NodeJs. Open your terminal, and use the following command to install the serverless framework:

mkdir my-serverless-api && cd my-serverless-api
npm init -f

npm install --save express serverless-http

So we installed the serverless framework, created our project directory and changed to it. We then initialised a NodeJs application, and lastly, we installed the express and serverless-http modules. The serverless-http package is a handy piece of middleware that handles the interface between your Node.js application and the specifics of API Gateway. While building my POC, I used a handy blog on the serverless framework site, and highly recommend that you go through that as well.

Now let’s open the project in VSCode and start creating our awesome serverless API. Open VSCode and add the my-serverless-api folder we created in the previous step. Your project folder should look like:

Now add a new index.js file to the route of your project and add the following code.

const serverless = require('serverless-http');
const express = require('express')
const app = express()

app.get('/hello-world', function (req, res) {
    var response = {
        message: "This is an awesome serverless API"
    }
    res.send(response)
})

module.exports.handler = serverless(app);

Your project structure should now also look like:

The code is fairly simple and straightforward, we are simply using standard express, and importing the serverless-http package. We exported a handler function which is our application wrapped in the serverless package.

Deploying the API

Now, with our awesome API code written, we need to start working on getting it deployed. In VSCode, add a new serverless.yml file and copy the following into it:

service: my-express-application

provider:
  name: aws
  runtime: nodejs8.10
  stage: dev
  region: us-east-1

functions:
  app:
    handler: index.handler
    events:
      - http: 
          path: /hello-world
          method: get

The configuration is pretty simple. We have created one function, called app, that uses the handler we exported from our index.js file created earlier. We also defined an HTTP event, that filters for the path we specified in our index.js function as well as an HTTP GET method call.

At the top of the file, we specify that we are deploying to AWS, using a NodeJs 8.10 runtime. The stage parameter will be appended to our API Gateway URL as well as our Lambda function. This is really neat for having separate API’s and functions across different environments. And we can manage that with our CI/CD pipeline that will be covered later on.

Head over to your AWS account and create a new user with programmatic access. For now, give the user administrative privileges, as AWS resources will be created on deployment. The serverless framework guys are working on providing a more specific list of exact roles required, but at the moment the features are being added so quickly, that keeping that list up-to-date is a challenge.

Run the following command using your newly created credentials. For more information around the setup and other options, reference the following serverless guide.

serverless config credentials --provider aws --key YOUR_KEY --secret YOUR_SECRET

Now that your AWS credentials are all set up, we can run our deployment. To do that, run the following from within your my-serverless-api directory:

sls deploy

If the deployment was successful you should receive a URL that we can use to test our new awesome serverless API. In my case, the deploy command produced the following output:

Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (680.16 KB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
...........................................
Serverless: Stack update finished...
Service Information
service: my-express-application
stage: dev
region: us-east-1
stack: my-express-application-dev
api keys:
  None
endpoints:
  GET - https://w3rayhfbj4.execute-api.us-east-1.amazonaws.com/dev/hello-world
functions:
  app: my-express-application-dev-app
Serverless: Removing old service artifacts from S3...

Copy and paste the URL from the output and paste it into your browser, and “voila”, you should receive your JSON payload:

{"message":"This is an awesome serverless API"}

If you want to remove all the resources from AWS, you can simply run:

serverless remove

Where is the rest?

In part 2 & 3, I will cover adding Auth0 custom authoriser functionality as well as the CI/CD setup.

Please feel free to leave me a comment or suggestion, but please be kind, this is actually my first blog post ever…