Node.js serverless microservice on AWS + SQLite database. Fully working example explained
In this tutorial, a step-by-step guide to create and deploy a RESTful Serverless microservice Node.js + SQLite on AWS lambda.
If you what you need is exposing some static data over HTTP and you are happy with a serverless architecture, this article is for you.
Project creation
npm init
npm install --save express serverless-http As you probably already know, Express.js is the lightweight node framework to handle HTTP requests that will be used in our index.js
Serverless-http is an useful middleware library that allows (with one simple call) to expose our Express.js routes to API gateway.
Hello world
This code declares one route returning “Hello” when the home URL (/) is hit:
const serverless = require('serverless-http');
const express = require('express')
const app = express()
app.get('/', function (req, res) {
res.send('Hello !')
})
module.exports.handler = serverless(app);Deploy (on AWS with serverless package)
Serverless, specifically on AWS, works by creating the app on the fly from a S3-zip file (containing this file above + the node_modules directory). You can find info on the AWS documentation. To deploy this infrastructure, it’s normally easier to use the serverless library. This library auto-deploys from the command line based on a configuration file (see below). Install and configure with your AWS credentials if you haven’t done it already. After it, you’ll have the CLI command sls available.
Add “serverless.yml” into your project root with the following content, where you specify the environment and the mapping between the route and the handler (the code being invoked).
service: YOURSERVICENAME
frameworkVersion: ^2
provider:
name: aws
runtime: nodejs12.x
stage: prod
region: us-east-1
functions:
app:
handler: index.handler
events:
- http: ANY /
- http: 'ANY {proxy+}'Launch the command sls deploy and the serverless tool will create the ZIP file, upload to S3, create the URL and configure the index.js code as handler.
When it ends, it’ll display the URL that you can hit (see “endpoints”). If there are no errors, it’ll return “Hello”.
curl https://xyz.execute-api.us-east-1.amazonaws.com/prod/hello
HelloIn the next steps, I’ll improve the service to do something useful: return data from a SQLite database. I’ll use some quotes from CSV files, organised by tag.
Alternative: Deploy with Terraform
If you prefer to deploy using a more generic tool, you can do that with Terraform, a more generic tool to declare the infrastructure you want and apply on any provider. See my full article on how to deploy a lambda function to AWS
Add the SQLite database
DynamoDB is probably your best choice if you need a NoSQL serverless autoscaling pay-what-you-use database. In case the data you handle is static (created at build time), you can instead just ship the data along with the app code. SQLite is probably the best choice if you need a SQL database in a single file. You could also attach a network file system in AWS where you could edit the SQLite file with the database, but I’ve never used that approach therefore not sure it could viable and reliable.
You can find the fully working application in my Github repository. I’ll explain the various files composing it:
- Script to create the SQLite database from CSV files: This script is launched only once when the container is built the first time (docker-compsose build). It parses the CSV files (I’m using vanilla ES6 for this, as CSV needs some fixing to be added correctly), then adds data into the SQLite database (db directory). This database will be included in the S3 zip file of course. To efficiently bulk-insert into SQLite, you need to wrap the queries into transactions (including the statement creation), refer to the library docs if needed.
- Dockerfile and docker-compose.yml, and why Docker should be used: You don’t necessarily need to use docker, but it’ll make easier to debug potential errors due to different runtime on the serverless platform. If you are developing on MacOS and you don’t have the exact node version running locally as the one in serverless.yaml, you’ll only find out of potential errors if you debug from Amazon Web GUI. For example, in my case I was locally running node 14 and the ES6 operator ?? was not supported by the JS version in node 12 (latest version currently supported in AWS), and also SQLITE node_modules didn’t work on Linux (serverless) when installed from Mac (different binaries).
Note that Dockerfile has the logic to create the database automatically, and to re-create it if any of the CSV changed. I’m sharing the whole app in the docker-compose, but you can customise this if you have performance problems in bigger apps. - Deploy config serverless.yml: Similar to what you saw in the example above. Note that I’ve excluded csv-sources from the package, you don’t need those to be uploaded to S3, the endpoints readcfrok the db. Exclude anything not needed for the app to run to make the ZIP smaller.
- App logic (index.js): Basically two Express.js endpoints, each one simply reading from the SQLite database, and returning records in JSON format. The second one supports a query parameter.
const dbPath = __dirname + '/db/quotes.db';app.get('/tags', function (req, res) {
const db = new sqlite3.Database(dbPath);
db.all("SELECT DISTINCT tag from quotes", (err, rows) => {
if (err) {
throw new Error(err);
}
const tags = rows.map(row => row.tag);
res.json(tags);
});
});app.get('/quotes', function (req, res) {
const db = new sqlite3.Database(dbPath);
const limit = req.query.limit ? req.query.limit : 1;
const tag = req.query.tag ? req.query.tag : false;
const sql = tag ?
`select * FROM quotes WHERE tag="${tag}" ORDER BY random() LIMIT ${limit}`
: `select * FROM quotes ORDER BY random() LIMIT ${limit}`;
db.all(sql, (err, rows) => {
if (err) {
throw new Error(err);
}
res.json(rows);
})
});
Thanks for reading !
What to do next:
- Clap if useful
- Buy me a coffee
- Follow me for more
- Read my other articles below or from my profile
- Keep in touch on LinkedIn
