Using ActiveRecord standalone with AWS Lambda and Serverless Framework

AWS Lambda is an absolute marvel. It allows us to build serverless applications, ship them, and forget about them. It covers a fair range of runtimes, and if you’re willing to get your hands dirty, you can make it run in almost any language. Its pricing model is very convenient, especially for apps that are event-driven.

On the other side of the puzzle, ActiveRecord is an Object-Relational Mapping (ORM) framework for working with databases in a way that allows developers to interact with database records as if they were objects in their programming language. It’s a key component of the Ruby on Rails web framework.

So, what happens if you want to use ActiveRecord with AWS Lambda for a small ETL project?

Why Use ActiveRecord and AWS Lambda Together?

I had to write a small app that reads data from an API and saves the response in a PostgreSQL database. For such a small app, I didn’t want to use a full Rails app, and knowing that this app had to run just once a day, serverless seemed like a nice choice. I know there are great tools for building ETLs, and I have used a few in the past, but given the size of the project and also the desire to experiment, I wanted to go down this path.

First Steps

First things first, I needed to create a serverless ruby app, this is very easy to do with the serverless framework.

 
npm install -g serverless
serverless create -t aws-ruby -p myservice

This 2 commands will install serverless as a global package using node and it will create a project ready to deploy tu AWS using the ruby runtime.

cd myservice

run the following command and you will land in the root folder of your project where you will find the following structure.

myservice/
├── handler.rb
└── serverless.yml

this is the basic structure that we need for our serverless app

Installing ActiveRecord and Package Gems

For our app, we need to install a couple of Gems to be able to connect to the database and save the records with get from the API

For that, we will create a Gemfile

# frozen_string_literal: true

source "https://rubygems.org"

git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }

ruby "3.2.2"

gem "activerecord", "~> 7.1"
gem "pg", "1.5.6"

... other gems in this list

Now we need to ensure that our gems are bundled as part as the deployment part of the deployment package. We can do that manually and that can be part of another tutorial, but we have a better way. Serverless have a plugin to create a layer for our lambda function that has all the Gems and libraries. To install it just drop the next command on the terminal.

sls plugin install -n serverless-ruby-layer

This command will generate a package.json file and will update our serverless.yml file

 
service: basic
 
plugins:
  - serverless-ruby-layer
 
provider:
  name: aws
  runtime: ruby3.2
 
functions:
  hello:
    handler: handler.hello

Running sls deploy automatically deploys the required gems as in Gemfile to AWS lambda layer and make the gems available to the RUBY_PATH of the functions hello.handler

Now update your serverless handler to require ActiveRecord like follows

handler.rb

require 'active_record'

def hello(event:, context:)
  { statusCode: 200, body: {
    "active_record_version": ActiveRecord.version.to_s
    }
  }
end

If you try to invoke this function you will get the following error

“Error loading the ‘postgresql’ Active Record adapter. Missing a gem it depends on? libpq.so.5: cannot open shared object file: No such file or directory — /opt/ruby/gems/3.2.0/gems/pg-1.5.6/lib/pg_ext.so”

Error on invoking the function

The reason for this is the pg Gem depends on native libraries, and you need to package those libraries with your Gems. Luckily the serverless-ruby-layer plugin handles this situation.

For this you will need to add a Dockerfile to your code base, this will be your new folder structure.

myservice/
├── handler.rb
├── serverless.yml
├── Dockerfile
└── Gemfile

Dockerfile

FROM public.ecr.aws/lambda/ruby:3.2

RUN yum install -y amazon-linux-extras
RUN amazon-linux-extras enable postgresql14
RUN yum group install "Development Tools" -y
RUN yum install -y postgresql postgresql-devel
RUN gem update bundler

CMD "/bin/bash"

And you will also want to update your serverless.yml file

service: basic
 
plugins:
  - serverless-ruby-layer
 
provider:
  name: aws
  runtime: ruby3.2
 
functions:
  hello:
    handler: handler.hello

custom:
  rubyLayer:
    use_docker: true
    docker_file: Dockerfile
    native_libs:
      - /usr/lib64/libpq.so.5
      - /usr/lib64/libldap_r-2.4.so.2
      - /usr/lib64/liblber-2.4.so.2
      - /usr/lib64/libsasl2.so.3
      - /usr/lib64/libssl3.so
      - /usr/lib64/libsmime3.so
      - /usr/lib64/libnss3.so
      - /usr/lib64/libnssutil3.so

This will create a bundle of your Gems and Libraries dependencies and create an AWS Lambda Layer with those dependencies.

Now if you invoke your library the previous error should disappear.

This worked for me for a couple of weeks, after that, I was updating my function and when I was testing I found the following error.

“Error loading the ‘postgresql’ Active Record adapter. Missing a gem it depends on? /usr/lib64/libnssutil3.so: version `NSSUTIL_3.82' not found (required by /opt/lib/libnss3.so) — /opt/ruby/gems/3.2.0/gems/pg-1.5.6/lib/pg_ext.so”

I first didn't understand the reason for this, but after some tests and errors I found that the lambda runtime already had a version of the library libnssutil3.so that was different from the version I had in /opt/lib

I needed that lambda to load my version instead of the version that comes by default in the runtime. after some debugging and help from ChatGPT, I found an easy solution.

We can use the LD_PRELOAD ENV to tell lambda what version takes precedent. for that, we just need to update our serverless.yml file

service: basic
 
plugins:
  - serverless-ruby-layer
 
provider:
  name: aws
  runtime: ruby3.2
  environment:
    LD_PRELOAD: '/opt/lib/libnssutil3.so'
 
functions:
  hello:
    handler: handler.hello

custom:
  rubyLayer:
    use_docker: true
    docker_file: Dockerfile
    native_libs:
      - /usr/lib64/libpq.so.5
      - /usr/lib64/libldap_r-2.4.so.2
      - /usr/lib64/liblber-2.4.so.2
      - /usr/lib64/libsasl2.so.3
      - /usr/lib64/libssl3.so
      - /usr/lib64/libsmime3.so
      - /usr/lib64/libnss3.so
      - /usr/lib64/libnssutil3.so

after this change, we can redeploy our function and test again, this time it should work like a charm.

and that's it, in this link you can find a Gist with the Dockerfile, Gemfile, and serverless.yml used in this article.