Databases on Soldier's 5

SQL Joins with Sequelize

Sun, 11 Oct 2020 00:00:00 +0000

While working on an API for my day job last week, I needed to do a SQL Inner Join with Sequelize and Typescript in a web API. Here’s how I achieved it.

Environment:

Things move quickly in the JS ecosystem. Here are the library versions used in the subject API at the time of writing:

"dependencies": {
    "@types/express": "~4.0.39",
    "@types/sequelize": "^4.27.21",
    "express": "~4.16.2",
    "sequelize": "4.38.0",
  }

Context:

I have an identifier in one system that I need to correlate with an identifier in a second system, but I could only do that via an intermediate identifier, also in the second system. A good case for a SQL Join.

System 1	System 2	System 2
Electronic Identifier	Animal Identifier	Legacy Animal Identifier
ABCD	555 WXYZ	1234

In System 1, I have enough information to send this request:

{
    "items": [
        "ABC",
        "DEF",
        "GHI"
    ]
}

And in response I want:

{
    "links": [
        {
            "rel": "self",
            "href": "https://api.com/123-i-am-a-uuid"
        }
    ],
    "reference": "123-i-am-a-uuid",
    "items": [
        {
            "legacyId": 123,
            "electronicId": "ABC"
        },
        {
            "legacyId": 456,
            "electronicId": "DEF"
        },
        {
            "legacyId": 789,
            "electronicId": "GHI"
        }
    ]
}

The initial approach was to make three repository queries to get the data I needed, and then go through three javascript map operations to splice the data into the above format. Using a series of javascript maps is inefficient, particularly as the data set grows.

A better way is to get SQL to do the grunt work, as it’s far more time and memory efficient than the javascript I would have needed to write.

As an aside, I’m not too fond of Javascript on the back end. I don’t feel it’s performant enough and as your codebase grows the lack of a robust type system hurts more and more. Typescript helps address the latter (assuming your libraries support it) but does nothing for the former. A typed language like C# or Java is preferable for web programming in a large enterprise due to better performance, more coherent ecosystem (see: left-pad) and robust type system.

I need to make a map of identifiers from an electronicIdEvent entity (that deals with electronic identifiers) and an animal entity that deals with animal information.

	electronicId	animalId	legacyAnimalId
electronicIdEvent	✓	✓
animal		✓	✓

I need the legacyAnimalId field. To find a legacyAnimalId given an electronicIdEvent, I have to:

Find all electronicIdEvent rows that match an electronicId.
Find all animal rows that match the animalId of the electronicIdEvent where an electronicId is present.
Return all legacyAnimalId rows that match the animalId of both the animal and the electronicIdEvent.

Hopefully, you can see why using javascript maps, and SQL selects for this would be tedious and inefficient.

Here’s a SQL query that demonstrates what I want if you’re more familiar with SQL then Javascript:

select event.eid, event.animalId, animal.id, animal.animalKey
from "electronicIdEvent" event
    inner join "animal" animal
on event."animalId" = animal.id

1. Create associations for Sequzelize, between the data you need to link.

For Sequelize to generate the right SQL, it needs to know about the relationships between your data.

Here’s the code that creates associations in Sequelize:

this.animal.hasMany(this.electronicIdEvent, {
  foreignKeyConstraint: true,
  foreignKey: "animalId",
});
this.electronicIdEvent.belongsTo(this.animal, {
  foreignKeyConstraint: true,
  foreignKey: "animalId",
});

Associations allow the ORM to understand your relations and map One To One, One To Many and Many To Many relationships to appropriate SQL code. The docs describe this here

2. Create a Sequelize repository method that uses this association to form a SQL query that returns the correct data:

There is some stuff in here that’s not great, but it works well enough. If I had more time, I’d figure out a way of using typescript’s not-null override (!), and ideally find a way of making event a typed response.

The pseudocode I want is:

First return all the rows for the electronicId and animalId columns in electronicIdEvent, where the electronicId is present in my electronicIdList
Next join the rows for those columns with the rows for the legacyAnimalId column, where the animalId is the same in both the electronicIdEvent and animal entities.
When the data comes back, drop the animalId column. I don’t need it once it has done the job of being the glue between the other entities and columns.

Or, as Javascript:

// imports omitted for brevity

export class AnimalRepository {
  constructor(
    private db: AnimalTimelineDbContext,
  ) {}

getLegacyAnimalIdentifierByElectronicIdentifier(electronicIdList: Array<string>): Promise<Array<IdentifierMap>> {
    return this.db.electronicIdEvent
      .findAll({
        attributes: ["electronicId", "animalId"],
        where: {
          electronicId: electronicIdList,
        },
        include: [ 
          {
            model: this.db.animal,
            required: true,
            attributes: ["legacyAnimalId"],
          },
        ],
      })
      .then((events) =>
        events.map((event) => ({
          eid: event.electronicId,
          legacyAnimalId: event.Animal!.legacyAnimalId,
        }))
      );
  }
}

Note: Setting required: true in the include block forces the query to only return records which have an associated model, this converts the query from an OUTER JOIN to an INNER JOIN.

Because we only return complete rows (no partials) event.Animal cannot be null, I can override the typescript hint, i.e. event.Animal!.legacyAnimalId.

I’m done at this point. We’ve submitted a list of electronic identifiers and gotten back an Array, which is syntactic sugar for Array<[string, integer]>.

3. Convert the raw data into my map object, and fire the response back.

Because Sequelize is doing most of the grunt work in formatting the data, all I need in my web layer is to add the UUID that the user-submitted and save that map. Once I’ve saved the map and returned the reference to the calling system, the calling system can ask for that map and get the data (this is the most RESTful way of handling the data exchange and I’m omitting that setup for a follow-up post).

Spring Boot - Comparing Liquibase and Flyway

Thu, 23 Apr 2020 00:00:00 +0000

What are Spring Boot, Liquibase and Flyway?

The Spring Ecosystem

When you need to create a web application or an API in Java, be that RESTful, SOAP or GraphQL; whether you’re looking at synchronous HTTP, Asynchronous or Reactive, Messages on Queues or Event Sourced with Kafka, it’s hard to go past the Spring Ecosystem. If nobody ever got fired for buying IBM, you could probably say that nobody ever got fired for creating a Spring-based application.

Spring is an extremely popular, well maintained and well known Java Web Application Framework. Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that you can “just run”.

Spring Boot is to Spring and Java as Ruby on Rails is to Ruby. So if you’re working in a Java environment, there’s a pretty good chance you’re using Spring or Spring Boot. Even if you’re using another framework, it’s worth learning about these - they’re so ubiquitous they’ll be around for a long time!

Database Migrations

If you’re using Spring, there’s a chance you’re using persistence tech like Hibernate , Jooq or Ebean to get data out of your database.

Features come and go; data models change; what happens when you need to change your data model? You run a database migration to add or remove columns or make other changes.

It used to be that this would happen manually; someone would log on to a database, run a bunch of SQL and rely on the fact that you’ve made the same changes in your application. There is a bunch of room for human error here, though. So we improve on this by trying to make our database changes safer. Version control, repeatability, and testability become applicable to more parts of the stack. Databases included.

Both Liquibase and Flyway help us with these migrations. Spring helps us with Liquibase and Flyway in turn. The result is a database changeset that does what we have specified every single time (of course, we might still specify the wrong thing - computers remain computers). All we need to do is write a migration and dump it into a folder in our Spring project to achieve that result. Spring handles the rest.

Liquibase

Liquibase is offered free and paid. They don’t share pricing on their website, though, which makes me suspicious. They don’t do an excellent job of describing why I would want to pay either. Liquibase offers migrations both through XML and SQL. The basic concept is that you have a master file that describes your database configuration and the changesets you want to include in your run. Spring comes in at that point, parses your config and your included changesets, and manages them appropriately.

Flyway

Flyway is offered free and paid, the same as liquibase. The pricing and feature breakdown is available here - and is a lot more detailed than anything I could find on liquibase.

Flyway focuses on your migrations as a first-class concept. You write SQL scripts, place them in a folder in your Spring project, add some config to your application.yml files and then Spring runs the migrations as per your config.

Using Liquibase

Note: I’m using Spring Boot 2.2.6 with Gradle here, but you can do the equivalent with a dependency block in Maven.

Step One is adding your dependencies to build.gradle:

    // Persistence
    implementation('com.h2database:h2')
    implementation "org.liquibase:liquibase-core"
    liquibaseRuntime "org.liquibase:liquibase-core"
    liquibaseRuntime sourceSets.main.compileClasspath
    liquibaseRuntime "org.postgresql:postgresql"
    liquibaseRuntime "com.h2database:h2"
    implementation('org.postgresql:postgresql')
    implementation('org.springframework.boot:spring-boot-starter-jdbc')

Next we need to make new folders in our project:

In src/main/resources add a liquibase folder
Add two subfolders, changelog and fake-data

Add some minimal configuration to your application.yml file and you’re good to go.

  liquibase:
    contexts: dev, faker
    change-log: classpath:liquibase/master.xml

At this point, you should be able to start writing some XML.

You’ll need a master.xml that describes your environment setup and the migrations that you want to run. Here’s an example:



        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.6.xsd">

     name="now" value="now()" dbms="h2"/>
     name="now" value="current_timestamp" dbms="postgresql"/>

     name="floatType" value="float4" dbms="postgresql, h2"/>
     name="clobType" value="longvarchar" dbms="h2"/>
     name="clobType" value="clob" dbms="postgresql"/>
     name="uuidType" value="uuid" dbms="h2, postgresql"/>

     id="00000000000000" author="alex">
         sequenceName="sequence_generator" startValue="1050" incrementBy="1"/>
    

     file="liquibase/changelog/20191024203226_added_entity_Company.xml" relativeToChangelogFile="false"/>
     file="liquibase/changelog/20191024203227_added_entity_Team.xml" relativeToChangelogFile="false"/>
     file="liquibase/changelog/20191024203234_added_entity_Project.xml" relativeToChangelogFile="false"/>

It looks a bit busy but let’s break it down. First, we need to tell Liquibase about our database setup, and what values it should use for various data types:

     name="now" value="now()" dbms="h2"/>
     name="now" value="current_timestamp" dbms="postgresql"/>

     name="floatType" value="float4" dbms="postgresql, h2"/>
     name="clobType" value="longvarchar" dbms="h2"/>
     name="clobType" value="clob" dbms="postgresql"/>
     name="uuidType" value="uuid" dbms="h2, postgresql"/>

Next, we need to describe the changes we want Liquibase to make; in our case, we want Liquibase to create a sequence and run three migrations. Note that if you write migrations but don’t include them in master.xml, they won’t run.

     id="00000000000000" author="alex">
         sequenceName="sequence_generator" startValue="1050" incrementBy="1"/>
    

     file="liquibase/changelog/20191024203226_added_entity_Company.xml" relativeToChangelogFile="false"/>
     file="liquibase/changelog/20191024203227_added_entity_Team.xml" relativeToChangelogFile="false"/>
     file="liquibase/changelog/20191024203234_added_entity_Project.xml" relativeToChangelogFile="false"/>

Here’s the company migration:



        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.6.xsd">

     id="20191024203226-1" author="alex">
         tableName="company">
             name="id" type="bigint" autoIncrement="${autoIncrement}">
                 primaryKey="true" nullable="false"/>
            
             name="name" type="varchar(255)">
                 nullable="false"/>
            
             name="web_page_address" type="varchar(255)">
                 nullable="true"/>
            
             name="billing_contact_email_address" type="varchar(255)">
                 nullable="true"/>
            
             name="primary_contact_email_address" type="varchar(255)">
                 nullable="true"/>
            

        
    

     id="20191024203226-1-data" author="alex" context="faker">
        
                file="liquibase/fake-data/company.csv"
                separator=";"
                tableName="company">
             name="id" type="numeric"/>
             name="name" type="string"/>
             name="web_page_address" type="string"/>
             name="billing_contact_email_address" type="string"/>
             name="primary_contact_email_address" type="string"/>

The first thing this migration creates is a table called Company. Since Liquibase uses XML as an abstraction on top of SQL, we don’t need to worry too much about compatibility, Liquibase will look at the XML file and translate our request to the appropriate SQL dialect.

     id="20191024203226-1" author="alex">
         tableName="company">
             name="id" type="bigint" autoIncrement="${autoIncrement}">
                 primaryKey="true" nullable="false"/>
            
             name="name" type="varchar(255)">
                 nullable="false"/>
            
             name="web_page_address" type="varchar(255)">
                 nullable="true"/>
            
             name="billing_contact_email_address" type="varchar(255)">
                 nullable="true"/>
            
             name="primary_contact_email_address" type="varchar(255)">
                 nullable="true"/>

Having defined our table, we can take advantage of one of the neat functions Liquibase can give us - Liquibase will load data into our database from CSV:

     id="20191024203226-1-data" author="alex" context="faker">
        
                file="liquibase/fake-data/company.csv"
                separator=";"
                tableName="company">
             name="id" type="numeric"/>
             name="name" type="string"/>
             name="web_page_address" type="string"/>
             name="billing_contact_email_address" type="string"/>
             name="primary_contact_email_address" type="string"/>

The loading of data is a changeset in its own right; this means you can flexibly load data when you create the table, or at a later stage. Loading data in this way can be useful if you’ve got complex relationships to model. Another trick here is the context="faker" statement above. If you want data to be loaded, make sure your application.yml’s liquibase entry contains that tag. If you remove that tag, the data won’t be loaded.

Here’s the CSV referenced above with some sample data in it:

id,name,web_page_address,billing_contact_email_address,primary_contact_email_address
1,mobileFish,HomeLoanAccountTableComputer,Trace,SavingsAccountalarm
2,TastyMetalBacon,redChipsSoap,HomeLoanAccount,web-readiness

That’s it. When you run your spring boot application, during startup Liquibase will apply this migration, and when your app connects to the database, the tables and data will be ready immediately.

How does liquibase know the database credentials? I’m not sure! It’s magic. I assume it hooks into your spring database config, an example of which might be:

datasource:
    type: com.zaxxer.hikari.HikariDataSource
    url: jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1;DB_CLOSE_ON_EXIT=FALSE
    username: sa
    password:
    hikari:
      poolName: Hikari
      auto-commit: false

Using Flyway

Flyway is a little bit different. The focus on SQL tidies things up, in my opinion.

First, you add the relevant dependencies:

// Persistence
    implementation('com.h2database:h2')
    implementation('org.postgresql:postgresql')
    implementation('org.springframework.boot:spring-boot-starter-jdbc')
    implementation "org.flywaydb:flyway-core"

Tell Flyway how to connect to your database (in my case, via application.yml):

  flyway:
    locations: classpath:db/migration/dev
    url: jdbc:h2:mem:testdb;DB_CLOSE_DELAY=-1
    user: sa
    password:

Make some folders, again in src/main/resources:

Create db/migration
In my case, I have H2 in dev and PostgreSQL in test and prod, so I have two sibling folders, dev and prod (test and prod scripts both live in prod as I want those to be the same, and they are the same SQL dialect).

The next step is to write some migrations in the appropriate SQL dialect. Where before our Company migration was in XML, now we’re in SQL, note that there is a naming convention here that allows us to cut down on boiler-plate. You have to name files like so V1.0__create_company.sql where the VX.x will define the order in which migrations are run. Anyway, here’s the file:

create sequence hibernate_sequence start with 1050;

create table company
(
    id                            uuid         not null,
    name                          varchar(255) not null,
    web_page_address              varchar(255),
    billing_contact_email_address varchar(255),
    primary_contact_email_address varchar(255),
    constraint pk_company
        primary key (id)
);

Dropping the XML leaves things a lot cleaner and readable, in my opinion. We load data as well, but this is just a SQL script rather than CSV:

INSERT INTO PUBLIC.COMPANY (ID, NAME, WEB_PAGE_ADDRESS, BILLING_CONTACT_EMAIL_ADDRESS, PRIMARY_CONTACT_EMAIL_ADDRESS)
VALUES ('1a689e52-f35b-4bda-934c-ea4f076bdc2c', 'Blue Fish Software Inc', 'bluefish.io', 'bills@bluefish.io',
        'hi@bluefish.io');

That’s it. When the application runs, it’ll load those SQL scripts and manage migrations and state for us. The disadvantage here is that we need two sets of scripts, one for H2 and one for PostgreSQL - this could be because my SQL is super weak and I can combine them but don’t realise it.

Comparing Liquibase and Flyway

I found Flyway to be a lot easier to use, despite having used Liquibase for a lot longer.

I like that Flyway is cleaner and less repetitive, and assuming I fix my dialect issue, it’ll be a case of 1000+ fewer lines of code with Flyway than with the same setup in Liquibase. I also found the docs Flyway provides to be a lot better, and I encountered far fewer undocumented tricks needed with Flyway. If crucial information needed to get going is easier to find on Stack Overflow than on your docs pages, you’ve done something wrong. Liquibase is bad for this.

Moving from bigint to uuid primary keys on all of my entities was impossible with Liquibase due to a bug around mis-detecting specific UUID values as strings.

Flyway had no such issues, and where I did make a mistake with Flyway, I got printed SQL exceptions rather than SQL exceptions that have abstracted by Liquibase. I encountered far less friction with Flyway.

u/hooba_stank via Reddit, makes the point that liquibase’s profiles grant it excellent flexibility. Using contexts in different environments, composing profiles for different testing requirements and conditional change sets and rollbacks make the free Liquibase offering compelling. It’s a great point.

Flyway’s requirement for sequential ordering of SQL files can lead to sprawl if you prefer small changes per file. Encountering failures becuase you’ve fat-fingered a SQL script version number is annoying too. u/koreth adds that this problem can be avoided by configuring a pre-commit hook and CI check to detect conflicting sequence numbers so this issue can be automated to a degree, though ideally it’s not something you have to deal with at all.

Wrap Up

Both Liquibase and Flyway are better than nothing by far. My personal preference is Flyway. Flyway is cleaner to implement and easier to use on an ongoing basis and more readable. That being said, Liquibase is a powerful tool. Chosing either of these tools to use in your development will give you good gains in productivity.