In it, I talk about the basics of Actions, how they work, what the language looks like and how you can get started. If you’ve never used Actions before, I highly recommend you have a quick read of that before reading this, as I build on some of the ideas presented there.

Continuous Integration (CI) versus Continues Deployment (CD) versus Continuous Delivery (unhelpfully, also CD)

Put simply, continuous integration is the act of continually moving your changeset into the wider codebase, as frequently as it makes sense for your use case. CI is an essential practice because it drives us as engineers to produce smaller changes more often, and these small changes tend to be less risky individually. Those small changes also lead us to deliver value to our users more rapidly, and more often.

Continuous deployment is the engineering decision to ship validated or verified changes to production environments as often as practical. The decision to continually deploy tends to be made for the same reasons as we choose to continuously integrate. It’s about reducing risk, making improvements available sooner, and building trust with users and with our less technology-focused colleagues.

The natural extension of continuous deployment is continuous delivery. While continuous delivery has become an umbrella term to describe all of the bits and pieces that contribute to it, I do see these things as separate concepts. The key difference then is that continuous deployment is an engineering decision to ship code to production. Continuous delivery is the product or broader business decision to make the features that our code provides, available to users.

Each concept requires the predecessor to work, but it can be a valid choice to stop at any of these points and not continue on. My own bias is to adopt continuous delivery as a fundamental practice, but your own operating context will have impacts on what parts of this you can and can’t do.

Continuous Delivery is a foundational skill that your organisation needs to be good at if it is to remain responsive and ’lowercase a’ agile.

In the previous article, I stated:

More concretely, a decent CI/CD solution will:

• Build your code
• Run the tests
• Deploy it to an environment
• Roll-back if there is an issue

My initial implementation of Github Actions saw me using the tool primarily for continuous integration. However, I’ve recently expanded that to include continuous deployment (and since I’m not using feature flags yet, this technically qualifies as continuous delivery if only as a side effect).

Github Actions now owns artefact build across all environments.

Previously, my create-rc.yml workflow built a container in dev, tested that and then told Heroku that the build is good, and to cut a release.

Heroku then would pull the code, build it and release it to my dev environment. I wasn’t actually running the code I tested, I was running code built from the code I tested. This has been the source of subtle but pernicious bugs for me in the past. My rule of thumb is that you want to build your release candidate once and pass it through environments. That way you know for sure, and can prove in an auditable way, that this particular code is running in this specific place.

Note: The Heroku CLI is bundled as standard on Github Actions runners. I couldn’t find this documented anywhere and literally only stumbled on it in a fit of frustration at midnight last night. My new heuristic is that if you need a specific tool, assume it’s on the runner but isn’t necessarily up to date. If your dependency isn’t there, then you should try to work around it.

I assumed that Heroku CLI wouldn’t be there and spent a long time trying to fight shadows. Don’t be like me.

Here’s a high-level summary of the work I did to make my builds work the way I’ve described above, with one RC being deployed to many environments:

1. Have create-rc.yml treat the docker image it makes like the one RC, rather than as a means of delivering integration tests.
2. Improve the tagging process so that we produce a docker image with the right naming conventions for Heroku.
3. Push the image to the Heroku docker registry, where we can persist it across builds and environments.
4. Introduce a new job to deploy the verified image to the right Heroku environment.
5. Introduce a new workflow with a different trigger, to take the already validated image and redeploy it to production.

Overall the changes weren’t massive, but I definitely cut myself on some rough edges (documentation being a big part of that).

These steps build my image, tag it and send it off to Heroku. That’s step 1, 2 and 3 done in ~20 lines:

- name: Login to Heroku Registry
  env:
    HEROKU_API_KEY: ${{ secrets.HEROKU_TOKEN }}
  run: heroku container:login

- name: Build and push
  env:
    HEROKU_API_KEY: ${{ secrets.HEROKU_TOKEN }}
  run: heroku container:push -a dev-api-spring-boot web

- name: Create tarball from docker image
  # Heroku has a naming convention for images, it's important to match that.
  run: docker save registry.heroku.com/dev-api-spring-boot/web > spring-boot-api-candidate.tar

- name: Upload image tarball for next job
  uses: actions/upload-artifact@v2
  with:
    name: spring-boot-api-candidate.tar
    path: spring-boot-api-candidate.tar

These steps close out the only actual building of code that I do. In every other case, I get code from one place to the other by redeploying code I already know to be good. It’s important to note here that I’m trying to keep build time low by using capabilities baked into the actions runner, rather than using externally published actions.

It’s also important to note that if you use Heroku commands the way I am, you lose visibility of fine-grained build actions because most of that code lives in your docker file. By that, I mean that all I can see in this code is that Heroku is pushing a container. I can’t see how it’s built. For that, I need to go to my Dockerfile, which in this case looks like this:

FROM gradle:6.5.1-jdk14 as build-stage

WORKDIR /build
COPY src /build/src
COPY gradlew /build/gradlew
COPY gradle /build/gradle
COPY build.gradle /build/build.gradle
COPY settings.gradle /build/settings.gradle

RUN /build/gradlew clean build -x test

# Running container build begins
FROM adoptopenjdk:14.0.1_7-jre-openj9-0.20.0-bionic

# copy application JAR
COPY --from=build-stage build/build/libs/spring-boot-api-*.jar /spring-boot-api.jar

# specify default command
# The command to run this in local mode would be docker run -p 8080:8080 -e JAVA_OPTS="-Dspring.profiles.active=local -Dserver.port=$PORT" registry.heroku.com/dev-api-spring-boot/web
CMD ["sh", "-c", " java ${JAVA_OPTS} -jar /spring-boot-api.jar"]

Now I have two sets of build logic in two places, which isn’t perfect, but given how docker works, it’s tolerable. An example of the impact this has is that if I want to stop running tests while I build the image (because I unit test and integration test within the workflow), I need to add -x test to the end of my Gradle command in the Dockerfile as seen here. It adds a little bit of cognitive burden. Okay for now but not ideal.

Speaking of unit tests, since I want to cut down my build time, I can run them in parralel! Hooray for Github’s jobs are parralel by default stance!

unit-test:
  runs-on: ubuntu-latest
  needs: build-image # build-image is the name of the first job in this workflow, it's where the Heroku code above lives.

  steps:
    - name: Check out code
      uses: actions/checkout@v2

    - name: Set up JDK 14
      uses: actions/setup-java@v1
      with:
        java-version: 14

    - name: Make gradle wrapper executable
      run: chmod +x gradlew

    - name: Run Unit Tests
      env:
        PORT: 8080
      run: ./gradlew clean test

Releasing a verified image to Heroku

The great thing is that we only need to add one more step to our create-rc.yml (although this file should probably be renamed to dev-build or something simillar). What I’m saying below is that at the end of my build, only when I’ve got green unit tests and green integration tests, I want that image that I’ve already pushed, to be activated in Heroku.

dev-deploy:
  runs-on: ubuntu-latest
  needs: [integration-test, unit-test]

  steps:
    - name: Login to Heroku Registry
      env:
        HEROKU_API_KEY: ${{ secrets.HEROKU_TOKEN }}
      run: heroku container:login

    - name: Release to Heroku dev
      env:
        HEROKU_API_KEY: ${{ secrets.HEROKU_TOKEN }}
      run: heroku container:release -a dev-api-spring-boot web

I don’t have to care about artefact handling or pass in a tarball. The image is always current, and I just choose whether or not I want to make it active. One of the drawbacks here though is that I could get an image pushed that fails unit tests or integration tests. This is mitigated by the image remaining inactive until I release it. There is an edge case here that if I had to rollback, I might roll back to a bad image but that’s a problem for improvement later - and I’ll never have a broken image that won’t start because if the image won’t build, the build job will fail.

So with the above changes, we have a dev build process entirely owned by Github Actions. In theory, I could rip out Heroku and point to AWS ECS or somewhere else quickly. I’m using my platform provider for its platform, not for the stepladder on to the platform!

Releasing to Production with Sentry Integration

My procedure is that a release to production is triggered on a push to main, and the only way to push to main is via a merged commit. This procedure needs to do a few things:

1. Take a validated image from an upstream environment and retag it.
2. Push that retagged image back to Heroku
3. Release the image
4. Create a new release version in Sentry

I have some intermediate steps here that I want to use to tie things together from an audit perspective, but I’m waiting on some feedback from elsewhere before I can do that. Here are the steps for my release job, note how similar it is to my dev release methodology.

- name: Login to Heroku Registry
  env:
    HEROKU_API_KEY: ${{ secrets.HEROKU_TOKEN }}
  run: heroku container:login

- name: Pull dev image
  run: docker pull registry.heroku.com/dev-api-spring-boot/web

- name: Re-tag dev as prod
  # Again, Heroku expects a naming convention here, so we just have to comply.
  run: docker tag registry.heroku.com/dev-api-spring-boot/web registry.heroku.com/api-spring-boot/web

- name: Push production image
  run: docker push registry.heroku.com/api-spring-boot/web

- name: Push Prod Image
  env:
    HEROKU_API_KEY: ${{ secrets.HEROKU_TOKEN }}
  run: heroku container:release -a api-spring-boot web

The post-processing I do on a release is much more interesting then docker jiggery-pokery:

1. I want to grab the entire codebase because I want that delicious commit history.
2. I want to create a new tag for this production release
3. I want to get all of the commit descriptions between the new tag, and the previous tag, and then filter out some commits that aren’t interesting for given values of interesting
4. Then cut a release for Github using the commits as a changelog
5. Finally, use Sentry.io for crash reports, and I want to tell Sentry about the new version so that I can track where issues might have been introduced.

- name: Checkout code
  uses: actions/checkout@v2
  with:
    # Fetch depth grabs the entire history, though I'm sure I could grab the commits since the last tag if I really wanted.
    fetch-depth: 0

# I encountered an issue here so have left this commented out. It's not a part of my workflow, but hopefully one day it'll create tags for me
# Leaving this out for now - to use the output of tag in the below release use ${{ steps.create_tag.outputs.new_tag }}
#      - name: Bump version and push tag
#        id: create_tag
#        uses: mathieudutour/github-tag-action@v4.5
#        with:
#          github_token: ${{ secrets.GITHUB_TOKEN }}
#          release_branches: dev

- name: Generate release notes
  # Important note: I use https://github.com/angular/angular/blob/master/CONTRIBUTING.md#commit for commit format. It makes it really easy to filter commits in the way I've done below.
  run: git log $(git describe --tags --abbrev=0)..HEAD --pretty=format:"%s" -i -E --grep="^(build|ci|docs|feat|fix|perf|refactor|style|test)*feat|perf|fix" > release-notes.txt

# I don't want to cut releases with no info in them! That's why this too is not currently part of my workflow
#      - name: Create Github Release
#        id: create_release
#        uses: actions/create-release@v1
#        env:
#          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
#        with:
#          tag_name: ${{ github.sha }}
#          release_name: Release ${{ github.sha }}
#          body_path: release-notes.txt
#          draft: false
#          prerelease: false

- name: Create Sentry release
  uses: tclindner/sentry-releases-action@v1.2.0
  env:
    SENTRY_AUTH_TOKEN: ${{ secrets.SENTRY_AUTH_TOKEN }}
    SENTRY_ORG: my-sentry-org
    SENTRY_PROJECT: spring-boot-api
  with:
    tagName: ${{ github.sha }}
    environment: heroku-prod

The net effect of the above is that I get to take a previously validated image. When I choose (via merged PR), I can automatically redeploy quality code to prod without rebuilding. The neat interplay here is that when I open that PR, my project will build, and use that same build to decide whether or not I can merge. As soon as I merge, the code is in prod without a rebuild. It’s nice.

With a new release being cut in Sentry each time, I can resolve errors that I’ve introduced down to at worst, a PR or at best, an individual commit. Also nice!

A full solution for continuous delivery tends to include (but isn’t limited to):

• The ability to create artefacts that we can release to production, across many products, teams, branches, environments and repositories.
• The ability to quickly validate an artefact as a candidate for release (through applying the testing triangle - unit, integration, acceptance, smoke and so on).
• The ability to release code to production without necessarily activating it for users (typically through feature flags, canary built and graduated rollouts).
• The ability to rapidly respond if a release candidate we later find to be problematic (again through canary builds, graduated rollouts, and by quickly rolling back software found to be problematic).
• The ability to audit or understand what code is where, what state that code is in, and why that code is where it is.

More concretely, a decent CI/CD solution will:

• Build your code
• Run the tests
• Deploy it to an environment
• Roll-back if there is an issue

Building an excellent build tool is no easy task, and because of the variety of use cases out there, you tend to find tools optimised for those use cases. They’ll be passable or useful in that niche and poor outside of it. Use-case variety and the complexity of the problem space is part of the reason why container-based builds have taken off. Where previously I’d have precious build agent boxes sitting somewhere that I’d need to feed and water and love and sell my children to, now I can have a dumb build box that knows about nothing. The build box just grabs a container off the internet (or I make a custom container) to fill my needs.

Github Actions

Github Actions is one such container based build offering, though at first they don’t tell you that!

The highest order concept in Actions is the Workflow. A workflow consists of several jobs, and each job can have several steps. Here’s an example we’ll become a bit more familiar with later:

As you can see, A workflow, is a set of jobs (that we run in series or parallel), with each job containing several steps. Jobs describe collections of steps that when together, perform some action that is useful to the business. Model your jobs in terms of things that are useful for you - it’s these aggregates that you’ll be watching most.

Steps are concrete actions that we perform, things like building, unit testing, renaming files and so on. A step is the smallest unit of work you can do in a workflow, and they don’t need to be useful in and of themselves.

The reason I think that jobs should be useful units of work for your business, is because if you plan them that way, it becomes trivial to re-use jobs to achieve new functionality quickly

Pipelines as Code with… YAML

Workflows use YAML as the format of choice, and I YAML in Github Actions as much as I hate any other white spaced language in a place it doesn’t belong. I’m sure Github chose YAML for a good reason, I’ve struggled to find out what those reasons were or are. Work continues in that area.

As with most things that involve YAML, the workflow file is easy to quickly grok while it’s small to medium length but challenging to troubleshoot or do anything with once it gets a bit bigger. Github Actions is still new, so I can understand why it might be a little rough around the edges. Perhaps there is tooling around the workflow file I’m ignorant of, but an equivalent to Bitbucket Pipelines’s pipeline validator would be an excellent addition.

Even better still (and this would be a feature far down the track), give me a fluent API akin to Bamboo's Java Spec Concept .

To save you a click, 90% of my warm fuzzies for Atlassian Bamboo come from being able to do this:

private Plan createPlan() {
    return new Plan(
            project(),
            "Plan Name", "PLANKEY")
            .description("Plan created from (enter repository url of your plan)")
            .stages(
                    new Stage("Stage 1")
                            .jobs(new Job("Build & run", "RUN")
                                    .tasks(
                                            new ScriptTask().inlineBody("echo Hello world!"))));
}

Getting access to this kind of API, and provide auto-completion, intellisense and everything that comes with that ecosystem is war-winning feature for pipeline builders.

I would prefer XML to YAML. There. I said it. We can move on now.

Workflows

What does a workflow file look like? Here’s the file for a Spring Boot application I’m currently working on. I was very focused on getting a minimal workflow up and running, so there are things to be added and things to be optimised.

Have a big read, and I’ll break it down below:

name: Create Release Candidate

on:
  push:
    branches: [ dev ]
  pull_request:
    branches: [ dev ]

jobs:
  create-rc:
    runs-on: ubuntu-latest

    steps:
      - name: Check out code
        uses: actions/checkout@v2

      - name: Build and tag release candidate
        uses: docker/build-push-action@v1.1.0
        with:
          username: ${{ secrets.HEROKU_USER }}
          password: ${{ secrets.HEROKU_TOKEN }}
          registry: registry.heroku.com
          repository: image-repository/heroku-process
          tags: latest
          push: false

      - name: Create tarball from docker image
        run: docker save registry.heroku.com/image-repository/heroku-process:latest > heroku-process-candidate.tar

      - name: Upload image tarball for next job
        uses: actions/upload-artifact@v2
        with:
          name: heroku-process-candidate.tar
          path: heroku-process-candidate.tar

  integration-test:
    runs-on: ubuntu-latest
    needs: create-rc

    steps:
      - name: Check out code
        uses: actions/checkout@v2

      - name: Download tarball
        uses: actions/download-artifact@v2
        with:
          name: heroku-process-candidate.tar

      - name: Load image from tarball
        run: docker load < heroku-process-candidate.tar

      # github doesn't allow us to gracefully expire artifacts (e.g. keep last 10 build artifacts and no more)
      # if we don't delete these, we'll hit the storage limit for actions quickly.
      # persisting docker images outside of builds should be done by pushing the image to a registry
      # we should rm the tarball as soon as we've loaded it as a docker image.
      - name: Tidy up artifact to avoid storage limit
        uses: geekyeggo/delete-artifact@v1
        with:
          name: heroku-process-candidate.tar

      - name: Run application server
        run: docker run -d -p 8080:8080 -e JAVA_OPTS=-Dspring.profiles.active=dev registry.heroku.com/image-repository/heroku-process:latest

      - name: Make server watch script executable
        run: chmod +x wait-for-heroku-process.sh

      - name: Wait for server to start
        run: ./wait-for-heroku-process.sh

      - name: Set up JDK 14
        uses: actions/setup-java@v1
        with:
          java-version: 14

      - name: Make gradle wrapper executable
        run: chmod +x gradlew

      - name: Run Integration Tests
        run: ./gradlew integrationTest

The above file is just under 100 lines of YAML to describe the first two jobs in the image above (Package for release and Run Integration tests). Now that I’ve finalised everything, it looks pretty clean and tidy, the process to get there was anything but. Again this is something that I’d attribute to new-ness. Still, Github Actions does an excellent job at giving you the low-level details about how a thing works, but a lousy job at describing how you can use what Actions provides you to create higher-order pipelines.

For example, the Gradle starter here does a good job of telling me how to declare jobs, steps and make them dependent on pull requests or branches. It also leaves me with questions. I want to package my application, and then run it, and then only when it’s running, run unit tests. How do I do that? I want to publish to my registry, but only if the unit and integration tests have passed, except for in dev, we can publish there with some failing tests. How do I do that?

The basic building blocks are there, but more complex pipeline logic is missing.

Compare this with the excellent documentation produced by VMWare and Pivotal for Concourse CI that gets much more into higher-order combinations of those basic building blocks.

Compare Gated CI Pipelines and Blue-Green Deployments with Concourse, to anything in the only workflow template repository I could find, Starter Workflows to see that high level / low-level split.

Concourse is a more mature product and more fully-featured, where perhaps Github Actions is still more of a CI tool rather than a CD tool. I’m sure it will get there in time. The portability and reusability that Github Actions provides is a significant advantage over other build tools.

I love how sturdy the modelling and pipelining features are for Concourse. It’s ironic though that at my current workplace we’ve just decided to get off Concourse CI in favour of Github Actions.

Jobs in detail

Here’s the same workflow as above, trimmed for easier reading.

name: Create Release Candidate

on:
  push:
    branches: [ dev ]
  pull_request:
    branches: [ dev ]

jobs:
  create-rc:
    runs-on: ubuntu-latest

    steps:
      - name: Arbitrary Step 1
        run: echo "Hello Muddah"

      - name: Arbitrary Step 2
        run: echo "Hello Fadduh"

  integration-test:
    runs-on: ubuntu-latest
    needs: create-rc

    steps:
      - name: Arbitrary Step 1
        run: echo "Here I am at"

      - name: Arbitrary Step 2
        run: echo "Camp Granada."

Important Concepts: Jobs and Steps

• A workflow has one set of jobs
• A job has one set of steps.
• Both of these sets can have one to many items in them. You can have ten jobs, each with 20 steps, or one job with 1 step.
• To create a sequential pipeline, add a needs statement in every job after the first one, and in that needs statement, refer to the previous job. For example, I have two jobs, create-rc and integration-test. In the second job, I have this statement: needs: create-rc.
• Importantly, Jobs run in parallel by default. A sequential run like the above is something you have to do specifically. Parallel by default creates advantages if you can decouple your build process.
• Right now, I have to wait for Actions to build my RC and image before I can integration test. I believe in pushing one artefact through the entire pipeline rather than rebuilding it each from source each time - but if I didn’t, I might be able to get super parallelisation on my build pipelines. I.e. rather than Checkout Source > Build > Test > Deploy with one artefact the whole way through, I could Checkout Source > Build, Checkout Source > Test, Checkout Source > Package > Deploy all at the same time. This means the time the build takes is the max() of the longest job, rather than sum(job_1 + job_2 ... job_n).
• I’m an absolute savage though and believe that the best approach is to checkout and built once, and then pass an artefact all the way through. The tradeoff for that certainty is a longer build time (about 5 minutes in this case, but for longer build processes, it probably stacks up quick).
• There is a whole ecosystem of Actions available on the Github Marketplace, but every action you use will cost you in build time. Actions are distributed as Docker images, and when your build runs, before you can build your build, you have to build all of your actions.
• Steps all run sequentially, and an action can be a step, but not all steps have to be actions. For example:

- name: Arbitrary Step 1
  run: echo "Hello Mother"

The above is a step that is not an action; the below is a step that is action. All actions are steps, but a step does not have to be an action.

- name: Build and tag release candidate
        uses: docker/build-push-action@v1.1.0
        with:
          username: ${{ secrets.HEROKU_USER }}
          password: ${{ secrets.HEROKU_TOKEN }}
          registry: registry.heroku.com
          repository: image-repository/heroku-process
          tags: latest
          push: false

Sharing and Storing Artifacts

Each job runs in parallel and runs in a separate container. If you have a job that builds, and you have a job that tests, you have to figure out a way to get the build output into the testing task. Github’s solution to this seems inefficient, but I can’t see any better way of doing it.

Your workflow could happen anywhere, multiple anywhere even. Because jobs are independent, in theory, you could have your build happening over in the US East region and your tests happening in Europe. You can’t rely on them being on the same machine or even in the same DC, so sharing artefacts via cloud storage is the only choice you have.

It gives rise to these two actions:

- name: Upload image tarball for next job
        uses: actions/upload-artifact@v2
        with:
          name: heroku-process-candidate.tar
          path: heroku-process-candidate.tar

- name: Download tarball
        uses: actions/download-artifact@v2
        with:
          name: heroku-process-candidate.tar

      - name: Load image from tarball
        run: docker load < heroku-process-candidate.tar

The way we share artefacts between jobs (or persist them outside of builds) is essentially by uploading and downloading from S3 or an equivalent (I assume they use S3 but haven’t done the research to confirm). I have a small workflow right now, but what this means is that passing my artefact from my first job to the second, accounts for a whopping 21% of the run time of the build. This behaviour pushes me to a perverse outcome where I’m better off starting from scratch in every single job because of the time it takes to trade artefacts! An alternative to that would be to have one enormous job, but that also flies in the face of the whole idea of Github Actions, which is that concept of composable, reusable workflows.

Further, you need to be careful with your usage of artefacts.

On a team or pro plan, you get 2GB of storage for actions. Artefacts associated with builds older than 90 days get deleted. To naturally stay at or under the 2GB limit, your builds need to persist fewer than 22MB per day - not per build, per day worth of artefacts. Github Packages, and storage for Actions, get very spendy, very quickly.

This behaviour gives rise to actions like this one. I consider this 100% required for anybody who has Workflows with more than one job, where you need to pass an artefact between them. Remember, every job that needs to share an artefact will incur the wrath of Github Packages.

      # github doesn't allow us to gracefully expire artifacts (e.g. keep last 10 build artifacts and no more)
      # if we don't delete these, we'll hit the storage limit for actions quickly.
      # persisting docker images outside of builds should be done by pushing the image to a registry
      # we should rm the tarball as soon as we've loaded it as a docker image.
      - name: Tidy up artifact to avoid storage limit
        uses: geekyeggo/delete-artifact@v1
        with:
          name: heroku-process-candidate.tar

As per the comment, I’d recommend you persist artefacts during builds and between jobs, delete them afterwards, and for audit, viewing test reports and things like that, find somewhere else to put them. Put containers into your registry, put test reports into S3 or similar. For more detail on how this all gets billed, see this

In closing, why use Actions?

I think for a lot of people (myself included), the main draw for Github Actions is that it’s conceptually quite simple (though as with anything there are gotchas), and tightly woven in with Github itself. I like that the build lives close to the code and everything that comes with it. Github Actions is ‘pipeline-as-code’, that isn’t that much of a burden to learn and use, and even having been newly released, it’s as capable as any other build tool I’ve used.

Where it starts to get powerful is the open-source ecosystem of Actions and the tie into wider Github. Be that the code repository or other apps I might have added to my repo or org. I can get this without having to deal with another tool, another login, or the overhead of another new thing. Thinking on it, maybe that’s why they chose YAML.

While I’ve spent a lot of time bagging it, I quite like it. It’s a good example of delivering just enough capability at the right time, and I’m sure Actions will continue to grow, if not via first-party development, certainly through that action ecosystem.

Yes, I’ve got gripes about YAML, but I have that most places I go. You can work around the storage issues, and it’s not unreasonable for Github to try and make money given everything else they provide for free or for cheap. Actions is a fully managed service, no servers to feed and water, no breaking updates so far. With versionable pipelines, there are very few moving parts that we the end-users have to care about. That’s worth more than it’s equivalent weight in gold.

If you’ve got a pet project or something new at work, I recommend you give it a try.

Context

We’re a Health IT company based out of New Zealand, and run an set of products that provide messaging integrations between Healthcare organisations across New Zealand and Australia. Our applications are a mixture of desktop and web service based, and we’re currently undergoing a massive change process from Waterfall to Agile and from a bespoke services company to a truly product focused company.

The Problem

Our time to market is poor, we want to shorten our release cycle and provide and awesome service to our customers (above and beyond the services that basically enabled us to become a monopoly in New Zealand).

The Solution

To enable rapid iteration, we needed to shorten our feedback loops, and the best way of doing that is by implementing continuous delivery.

The Principles of Continuous Delivery

There was quite a but of work to do to enable our solution, continuous delivery has a bunch of prerequisites and we wanted to abide by a set of core principles that would guide the technology and process decisions we made.

The principles that guided us are:

1. Processes MUST be repeatable and reliable.

   • The process by which we deploy our software has been automated, but often full automation of a release can make the business uncomfortable. To help the business get used to this, we’ve introduced some decision points in our release flow that require manual intervention.

2. EVERYTHING has to be automated.

   • The manual decision points we’ve introduced don’t violate this because no process is actually being done manually, we’re just providing business the opportunity to control the process in more discrete steps. All of our build, test and deployment steps are automated though, that in itself has reduced our time to any arbitrary environment to about a minute, with full unit, integration and system tests conducted every time we build and deploy.

3. You must be able to guarantee quality.

   • The bedrock of any automated process is quality. We guarantee the quality of our code by monitoring our code metrics in Sonarqube - things like test coverage, technical debt and best practice conformance. Sonarqube breaks builds if they introduce issues or lack test coverage on new code. Further, in tracking unit test coverage, we also ensure that those tests are run and pass - if you don’t use them, there’s little point spending time and money creating and maintaining them.

4. If something is difficult or painful, it must be done more often. You can’t build muscle without exercise.

   • In forcing ourselves to do painful things often, we worked out the issues in those processes and were able to mitigate or resolve them entirely, leading to a much more robust process.

5. Everything must be expressed as code and versioned.

   • Obviously our application code is all safely in git, versioned and lovely, but it’s important to version and code our build steps and infrastructure as well. We achieved this using Puppet with Hieradata within a git repository managed with Puppets’ Code Manager, and docker allows us to store our image creation steps in a Dockerfile, which is stored side by side with the application code itself.

6. Things are only considered DONE when they are released to Production.

   • Setting up triggers such that successful deploys off a dev branch trigger builds and deploys on our master branch enabled us to smoothly flow code through our environments in and automated way. Bamboo gives great deployment functionality that we’ve made good use of.

7. Measure quality throughout the process.

   • Tying all of our builds back to Sonarqube enabled us to keep track of quality, we also track metrics around the amount of time it takes to complete the process, ratios of successfull builds and deploys to failures, and a bunch of other stuff.

8. The Continuous Deployment process needs to be continuously improved.

   • We’re still working on the process. It’s important to not forget that there is always another issue we could be improving.

The Tools we chose

• Docker

  • Docker enables us to have superfast builds by building our applications in layers. By building these layers separately, when we build the application, we only have to build the application, rather than also having to build all of our dependencies and other bits and bobs.

• Rancher

  • We use Rancher for container orchestration and management, it enables us to keep a high level picture of what our applications look like somewhere easy to reach. It enables us to do a whole bunch of really cool things like load balancing, health checks and the bread and butter of container orchestration. Further, it also has a great repository of applications, stuff like load balancers, CTF stacks, container discovery tools, even a containerised version of the famous ELK stack. The capability to create your own catalog also exists, so once we’ve got all of our tools and services in to containers, we can spin up production ready software in no time flat, with all of the quality guarantees built in to our building and deployment pipelines. This is the part of our solution that I’m most excited about and I would recommend Rancher to anyone interested in container orchestration and management.

• Nexus

  • We use Sonatype Nexus 3 for a bunch of stuff. We have two instances of Nexus, one for integration ready blobs and one for deployment ready blobs. That means that all of our base docker images, sub-images, and layers we don’t want to send to prod live in our CI Nexus, and the only images that live in our CD instance of Nexus are images that are ready for deployment at a moments notice. We also use Nexus to mirror Maven Central and the NPM Registry in order to reduce our build times by removing as much network variance as possible. We also keep build dependencies in our CI Nexus, blobs like Servicemix archives, SoapUI archives and others.

• Bamboo

  • We use Bamboo to wire up our builds, tests and deploys. One thing that irks me about Bamboo is it’s lack of ability to verify a deployment and conduct system and smoke tests. That being said, it’s got an awesome build system and the a strong deploy system that allow us to tie any image running in production all the way back through the pipe, to the commit that triggered the build. That provides us with a great deal more visibility as to what code is where when compared with our old process.

• Bitbucket

  • Git and Bitbucket are the solid but silent performers. We use a Feature branching workflow with pull requests - we require that the only change to master is a pull request from dev, and code only gets in to dev via pull requests. Each request must have a minimum of two reviewers, no issues or tasks against them, and a passing bamboo build.

• SonarQube

  • Sonarqube enables us to track issues with our code, and break builds if quality gates aren’t met. It’s also a great tool to communicate the health of our code base to the rest of the business.

System Map

Here’s the system we built, end to end. It’s a bit busy but it is quite compact, and looking at it, I can’t see anything I would take away to make it simpler, which I think is a good thing. It’s as simple as it can be while still being powerful enough to deal with any deployment requirement we might have.

What value has the business derived from this project?

Applications that use our CD Pipeline gain the following benefits:

1. The application can be built and deployed to any arbitrary environment in less then two minutes in most cases, and we’ve yet to see a build and deployment take longer than four.
2. Batch sizes are smaller, so deploys are safer
3. Deploys can be rolled back in a single click, in the unlikely event of a failure

The business benefits derived from this then are:

• Vastly cheaper deployments, we’re saving about $800 every deployment on every project that uses this flow.
• Our time to production, where customers use our software, is under half an hour. The benefits of being to get feedback and iterate quickly provide us with an unfair advantage over the competition. I like unfair advantages.
• Software Quality is way up, which means less issues in production, less time spent supporting that software and less time spent maintaining that software.

I feel like that’s some pretty compelling stuff.

It was an awesome experience, and the first time I’ve spoken on technical and non-technical topics at an industry conference.

The talk I gave is available on Speakerdeck . I’ve also embedded it below:

I think that the one thing that I took away from Devops Days NZ 2016 was this.

It’s very easy to get wrapped up in your own problems in this industry. It’s very easy to feel like your problems are unique, new or otherwise unreproducible. Unless you’re Google, Netflix or of that ilk, this is very unlikely. You and I, we’re not so different. Maybe it’s health, maybe it’s media, maybe it’s any other industry. We’re all going through the same things, at different times. It’s important that we continue to work through these diverse issues as best we can, while knowing that someone out there has felt, is now feeling or will feel your pain.

You are not alone!

In the spirit of DevOps and all the baggage that that word contains and frees us from, let’s keep helping each other, let’s keep sharing, we’re a community of like minded people.

So without Further ado, here’s what we’re running at the moment.

I’ll go through our components technology by technology:

Most importantly, Docker

We couldn’t do what we wanted to do with Overtime’s CI / CD without docker. Initially I created docker-compose containers for instances of Docker Registry, Nginx, and MongoDB and our Web App. The plan was to containerise those components because it would allow us to scale up instances of our web app and MongoDB in case our load got crazy high. An ambitious design requirement, but much better for the long run. In fact nearly every design decision we’ve made technologically has been focused on “What will make our lives easier in the long term”. We do this voluntarily so making money immediately and as quickly as possible isn’t such a huge issue for us! It’s nice.

We had a bunch of technical issues with Nginx not liking being containerised and not linking to our other containers properly. That resulted in us shifting it to the VPS itself and using it for reverse proxy and routing, it’s working pretty nicely so far.

We run separate production and test environments. Local builds are automatically built and dockerised on commit, are automatically deployed to our test environment with 0 loss of service. When we’re ready to go to prod, we just retag the docker image in question as prod rather than test. Docker picks up that there is a new image available and redeploys in the same way as our test environment, meaning we also get 0 down time. We still plan our deployment windows for early in the morning just in case something breaks but the process seems pretty solid so far.

Our Web App

Overtime uses a Java 8 back end with an Angular front end. Once the application is complete, players will be able to register themselves and their teams, sign up for tournaments, track points, rankings tables identities, and communicate with organisers and other teams. We have a dedicated slack team for overtime and we’re planning on our players being automatically enrolled into slack on sign up. I also want the enrollment process to add them in to relevant channels (they might get a private team channel and enrollment into open game and interest channels) - but we’re still thinking about how that would work!

We’re using MongoDB because we really like it’s object paradigm, it’ll make it easier for us to manage our data long term.

Speaking of building:

We use Bamboo for our CI/CD work, initially we used Jenkins but found it unreliable, prone to random failures and difficult to set up, maintain and extend. Bamboo is a much more user friendly solution. We had set up our build server as an additional docker container on the first VPS but thinking about it, we wanted to ensure that development work could never impact production environments. Because of the size of our VPS it was possible that our build container could cause production issues by hogging up resources while building things. An edge case but with the focus on the long term we decided on separating that out on to it’s own VPS with its own resources.

Project Management

We use Taiga.io for project management. It has the option for sprint or kanban set ups, it’s a hosted solution and it’s free, which fits our budget! Taiga is also open source which means that we can take it, self-host and extend it should we ever need to provided we comply with the licence, which is another bonus.

Wrap Up

That’s all the detail I’ll go in to for now. I hope this helped you, if you want any more detail or you’d like help with your own projects, let me know, I’d love to see how I can help!

That’s a pretty huge productivity increase

Especially given that it was two juniors and two seniors across three months that did this work.

We’re not done yet, we’ve still got some UAT and final bits and pieces to deal with, but I’m proud to say that we’re mostly there.

Here’s a basic run down of how we did it:

We chose Atlassian’s Bamboo as our general purpose task runner. It’s the backbone of our process. We chose it because it’s got a nice interface, awesome integration with Bitbucket (which all of our repos are hosted on) as well as Jira and Confluence.

It can provide us with automatically generated information about the issues addressed by a deployment, link back to repos if we need to sanity check source code and a bunch of other cool stuff. We’ve also added webhooks to our deployment channel within Slack, so Bamboo posts alerts to that channel when a deployment completes.

We’re stuck with a bunch of technologies that aren’t being used correctly (Like using Puppet to manage individual machines rather than types of machines) and Bamboo is going to give us freedom to swap out discrete pieces of the process as and when we need.

We’ve got a bunch of scripts and SSH tasks that reach out to our repos, check file versions, run puppet agents (yes, manually running rather than every n minutes), stop queues, dump databases for backup purposes, start queues and all sorts of other nebulous tasks.

The end result is that to build a venture project, all a developer needs to do is commit code. To cut a release they need to select a branch to build. To deploy to a dev, test or staging environment they need to select a build result and click deploy, same deal for pre-prod and prod (with a few additional caveats).

The great thing is, the teams are onboard with it.

This is a huge amount of disruption to the way 80% of our team does business. It affects developers, testers, BAs and product owners. Without exception, everyone that I’ve talked to is pumped. It’s a credit to the guys here that they aren’t stuck in a process but are always looking to improve. It makes me happy.