Postgres Partitions in Elixir Migration


Postgres 11 has a nifty feature around partitions. When a partition exists for a range of values, when you insert into the parent table it’ll get routed to the correct partition. When you update that record’s partition key it will get moved to the correct partition. A default partition feature exists as well so that if you do try to insert something that doesn’t belong in any available partitions, it will be put there instead.

I wanted to use this feature to track transactions in a game economy. Where all transactions were stored with their transacted date as the partition key.

Here is the Wallet model.

defmodule Game.Wallets.Wallet do
  alias Game.Wallets.Transaction
  alias Game.Wallets.Wallet

  use Ecto.Schema

  import Ecto.Changeset

  @type t :: %__MODULE__{
          id: Ecto.UUID.t(),
          name: String.t(),
          balance: Decimal.t(),
          locked_at: DateTime.t(),
          deleted_at: DateTime.t(),
          updated_at: DateTime.t(),
          inserted_at: DateTime.t()
        }
  @primary_key {:id, :binary_id, autogenerate: false, read_after_writes: true}
  @foreign_key_type :binary_id
  schema("wallets") do
    field(:name, :string)

    field(:balance, :decimal, default: 0.0)

    field(:locked_at, :utc_datetime)
    field(:deleted_at, :utc_datetime)
    field(:updated_at, :utc_datetime)
    field(:inserted_at, :utc_datetime)

    has_many(:transactions, Transaction)
  end
end

Here is the Transaction model.

defmodule Game.Wallets.Transaction do
  alias Game.Wallets.Wallet

  use Ecto.Schema

  import Ecto.Changeset

  @type t :: %__MODULE__{
          id: Ecto.UUID.t(),
          occurred_at: DateTime.t(),
          credit: Decimal.t(),
          debit: Decimal.t(),
          description: String.t(),
          wallet_id: Ecto.UUID.t()
        }
  @primary_key {:id, :binary_id, autogenerate: false, read_after_writes: true}
  @foreign_key_type :binary_id
  schema("transactions") do
    field(:occurred_at, :utc_datetime)

    field(:credit, :decimal, default: 0.0)
    field(:debit, :decimal, default: 0.0)

    field(:description, :string)

    belongs_to(:wallet, Wallet, type: :binary_id)
  end
end

A fairly simple model, where I expect there to be millions of transactions. Something that can not live well on a single partition and be performant. So we need to make a migration that can handle this.

defmodule Game.Repo.Migrations.CreateWalletsAndTransactions do
  use Ecto.Migration

  def up do
    execute """
    CREATE TABLE wallets (
      id          UUID NOT NULL DEFAULT uuid_generate_v4(),
      name        TEXT NOT NULL,
      balance     DECIMAL(20, 2) NOT NULL DEFAULT 0.0,
      inserted_at TIMESTAMP NOT NULL DEFAULT now(),
      updated_at  TIMESTAMP NOT NULL DEFAULT now(),
      locked_at   TIMESTAMP,
      deleted_at  TIMESTAMP,
      PRIMARY KEY(id),
      CONSTRAINT wallets_balance_ck CHECK(balance >= 0)
    )
    """
  end

  def down do
    execute "DROP TABLE wallets CASCADE"
  end
end

We need to create the base paritition and default partition.

execute """
CREATE TABLE transactions (
  id          UUID           NOT NULL DEFAULT uuid_generate_v4(),
  wallet_id   UUID           NOT NULL REFERENCES wallets(id),
  occurred_at TIMESTAMP      NOT NULL DEFAULT now(),
  credit      DECIMAL(20, 2) NOT NULL DEFAULT 0.0,
  debit       DECIMAL(20, 2) NOT NULL DEFAULT 0.0,
  description TEXT
) PARTITION BY RANGE (occurred_at)
"""

Then we need to create the default partition.

execute "CREATE TABLE transactions_default PARTITION OF transactions DEFAULT"

Now the fun part is I needed a bunch of partitions created but didn’t want to type them all out by hand / copy paste.

start_date = beginning_of_month(~D[2019-04-01])

for months <- 0..47 do
  create_partition("transactions", calculate_next_month(start_date, months))
end

The #create_partition/2, #beginning_of_month/1, and #calculate_next_month/2 are defined as follows.

defp create_partition(table, date) do
  start_date = date

  stop_date =
    date
    |> Date.add(Date.days_in_month(date))

  month =
    start_date.month
    |> Integer.to_string()
    |> String.pad_leading(2, "0")

  execute """
  CREATE TABLE #{table}_p#{start_date.year}_#{month}
  PARTITION OF #{table} FOR VALUES
  FROM ('#{start_date}')
  TO ('#{stop_date}')
  """
end
defp beginning_of_month(date) do
  if date.day == 1 do
    date
  else
    Date.add(date, -(date.day - 1))
  end
end
defp calculate_next_month(date, 0), do: date

defp calculate_next_month(date, months) do
  next = Date.add(date, Date.days_in_month(date))
  calculate_next_month(next, months - 1)
end

I defined these methods in the Game.Repo.Migrations.CreateWalletsAndTransactions but definitely will extract these into a utility function to be used later.

Feel free to use it or manipulate it how ever. If you come up with a better solution I’d really like to see it.

Service Objects for Good


Service objects are a handy tool to use in any ruby application, that has complex logic that needs to be extracted out of a controller or model. There are some nice benefits to extracting complex things into a more testable interface.

To avoid providing a contrived example, I will use similar code to how uploads on rubyfm are handled.

class UploadService
  attr_reader :user, :logger, :upload

  def initialize(user, options = {})
    @user   = user
    @logger = options[:logger] || NullObject.new
    @errors = {}
  end

  def start(params)
    @upload = build_from_params(params)

    unless @upload.save
      logger.error { 'failed to save the upload' }
      # more logging info on why
      return false
    end

    unless transcode
      logger.error { 'failed to transcode upload' }
      return false
    end

    true
  end

  def transcode
    AmazonTranscodeService.new(@upload).start
  end
end

And the UploadsController that utilizes the UploadService with the following.

class UploadsController < AuthorizedController
  def create
    service = UploadService.new(current_user, logger: Rails.logger)

    if upload_params[:episode_id]
      episode = Episode.find_by(id: upload_params[:episode_id])
      authorize!(episode, :update?)
    end

    if service.start(upload_params)
      flash[:info] = I18n.t('upload.processing_started')
      redirect_to(upload_url(service.upload.id))
    else
      flash[:error] = I18n.t('upload.failed')
      redirect_to(new_channel_episode_url(primary_channel))
    end
  end
end

Attempting to shove all of that logic into a controller action would simply be unmantainable and probably not well tested. With this setup you can isolate the service object from the request / controller tests and really excercise it at all of the potential fail points.

For example, we use Amazon to handle transcoding and I needed a test to ensure all issues would be caught and wrapped appropriately.

RSpec.describe UploadService, '#start', type: :service do
  let(:user)    { Fabricate(:user) }
  let(:service) { described_class.new(user) }

  context 'when amazon fails to transcode' do
    it 'returns false' do
      allow(service).to receive(:transcode).and_return(false)

      expect(service.start(episode_id: SecureRandom.uuid)).to eq(false)
    end
  end

  context 'when amazon transcode is successful' do
    it 'returns true' do
      allow(service).to receive(:transcode).and_return(true)

      expect(service.start(episode_id: SecureRandom.uuid)).to eq(true)
    end
  end
end

Go forth and use services where complex controller actions exist.

Error Helpers in C


I’ve been writing a lot of C lately for a game I am working on. I am not a perfect programmer and I would like to catch my bugs before they arise. Thus, I am attempting to learn Rust in my free time.

Backtrace

Printing a backtrace in C is not incredibly difficult to accomplish. Although the following is fairly primitive, it will aid in your ability to discern what is happening in your program

/// @file core/backtrace.c
#include <execinfo.h>
#include "backtrace.h"

/// @brief Prints a backtrace of up to the last 256 calls
///
/// @param [out] fd The file descriptor ID to write to
void print_backtrace(int fd) {
    void* array[256];
    size_t size = backtrace(array, 256);
    backtrace_symbols_fd(array, size, fd);
}

The output will look like the following

bin/example(print_backtrace+0x22)[0x403002]
bin/example(test_block_insertion+0x218)[0x402b68]
bin/example(main+0x2f)[0x402eef]
/usr/lib/libc.so.6(__libc_start_main+0xf1)[0x7f3e3cb5c291]
bin/example(_start+0x2a)[0x4027fa]

As you can see, in a method test_block_insertion I call print_backtrace. This is incredibly helpful if you need to determine how deep in the program your issue occurred.

Panic

I’ve been tinkering with Rust in my free time to get a better understanding of it and I have come to enjoy the panic! macro it employs. I like the verb and decided that it would work perfectly in my daily use.

With some alterations, I just wanted panic to do exactly what its name suggests, I want the program to panic with a message and abort.

/// @file core/panic.h
#pragma once

#include <stdlib.h>
#include <stdio.h>
#include "backtrace.h"

#ifdef NDEBUG
#define panic(message)
#else
/// @brief Causes the program to abort and print a message
///
/// @param [in] message The error message you wish to spit out to stderr.
#define panic(message)                                               \
    do {                                                             \
        fprintf(stderr, "panicked at: %s:%d\n", __FILE__, __LINE__); \
        fprintf(stderr, "--> %s %s\n", message);                     \
        fprintf(stderr, "--> STACKTRACE START\n");                   \
        print_backtrace(2);                                          \
        fprintf(stderr, "--> STACKTRACE END\n");                     \
        fflush(stderr);                                              \
        abort();                                                     \
    } while (0)
#endif

Example usage would be like this

int some_function(uint8_t* data, size_t length) {
  if (length == 0) {
    panic("Zero length buffer provided!");
  }

  //
  // consume data
  //

  return 1;
}

Output from this macro will look like the following.

panicked at: /home/warmwaffles/code/example/rbp_test.c:173
--> message: Failed to insert the block correctly
--> START STACKTRACE
bin/rbp_test(print_stacktrace+0x22)[0x403002]
bin/rbp_test(test_block_insertion+0x218)[0x402b68]
bin/rbp_test(main+0x2f)[0x402eef]
/usr/lib/libc.so.6(__libc_start_main+0xf1)[0x7f3e3cb5c291]
bin/rbp_test(_start+0x2a)[0x4027fa]
--> END STACKTRACE
Aborted (core dumped)

I wanted the core dump to take place so that I can inspect it if I need to. Luck favors the prepared, and I always like to be prepared.

Assert

I use assert(expr) liberally through out my code to ensure that my program operates as I intend it to. Sometimes I make a mistake and would like to be notified where it happened and how deep in the call stack it did.

Unfortunately vanilla assert(expr) does not do this. But it is a simple enough macro to override and provide a little more meta information about where it failed and why.

/// @file core/assert.h
#pragma once

#include <stdlib.h>
#include <stdio.h>
#include "backtrace.h"

#ifdef NDEBUG
#define assert(expr)
#else
#define assert(expr)                                                                     \
    if(!(expr)) {                                                                        \
        fprintf(stderr, "assertion (%s) failed at: %s:%d\n", #expr, __FILE__, __LINE__); \
        fprintf(stderr, "--> STACKTRACE START\n");                                       \
        print_backtrace(2);                                                              \
        fprintf(stderr, "--> STACKTRACE END\n");                                         \
        fflush(stderr);                                                                  \
        abort();                                                                         \
    }
#endif

As you can see it looks almost exactly the same as panic(message) does. However, I want the expression to be spit out into stderr so that I can see what expression failed.

assertion (1 == 0) failed at: /home/warmwaffles/code/example.c:170
--> STACKTRACE START
bin/example(print_backtrace+0x22)[0x4015a2]
bin/example(test_block_insertion+0x1cc)[0x40134c]
bin/example(main+0x2f)[0x40148f]
/usr/lib/libc.so.6(__libc_start_main+0xf1)[0x7f989aaec291]
bin/example(_start+0x2a)[0x40102a]
--> STACKTRACE END
Aborted (core dumped)

Found these little bits of code to be useful, and figured others would probably find it useful as well.

Ruby Mixins


Mixins are a bit of a touchy spot for me. I am in a love hate relationship when it comes to them. In some cases, they work brilliantly and in other cases they hide complexity.

I have a few rules that I try to follow when I am considering using mixins.

  1. Does it hide complexity / indirection?
  2. Will it benefit the code base to share code?
  3. Will it override methods or will methods need to be overriden?

Moving methods into mixins to just move them is not sufficient enough to warrant the need for mixins. It is something that should be used to refactor once a pattern is established. Mixins should only be used for adding abilities to classes.

Let’s take a look at an example. We need an ability to revoke tokens mixed into three separate classes.

module Revokable
  # Sets when the token was revoked
  # @return [void]
  def revoke
    @revoked_at = Time.now
  end

  # Check to see if the token was revoked
  # @return [TrueClass,FalseClass]
  def revoked?
    !!@revoked_at
  end
end

The personal token represents a token that belongs to a user and never expires, but it can be revoked.

class PersonalToken
  include Revokable

  attr_accessor :id, :token, :user_id
end

The access token represents a token that is only available for a limited time.

class AccessToken
  include Revokable

  attr_accessor :id, :token, :user_id, :refresh_token_id

  def expire
    @expired_at = Time.now
  end

  def expired?
    !!@expired
  end
end

The refresh token never expires but is only used for getting another access token.

class RefreshToken
  include Revokable

  attr_accessor :id, :token, :user_id
end

We have accomplished is adding an ability to three classes without hiding a lot of complexity. If you find that your classes have too many methods defined, that should be a sign that it is too complex. But, do not immediately reach for mixins just because it makes the class less cluttered. It actually hides the mess as opposed to solving the issue.

Use mixins wisely!

A Ruby Yukata


Let me introduce you to my new library called Yukata. It is a light weight Ruby attribute library that is configurable and extendable.

Virtus has been a huge inspiration for this library. I enjoyed the DSL it offered, while allowing me to have a quick way to make data objects.

Here is an example on how to utilize Yukata:

class Person < Yukata::Base
  attribute :first_name, String
  attribute :last_name,  String
  attribute :born_on,    DateTime
  attribute :married,    Boolean, default: -> { false }
end

The #attribute method is straight forward with its meaning. It is dynamically creating both getter and setter methods for the object. It can be thought of as a fancy attr_accessor but with a few extra features. It provides a fast way to discover what data type can be expected for that attribute.

Example Usage

When using Yukata, the the initializer expects a hash to be provided or a class that behaves like a Hash.

john = Person.new({
  :first_name => 'John',
  'last_name' => 'Doe',
  :born_on => '1969-01-16T00:00:00+00:00'
})

Yukata will take the hash and assign the values to their respective attribute keys. If a setter method is defined, then a corresponding value can be passed as well.

class Foo < Yukata::Base
  attr_accessor :bar
  attribute :qux, String
  attribute :baz, String, writer: false

  def baz=(value)
    @baz = value.to_s
  end
end

foo = Foo.new({
  bar: 'woot',
  qux: 'herp',
  baz: 'derp'
})

foo.bar # => 'woot'
foo.qux # => 'herp'
foo.bas # => 'derp'
foo.attributes # => { bar: 'woot', qux: 'herp' }

If a :coerce => false is passed, then Yukata will not attempt to coerce that attribute and leave it as is. This can be handy if a custom coercion is desired for the specific model. Here is an example:

class Episode < Yukata::Base
  attribute :season, Integer
  attribute :number, Integer
  attribute :name,   String, coerce: false

  # @override overides the yukata definition
  def name=(value)
    @name = '%sx%s - %s' % [@season, @number, value]
  end
end

episode = Episode.new({ season: 1, number: 1 })
episode.name = 'Foo Bar'
episode.name # => '1x1 - Foo Bar'

Now, remember just because there is access directly to instance variables does not mean it is okay to abuse them. With great power comes great responsibility, this means I am not responsible for your mistakes.

Setting Attribute Defaults

Sometimes the objects need default values if it is not set. Defaults are lazily loaded. They will only be set once the getter method is called.

class Book < Yukata::Base
  attribute :name,       String
  attribute :created_at, DateTime, default: -> { DateTime.now }
end

Registering Custom Coercions

This library only comes with basic coercers. I tried to make as little assumptions about the data coming in as I could. I believe that the consumer of the library should be the one who defines the coercions.

If the value can not be coerced, it is simply passed through and left alone.

Yukata.coercer.register(String, Array) do |string, target|
  string.split(' ')
end

Optional Readers and Writers

When declaring an attribute, both the reader and writer can be skipped. There is a use case where this would be handy.

class Book < Yukata::Base
  attribute :title, String, writer: false, reader: false

  def title=(value)
    @title = value.to_s
  end

  def title
    @title
  end
end

This is a bit contrived, but it demonstrates the following:

  1. The expected return data type for #title is a String.
  2. Custom coercer is defined.
  3. The attribute will be included when #attributes is called on Book.

If :writer => false is provided, there would be no need to include :coerce => false since the coercion only takes place when the value is being set on the object.

Conclusion

I wrote this library becaues I wanted to see how Virtus accomplished this task and how I could go about doing it differently. This is a highly configurable library that can be used to put your fat models on a diet.

References