Services and Pub Sub


The Publish-Subscribe pattern is a great way to make code modular and decoupled from the rest of the architecture. It also has a nice side-effect of making testing easier and classes smaller.

In a pub-sub pattern, publishers don’t know anything about its subscribers and subscribers don’t know anything about who is publishing to it. Subscribers are simply listening for specific messages and handling them accordingly and publishers are simply broadcasting messages.

Subscribers are meant to be reused with other publishers. Though often some subscribers will only be used in one place. The value add with this approach is that the subscriber can be removed at anytime and readded. If one were to implement a feature flipper, dynamically adding subscribers can be a big win.

Let’s take a dive into the Signals gem.

At a basic level a publisher can be considered a service object and any calls to and external service can be a subscriber. Subscribers need a way to listen for specific events and fire off the appropriate actions.

class ActivityListener
  include Signals::Subscriber

  listen_for :failed_login => :log_failed_attempt

  def log_failed_attempt(user)
    # ... some security audit stuff ...
  end
end

The ActivityListener is now isolated away from what other subscribers are doing. This makes testing really simple. A mocked user could be passed in and test spies could be used to ensure that the proper methods were invoked.

Sometimes there are events that will take place that need to have multiple actions taken. For example, when a user creates a subscription the application will probably need to send a confirmation email along with talking to a payment gateway while logging the request to some security tracker. This is a strawman example, but it is something that can happen.

class ActivityListener
  include Signals::Subscriber

  listen_for [:logged_in, :logged_out] => :log_activity

  def log_activity(user)
    # ... something ...
  end
end

Example

This is something similar to how I use Signals in production. This does assume you are using Delayed Jobs, however if you are using a different background job tool, I’m sure the conversion is easy enough.

# app/services/create_user.rb
class CreateUser
  include Signals::Publisher

  def initialize(params={})
    @user = User.new(params)
  end

  def execute
    if @user.save
      broadcast(:create_user_successful, @user)
    else
      broadcast(:create_user_failed, @user)
    end
  end
end
# app/jobs/welcome_email_job.rb
class WelcomeEmailJob
  def initialize(id)
    @id = id
  end

  def perform
    user = User.find(@id)
    EmailListener.new.user_created(user)
  end
end
# app/listeners/email_listener.rb
class EmailListener
  include Signals::Subscriber

  listen_for :create_user_successful => :enqueue_user_created

  def enqueue_user_created(user)
    Delayed::Job.enqueue(WelcomeEmailJob.new(user.id))
  end

  def user_created(user)
    WelcomeMailer.welcome_email(user).deliver
  end
end
# app/controllers/users_controller.rb
class UsersController < ApplicationController
  # ...
  def create
    service = CreateUser.new(user_params)
    service.subscribe(EmailListener.new)
    service.on(:create_user_successful) do |user|
      redirect_to root_url
    end
    service.on(:create_user_failed) do |user|
      @user = user
      render action: 'new'
    end
    service.execute
  end
  # ...
end

Resources

These are some resources I found extremely helpful as I wrote the library to further my understanding of the publish-subscriber pattern.

Task Based Threading


Task based thread execution is a great way to do threading in an application. Taking the same idea that is used in web development can be applied directly to any application that can utilize threads.

When working with a web application, often the application will need to send an email to a customer without slowing down the user’s request. Background jobs are typically used for slow; long running tasks that can be processed later.

These background jobs are normally added to queue and multiple workers are processing them simultaneously. Often I have been guilty of writing baddly threaded Java applications (C is not exempt from this either). I would make a few Runnable tasks that would have an infinite loop and just spawn threads to process the tasks. This was my naive way of thinking.

// SomeBadJob.java
public class SomeBadJob implements Runnable {
    public void run() {
        while(true) {
            // Some loop to crunch numbers
        }
    }
}

In reality, the maximum number of threads running concurrently is based on the number of cores available on the CPU. So if I spawned five threads, one of them will be context switching and that is not free.

Solution

Stop writing individual tasks and break it down to a smaller tasks. Each runnable task should be executed once, and then return. For example:

// MovePlayerTask.java
public class MovePlayerTask implements Runnable {
    private Player player;

    public MovePlayerTask(Player player) {
        this.player = player;
    }

    public void run() {
        player.move(Direction.SOUTH, 1);
    }
}

Testing these runnable tasks becomes very easy and the overall complexity of the runnable job has been reduced significantly.

How this task can be rans is by wrapping Java’s ThreadPoolExecutor into a class called Scheduler and is setup as a singleton. This is because there is no need to have multiple schedulers around.

// Scheduler.java
import java.util.concurrent.LinkedBlockingQueue;
import java.util.concurrent.ThreadPoolExecutor;
import java.util.concurrent.TimeUnit;

/**
 * A very thin wrapper around ThreadPoolExecutor
 *
 * @author Matthew A. Johnston
 */
public class Scheduler {
    private static Scheduler instance;

    private ThreadPoolExecutor executor;
    private LinkedBlockingQueue<Runnable> queue;

    public Scheduler() {
        int processors = Runtime.getRuntime().availableProcessors();
        queue = new LinkedBlockingQueue<Runnable>();
        executor = new ThreadPoolExecutor(processors, 10, 10, TimeUnit.SECONDS, queue);
    }

    public void schedule(Runnable runnable) {
        executor.execute(runnable);
    }

    public static void scheduleTask(Runnable runnable) {
        getInstance().schedule(runnable);
    }

    public static Scheduler getInstance() {
        if (instance == null) {
            instance = new Scheduler();
        }
        return instance;
    }
}

To execute the MovePlayerTask all that is necessary is the following:

player = Game.getPlayer();
Scheduler.scheduleTask(new MovePlayerTask(player));

The MovePlayerTask will be executed once and to execute it again, all that is necessary is to re-enqueue the task. I have found that this approach will scale very well as computer hardware progresses. As the number of cores available increases, so will the number of workers.

Limitations

It is possible to fill the Scheduler queue up with jobs faster than it can process. Though I have not run into this issue yet, however the LinkedBlockingQueue that the ThreadPoolExecutor uses will not allow an new task to be scheduled until it is able to do so. If the application is time critical, a time delta should be used in the task’s run() method.

Resources

Null Objects


Rails is a wonderful library. It’s a framework that is opinionated and requires some footwork on the developers part. One such issue is returning nil when the return type is an object. Returning nill doesn’t provide the view enough information about what actions to take and leads to complex whack-a-mole scenarios.

The solution to this problem is to introduce a NullObject into the application. There are null object libraries already available, but they are so dead simple to write that it’s not necessary to use one. For example, it would be trivial to declare this basic NullObject.

class NullObject
  def to_s
    ""
  end

  def method_missing(*args, &block)
    self
  end
end

object = NullObject.new
object.this.doesnt.exist #=> <NullObject>

This is extremely handy when query objects are used.

Maybes

Maybes are really handy. It’s similar to the #try method that is in Rails.

user.try(:non_existent_method, NullObject.new) #=> <NullObject>

Instead, when the foreign key is not set, the method will return the NullObject.

class User < ActiveRecord::Base
  belongs_to :subscription

  def subscription
    self.subscription_id? ? super : NullObject.new
  end
end

This allows us to use the null object while still providing the original ActiveRecord functionality.

user = User.find(1)
user.subscription #=> <NullObject>
user.subscription.product #=> <NullObject>
user.subscription_id = 1
user.subscription #=> <Subscription>
user.subscription.product #=> <Product>

Falsiness

An issue that will be encountered is testing the falsiness of a NullObject. The following will not work:

obj = NullObject.new

unless obj
  # Never executes
  puts "I'm true"
end

A work around that should be used is defining nil? on an instance of NullObject.

class NullObject
  def to_s; ""; end
  def nil?; true; end

  def method_missing(*args, &block)
    self
  end
end

obj = NullObject.new

if obj.nil?
  puts "I'm true"
end

This makes the code readable at a glance. When if some_val is used then it can be confusing at first and one must recall the old Perl style of programming. In Perl nil and null were equivalent to FALSE.

The final NullObject that I ended up using looks like the following:

# app/models/null_object.rb
class NullObject
  def initialize(*methods, &block); end
  def to_s; ""; end
  def to_str; ""; end
  def to_i; 0; end
  def to_f; 0.0; end
  def to_c; 0.to_c; end
  def to_r; 0.to_r; end
  def to_a; []; end
  def to_ary; []; end
  def to_h; {}; end
  def nil?; true; end
  def present?; false; end
  def empty?; true; end
  def !; true; end
  def blank?; true; end

  # @return [NullObject]
  def tap
    yield(self)
    self
  end

  # @return [NullObject]
  def method_missing(*args, &block)
    self
  end
end

Custom Null Objects

This goes hand to hand with the “Maybes” section of this article. When trying to clean up view code conditionals, this is a great opportunity to use a NullObject specific to the model expected.

class NullSubscription < NullObject
  def product
    NullProduct.new
  end

  def active?
    false
  end
end

```haml app/views/users/index.html.haml

  • @users.each do |user| %tr %td= user.email %td= user.subscription.active? ? ‘Subscribed’ : ‘Not Subscribed’ ```

The benefit of returning a null object instead of nil is that crazy if checks wont be necessary in view code or model code. The ultimate goal of a NullObject is to get it to behave like the object it is trying to imitate. This adds predictability to the application and makes testing much easier.

Resources

Query Objects


I’m trying to migrate my models to be dumb data bags that have some useful state altering methods. I’ve found that using query objects, I was able to reduce the complexity of some of my models and controllers.

The Problem

My User model is gigantic. It’s well over 600 LOC, and it’s growing at a scary rate. Testing is becoming a nightmare because of all the methods. A few methods I have realized, I could remove completely and extract them into query objects.

# app/models/user.rb
class User < ActiveRecord::Base
  # ...

  def self.search text
    where {
      (users.email.like "%#{text}%") |
      (users.username.like "%#{text}%") |
      (users.first_name.like "%#{text}%") |
      (users.last_name.like "%#{text}%")
    }
  end

  # ...
end

Now this method isn’t exactly all that complicated but my User model is rittled with these methods. They are just there to be used by the controller like so:

# app/controllers/users_controller.rb
class UsersController < AuthorizedController
  def index
    @users = User.search(params[:q]).page(params[:page])
  end
end

All that method does, is execute a query. It doesn’t alter the state of any model. This could be extracted out into its own class and be isolated enough that testing becomes very simple.

The Solution

The solution is simple, kill The Batman. First, I need a query class. When creating query classes, keep in mind that you need to interact with ActiveRecord::Relation objects. This will allow for the query objects to be chained and used again later.

# app/queries/user_search_query.rb
class UserSearchQuery
  # @param [ActiveRecord::Relation] relation
  def initialize relation=User.scoped
    @relation = relation
  end

  # @param [String] text
  def search text
    @relation.where {
      (users.email.like "%#{text}%") |
      (users.username.like "%#{text}%") |
      (users.first_name.like "%#{text}%") |
      (users.last_name.like "%#{text}%")
    }
  end

  # @param [String] text
  def self.search text
    new.search(text)
  end
end

Second I need to replace the call to User.search from the UsersController

# app/controllers/users_controller.rb
class UsersController < AuthorizedController
  def index
    @users = UserSearchQuery.search(params[:q]).page(params[:page])
  end
end

Now looking at this, it is easy to say, “Well that didn’t change much”, but it did. I can remove the old method from the User model, and reduce the complexity a little bit by doing this.

Chaining

Here is an example of chaining some queries together

# app/queries/new_users_query.rb
class NewUsersQuery
  # @param [ActiveRecord::Relation] relation
  def initialize relation=User.scoped
    @relation = relation
  end

  def today date=DateTime.now
    between(date.beginning_of_day, date.end_of_day)
  end

  def last_seven_days
    date = DateTime.now.begninning_of_day
    between(date - 7.days, date.end_of_day)
  end

  def between starting, ending
    @relation.where {
      (users.created_at >= starting) &
      (users.created_at < ending)
    }
  end
end

Now this may sound a little dumb, but this is merely just an example.

query = UserSearchQuery.new(NewUsersQuery.today)
query.search('someone').each do |user|
  NotificationMailer.welcome(user).deliver
end

References

Ruby Mixin Exploration


Recently I’ve been playing with Ruby’s class_eval and define_method to append methods to an object. Meta programming is something new for me and I had a problem that would benefit from using some of Ruby’s cool features.

The Problem

There was a model that had 6 fields that needed to be generated at some point in time. Some fields had to be unique and others had to be generated a bit differently. Here is a contrived example of what I was doing:

class User < ActiveRecord::Base
  def generate_api_token
    begin
      self.api_token = SecureRandom.urlsafe_base64(32)
    end while self.class.exists?(api_token: self.api_token)
    self.api_token
  end

  def generate_new_password
    self.password = SecureRandom.urlsafe_base64(16)
  end
end

class Widget < ActiveRecord::Base
  def generate_slug
    begin
      self.slug = SecureRandom.urlsafe_base64(32)
    end while self.class.exists?(slug: self.slug)
    self.slug
  end
end

If these generators are spread across multiple models, it will cloud the source code and a test will need to be written for every generator. If we utilized shared examples, there wouldn’t be a need to write a test for every single generator.

The Goal

In the end, I wanted an easy to read DSL. Adding more DSL to a class can mean a “code smell”, however in this case, I do not think it is a code smell.

# app/models/some_model.rb
class SomeModel < ActiveRecord::Base
  include Generatable

  generatable :some_field,   lambda { SecureRandom.hex(32) }
  generatable :unique_field, lambda { SecureRandom.hex(32) }, unique: true
end

s = SomeModel.new
s.generate_some_field #=> some 32 character string
s.generate_unique_field #=> some 32 character unique string

The pros

  • Easy to read
  • Easy to figure out where generatable is being loaded from
  • Compact
  • Testing becomes easier (more on this later)

The cons

  • Can’t use the model in the lambda
  • Adds to model complexity
  • Lose some control on testing (more on this later)

Solution

This is a perfect case where ActiveSupport::Concern applies. This is functionality that multiple models can utilize and thus, should be put into a mixin and included to those models. My initial implementation only made a simple generator but, it does show how class_eval works and is an easy example to grok.

# app/models/concerns/generatable.rb
module Generatable
  extend ActiveSupport::Concern

  module ClassMethods
    # @param [String] field the name of the field you want to generate
    # @param [Proc] generator the lambda or Proc that you wish to use as the
    #   generator
    # @param [Hash] options
    # @option options :unique
    # @return [void]
    def generatable field, generator, options={}
      field = field.to_s
      class_eval do
        define_method "generate_#{field}" do
          self.send("#{field}=", generator.call)
        end
      end
    end
  end

end

The next requirement I had to fulfill was generating unique values. This took a little bit more thought.

# app/models/concerns/generatable.rb
module Generatable
  extend ActiveSupport::Concern

  module ClassMethods
    # @param [String] field the name of the field you want to generate
    # @param [Proc] generator the lambda or Proc that you wish to use as the
    #   generator
    # @param [Hash] options
    # @option options :unique
    # @return [void]
    def generatable field, generator, options={}
      field = field.to_s

      if options[:unique]
        class_eval do
          define_method "generate_#{field}" do
            begin
              self.send("#{field}=", generator.call)
            end while self.class.exists?(field.to_sym => self.send(field))
            self.send(field)
          end
        end
      else
        class_eval do
          define_method "generate_#{field}" do
            self.send("#{field}=", generator.call)
          end
        end
      end
    end
  end

end

This isn’t exactly pretty. Infact, one could argue that this is a bit difficult to follow due to the if options[:unique] statement. Another possible solution could be to break this up into three methods like so:

# app/models/concerns/generatable.rb
module Generatable
  extend ActiveSupport::Concern

  module ClassMethods
    # Appends a generator
    # @param [String] field the name of the field you want to generate
    # @param [Proc] generator the lambda or Proc that you wish to use as the
    #   generator
    # @param [Hash] options
    # @option options :unique
    # @return [void]
    def generatable field, generator, options={}
      if options[:unique]
        generatable_unique field, generator
      else
        generatable_simple field, generator
      end
    end

    # Append a unique generator
    # @param [String] field the name of the field you want to generate
    # @param [Proc] generator the lambda or Proc that you wish to use as the
    #   generator
    def generatable_unique field, generator
      field = field.to_s
      class_eval do
        define_method "generate_#{field}" do
          begin
            self.send("#{field}=", generator.call)
          end while self.class.exists?(field.to_sym => self.send(field))
          self.send(field)
        end
      end
    end

    # Append a simple generator
    # @param [String] field the name of the field you want to generate
    # @param [Proc] generator the lambda or Proc that you wish to use as the
    #   generator
    def generatable_simple field, generator
      field = field.to_s
      class_eval do
        define_method "generate_#{field}" do
          self.send("#{field}=", generator.call)
        end
      end
    end
  end

end

Testing

Earlier I mentioned that this makes testig easier and then followed up with a loss of control in the tests. The Generatable module will need to be tested in some fashion, whether it is directly testing it via including it on a dummy model or making a shared example where the test would have it_behaves_like somewhere.

Testing a module directly:

# Module Testing
module Say
  extend ActiveSupport::Concern
  def hello
    "hello"
  end
end

class DummyClass
  include Say
end

describe Say do
  let(:dummy_class) { DummyClass }
  let(:dummy) { dummy_class.new }

  it "get hello string" do
    dummy_class.hello.should == "hello"
  end
end
# spec/support/shared_examples/generatable_fields.rb
shared_examples 'generatable fields' do |field|
  describe "#generate_#{field}" do
    subject { described_class.new }

    it "should change #{field}" do
      expect {
        subject.send("generate_#{field}")
      }.to change(subject, field.to_sym)
    end
  end
end
# spec/models/user_spec.rb
describe User do
  include_examples 'generatable fields', :api_token
  include_examples 'generatable fields', :password
end

Resources