SOA - using HTTP

Tue, Sep 2, 2014
Service Oriented Design with Ruby and Rails - Paul Dix

In the introductory article we talked about why we might implement a Service Oriented architecture in order to:

We also talked a little bit of theory on what an SOA looks like, how to define the boundaries for our services, the practical implications of building a distributed system, and the 3 typical communication patterns we might need:

There are many ways to approach an SOA infrastructure, but within the Ruby and Rails community we tend to see the following two choices being considered:

This article shows how to implement an SOA using HTTP.

Table of Contents

Synchronous Request/Response

Once we have decided on the boundaries to partition our services we need to implement a technology for communication between those services. There are 2 primary components when thinking about synchronous request/response using HTTP:

Hosting the Service

Let’s build a simple weather service whose only job is to return todays (random) weather forecast. We can build this as a rack application in config.ru:

class WeatherApp

  def self.call(env)
    [ 200, {}, [ "Today will be #{forecast}" ] ]
  end

  def self.forecast
    [ "sunny", "cloudy", "rainy", "windy" ].sample
  end

end

run WeatherApp

We can run this service from the command line:

$ rackup --port 9292 config.ru

However, in a production environment we would want to run multiple instances of the service and would turn to a robust application server such as unicorn:

$ unicorn --port 9292 config.ru

Using a unicorn configuration file we can run multiple workers, daemonize the master process, configure logging, and more. We can manage the master Unicorn process using traditional Unix signals.

Making Client Requests

We can start making requests by testing our service from the command line:

$ curl http://localhost:9292/
Today will be sunny

In order to make client requests from within our applications (or other services) we would choose an HTTP client library.

For example, using Faraday:

puts Faraday.get 'http://localhost:9292/'  # "Today will be cloudy"

The advantage of Faraday over other http clients comes from it’s extensibility options using a middleware pattern similar to Rack itself. If our application needs to make multiple independent service requests in order to fulfil a user request then we should make those requests in parallel:

A Better Service with Sinatra

For a single action service, building a low level rack application might be all that we need, but any non-trivial service will need to support a more extensive API with multiple, parameterized actions. The Sinatra DSL provides a great lightweight framework for building services.

Consider this sinatra app in config.ru:

require 'sinatra/base'

class MyApp < Sinatra::Base

  get "/hello" do
    "Hello World\n"
  end

  get "/weather/in/:location" do
    "It's #{forecast} in #{params[:location]}\n"
  end

  post "/submit" do
    "Submitted #{request.body.read}\n"
  end

  private

  def forecast
    ['hot', 'cold', 'warm'].sample
  end

end

run MyApp

Run the service:

$ rackup

Test the service from the command line:

$ curl http://localhost:9292/hello
Hello World

$ curl http://localhost:9292/weather/in/london
It's cold in london

$ curl http://localhost:9292/submit --data 'stuff'
Submitted stuff

Make service requests from our application:

conn = Faraday.new(:url => 'http://localhost:9292/')

puts  conn.get("/hello")                 # -> "Hello World"
puts  conn.get("/weather/in/london")     # -> "It's warm in london"
puts  conn.post("/submit", "stuff")      # -> "Submitted stuff"

For making synchronous request/response services using HTTP, using Faraday to connect to a Unicorn hosted Sinatra app is a nice combination. Combine Typhoeus with Faraday for making multiple requests in parallel and we have a very powerful tool for combining data from multiple services into a single web application request.

Asynchronous Worker Queue

If we have chosen HTTP as our transport for making synchronous calls, then we must turn to another technology for our asynchronous calls. Implementing a worker queue is already a common practice in the RoR community and we have a number of technologies to turn to:

If we already have Redis as part of our platform (e.g. for caching or session storage) then using Resque as our worker queue can be convenient. Alternatively using Beanstalkd as a simple, fast, work queue is a great option. Finally if we can be sure our workers are thread safe then using Sidekiq is another possible solution.

The following example shows how to use Resque for our asynchronous worker queue:

Start by installing both Redis and Resque:

$ sudo apt-get install redis-server
$ sudo gem install resque

Create a class in file worker.rb that can #perform some work:

class Worker

  @queue = "example"

  def self.perform(amount)
    amount.times do |i|
      puts "working for #{i} seconds"
      sleep 1
    end
  end

end

Create a Rakefile to contain tasks for our worker processes:

require 'resque/tasks'

task :environment do
  require File.expand_path("worker.rb")
end

task "resque:setup" => :environment   # ensure resque loads our env

Run a worker process:

$ QUEUE=example rake resque:work

Enqueue a worker process from our application:

Resque.enqueue(Worker, 10)

… or to test using IRB:

$ irb -I.
irb> require 'resque'
irb> require 'worker'
irb> Resque.enqueue(Worker, 10)

Asynchronous Publish/Subscribe

If we have chosen HTTP as our transport for making synchronous calls, then we must turn to another technology for our asynchronous calls. Implementing a publish/subscribe pattern is, unfortunately, not very common in the RoR community but we can turn to Redis for our solution:

The following example shows how to use Redis asynchronous PubSub:

Start by installing Redis and it’s Ruby client library:

$ sudo apt-get install redis-server
$ sudo gem install redis

Publishing an event to a channel from our application is as easy as:

require 'redis'

CHANNEL = "my.channel"

redis = Redis.new
redis.publish CHANNEL, "an.event"
redis.publish CHANNEL, "another.event"

The event can be a simple string, or it can be a serialized Hash or JSON object.

Creating a subscriber process can be done with a Ruby script:

#!/usr/bin/env ruby

require 'redis'

CHANNEL = "my.channel"

redis = Redis.new(:timeout => 0)
redis.subscribe(CHANNEL) do |on|
  on.message do |channel, msg|
    puts msg
  end
end

For a production environment we might also need to extend the script to:

I will leave those topics (daemonizing Ruby processes) for a future article.

Up Next…

This article was a (quick) look at how to implement the 3 common communication patterns of an SOA using HTTP related technologies such as Rack, Unicorn, and Faraday, along with other technologies such as Redis and Resque for asynchronous communication.

In the next article we will look at how to implement the same 3 communication patterns of an SOA using a single technlogy - AMQP using a message broker such as RabbitMQ.

HTTP Servers

(Lightweight) Web Frameworks

Client libraries

Worker Queues

Publish/Subscribe