SOA Series Part 2: SOA Sample Apps and their Development and Deployment Flow

This is the second in a series of seven posts on service-oriented architecture derived from a workshop conducted by Cloves Carneiro and Tim Schmelmer at Abril Pro Ruby. The series is called SOA from Day One – A love story in 7 parts.

In the previous article in this series, we laid out some of what we understand about Service-Oriented Architecture, and why we think starting with SOA from day one is a good idea.

To bootstrap things, and to get us focussed on some key areas of developing in a SOA, we have gone ahead and built out a set of apps at varying stages of completion. This, and the following 5 posts will focus on showing some of this code. Exercises to add features, or to refactor will be interspersed.

In this part, we introduce the applications that will constitute our tiny SOA. We will show how to set these applications up locally, how to deploy them to heroku, as well as explain a viable development workflow for building applications in a service-oriented set-up.

The functionality of the entire system is to show deals (aka. ‘inventory items’) available in or near a given city. These items can also be organized by a ‘category’ (aka tag), to cater to the various customers’ areas of interest.

Let’s start out by introducing the sample applications, all of which can be found on my github account.

The Consumer-Facing Application

The Deals app

This application is a classic (albeit entirely DB-less) Rails 4 web app, based on Twitter’s bootstrap. The main features it will be exposing are:

  • a list of deals (aka. inventory items) in a market (city)
    • links to the pages for nearby cities
  • a list of inventory items nearby (i.e., in cities within a given range in miles)
  • details about a single inventory item
    • shows an item’s image, price, and description
    • also shows a section with deals nearby
  • a page for deals of a particular category (i.e., inventory items that are tagged as belonging to a particular category)

Three services

The above Deals application retrieves all its data from the following backend applications.

Inventory Items Service

This is the application’s core service, and it vends, amongst other things

  • general inventory information (price, market, title, image url, description)
  • a list of all inventory items
  • all items anchored in a particular city
  • all items in cities nearby a given city
  • plus, the various endpoints to add or manipulate inventory items

Tagging Service

This service application is responsible for storing tags, and for associating them with inventory items.

It has the following endpoints for tag management:

  • show a list of all tags
  • show information about a single tag (which includes list of items tagged)
  • create / manipulate tags

As a second set of functionality, it exposes endpoints for managing tagged inventory items, namely:

  • to return a list of all inventory items tagged with a given tag
  • to return a single tag + item combination
  • to create an “item is tagged” relationship (i.e., tag an item)
  • to delete an “item is tagged” relationship (i.e., ‘untag’ an item)

Cities Service

Finally, this service’s responsibility is to own all city (aka. market) related data (like its name, country, state, lat/long, and cities nearby). It offers endpoints to:

  • find a city by ID and show its full information,
  • list all cities
  • list all cities in a given country
  • list all cities near a given city (in a given range)
  • create / manipulate cities

Setting up the apps locally

We know, we just told you in part I that we don’t want to ever have developers have to set up all the services on their local machines. This general rule came with a footnote, though: if you would like to make changes to service apps, you will still need to have them set up locally … which is exactly what we are going to be doing from now on.

  • Clone the repositories for all four applications listed above.

  • In each of the 3 ..._service apps, execute the following:

1
2
3
4
5
bundle install
bundle exec rake db:create
bundle exec rake db:migrate
bundle exec rake db:fixtures:seed
rails s -p <unique_port_of_your_choice>
  • In the front-end deals app:
1
2
bundle install
rails s -p <unique_port_of_your_choice>

As these applications depend upon each other (as described in the previous section), there needs to be configuration to declare these dependencies in each of the apps.

For the deals application, the only dependency is inventory-service, as it is (quite inelegantly) hardcoded in the application’s ruby code

1
RemoteInventory.host = 'http://inventory-service-development.herokuapp.com'

The service with the most dependencies is inventory-service, as it relies on cities-service and tags-service

The cities-service will initially have no external service dependencies, but will add reliance on tags-service at a later stage in its lifecycle. Lastly, tags-service is entirely without external service dependencies, and will remain so for the rest of this series of blog post.

As you can see in this example of inventory-service’s configuration for accessing tags-service, the application reference heroku instances (owned by me) in the development environment by default:

1
2
3
4
5
development:
  host: tags-service-development.herokuapp.com
  port: 80
  api_version: 1
  max_concurrency: 4

The clear advantage for developers who are just concerned with making changes to inventory-service is that they will not need to run any of the other dependent services (in this case cities-service or tags-service) locally on their development machine: all communication happens into the heroku ‘instances in the cloud’.

However, if you ever need to make changes in several services that depend upon each other (e.g., tags-service and city-service), then you can start both services locally and simply change the service configuration in city-service to point at a locally running instance of tags-service (which in this example running on port 3004), like so:

1
2
3
4
5
development:
  host: localhost
  port: 3004
  api_version: 2
  max_concurrency: 4

Using Heroku as your development environment

While there are many ways to get started on a cloud-hosted infrastructure, the rest of this series will use heroku’s infrastructure to show the basics of how a development workflow in a service-oriented environment can work out.

If you are following along and want to try some of the examples in this series of posts by yourself in a cloud-hosted environment, you can get started by following these steps:

  • Unless you already have an account with heroku, visit Heroku and sign up for a free developer account
  • Next, we are loosely following the “Getting Started with Rails 4.x on Heroku” guide:
    • heroku login
    • in each of the local git repos for cities_service, inventory_service and tags_service, do:
1
2
3
4
5
6
heroku apps:create $MY_UNIQUE_HEROKU_NAME
heroku git:remote -a $MY_UNIQUE_HEROKU_NAME -r development
git push development master
heroku run rake db:migrate
heroku run rake db:seed
heroku  ps:scale web=1

NOTE: the cities-service DB seed exceed the limit for the “Hobby Dev” pg instance; either trim down the seeds.rb file to < 10k entries, or upgrade to “Hobby Basic” (~ $9 / month)

Now visit your app via a browser hitting http://$MY_UNIQUE_HEROKU_NAME.herokuapp.com, or via heroku open. You should see a swagger UI index page of your service

Here is a list of miscellaneous, useful commands that we have found to come in handy when working with heroku:

  • heroku logs : shows your app’s rails logs
  • heroku run rails console : bring up a rails console on your application server
  • heroku maintenance:[on|off] : dis-/enables your app and shows a maintenance page
  • heroku ps : lists all your app’s dynos, and what they are running
  • heroku ps:scale web=[0|1] : dis-/enables your web worker dynos
  • heroku config:set MY_ENV_VAR=foo --remote development : sets an environment variable (e.g., RAILS_ENV) on your app’s server
    • We set the RAILS_ENV and RACK_ENV variables on some of the heruko apps to development this way … more later.

So, … where are the tests for this code?

Well, there are no tests. Yeah, yeah, we know … we will go into TDD hell, and all.

But seriously, why are there no tests? Here are the main reasons:

  • These projects will never see any production traffic.
  • We have done a fair amount of manual testing of the services via a cool ‘service explorer’ JavaScript UI
  • The underlying service framework code is well-tested in our production projects; much of what you see in the service repositories is almost exactly the same code as what we are using in our day-to-day jobs
  • we are lazy, and we want you to do all of you work for us in a later part of this series where you will be adding some test coverage.

Development and Deployment Workflow

Manage several environments

To effectively develop, test and deploy in a SOA, you will need (at least) three environment ‘fabrics’

  • a local development machine (running tests and used for developing)

    • all your local development is done here
    • only check out the applications and services you actually are planning to change
    • configuration files (e.g., yml files in your Rails app’s config directory) for all services your application depends upon pointing into the development fabric (next)
  • a remotely hosted development (or staging) fabric

    • remote here means that these services do not run on your local development machine
    • all services your local machine depends upon run here, at a well-known domain names; in our example set of applications, the three services run at cities-service-development.herokuapp.com, tags-service-development.herokuapp.com, and inventory-service-development.herokuapp.com
    • once development is completed locally, new (feature) branches get promoted to here
    • data held in data stores for applications deployed here is ‘production like’, so that production conditions can be simulated; it is most likely manufactured seed data, sometimes generated from production data dumps.
    • after some quality assurance, code will be promoted to the next (production) stage
  • and finally, the production fabric

    • this is your stable production level code
    • this is running on server instances that the end customers will connect to and exercise
    • the data here is the actual production data all your live applications produce and depend upon

How is this implemented?

Looking at the sample applications we have coded up for this series, every app has (yml) configuration files that declare where its dependent services are located:

  • for the test, development and production environments (e.g., the respective sections in tags_service.yml and cities_service.yml in the inventory_service repository)
  • the development sections point at the well-known development fabric instances of the dependent service, while production sections point at production services
  • the test sections most likely point at the well-known development fabric instances, or the actual production services
    • this way, tests can retrieve (test) data from such dependencies without having to run them all locally
    • the service responses can be stored in canned responses for future runs (e.g. using gems like vcr … about which we will talk later)

Working with Heroku

First off: there is no particular reason why we chose heroku over similar public cloud offerings such as straight AWS EC2 instances, or DigitalOcean, or even private clouds like Cloud Foundry Community.

We like heroku for the reason that it offers comprehensible pricing, as well as good documentation for working with multiple environments.

Here are a list of steps you can follow to set up your own development and deployment workflow when using heroku:

  • Create your development fabric instance of a service via the heroku command line tool

    • heroku create --remote development

    …or rename your current heroku remote via

    • git remote rename heroku development
  • Create your production fabric instance of a service

    • heroku create --remote production
  • By default, heroku apps run in production environment mode; to have your development fabric instances all easily point at each other, change their RACK_ENV and RAILS_ENV environment settings to development, like so:

    • heroku config:set RACK_ENV=development RAILS_ENV=development --remote development --app <my_dev_heroku_appname>
  • As heroku ignores all branches that are not master, you need to push you local feature branches to the remote master branches

    • I.e., to push a branch called feature to your development fabric instance, you need to do:

    git push development feature:master

  • As an example of my current git set-up for inventory_service:

1
2
3
4
5
6
7
$ git remote -v
development     git@heroku.com:inventory-service-development.git (fetch)
development     git@heroku.com:inventory-service-development.git (push)
origin  git@github.com:timbogit/inventory_service.git (fetch)
origin  git@github.com:timbogit/inventory_service.git (push)
production      git@heroku.com:inventory-service.git (fetch)
production      git@heroku.com:inventory-service.git (push)

Exercise “Deploying Code”

If you followed along, and want to play around with deploying the sample apps in a distributed multi-fabric environment, we suggest you try tackling the following exercises based on the sample code:

  • Create production and development instances on heroku for all three sample services, and adapt your git remotes respectively.
  • As inventory_service depends on cities_service and tags_service, its services’ configuration yml files (config/tags_service.yml and config/cities_service.yml) will need to be pointed at your production and development fabric instances respectively. Keep your test yml file entries pointed at your development fabric instances.
  • Make similar changes to the cities_service repository, so that it will point at the dependent tags_service in its respective development and production fabrics
  • push these changes (e.g., as new git branches) first to your development fabric boxes, and then – after some initial testing – through to the production fabric services

Comments