Monthly Archives: January 2014

Setting up a Read Only Follower Database on Heroku

As CoachUp has grown, the amount of statistics and reporting needed to support the core business has also grown.  Although it may not be evident to passersby, every business has back end reporting and statistics needs in addition to customer facing features.  There are some tools and services that make reporting easier, but I have yet to find one that actually covers all of our use cases (If you don’t believe me, I challenge you to leave a comment and make a recommendation or tweet me up @aradmand).  This means that we need to store metadata and statistical information ourselves.  If your application is backed by a database, you’re probably using the same database to store both customer and statistics related data.

When a business is just getting off the ground this is perfectly acceptable. However, this presents a couple of problems:

  1. As the business grows, the amount of application data/logic and statistical data/logic also grows. It makes sense to decouple and separate application data from statistical data and the larger each set grows, the more difficult this will become.
  2. Reports can often be taxing on the database.  If the same database is being used to run your application as well as statistics reports, it means that application performance can be degraded by running those reports.  Not ideal for your customers!

Solutions:

There several options when you get yourself into the aforementioned scenario (note: this is not an exhaustive list, but it enumerates the practical options that we realistically have on the table given time/effort/resources):

  1. Move all reporting and statistical data/logic out of your original application to a completely separate database and create a separate reporting application.
  2. Leave the statistical data/logic coupled in your database; set up a follower database; and alter your application code to have all reports pull data out of the follower database.
  3. Leave the statistical data/logic coupled in your database; set up a follower database; and create a separate application that runs using the follower database.

There are pros and cons of each approach, which I’m more than happy to discuss in the comments below or on social media.  Ultimately, we decided to take approach #3.

Why?

I’m glad you asked! We decided approach #3 was best for us because we ultimately want to implement scenario #1, with a separate application AND completely decoupled statistics data.  However, given time an resources available, we determined that the last approach would be a move in the right direction and could be achieved relatively quickly.

How?

Setting up this configuration was fairly straightforward:

1.  Create a follower Read-only database that mirrors your production database

Heroku makes this super simple.  A follower can be created for any Standard, Premium, or Enterprise tier database that is itself not a follower (that is, followers cannot be chained).

Use the heroku pg:info command to ensure your database is capable of supporting a follower. You also need to know the add-on name of the master database (i.e. HEROKU_POSTGRESQL_*COLOR*_URL):

heroku pg:info
=== HEROKU_POSTGRESQL_PURPLE_URL (DATABASE_URL)
...
Fork/Follow: Available
Status: available
...

Create a follower database by provisioning a new heroku-postgresql Standard, Production, or Enterprise add-on database and specify the master database to follow with the --follow flag.

heroku addons:add heroku-postgresql:ronin --follow HEROKU_POSTGRESQL_CHARCOAL_URL

That’s all!  Give the database some time to catch up to the master DB (depends on the amount of data you have).

For full details, you can visit the following article:

https://devcenter.heroku.com/articles/heroku-postgres-follower-databases

 

2. Provision a second Heroku app, which will run off the follower database just configured

Creating a new app on Heroku is fairly straightforward. Follow the instructions listed here:

https://devcenter.heroku.com/articles/creating-apps

 

3. Configure the new app to use the follower database

On Heroku, applications use the DATABASE_URL environment variable to determine the database to which they should connect. So setting up the new application to use the follower database is just a matter or setting the DATABASE_URL variable to the location of the follower database:

Use heroku config to determine the URL of the follower database:

heroku config
=== Config Vars
...
HEROKU_POSTGRESQL_GOLD_URL:  postgres://u83npgc802nbpu:p4vj4nee924qcc5b8d21dp9m9pf@ec2-54-235-122-116.compute-1.amazonaws.com:5682/d347pnrm01llrm
...

Set the value of DATABASE_URL in your new app to the URL of the follower database:

heroku config:add DATABASE_URL='postgres://u83npgc802nbpu:p4vj4nee924qcc5b8d21dp9m9pf@ec2-54-235-122-116.compute-1.amazonaws.com:5682/d347pnrm01llrm' -a new-app-name

Voila! Done!

You now have a new application running off a follower database. Now, just deploy and push code to this new app as normal.

The result?  We can now run reports on production data using this follower database and not put any burden at all on our production site. Perfect!

 

Caveats and Unexpected Issues

One unexpected issue encounter during this project was our use of the DelayedJob gem. We rely on this gem to asynchronously execute longer tasks for us.  However, due to the implementation of DelayedJob, it required a DB write to the Job table each time a task was scheduled. Since we were now running off a Read-only database, this was not an option.

The solution? Resque to the rescue! Resque is a Redis-based job queuing system.  Switching to Resque allowed us to maintain the same functionality we already had in our application (with minimal alterations) but also worked in the new Read-only DB environment.

Stay tuned for a follow-up blog post describing how to set up Resque for your Heroku application!

Has anyone else attempted similar solutions? Any suggestions for how to make this configuration better?

HackFit Boston Hackathon

Last week I posted about the upcoming On Deck Cup sports + technology hackathon that is being held in CoachUp’s office.  We’re also super excited to announce that CoachUp will be sponsoring the first ever HackFit hackathon the following weekend, which is an event that combines engineering and fitness in a way that has never been done before. Read on for details!

HackFit promotes a different way of thinking for entrepreneurs and startups that reflects a balance of healthy living and innovation.  The fitness-technology hackathon is the first of it’s kind and is supercharged with with exercise sessions (climbing, yoga, crossfit, running, and more) healthy food, and active challenges!

I’m happy to say that in addition to sponsoring the overall event, CoachUp Martial Arts coach Amanda C. will be there to provide a breakout 1 hour martial arts session for anyone interested.

If you’re interested in sports, technology, being active, hackathons, or having a good time with fun people, definitely come check out HackFit and say ‘hi’ to the rest of the CoachUp team!

Details:

What: HackFit

When: September 20th – 22nd, 2013

Where: The event will be hosted jointly between Microsoft NERD Center and Brooklyn Boulders Climbing gym in Somerville

On Deck Cup Sports Hackathon

I wanted to write a quick blog post to let folks know about a couple of upcoming events in which the CoachUp engineering team will be involved.

The first event is the On Deck Cup regional sports hackathon.  Come join the CoachUp engineering team as well as others from the tech and sports industries for a weekend of innovation, hacking and general merriment!  The event is going to be held the weekend of September 7th-8th right here at CoachUp’s office!

The On Deck Cup is an offshoot of the On Deck Sports and Technology Conference.  The goal of the conference is to spotlight some of the coolest and most innovative technologies improving the life of the sports fan.

Here is a brief synopsis from the On Deck Cup website:

“… we’re organizing the On Deck Cup to provide support and attention to some of the most talented innovators in the sports technology space. Our hope is that the On Deck Cup series will foster innovative new projects and collaborations that, when fleshed out over time, could materially impact the sports industry in years to come.

We also aim to give the best of the best projects to emerge from the On Deck Cup series additional exposure by putting these projects (and their creators) front and center at the On Deck West conference, where top teams will demo their projects in front of hundreds attendees from the sports technology and sports business worlds. And of course, we’ve got some great prizes on offer for the best projects of the On Deck Cup series.”

Summary:

What: On Deck Cup Sports Hackathon

Where:

CoachUp’s office
355 Congress St.
4th Floor
Boston, MA 02210

When:

September 7th-8th

How to get involved:

Visit the eventbrite website and sign up! Click here for the On Deck Sports eventbrite sign up.

Have any great ideas for sports related apps?  Would love to hear of any in the comments section!

Continuous Integration – Tddium

The CoachUp engineering department is constantly refining its development process for the sake of efficiency. I wanted to spend some time talking about one change we’ve recently made that I really feel has maximized our development speed: setting up Tddium’s continuous integration environment (tddium.com).
I should begin by talking a bit about our development process at CoachUp. First, we attempt to get new features into production as quickly as possible. We push new releases of our code into production every single day. To ensure that we do not break existing functionality, we put a heavy emphasis on testing. More specifically, we put a heavy emphasis on automated testing. We’re a Ruby on Rails shop, so we leverage the great built-in testing functionality that comes with Rails (primarily rspec, but a variety of other utilities as well). Although none of our engineers subscribe to TDD fully, we are meticulous in ensuring that every piece of functionality that enters our codebase is accompanied by a corresponding set of tests. We have operated in this way since the company was started.

As a result of our practices, the test suite has grown larger each day, which has been both a blessing and a curse at the same time. The engineering team has a policy of not pushing any release to production if our test suite is not completely green. We realized that it was great that we were maintaining adequate test coverage across our application. But with a large test suite, the problems we ran into were twofold:

1. The amount of time in which our test suite ran constantly increased
2. As it took longer and longer to run our test suite, the frequency of test suite runs began to decrease (after all, we were running everything manually)

Our solution: Tddium continuous integration environment.

For those of you unfamiliar with Tddium and continuous integration, I’ll explain a bit about how we’ve integrated Tddium into our dev process to make us faster and more efficient. At CoachUp, we used Tddium to address the two problems mentioned above. We signed up for a Tddium account, which involved hooking up our github account and selecting a plan in accordance with the size of our test suite. After we were set up, the rest was really effortless!

From our perspective, we basically just develop as normal: create a new feature branch from our github repository, develop, push to github, issue a pull request, and merge when ready. In the background, Tddium works on our behalf to do several things. It will monitor our github repository and when a new feature branch is pushed up to github, Tddium springs into action by grabbing the new feature branch and cranking through our test suite. We then conveniently get an email report sent to us detailing the results of the test run. If the new feature branch introduces a regression bug, we know about it immediately and can fix it well before it even has a chance to become a problem. Further, Tddium makes it super easy to switch plans and add/remove workers based on your scaling needs.

For us, the move to Tddium greatly cut down on development time by letting us really step on the gas and develop at a face pace, knowing all the while that we would be notified immediately if we introduced any regressions into the codebase.

We’re constantly trying new tools and processes here at CoachUp to make us more efficient, and Tddium has been one of our biggest successes.

Bottom line: if you’re looking for a quick, easy, non-intrusive way to speed up your test suite and make your dev team more efficient, definitely check Tddium out!

This is what has worked for us. What tools have others used to boost the efficiency/productivity of their team?