A new mentorship program

I recently returned from RailsConf 2012 in Austin.  I had a terrific time,  learned a ton, and most importantly: I was inspired.  I was inspired to return to making more contributions to Open Source and to help introduce others (kids and adults) to the computer world.

RailsConf had three double-sided whiteboards that (as far as I could tell) with nothing but “We’re hiring” ads on them.  I also stopped by a job fair where there were more companies hiring than applicants.  And finally, when all the attendees were asked if they worked for a company that was hiring Rails developers, at least two-thirds raised their hands!

So my first idea to get wings is a new mentorship program I’m starting in my town.  This is specifically a locally-oriented program.  I’m currently searching for 1-3 high school or college students who are interested in learning how to be the kind of well-rounded & highly-knowledgeable developers that these businesses are looking for.

Over about the next four months, we’ll cover the types of topics not typically covered at university, such as: git, TDD, setting up your own servers, and a complete understanding of the Rails stack.  This isn’t an internship (they’re doing their own work, not work for some other company), and it’s not a class (I’m not getting paid).  It’s just a chance to get some young new developers off on the right foot.  There will be a ton of material to cover, and there definitely will not be a speed limit!  Along the way, I hope to have each student come up with their own web app idea and fully implement it, start to finish.  In the end, I hope they’ll wind up with something even more impressive than a portfolio to be able to show potential employers.

I’m pretty excited and hope to find some equally excited students.  I’ll be periodically offering updates on how the program is going.  I also hope to open source my lesson plan as well for use by others, so check back for more!

Where do you get hosting support?

For quite some time now, I’ve found that the options for good Rails hosting have been significantly lacking.  As a consultant/contractor on a huge range of projects, I’m often asked for advice, guidance, or help in choosing and setting up servers for a client.  Nearly every client or customer wants the same thing:

  1. Stability/reliability
  2. Flexibility/room to grow
  3. Someone to keep things running
  4. Someone to call when they need help

The first two options are met by a lot of providers.  Tier IV datacenters, hardware redundancy, and virtualization are a dime-a-dozen nowadays and building a good Rails stack is just about the same for everyone.

However, the rub is in #3 & #4.  As my colleagues, peers, and I are already work full-time writing new applications, there is precious little time for system administration and support of completed projects and old apps.  Even with extensive automation, a small 1-3 person team can write many more Rails apps than they can support long-term.

Inevitably, the client wants to know “Who is going to keep things going once development is done?” and “Who can I call when things stop working?”.  Set them up on a physical machines, VPSes, EC2, or anything else and the developer is left with little choice but to help keep that server running long-term.  Including late night phone calls when something goes wrong.  And can you honestly say you are regularly doing all the little extra things that need to be done?  General maintenance?  Security patches?  Tuning?

Want an alternative?  AWS Premium support won’t touch your software stack.  Rackspace won’t support Rails.  Slicehost: no managed option at all.  There’s really only one player: just google ‘rails cloud support’.

So my question is: If you can get easy, scalable, on-demand hosting, why can’t you get easy, scalable, on-demand support? My answer to this issue is to launch a service that lets developers keep developing while someone else takes care of the system administration long-term:  RoundHouse Support.  Please read my public release announcement and then come check us out!

Announcing RoundHouse – Managed support for your host

I’m very happy today to announce a new community service available for Rails shops: RoundHouse – Server Management and Support.

RoundHouse is a cooperative solution for getting managed servers and system administration for your Rails stack, no matter what host you use.  We’re gathering a pool of specialists that you can call upon to get the help you need.  Whether that’s emergency support when you’re having server problems, regular day-to-day duties, or assistance in configuring a particularly difficult piece of software.

This is a service that provides freelancers, development shops, and companies alike an opportunity to focus on their product instead of on their hosting.  For developers this means freeing up more time to code.  For those running a website it means reliable service from a great group of experts.  For everyone it means having someone available whenever you need it.

Obviously this is a new offering and system administration (much like your hosting provider) must be utterly reliable.  So we’re beginning to establish a base set of clients to try out our service for free.  This gives us the chance to get established, continue to develop a solid organizational structure, and to expand our brand.  For our customers, it means you’re going to excellent sysadmin support at no charge while you learn about all the great things we can offer (and then hopefully recommend us to all your friends!).  So if you’ve been needing help setting up or running your Rails app, please contact us to get started.

We’re also looking to add additional members to our team as we continue to grow.  If you have expertise in system administration, elements of the Rails stack, or are just a great DevOp, please e-mail us at jobs@roundhousesupport.com and we can talk more!

Finally, please feel free to read my expanded rationale for how this service fits into the Rails ecosystem.

My new love: Codebase-HQ

I have discovered a great new project management service called Codebasehq.com, which merges Github and Lighthouse with a pleasant and seamless integration.  It answers a lot of my wishes for the features that are needed when dealing with a “whole-project” lifecycle.  I’ll talk about the basic features and subareas of the site, but first I want to highlight what truly makes Codebase a game-changer:

Project management

When you create a project in Codebase, you are given access to a number of critical features that disjointed services like Github and Lighthouse just can’t offer.  To start with, the unified user management is terrific.  If needed, you can get into some very decent ACLs to custom each user’s interaction.  If that’s not important to you, then the standard user setup is fine and the ACLs won’t get in your way.  Each project can also have several repositories, which is great for for when you have separate repos for Rails, design files, data files, etc.  Having your wiki, repos, tickets, and more aggregated into a single unified structure is awesome and really transforms how everyone understands the state of a project.

The individual featuresets that go into creating such great project management are:

Git hosting

Git hosting on Codebase is nearly identical to Github.  And that’s a very good thing.  Repo browsing, navigation, etc is pretty much the same (if not better) and you get the important extras like deploy keys and whatnot.  You can also set a repo to constantly sync/mirror another.  It is missing some of the more flashy Github features, like the forking integration and network/commit graphs, but overall is quite polished and very capable.

Ticketing

Ticketing is similar to Lighthouse in its simplicity.  In some areas its a little less feature rich:  email integration is not so great right now (should get better) and you can’t customize your statuses for tickets.  In other areas it’s vastly superior, especially in the general gestalt of the interface which is much easier to use and navigate.  It is still missing the one feature that I wish all ticketing systems had: a public support interface where non-users can submit and track tickets.  But since no one else offers it either, I can’t complain much. (Tender is getting close, but they’re taking their sweet time and so far I’m not a fan).

Milestones

Pretty much the same as any other milestone system.  It has one KILLER feature though: milestone/branch integration.  If you start off a git branch for your milestone, you can tie them together so you can track commits specific to that milestone. It’s incredible and stands as a great example to what can be done with true integration.

Wiki

Github offers a wiki, but it’s a pain to edit and I generally don’t bother with it.  Especially because for an ‘enterprise’ project, many of the people who need to read the project’s wiki don’t even have github access (or even know what github is).  Codebase’s wiki is more traditional in its setup and works great.

Time tracking

A very nice addition that is important on many many projects.  You might not always need it, but if you do, you’ll love it.  It is simple, straightforward, and works well.  I’d like to see it integrate even more closely with tickets, but otherwise a small to medium sized team will have little to complain about.

Summary

Although both are very similar and center around git repositories, Codebase and Github do not fill the same role.  For your open-source projects, Ruby gems, and such: stick with Github.  The forking/networking features and wide community acceptance make for an unbeatable setup.  But when it’s time to get a whole dev team together on a project, Codebase definitely has the upper hand.  They have a great set of features (and are coming out with more all the time) and the overall integration and polish makes it a joy to work with.  Their pricing is also incredibly good (even if it is in Euros), so please check them out.

Big changes in Rails coming

I prefer posting brand new content and usually avoid “re-propagating” news, even in the Rails space. However, the major happenings of the last two weeks in the Rails community have prompted me to speak up. It all started with some rather unpleasant video exchanges between Rails-ists and Merb-ists about their differing theories in framework development. Out of that came a big discussion about where Rails and Merb were headed and before anyone knew it, The Merge was announced.

At first I was amazed.  I’ve always loved having both Rails AND Merb.  I felt that having two different philosophies side-by-side allowed the community to try out new ideas in a very small and agile way (in Merb), and the ones that were really good surely found their way to Rails shortly after.  Let’s face it, Rails is growing up fast and doesn’t have the luxury of breaking backward compatibility like it used to and it’s no longer able to explore the “what-ifs” like before.  But we still had that ability in Merb.  So at first I was a bit concerned.  But I still think this is a great new direction to take.  And even with the reduced flexibility, I’m really excited to have all this awesome brainpower concentrated in one place.  I can’t wait to see (and use) the end result.

Today an addendum was added to this new revolution.  It’s been announced that there will be a Rails Activist group.  The Rails Activists will champion Rails (and of course Merb) across the universe.  They will act as a voice for the community both to the Rails Core team and to the world-at-large.  There’s a UserVoice site, a Google group, and more for helping the community express their opinions.  This is a truly awesome idea and I’m even more excited to see what these guys can do for Rails.

Bad Ruby/Rails interviews

A few of my colleagues from a previous project were recently laid off.

I won’t go into the absurdity of the fact that their company was bought out and the new owners decided they didn’t like the project the developers were working on and showed them the door the very next day with no severance (seriously, who ever negotiated a buyout where the employees are left completely unprotected from the new owners?!).

What I’d like to talk about is what makes a good interview.  One of the guys recently went to an interview for a Rails job with a big-name magazine.  Here is the account of what happened (as told to me).

First, no Rails specific questions were asked, they were all Ruby questions.  That’s a little off-putting, but not a big surprise considering the unique boat that Rails is in.  The nature of the questions, however, were equally confusing.  Here’s an example:

Make a method that reverses a string…without using reverse

Okay, I’d say that this is a relatively easy question, and is probably something that a prospective candidate should know.  But this question still has some major problems when being used to gauge a developer.  Here are my issues with it:

  1. Arrays, Enumerables, and Strings have huge numbers of helper methods.  I would want any Ruby developer to instantly to be able to tell me uses for map, inject, etc, but each_char is not high on the list.  I can’t even recall the last time I used each_char.  Should an interview really hinge on such a random request?
  2. There is no good way to write a replacement for reverse, so why are you asking someone to do so? This question was obviously asked by someone with a C/Java background and formal CS training (don’t yell at me yet…I’m a former C programmer with formal CS training too!).  The point is that this question clearly shows a mindset that doesn’t truly apply to Ruby.  If this were C, there would be an in-depth discussion here about manipulating the char array and the most efficient way to do so.  In Ruby, you don’t really have a choice without access to the underlying representation of the string.  You have to iterate through the object and then stuff it into a new string.
  3. Ruby 1.8 has weird string handling.  If I couldn’t immediately remember each_char, my next approach would be to just iterate backwards with a more normal C style loop.  However, as we know, doing some_string[i] returns an integer, not a single char string.  Great.  Now I have to remember how to convert that int back into a string (ah yes! chr!)
  4. Ruby 1.9 changes the whole game.  String iterators are changing.  As I recall from Matz’s keynote last year at RubyConf, there are so many ways to iterate through a string (char, word, line, etc) that a new paradigm is needed.  How much do you want to bet the interviewer didn’t even know that things were changing?
  5. This question doesn’t tell you anything about the person’s knowledge of Ruby.  Get into some procs, send, mixins, alias chains, or some meta programming if you actually want to test for an understanding of the language.

Unfortunately, our story doesn’t have a pleasant ending.  Not only were his chances at the job ruined because of this one question, it turns out that this is the process for all applicants.  They are progressively given a series of questions throughout the interview process.   Each question has exactly one right answer (yea right!) and the applicant must get every one right in order to be offered the position.  They stopped the interview right there and showed him the door, no chance for any other questions.  I think they’ll be looking for quite awhile.

Rails exception monitoring

There has been an explosion of late on new ways to deal with tracking exceptions thrown in a production Rails app.  It used to be that you put in exception_notifier and went about your business.  But not too long ago I decided that I needed more.  Two things started this push:

  1. There was more than just me working on a project.  Several people needed easy access to the exceptions (at times) but I didn’t want to clutter their inboxes with e-mails for every exception.
  2. I would not be working on the project forever and others would be handling long term maintenance.  This meant changing the e-mail addresses in the config file quite often and just felt like ‘the wrong way’ to be doing it.

So I started looking for other alternatives.  Here’s an overview of the three I found:

Exception Logger

Let’s start with the positive.  Exception logger is exactly what I was looking for.  It provides the functionality I was looking for and makes tracking exceptions with multiple users easy.  Unfortunately, the install procedure is a mess.  The standard plugin just would not work for me.  Rails is notoriously bad at handling controllers/models/views that are inside a plugin so exception logger really needs to be built on Engines for that kind of functionality.  Instead, it uses a number of hacks to try and get Rails to recognize the code in the plugin and it just doesn’t work well.  Plus, in order to get added functionality (such as authentication) into the controller, you’re supposed to use a config file!  It’s a crazy unclean mess.  So in order to get it operating, I was forced to simply take the controller, views, etc and put them into the normal app tree.  This took a lot of setup and debugging, but I was able to get it to running and now it works extremely well.  I have it in one high-use production app and I’m very happy but I don’t expect to use it much more.  You lose the ability to e-mail notifications (at least not without hacking some more of the plugin) as well, so it’s only good in a few select cases.

Exceptional

Now we start getting to the fun stuff.  Several new exception monitoring applications have sprung up recently and I decided to check them out.  Exceptional is a hosted service and although I’d really prefer to have my exceptions tracked locally on a per-app basis, having them aggregated does have its benefits.  It’s totally free, so sign up for a username, then create a new app profile to get an api key.  Install their plugin and paste in the key and you’re set.  It runs in production mode and sends the exceptions off to their logging service.  It integrates with lighthouse, campfire, and twitter (none of which I use, but I’m sure it helps others) and will also e-mail you the notifications.

Once again, though, some issues made it unusable.  It appeared to work great, but as I started doing some system administration, I started running into a number of problems.  Whenever the plugin loads (when in production mode) it dumps a set of debug messages to stdout.  Every time time I’d load a production console (for working on a few issues that could only be tested on the production/staging server) I’d get messages about its attempt to connect.  Then a few seconds later, interrupting whatever I started, there would be more messages about the successful connection.  Unfortunately it does this when running rake tasks as well, so all my cron jobs that use rake tasks were now littered with these messages.  I was prepared to just edit the plugin to stop these messages, but instead I came across our next entry.

Hoptoad

Hoptoad is very simple, free, and nearly identical to Exceptional.  Sign up (you get your own subdomain), add an app profile, install the plugin, and copy the api key.  It doesn’t have the extra integrations of exceptional (though I’ll bet they’re coming), but it does everything I need and without the annoying messages.  It also lets you give extra users access to errors from certain apps only (as does Exceptional, although it wasn’t immediately obvious whereas Exceptional appears to be one username only).  One thing I will really miss from Exceptional is that it tracked 404 errors as well.  Although 404s usually come from scan bots, some may very well be legitimate broken links on or to your site, so it’s nice to track them.

Summary

Overall, I would highly recommend Exceptional and Hoptoad for everyone for all their exception tracking from now on.  Which one you use just comes down to a matter of taste right now and maybe your specific requirements.  I’m very excited to see what further features they add to differentiate themselves and I am REALLY hoping to see integration with scoutapp somehow, as I think exception handling and performance monitoring go hand-in-hand.

Finally, please feel free to comment on your own experiences with any of these projects!

Presenting Overloadr: The Great Rails Host Shootout

or: How I learned to stop worrying and love scaling Rails.

There has been an explosion in Rails hosting providers: Boxcar, Mosso, MorphLabs, EC2, Heroku, etc. To make matters even more difficult for the typical Rails developer, many of these new providers have very unique properties: scaling/clustering/cloud computing/and more. It’s no longer as simple as Shared/VPS/Dedicated.

Therefore, I have started a new pet project: Overloadr. The goal of Overloadr is to test two things:

  1. The ease of deploying a basic Rails app to a hosting provider
  2. The relative performance of ‘specialized’ hosting platforms.

To start this off, I’ve written a Rails app that can be used with ab (Apache Benchmark) to test the performance of the Rails engine and the database behind it. The project is available on GitHub, as is fashionable… Feel free to fork, contribute, etc. Eventually I may set up bug tracking, discussion groups, etc. but for now, this is what I’ve got:

http://github.com/isotopetech/overloadr

My list of prospective hosts so far is:

VPS

  • Boxcar
  • Slicehost
  • RailsMachine

Cloud

  • MediaTemple
  • MorphLabs
  • Mosso
  • Heroku

Shared

  • Phusion Mod_rails (Dreamhost?)

EC2 (Testing ease of deployment with different gems)

  • PoolParty
  • ec2onrails
  • rubyworks-ec2
  • Rubber

A Dedicated server

You can expect the first of the posts of the deployment experience and performance shortly.

Comparison of Rails monitoring apps: FiveRuns vs NewRelic RPM vs Scout App

Monitoring your production Rails application is a very important part of deploying and operating a web app. There are several more general solutions that work very well: Nagios, Munin, etc. As of late, however, several Rails specific options have come into common use. I’d like to discuss the three big players here:

FiveRuns RM-Manage

The FiveRuns client has been out for about a year and offers a terrific suite of monitoring: Server load/memory, MySQL queries, Rails errors, etc. As of version 2.0 (which is in open beta is and will be released for customers during RailsConf) it also supports monitoring your mongrels. It works great, but can get a pretty expensive ( They don’t publish their prices, but I’m paying $30/server ). It is an good choice for most users.

NewRelic RPM

I’ve been beta testing the NewRelic RPM service for the past few months. It’s a decent service, it’s very easy to install, but is very limited. It will monitor your server load/memory, slow queries, etc as will all the other monitoring tools. But beyond that it doesn’t offer much. You are limited to graphing only a 24 hour period of data, so you can’t see any kind of long term trends.  They have an amazing backend system for collecting data and their site is the fastest and most responsive I’ve ever seen.  Once they get their UI front-end featureset to match their amazing data collection system, they’re going to be awesome.  As of today, they opened to the general public and released their pricing. It is based on the number of mongrel/thin instances no matter how many servers (at least that’s my understanding). For a small to medium app running up to 40 mongrels (which would probably be 2-4 servers), you’ll wind up paying $250 / month, as compared to $60-120 for FiveRuns. Overall, given the limited functionality and hefty price, I can’t yet recommend NewRelic. I hope to see it grow and quickly add more features to change my mind.

Scout App

The third option is Scout App. It offers the same suite of monitoring features as the others, but goes a step further by offering a huge range of additional plugins that will allow you to customize its functionality and easily set up extra functionality such as restart dying mongrels. You can also write your own plugins. To add another scoop on this already monstrous sundae, Scout App is the cheapest of all. It will run you only $29.00 / month for four servers.

If I had to pick one service to recommend, it would likely be Scout. They provide just about everything you can ask for out of a monitoring app at the lowest price point. If anybody else has experience with these services, please add your own comments!

How to scale Rails

So we established previously that Rails really can scale!  Now it’s time to look at how to do it.  I’ll list here the basic steps and then follow up with a post on each of these:

  1. Of course, Memcached helps a ton!
  2. Start with a master database and add two or three slave databases.  Split your reads from your writes. Rails has a great plugin for this that makes it about as easy as it can be (certainly easier than PHP). This will move you to being able to handle 15 or so app servers.
  3. Denormalize some of your data to reduce query intensity.
  4. Shard your data. Put statistics/usage data on a totally separate set of DB boxes from your user’s data.  If you really are the next MySpace,  check out AsterData for offloading some of that work.
  5. Switch to a commercial DB setup. If you’re still desperate to expand, then a more robust DB is probably in order. The great part: Rails is database agnostic (at least far more so than PHP or ASP) and will happily switch without too much effort.