CI server using Firefox

If you are like me, you always setup a CI to ensure things that you already finished are not broken by things you are currently doing.  And if you are like me, you use capybara, selenium-webdriver, and firefox to actually test the results in a browser.  The rub comes when the CI server (running headless) needs to run firefox, which needs a screen.

Virtual Frame Buffer (xvfb) is a great package.  All it does is provide an X11 interface in memory instead of requiring a screen.  Below I detail what I do on Ubuntu, but it should work the same on any Linux distro.

Installing Firefox

Testing Firefox
Once firefox is installed it can be tested on a X11-forwarded ssh connection.  X11-forwarding has to be enabled on the server and client and can be tested by checking DISPLAY.

Installing Xvfb

Testing Firefox again
The easiest way to test is to use image magick to take a screen shot of the VFB and save it to an image that can be opened in Firefox.  If the image opens and it looks like a firefox browser is running on the google page then you did it correctly.

Create a xvfb startup script
On ubuntu no start script is provided, but a simple one put in /etc/init.d/xvfb is enough.

Getting the CI server to use the VFB
At this point getting a CI server to use Xvfb will depend on how it is started.  If you use the Xvfb startup script then the display is :1.  All you have to do is export DISPLAY=:1 before the CI server starts.

Ruby module instance and class methods

A very common idiom especially when code starts to become complicated is to put functions in modules and mix in that behavior to several classes.  And a common expansion is to have a single "include" add both instance and class methods.

Here is how to do it:

"include" is used to add a module's instance methods as instance methods to the including class, but it doesn't traverse sub modules.  "extend" is used to add a module's instance methods as class methods to a class.

When a module is "included" it can register a callback which is passed the class that is doing the including.  And calling "extend" on the base and passing a submodule will cause it to put those methods at the class level.  This causes a chain reaction of loading both instance and class methods using a single "include".

Pro Tip: It is possible to cause the same behavior using "extend" by using the "extended" callback; however, this is not common and "include 'x'" reads better then "extend 'y'".

The 3N Rule for deployment

As a programmer I am lazy, but in a productive way. It is not that I don't do anything. It is that I only do things a set number of times. I have a simple rule, which I call the "3N rule", which states "I will doing something once, script it the second time, and use the script from then on". Therefore, if I have 3 to N devices to manage then I have a script that takes minimal input and replicates the change on all devices.

I also have a 3N rule for programming, but that is a matter for another time.

By using a script I eliminate a lot of the human error and also increase my productivity because computers are, now days, faster at responding to stimuli then I am. But computers are "swift idiots" so for this to work the environment has got to be controlled. I recommend doing the following:

1. Use VMs
2. Run headless, enable SSH
3. Use ClusterSSH or csshX for learning what to script
4. Use Capistrano or Fabric for automating change

By using a VM and running headless you remove the temptation to sit at a physical terminal.  It also eliminates the fear associated with screwing something up as it is trivial to snapshot a machine and restore its state.

By using SSH and ClusterSSH you enable yourself to start thinking about the cluster as a whole, and more importantly start acting on the cluster as a whole.  If some of the servers get out of sync then it makes management more difficult, so by using ClusterSSH you eliminate that problem.

By using Capistrano, or its like, then you eliminate the human error entirely.  This is just good policy.  Also, a benefit of a deployment script is that it is self documenting (assuming you know how to read).

ExtJS 3 Views and Controllers

In a previous post I wrote about my impressions of Ext4.  The overall impression were that the ideas were sound, but the implementation was still buggy.  So instead of waiting (which is not possible in a business sense) I have back-ported the ideas to Ext3.  I have not yet figured out how to get stores and models working well, but I have gotten views and controllers works very well.

Layout
Views basically work as is, because that is how ExtJS was designed.  Any subclass of a UI widget will work as a view.  However, to make the concept of a view work well they should only be instantiated once and have their modification logic in a controller.  For this reason I recommend using the following directory layout:

• app.js
• ext/
• app/
• controllers/
• Controller.js
• ContentController.js
• lib/
• view/
• layout/
• Content.js
• Viewport.js
• OtherView.js

app.js
This should just a Ext.onRead function that instantiates the Viewport.js class.  It should also setup any globals that will be used throughout the app and should be included last.

lib/
Since views should only be instantiated once this is the place to put reusable code.  Items here will not usually have an xtype because they will be extended by view items.

Controller.js
This is the base class for all controllers.  It has the helper to bind to views and look them up afterwards.  A small override to Ext.ComponentMgr is required to make everything work.



Content.js
This needs to be a simple card layout so that you can add new items to it.


ContentController.js
This class binds to a something that would change the content panel, like a TreePanel, and then using Ext.ComponentMgr creates a card, adds it to the content, and shows it.

Minimum Rcov

Uncle Bob, has stated that all code should be 100% covered, but 100% is an asymptote.  Therefore he expects something less then 100%.  This might be true in the Java, or C version of Rcov, but not for Ruby Rcov.  Also, I personally think that 100% is a bit low.

All 100% coverage means is that at least one test touches every line.  And in truly simple code that is usually enough.  But I work on systems, not simple code.  Rarely do I write single use code.  In most cases I write code that does one thing well, but is used by a lot of different part of the system.

Errors are very rarely in core modules, but rather in the runtime combination of core modules.  This is covered by integration testing which is not usually pulled into rcov.  I think this is a mistake.

The following is how I think about a minimum level of Rcov:

1. Exclude libraries from rcov metrics. That is unless you decide to include their unit testing within your own.  Either way this will give you a more accurate reading of coverage.  Seeing rcov at 90%, but thinking it is 100% because rcov can't see libraries isn't a valid assumption, IMHO.
2. Include integration testing in rcov.  Rcov has an option to aggregate several runs into a final number.  You should include both unit and integration testing in the final number.
3. Expect greater then 100% coverage.  Actually, including edge cases, positive, and negative testing the number will be closer to 1000% percent covered.  You read that right, every line of code should be covered by at least 10 tests.
4. Include the rcov threshold test in CI.  You should, before committing, always run all unit tests, integration tests, and rcov threshold.  At first set the threshold low (about 80%) and move it to 100% ASAP.  And by including threshold test in CI then the code can't go out the door without the correct amount of coverage.
5. Review tests first and last.  If your teams doesn't already do code reviews, it should.  If it does then start with the tests and then review the code.  This lets you ensure the obvious edge cases are covered without being swayed by the code.  After reviewing the code re-review the tests to ensure there are no new edge cases need to be covered.

Ext4 Impressions

I have been using Ext for years, starting with 2.0.  In most cases this is paired with Ruby on Rails on the server and has generally been nice.  There were some growing pains in understanding that Ext is not like jQuery.  It isn't meant to enhance the page it is meant to control it.  Once it is understood that ExtJS is an event driven UI layer much like MFC (Microsoft) then things are rather easy to grasp.

In fact, Ext (pre 4) is almost just like MFC in that it is primarily a collection of UI widgets that you can put together in any way you wish.  This is great for very small UIs that do not have a lot of interconnected parts, but once the number of interconnected item reaches more then a few (say 5 or 6) then it is nearly impossible to map all the emergent properties mentally.  Enter MVC.

MVC is a design patter created over 30 years ago, whose sole purpose is the separation of concerns.  Business logic (model) is separated from UI (view) and they communicate via a defined set of actions (controller).  Anyone familiar with UI programming should be familiar with this pattern.  It is used in almost all major UI libraries except ExtJS, until now.

Ext4 aims to fix this flaw by retooling the framework to be MVC.  It is almost successful.  There is a major problem in separating models from collections (which is understandable given Javascript's nature) but there are more elegant solutions then the one Ext4 uses (see SproutCore and Backbone).  There are far too many annoyances at the model level for me to detail here, but as an example the LocalStore and MemoryStore are broken as of this writing and cannot update or remove models since they do not fire events.  This problem can be seen is a number of their own grid demos.

I also tried to use the compatibility mode to convert an older Ext3 app to Ext4.  I was successful in removing all of the errors and most of the warning, but could not get the page to render.  Each component thought it was rendered, but no DOM was presented to the browser.  Lots of digging later and it turns out you can't mix MVC Ext4 objects with non-MVC Ext3 like objects.  Not sure why, and without errors it is hard to tell what went wrong.

Finally I started from scratch and created extjs4-frame so that I could dig in without having to worry about Ext3 compatibility screwing things up.  Thing did not go much better.  I watched the videos and then dove right in.  As expected what was shown on the videos (obviously dumbed down) worked as expected, but as I started to add complexity things would break in mysterious ways.  Turning to the documentation was only marginally helpful, since it appears that it has not been full updated.  Also, some of the documentation does not match the JDoc comments in the source (probably human error in generating the docs).  This is an annoyance to be sure, but not a show stopper since I know how to read Javascript (as anyone that would be using ExtJS should).  However, documentation error (especially when generated from embedded comments) are indicative of a lack of attention to detail and since they do not provide their unit tests I can only guess that their storage engines are not well tested either.

Pros

• Documentation videos are a nice touch
• Simplified code due to the MVC nature
• Sensible file structure (forced)
• Auto-loader (turn off in production, but great for development)
• Better stack and error handling (as compared with Ext 2/3)
• Trees use real models (meaning the server can respond with any hierarchical objects)
• Non-flash based charts
• Smaller size (as compared to Ext 2/3)

Cons

• Difficult to understand (even for those with MVC and ExtJS experience)
• No backwards compatibility (the compatibility mode is a joke)
• 4.0.X versions are not guaranteed compatible (there are still a lot of internal changes going on)
• Documentation does not always match source comments and is not always correct (save some time and just start with the code)
• Inconsistant Model/Container behavior
• No generator scripts provided for aiding design patterns
• No unit testing framework provided
• No JS compiler/minifier provided (one that can read the requires directive is almost a must now that they exist)

Overall

Wait for 4.1 to see if things stabilize.  Or switch to SproutCore.

Cucumber with using Rspec should_receive

I recently came across a case where I wanted to use a should_receive test in cucumber. The basic idea is that I was attempting to prevent Apache library code from calling out to the system, while simultaneously testing that the correct function was actually called. In Rspec (which backs cucumber in my setup) it is simply:



At the end of the test if restart has not been called then the entire test fails. Because of the way that cucumber works with rspec 1.3 (rails 2.x, which is required by the project) registering a should_receive has no effect (seems this is fixed in the cucumber/rspec versions for rails 3). Even if it were to work it would mean putting a Then before the When or putting a test in a Given. To me that just seems wrong.



What I had to do to simulate a should_receive but still maintain the flow of the scenario was break it up so that I stub the function and when that function is called I set a flag. Later I test that the flag is set.



The step that sets the flag is Apache can restart and the one that tests is Apache should restart. Here are the step definitions:

Heroku

At first it was not clear to me that Heroku was free for very small apps, but it is.  There are also a few restrictions that Heroku has:

1. The project has to use Git
2. The project has to be a Rails App (or have a config.ru)
3. The project has to be in the repository root (good practice)
4. The project has to list all the dependent gems (good practice)

If you meet all these requirements it is as simple as installing the heroku gem, creating a new app, and pushing the to heroku.

Setup a heroku ENV

$ mkdir heroku

$ cd heroku

$ rvm use 1.9.2@heroku --create

$ gem install heroku

Pushing to Heroku

$ git clone git@github.com:jkamenik/Laser-Landmarks-Prototype.git

$ cd Laser-Landmarks-Prototype

$ heroku add laser-landmarks

$ git push heroku master

Continuous Integration with CIJoe

CIJoe lacks a great many things that I usually expect in a CI server.  But it is now unclear if they are really all that needed, or just added by CI marketing.  Here is now I usually get CIJoe working with the least Effort.

The abridged version:

# rvm
$ bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )
$ rvm install 1.8.7-p330
$ rvm use 1.8.7-p330@foo --create
$ gem install bundler
$ gem install cijoe
$ git clone git://github.com/XXX/foo.git
$ nohup cijoe -p 4000 foo &
$ git config --add cijoe.runner "bundle install 2>&1 && rake ci"

  1. Install RVM
Docs on installing RVM are here.  I use this to isolate my version of Ruby and Gems from other CIJoe servers that I will setup on the same machine.  This reduces the headaches later

1.1. Install Ruby
With RVM installing new versions of ruby is easy with RVM.  For example I use ruby 1.8.7-p330 on one of my projects so I run rvm install 1.8.7-p330.

1.2. Create a Gemset
For every CI Server I run I use a different gem set.  Yes it uses more disk space but it saves so many headaches if two repos need different versions of different gems it is worth it.  You can run it with the "rvm use" command.  Lets say my project is called "FOO" I would run rvm use 1.8.7-p330@foo --create to switch to ruby 1.8.7-p330 using gemset foo and create the gemset if it doesn't already exist.

1.3. Install Bundler
Bundler is a manager for RubyGems that aids in getting the correct version of the correct gems installed.  I install it so that before every make I run "bundle install" to make sure that the CI server has all the gems it needs for the test to pass.  And if it fails because of a gem problem then it means the correct gems were not noted in the GemFile and therefore it will break in production.

2. Install CIJoe
CIJoe is a gem and it should be installed in the same gemset that the repo will be run out of.  So if the code to test is Foo then run rvm use 1.8.7-p330@foo --create; gem install cijoe.

2.1. Clone your repo
git clone git://github.com/XXX/foo.git or something like that

2.2. Run CIJoe
I run it with nohup and & so it is in the background and always running.  I also give it a known port so that I blow a hole in the firewall for it (see below for why).

nohup cijoe -p 4000 foo &

3. Configure your repo
This is both a strength and weakness of CIJoe: all the configuration for what to run and how is stored in the Git repo.  The strength is that it is stored with the repo, the weakness is that .git/config and .git/hooks are not pushed to remote repos so if you need to start over you will need to recreate that config from scratch.

3.1. Add the runner command
CIJoe only allows running of a single command but it executes it in "sh" so you can use "&&" to chain them.  However, to make things easier on myself I always create a rake task called "ci" and execute that. Also, to make sure that all the gems are correctly installed I run bundler first.

Note: When chaining commands via && make sure you redirect stderr to stdout since CIJoe only reads stdout, otherwise you will get nothing in the output file.  The last command will already have stderr redirected so do not do it twice.

git config --add cijoe.runner "bundle install 2>&1 && rake ci"

3.2. Adding Notification
CIJoe just executes .git/hook/build-failed on a failure and .git/hook/build-worked on a success.  The examples show how to setup these up.

4. Auto build
Any HTTP PUSH to CIJoe will cause a build to be kicked off.  This is useful for auto-kicking a build when a new commit is pushed.

4.1. Blow a hole in the Firewall
This is so that github can get into your build server, since I assume you have your build server behind a firewall.  Your config will depend on Firewall/Router.

4.2. Configure Github
Just add a Post-Receive hook via the github admin screen.

BDD and TDD

Apparently I have been doing TDD wrong for years (who knew?).  There are plenty of resources out there to help, but here are the ones I found useful in the order of most to least useful.

1. BDD - TDD done well?  Basically explained away my misconceptions about BDD.  I thought it a replacement to TDD (unit testing) and it is not.
2. Roman Numeral Kata in Ruby.  I had thought the point of testing was to test working code and never really bridged to understand how good code comes from bad code that is tested.  The video shows by adding tests that define the "correct" code 1 at a time and coding only to the current tests that elegant solutions are likely.
3. TDD Problem.  It is nice to read articles and watch movies, but until you have actually attempted to solve a real life TDD problem things are not going to make sense.  This site is full of problems.
4. Gherkin.  This is the language used by Cucumber to define BDD.
5. How I Learned To Love Testing.  Demo of by testing up a test env things are easier.
6. How to adopt TDD and ensure adherence?  The question posed is strikingly similar to oppositions that I have heard in the past; mostly stemming from misconceptions about testing.  The selected answer is well formed, and is the main reason to read the question.
7. Ruby Koans.  Helps people learn ruby by presenting tests that all fail and requiring the user to code until they pass.  The tests are very basic, but it is a good foray into TDD
8. Cucumber Tutorials and Related Blog Posts.  Just a list of other resources.

Sinatra Proxy

Anyone that has developed for the web has come accross the Same Origin Policy problem.  Basically it says that for an AJAX call to be valid it must come from the same host as the page, using the same port, subdomain, and protocol.  I find there are many cases where I need to gather information from remote hosts, which leaves me two choices: proxy the request through my server, or use an iframe.

In general since it is JSON/XML data I am interested in then an iframe will not work so I have to create a proxy.  Below is what I do using Sinatra.

Required Gems

1. Sinatra
2. Net::HTTP
3. CGI
4. URI

Directory Structure

This is just the default that I use for all sinatra apps because it makes things easier.  Sinatra will automatically serve files in the public directory if a file by that name exists

• index.html - static file that includes all other files
• public - place where other static files will go
• javascript
• stylesheets
• images
• server.rb - the sinatra app

Server.rb

/remote/*
The * tells sinatra to blindly take everything up to the "?" and put it in params[:splat].  I then split out the first item for the server name and port.  You could also split out username/password if you want, but I rarely use HTTP Basic Auth so I never do.

I delete the key 'splat' and anything I do not want to forward so that they are not URL encoded to the remote.

urlencode
A simple function that re-encodes params that were placed in the params hash.  Sinatra does a good job of taking params that look like color[]=red&color[]=blue and turning them into {"color" => ["red","blue"]}.  Therefore I check the value's class for Array and if so I add the "[]" to the param name.

Proxy
The proxy code is a call to Net::HTTP returning the response body.  I could also have returned the headers, but in this case I do not care.



In Action

/remote/medic/this	->	/medic:80/this
/remote/medic:4567/this	->	/medic:4567/this
/remote/medic:4567/this?_dc=1	->	/medic:4567/this
/remote/medic/this?foo=bar	->	/medic:80/this?foo=bar
/remote/medic/this?foo[]=bar&foo[]=baz	->	/medic:80/this?foo[]=bar&foo[]=baz

RVM

It looks like I am the last hold out for the use of RVM.  What prompted my move?  Well Macs come with ruby 1.8.7, but I always use ruby 1.9 in my projects.  I replaced the standard system ruby with my own 1.9 version.  The problem came when starting a new job since they used ruby 1.8.7 (a patch level greater then supplied my Apple), and JRuby.

I spend a few hours just trying to get JRuby to play nice with the 1.9 version that I had installed.  Things where not going well.  The problems were almost entirely related to path-ing, but I am ruby programer not a system administrator.  I was almost ready to give up when I remembered a lightning talk given at the B'more on Rails group on RVM.

In case you, like me, have been hiding under a rock for the last few years RVM is a Ruby Version Manager.  In a nutshell it lets you install any or all versions of ruby in their own sandbox (you would be insane to install "all", but you can!).  It solves all the problems associated with having different ruby versions installed and allows you to easily switch ruby versions with rvm use XXX where XXX is the version of ruby of you want.

More then just being able to switch ruby versions; you are able to create arbitrary sandboxes for gems called GemSets.  This is most useful for me in trying to figure out what gems are used in what projects.  In most of my projects now I have a .rvmrc files containing the version of ruby and the GemSet to use for that project: echo "rvm use XXX@YYY --create" > .rvmrc, where XXX is the ruby version and YYY is the GemSet to use.  --create just creates a GemSet if it doesn't exist.

Your results may vary, but for me all I had to do was:

1. Follow the quick install: bash < <( curl http://rvm.beginrescueend.com/releases/rvm-install-head )
2. List the rubies I could install: rvm list known
3. Install a few rubies: rvm install XXX
4. Switch to a ruby and test it: rvm use XXX
5. Setup and a .rvmrc file: echo "rvm use XXX@YYY --create" > .rvmrc

I have not yet played around enough to see if I could use a GemSet to create a Bundler package.  It seems that they are made to play nice with each other, so I foresee no reason it will not be as straight forward as this.

Problems with Agile Implementation

I really like agile programming.  It keeps me close to the action, and makes me have to think about my next moves.  It also keeps me informed as to what is going on around me.  But in my many years of using agile I realize that, though the process itself is very nice, its implementations can tend not to be.

Problem don't arise from agile itself, but who and how it was implemented.  If the implementer's goals do not match the Agile Manifesto there is little chance of success.  I have used scrum many times and the most common problems I see are:

1. Agile as Micromanagement
2. Agile as a Whip
3. Agile as an Excuse

Agile as Micromanagement

It looks like:

1. Having to break task down into hourly segments of work
2. Having to break task that logically have to be done by a single person
3. Having to account for ALL time taken, even time not related to code like attending meetings

Agile tenet misused:

1. Individuals and interactions over process and tools
2. Working software over comprehensive documentation

This happens when a manager (chicken role) is the Scrum Master or when the Product Owner has say over implementation specifics.  It is a confusion of roles, which in turn leads to a confusion of goals, which in turn leads to over documentation.

Agile as a Whip

It looks like:

1. Filtering a burn-down on an individual basis
2. Placing more in the sprint then can be done (but still requiring it all to be done)
3. Associating points with people (publicly)
4. Associating number of tasks done with effort
5. Associating points with hours
6. Basically anything where measured output is more important then people

Agile tenet misused:

1. Individuals and interactions over process and tools

Anytime you associate numbers with people you have created a crab mentality.  Their focus will stop being on software, but on making their numbers better.  Those that are better at number games will succeed, those that are better at software will fail.

Anytime you put your people under undo pressure then simple mistakes are made.  This is going to later erode confidence in the team.  It is going to happen like this: "you missed a comma in a Javascript file which causes it not to work in IE.  That was such a simple mistake to have tested for that I am not sure you are testing any of your code."  The problem was caused by 4 hours of sleep in 72 hours of coding at the end of an over-extended sprint.  The programmer was nearly delirious.  It is shocking it was the only mistake, not that it was a simple mistake!

Unfortunately I have seen this situation start innocent enough, with comments like "we don't want to over work the staff" or "we want to make sure they always have something to do" or "we want them working on the correct things".  If the "we" in question is management (chicken role) then there is probably already micromanagement going on, and Agile is being used as whip to solve the problems created by the bad implementation.

Agile as an Excuse.

It looks like:

1. "You said it would take XXX.  It took YYY.  You need to make up the difference out of your own time"
2. "We cannot slip these date, and you have already pared back the release N sprints ago.  You need to put in extra effort"
3. "Agile is about being agile.  Even though we are mid sprint we are radically changing direction, but we are not canceling the sprint or doing sprint planning.  We are just swapping out some tasks for others."
4. "You picked the language.  It is now your problem to bring this project to conclusion and under budget."

Agile tenet misused:

1. Individuals and interactions over process and tools
2. Responding to change over following a plan

These are all excuses I have heard.  Each time given by a person in chicken role (managers) because they are ignoring changes in the field (military term).  Every choice has a set of outcomes: some good, some bad.  The attempt with agile is not to mitigate bad outcomes, but to allow those outcomes to contribute to the overall direction.

Sometimes the bad outcome will be that something took to long, or that one language/tool was not the correct choice given the problem set.  If for every problem that happens the developer has to take their own time, or face embarrassment, to solve the problem then they will stop making choices.  Not just choices that might have bad outcomes, but choices altogether.  At which point someone in a chicken role will start making more choices then they should.