The Rob Test… A Craftsman’s Version of the Joel Test
Almost 10 years ago I stumbled onto Joel Spolsky’s site “Joel on Software” and found a bunch of really great insight into the development process from an ex Microsoft developer with a great sense of humor. One article in particular “The Joel Test: 12 steps to better code ” I adopted immediately and added to my virtual toolbox of great ideals for the development process.
Since that time there have been a few shifts in the details of how software is built (or how I like to have it built), for instance Joel speaks about daily builds whereas I prefer the continuous build process, Joel mixes in the details of having developers write code during the interview (which I also do, but don’t think it belongs on this list per-say), and Joel speaks of Specs and I have included the Backlog, etc…
So, 9 years after Joel posted his original “Joel Test”, I thought I would share my variation of this “classic” list I have toted around with me and strived to implement in all of my development endeavors to this day. I use this list whenever I’m sizing up a client for consulting work, not necessarily to force them into “my” way of thinking, but to use as a guide for them to be sure, and to give me a clear indication of exactly how big the mess is that I’m walking into.
The Rob Test
1. Do you use source control?
2. Do you run automated builds?
3. Do you have a bug database?
4. Do you fix bugs then write new code?
5. Do you have an up to date schedule?
6. Do you have a spec?
7. Do you have a backlog?
8. Do developers have a good environment?
9. Do you use the best tools for the job?
10. Do developers write unit tests first?
11. Do you have testers?
12. Do you do hallway usability testing?
Not that far off from the original “Joel” test, but enough of a variation that I thought it would be worth mentioning and breaking down for todays developers, project managers and general geeks of IT.
As Joel points out 12 is perfect, 11 is manageable, but 10 or less and your heading into uh … interesting territory. As Joel also points out this is not the only hard and fast list to use, but I can tell you from personal experience and almost a decade of use, this list of 12 things will provide you a very good indication of what to get fixed first in your process to make the development iterations manageable and on time.
1. Do you use source control?
CVS has all but been replaced with SVN (Subversion) now, but CVS is still in use and there are plenty of alternatives to SVN from Microsoft Source Safe (not recommended to anyone), Git for the mainstream Rails community (although still not as accepted as SVN), and Mecurial (may actually become the new SVN one day). The bottom line is still the same though, if you don’t use source control (even for a single developer), you will loose working code, waste hours re-writing things, and not have a clear log of what you did and when. Everyone should use source control.
2. Do you run automated builds?
Originally this was “Can you make a build in one step?” but I prefer the idea of continuous builds. Anytime a developer commits their code to the repository there is a “build box” that sees the checkin and checks out the code and runs through the whole build process in a single step, automatically. Note that for ridiculously large or complex builds that take an excess of 20 minutes to get from end to end, you may want to step down the process to individual modules of code (only replace what was committed), or take a look at the architecture of your build process. or confine yourself to daily builds as Joel originally recommended as his #3. In 2002 Joel and I had a conversation about Final Builder as a Windows tool for automating daily builds, I was at Rogers Cable at the time and welcomed the advice, it worked great!!! (Thanks for the tip Joel, it was a great recommendation) 
.
3. Do you have a bug database?
There are plenty of clients I have been involved with, that do not track their issues formally as Joel recommends. This is usually a good indication that your walking into a shit storm, my personal preference today is Redmine for defect tracking, although when I was @ v.1 we did use Fog Creek’s FogBugz and it is a great product that I totally endorse, Redmine is “today’s” favorite for me just because it’s written in Ruby, and Free. I have also used Mantis extensively (PHP based but isn’t all that pretty), and have tried to tinker with bug genii. I would have to say for commercial trackers, go with FogBugz, for Rails development Redmine is a winner, and if your confined to a php environment and need free, look at Mantis. The bottom line, is get all of your issues into a tracker if you want to release the best code possible. There is no other way to ensure that your issues aren’t falling through the cracks, and to think there is, or to think that your code will be bug free is just plain insanity.
4. Do you fix bugs then write new code?
Hank Rainwater pointed out the managing developers is like Herding Cats, and he is absolutely right. Making sure that developers fix the bugs assigned to them before they write that cool widget can be a fun exercise in developer management. It’s best managed (IMHO) by ensuring that anytime the bug count for any developer is greater then 10, they are not allowed to commit anything except bug fixes for a day. Joel prefers a zero count and I couldn’t agree more, but the honest truth of the matter is if you have a feature that has to be implemented and 3 open but low priority issues that are obscure and near impossible to replicate, I say let the developer add the new functionality to appease the stakeholders in the project. BUT if the bug count for any developer hits 10, thats an instant “your working on bugs today” conversation during the daily scrum.
5. Do you have an up to date schedule?
Joel has a great article for Painless Software Schedules and I highly recommend it as reading to see just how simple scheduling can be (despite the don’t read this message), read it, and then read the Evidence Based Scheduling article too… That said I prefer the Agile approach to development and the use of daily scrums while running through short sprints of software development. This moves the actual management of the daily schedule out to the scrum master and only the items in the sprint are managed. The scrum meeting should never take more then 10 minutes and only needs to answer 3 questions for each developer. 1) What have you done since yesterday? (last meeting). 2) what will you have done by tomorrow? (next meeting). 3) what (if anything) is standing in your way? This puts the onus on the developer for managing their time and helps them get better at time estimates. I will have to write out a future article on my methodologies for software scheduling but I think you get the just of it.
6.Do you have a spec?
The software specification for what your developing (in it’s current iteration) is usually the one thing that most clients I deal with are missing, sure they have a spec from what they originally wanted, but tis document is almost always out of date. Joel has a 4 part series on writing software spec’s that I can’t endorse enough, and while writing a spec is a necessary step in ensuring that your developing something that has had thought put into it’s design, keeping this document up to date is the whole key to making it a successful tool. There are different types of specifications that go hand in hand with development, and they all need to kept in sync with the actual development cycles, but specifically I’m referring to the 100,000 foot view of the project. The features that are in or out, and the general design ideas that make up the project as a whole. After all, we have automated tools that generate the technical documentation of your code based on your comments, you do comment your code (or ensure your developers do) right?
7. Do you have a backlog?
There are really 2 backlogs (not covered by Joel’s methodologies) and the concept is taken from the Agile approach to software development, Put simply, all of the high level features that will go into the project make up the project backlog. Each item is spec’d out for priority and time, and then specific items from the project backlog make it into the current sprint backlog, and sprint schedule. As new features creep in (scope creep) they should only ever be added to the project backlog and then reviewed at the beginning of each sprint iteration. This ensures that developers can focus on the tasks at hand in the sprint backlog and don’t get sidetracked by management or executives for “must have” features that really don’t make the grade. Each week (or iteration) the project backlog is reviewed and a new sprint backlog is generated.
8. Do developers have a good environment?
This is updated from the “quiet working conditions” Joel suggested, while technically the same thing, no two environments are created equally, and some developers (like myself) prefer to have background noise like my coding playlist playing, some studies show that different types of music can increase or decrease productivity. So simply stating that all developer environments should be quiet doesn’t really fit with my personal views. How can you determine if the developers have a “good” environment? Ask them. If they have phones on their desk that the company uses to page everyone else in the company, which causes a distraction they should disable this on the phone and place it in do not disturb mode (Feature #85 on most of the interoffice systems I’m in). If the developers desk is in the hallway, expect them to be distracted constantly, and in turn, expect things to take longer to finish. Once a developer has lost their focus it is a very real problem to get back into the “zone” and you should work directly with your developers to get them into an environment that makes them as productive as possible.
9. Do you use the best tools for the job?
While the “best that money can buy” (Joel’s #9) sits along the same line, I know of a few places that were using expensive tools the just weren’t right for the job, but the developers were practically forced to use them anyway because the company had paid so much money for them. Management had made the case for a tool and after it had been purchased the developers discovered that it didn’t provide any value add what so ever, and using it caused more slowdowns then anything, but because management had gone out and made the case for the tool there it was. This is incredibly daft (IMHO) and developers should always be on the lookout for the best tools for any job. Having developers that are on the pulse of the development community will usually yield the right kinds of decisions about tools, and as methodologies or processes change they can update or implement new tools that will help in the process without arguments from management if the tools are free. sometimes the best tool is the most expensive, and if thats the case, cost shouldn’t be an object. Just make sure that the tool is a requirement for the task at hand and not a nice to have, or something that the developers want for the sake of having it.
10. Do developers write unit tests first?
While not 100% accurate, the point here is really do the developers write unit tests at all? I won’t go into the debate over test driven development vs. development that has unit tests added regularly, I believe that the former is possible only when all of the developers have seen it work, and/or believe it will work. But if you have a code base that has no unit test, and no integration tests, I bet your re-creating the same issues over and over again in your development cycles. Developers should always be responsible for writing unit tests to ensure that what they just coded works as expected, but here in lies part of the testing problem. Developers test differently on their own code. They usually will only code to test that the code does specifically what they intended it to do, wherever possible have a co-developer or alternate developer test the bounds of the code (this will yield best results), the same thing applies to integration testing, the developer integrating the code will only test it is limited ways. Never have the developer writing the code responsible for the end user testing, this leads me into point 11.
11. Do you have testers?
My personal rule of thumb is that for every 2 developers you need 1 dedicated tester, Joel is right on the money with this, the tester may be able to handle up to 3 developers, but once you hit 4 developers, get another tester. The best testers are usually jr. developers that want to get better at coding, they enjoy breaking things and learning how to fix things, and you can usually give them the ability to hunt for the root cause in code as long as you have a more senior developer reviewing any changes a tester or jr. developer makes. Make sure that your testers are adding to the issue tracker (bug database) and make sure that at a minimum every bug found (regardless of how simple the fix is) contains the following 3 things. 1) A clear summary of the bug. 2) A more detailed description of the bug including a) exactly what happened, and b) what was expected to happen (i.e. why is this a bug). and 3) most importantly, what are the steps to reproduce the bug. If a developer can’t re-produce the bug then it won’t get fixed. Then make sure that it gets assigned, but I prefer to let the developers the bug gets assigned to set the priority. All to often a beta tester will enter everything as a show stopper and it takes time away from fixing things to have to re-prioritize issues in the tracker.
12. Do you do hallway usability testing?
I love this one which is why I have made sure it has stayed on my list over the years. So many times I have seen someone nabbed out of the hallway and told here, use this for a second to see them totally break it and wreak havoc on the system that the developer just told me minutes earlier was “bullet proof”. How does this happen, simple. Developers only test that things work the way they do them, if the developer always uses the keyboard shortcut for a button event and the nabbed hallway tester user only uses a mouse guess what … that onRelease code that never got tested in development just got some exercise. While Joel points out you also get invaluable user interface design issues from this testing, the number of times I have seen people break something is seconds makes this a great tool .
There you have it, my updated “craftsman” version of the Joel test… A day or two ago I was thinking that this material would make a great crash corse for managers and developers alike, not to mention a chance to play with my moodle setup. I would want to get permission and blessing from Joel of corse to refer people to his material, so Joel, if your reading this give me a thumbs up .
Comments are welcome ladies and gents.
