Life is suffering.
The origin of suffering is attachment, due to ignorance.
The cessation of suffering is attainable.
The eight-fold path leads to liberation.

I was coding happily along, and realized in a flash of insight that this applied to what I was working on.

In coding, suffering comes from:

• not being comfortable making a change because you don’t quite understand how the system works
• working hard but realizing your code is still buggy
• a client being less than impressed by “a change that couldn’t break anything”, but did
• not being able to refactor because you can’t see all of the implications
• wondering if this ever really worked at all
• having that bug pop up again, although we thought it was fixed
• not delivering with quality and on time

The Four Noble Truths of Coding

Coding is suffering.
The origin of suffering is attachment, due to ignorance.
The cessation of suffering is attainable.
The path of executable specifications leads to liberation.

This is a bold statement to make. By using automated means of capturing the assumptions present in code, one breaks the painful cycle that comes from ignorance about the code base. Coding then becomes not suffering, nor not-not suffering, but just coding. Instead of coding in fear, it again becomes a creative process that encourages working with others in harmony. With automated tests of some sort, instead of having the law of cause and effect operating over the course of months, weeks, or days, feedback operates instead in terms of minutes.

However, there is a middle path to take here. If one is entirely safe, one loses the creative edge that makes the project exciting and lessens nimbleness with too much process and boilerplate. If one is too loose, one stands to let some quality slip. Experience seems to be the best guide.

This thought probably emerged from reading philosophy and Working Effectively With Legacy Code.

Original article: The Four Noble Truths of Coding

My biggest accomplishment of the year so far has been creating a new website for Ultimate (Frisbee) in the Indianapolis area. I got a basic concept going one Saturday morning, took feedback and talked to stakeholders, and then made a final version with the help of others in Indy. Notable improvements include:

• pictures from Zach Dobson, an professional photographer in Indianapolis, on the front page
• maps of all the fields that are used for pickup games, with up-to-date information about when they are played
• integration with a league manager that has online signups, payments, and Google Maps support to show where fields are
• live streaming of the 2010 Indiana Ultimate high school championships on June 20th

For comparison, you can check out the old site. I’m really happy that people helped out and offered suggestions and improvements to the site. This project is on autopilot now.

Other projects

A few months ago I created an eight-session class about developer testing that I taught to people at work. I was surprised at how much work it was, but it seemed like the participants got a lot out of it, and I learned a ton putting it together.

I recently made a simple Rails app that tracks personal events. I started this to keep track of soda consumption, but use it for many things now (showering, wake and sleep times, work transit times, standup lengths, etc.) It’s handy to be able to do this from anywhere, so I created a bare-bones Android app that can send events. I’d like to put together some neat graphs and histograms. I plan on implementing some basic parsing to override the dates and times that things were submitted (think: ‘soda 12oz at 12:30′ or ‘exercised yesterday afternoon’). Right now the app would at least be useful for tracking how many days in a row I’ve flossed or exercised.

Lately I’ve been reading feverishly. Recent books have mostly been from the Personal MBA program. I have read 22 read out of 100 and expect to get to 30 within a few months. I’ve found this undertaking to be helpful in understanding how businesses work. I supplement this reading with blogs to ensure I’m not only consuming out of date material. Reading marketing and sales books is useful because I do not believe that merely creating a superior product is enough to be successful.

Finally, I am organizing a Code Jam at my place of work on Saturday morning. Basically it’s a time for people at work to come together and hack on whatever cool ideas they have or learn about interesting things. If anything cool happens, I’ll post some pictures.

Original article: So what have I been doing the past few months?

I heard Mike Kelly talk a bit about exploratory testing, and it seems like what I have recently been trying is similar to this technique. Essentially, exploratory testing is ad-hoc testing’s bigger, more refined brother. Instead of just willy-nilly testing things, you have a designated goal and you create a plan that you try to stick to. When you see something interesting or have a clever thought on how to test that things are well done, you can venture off of the path for a bit and explore that idea. In this way, you can use your intuition as well as having a way to document and hold yourself accountable for the testing that you are doing.

Of course, this doesn’t work for all applications, but it seems like it’s useful for my case. I just realized that this is a pretty cool subject, something that gives ad-hoc testing more value and more credibility as a testing methodology. This is vague and my understanding limited, so if you want more information, check out the Wikipedia entry for exploratory testing for starters, along with James Bach’s PDF that’s there.

Image recognition

It seems like we could improve on verifying the look in all of the browsers we are tasked with using (IE6 & 7, FF 2 & 3.) It would be best if the site looks essentially the same on all browsers at the supported resolutions. At this time, however, the CSS is not very well decoupled, so what happens is that someone changes the CSS on one page, and somewhere else something is misaligned on one of the browsers or something. It’s time consuming to verify just one page on all browsers, let alone pages that you don’t think might have been modified. This leads to mistakes because we are not getting feedback quickly enough.

To mitigate these mistakes, we might employ a technique mentioned at the September IWST. You create a script that goes through the site for the supported browsers and takes screenshots of them at the supported resolutions for all of the pages on the site. You have a baseline of some sort, which is how things looked when you last got latest from the repo. Then, if you make any style changes, you run the script and look to see if anything changed. If things look good, you check in. If not, you know and fix them. By making the right choice easier, we improve the quality of our product.

Pop Quiz

So, which one of these blogs is not like the others?

Which one of these is different?

Well, it was kind of a silly quiz, you probably already knew that it was going to be IE that was going to be different. (RSS icon) Kind of reminds you of those “find the six things that are different between these pictures” things in the Sunday comics?

The point here is that it’s pretty quick and painless to go through the site and see if there are any major differences between what the site used to look like and what it will look like after you check in. It should ideally take less than ten minutes to go through the entire site if you didn’t mess anything up. This is much better than using some fully automated system, because you can disregard changes that are irrelevant (for example, the changes that you presumably made for this check-in.) I would envision the images being considerably larger so that you could see differences, but you get the idea.

It seems like this should be something someone has already solved. Is there an open-source product that you know of that does something like this? I don’t think that a solution where we allow an external service to view the pages is applicable. Perhaps a Selenium solution of some sort, but this seems somewhat expensive to create. One thing that would be very handy would be to be able to see the entire contents of the page (not just one vertical screen’s worth of information.) This combats the problem where the top half of a long page looks correct but the bottom half is messed up.

Original article: Exploratory Testing and Visual Browser Diff

We’ve posted the 2009 schedule for the Indianapolis Workshops on Software Testing. We’re looking for people willing to share experience reports and people who just have a general interest in the various topics and would like to attend and ask questions.

Here’s a summary of the lineup:

February 26, 2009: Presenting test results

March 27, 2009: Test-driven development

June 27, 2009: Test design and development

August 28, 2009: Testing mobile devices

September 26, 2009: Automation and performance logging

October 29, 2009: Time management for testers

November 21, 2009: Managing your focus while doing exploratory testing

If you’re interested in attending any of the workshops, please drop me a line at mike@michaeldkelly.com.

I sent him an email saying that I would be interested in the March and October sessions. Perhaps you have some interest in the same/other sessions?

Original article: Testing Seminars

I’ve edited liberally, so here goes…

Workshop on Regulated Software Testing

May 15, 2009

The WREST is coming to Indianapolis again in 2009. The topic for the workshop is “Beyond Scripted Testing.” You can learn more about the workshop here.

…

Major Questions of Interest

How can we better test software in regulated environments while retaining processes and documentation that will pass regulatory audits?

How can we prove the value of implementing progressive test approaches within the context of the necessities of regulation (FDA, SOX, CMM, etc.)?

Workshop on Performance and Reliability

April 16-1716-18, 2009

Through much arm twisting and whining, the WOPR 12 organizers are hosting WOPR next year here in Indianapolis. The theme for the workshop is “Resource Monitoring During Performance Testing.”

You can find more details here.

…

The ultimate goal of performance testing is identifying and fixing performance bottlenecks before production deployment. While a large percentage of bottlenecks are ultimately traced to the application code, they may also occur in servers, system software, and network components. Critical to identifying their source and diagnosing their root cause is monitoring the “bellwether metrics”. Let us define “bellwether metrics” as the “key indicators of future developments or trends”, and thus the minimum set of system vitals that we should be measuring for each hardware and software component.

Thoughts

The process for applying to share an experience report was pretty easy. Just have some relevant experience and know enough about it to share. Come up with a presentation. Kind of like a brown bag really. If either of these two are similar in size to IWST, they are enjoyable because you get to know the people at the meeting. If you’re jazzed about this kind of stuff, you should totally do it, for both the knowledge and the experience.

Original article: Indianapolis Conferences

One thing that has piqued my interest of late is theories in JUnit 4.4+.

Overview

In JUnit, the still experimental theory functionality is derived from the open-source Popper project. The name comes from Karl Popper’s work on the philosophy of science. He was a major proponent of saying that something isn’t scientific unless it is falsifiable, and that theories cannot be proven correct, merely shown to have exceptions. If someone can provide an exception, it means that the theory does not match all of our observations. Theories are models of the world, so while a model may not be correct, it can still be useful.

So what are theories in software good for? When using test-driven development, the tests serve as a form of code documentation. However, there are certain properties of code that are known or assumed by the developers that are difficult to codify as simple tests. Theories provide a simple and expressive means for at least the following cases:

For example, you might have a theory that data that is serialized into XML and then unserialized should always result in the original object. This seems to make sense, but it might be time-consuming or error-prone to specify all of the different tests that you have to consider. A single theory can codify this assumption.

In JUnit, theories are a special case of parameterized test. So instead of using the standard test that has no parameters, you add parameters to make testing various combinations of values shorter and easier. You can then take these parameterized tests further by specifying data points to test for by default in an expressive manner.

Example

Alright enough theory. What does a JUnit theory actually look like? Given that you have a theory about two methods on a Currency object that should be inverses of each other, one nice example is:

    @Theory public void multiplyIsInverseOfDivide(
            @TestedOn(ints={0, 5, 10}) int amount,
            @TestedOn(ints={1, 2, 3})  int multiplier) {
        assertThat(new Currency(amount).times(multiplier).divideBy(multiplier).getAmount(), is(amount));
    }

On the first line, we declare that we are going to postulate a theory, and give it a unique name. This method would typically live in a test suite or test file of some sort. The next two lines are the parameters to the test method. The parameters are prefixed by annotations to put in sample test values. All permutations of these values will be attempted when running the theory, and the values will be reported for any failures so we know what happened when the theory failed. On the last line, we test that the functions are indeed inverses by invoking them and making sure that the original value is the same as the end value.

Going further

So big deal, we could easily just create a helper test method that does something similar and we could pass whatever values to the method that we wanted. Well, hold on a second. One thing I haven’t mentioned yet is that there are some nifty tools that support automatically inserting values in to test theories. The tools automate searching the theory space for members that fit the theory assumptions but cause the theory to fail. Something like this might be useful in detecting an off-by-one error or corner case that you hadn’t considered while implementing. Typically for development you would just run the test cases presented with the method, and then would search the theory parameter space with spare cycles. This enables you to have great power as well as having blazing unit tests, which is key for any test-driven development.

If we allow automated searching of the theory space, there is one small problem with the example theory listed above. Can you see it?

The assert attempts to multiply by the multiplier and then divide by the multiplier, so a zero value for the multiplier will cause a divide by zero error. When we use automated tools for searching the theory space, we want to say that this theory only holds when the multiplier is not zero. We can easily do this:

    @Theory public void multiplyIsInverseOfDivide(
            @TestedOn(ints={0, 5, 10}) int amount,
            @TestedOn(ints={1, 2, 3})  int multiplier) {
        assumeThat(multiplier, not(0));
        assertThat(new Currency(amount).times(multiplier).divideBy(multiplier).getAmount(), is(amount));
    }

Similar expressions exist for ensuring the value is positive, and so forth. You might create a different test or theory to show the expected behavior when the multiplier is zero.

I really liked the use of Java annotations to make this code considerably more readable. You would probably need a runner of some sort to specify the values if you didn’t do it right there in the method signature.

It seems like there is some overlap between theories and contracts, but I won’t go into that at this time. Theories exert positive design pressures like tests do, in that they tend to lead to leaner code that just does what the theory asks for.

Extensions

In Ruby on Rails, it would be nice to be able to easily test methods with multiple models that you have defined. For example, you might have a theory:

  fixtures :users

  def test_admin_user_can_access_this_page
    login(users(:admin_user_1))
    get :this_page
    assert_response :success
  end

  # the previous test, but better
  def theory_that_admin_users_can_access_this_page(User user)
    assume(user.is_a? AdminUser)
    login(user)
    get :this_page
    assert_response :success
  end

So, instead of using just one user or manually going through a set of users, you could just specify a whole bunch of fixtures and go to town. This might make fixtures less brittle by pointing out subtle interactions between parameters that would cause the theory to fail. I’m not sure how the syntax would look, and I’m not sure how you could use something better than annotations in Ruby.

Another thing that might be nice would be to say: I have a theory that any strings we put to the page from the database will always be escaped. The automated theory explorer could step through the pages that you define and try once with some basic data and then start generating HTML and JavaScript garbage and ensure that nothing gets through to the users. This might be a nice way of preventing cross-site scripting attacks (although I would imagine that there are tools that already exist for just this purpose.)

I looked, but could not find a Ruby plugin to handle theories. Hmmm…

Further reading

Some excellent resources are:
The Popper tutorial
A great paper with examples of use and benefits (see the pdf on that page)
General overview of Theories in JUnit [pdf]

Original article: Introduction to Theories

The first experience report probably had the most relevance to things that I have directly worked with. In the production Rails project that I worked on, we used Selenium for automated integration testing as verification of the development process. Dave Christiansen covered using Selenium to do automated integration testing on a Rails app. However, what was noteworthy to me about the approach that he took was that instead of writing and executing these tests only in the test phase of the development process, they were written and executed throughout the development lifecycle. He wrote a wrapper to generate RSpec tests from the Selenium IDE, refactored to DRY up the code, and then had the team write their own integration tests as they were developing. Since the tests were written in RSpec, they would actually be run whenever code was checked in (continuous integration.) After awhile, they built up enough helpers to stop really using Selenium at all except for replicating bugs.

This seems like a nice contrast to the approach that we took for two reasons. First, integration tests would automatically keep up with the development process because the team could run these tests within their own environment. If an integration test was broken, it would be immediately visible. Contrast that to having to rewrite tests several days after the structure changes for no apparent reason. Second, they ran a headless version of the Selenium Grid, which had the effect of getting their integration tests run in about five minutes. I remember this process taking quite a bit longer for us. Part of this was likely due to supporting IE6, IE7, and Firefox, but it seems like this still could have been sped up.

But overall, it’s nice to take the integration testing process and decentralize it. This might not work with all types of projects. Obviously having the developers in charge of their own integration tests seems like conflict of interest, but I suppose that solid reviews and other testing would reduce defects. But depending on the project, I can see that allowing the development and test stages to work more closely together is a plus.

Lunch was a nice experience, it was fun hearing about some things that people have done in the past and how that affected their current views on software development and testing.

Recurring themes

It seemed like people liked using Ruby for data munging and actual testing. Over half of the presentations given were using some form of Ruby for development, testing, or tools. You could probably get away with using Perl or something for this, but you can also probably pick up most of Ruby in a couple of days if you have used Perl before.

There was emphasis on abstracting tests to the largest degree possible to facilitate agility of development. This made a lot of sense to me, because one of the common pain points people expressed was having tests that were very brittle. When the application changed, the tests had to be scrapped. Considering testing to be a valuable software architecture practice in itself makes more sense than seeing testing as isolated.

Ideas and technologies

One interesting idea given in the “Open Season” part of each presentation was the idea of thumbnails for quick manual content tests. Instead of specifying a test that used screenshots or some sort of brittle method, instead print out an expected screenshot of each of the screens and the current screenshot on a single sheet of paper. Someone can then easily scan a whole page of these relatively quickly, as humans are naturally good at picking out differences between images and determining whether they are important or not.

Exploratory testing and pair testing were topics that were mentioned a few times, and I had never heard of them before. James Bach was suggested as a resource for this and pair testing. His site contains some newer articles that deal with these subjects.

Technologies discussed included:
Ruby and Ruby on Rails
Selenium
Watir
RSpec
soapUI
Groovy
Hammer/QTP

Overall thoughts

I really appreciated the constructive nature of the conference. Everyone was very knowledgable and had something to contribute to the process. All questions were asked kindly and there was a feeling of building on things others had said. There was a perceptible energy in the room during the concluding processes. People said that they were fired up to go to work on Monday and solve a problem that they had been facing using new approaches and technologies.

Something that stood out to me was the vast differences in terms that people had for testing practices. Obviously I am not an expert at testing, so there were some word hangups that I had. For example, the decision to call something a unit test versus an integration test was seemingly person-specific, if they used those terms at all. I can see how having a good stream of communication with team members, especially new ones, is very important in testing. Obviously this applies to software development in general.

While I have never been a dedicated tester before, I enjoyed the problems that people presented and their creative approaches to solving those problems. I think that I walked out of the conference having a lot more respect for software testers because of the challenges they expressed, both technical and organizationally.

One meme that I took away from this was the pirate heuristic: When you are out of ideas, steal someone else’s. I was out of ideas, and hence stole this one myself.

Original article: IWST Summary