Steve Portigan and Dan Soltzberg on Noticing.

Good article on the lost art of Noticing. I’ve had the good fortune of working closely with Kristan Vingrys in Australia, and I’m constantly in awe of his ability to notice things. It’s amazing what I sometimes don’t see, directly in front of my face.

Portigal: Someone showed me a great user research training activity: circulate through an environment and note everything you observe, but using only one sense. First, observe from a distance—say, from on high—so you can’t hear what people are saying. Then sit in the middle of an active zone, but close your eyes. Students have told me how rapidly one sense fills in for the other. Of course sometimes that filling in isn’t accurate, so it also illustrates the importance of triangulating observations from a few different perspective.

Some great stuff here on noticing, and using what you witness to drive conversation and action.

Portigal: That reminds me of improv. Newcomers expect that improv is a very active, concerted effort to be funny. But what’s so stimulating about doing improv is that it’s not (necessarily) about being funny, but that the whole approach of saying, “Yes, and…,” guides us to notice and act in response to what the rest of the team is doing. It becomes this collaborative problem-solving activity that happens to generate a performance, rather than the typical “stuff from the inside comes out” model of performance. And the key to making that performance flow is that everyone is paying close attention each other.

Soltzberg: It’s funny you say “notice and act.” To reference Zen again, one of the maxims of Zen practice is “notice and allow.” In both practices—improv and meditation—I think giving yourself permission to “just be,” to receive without transmitting, makes it possible to really drink in sensory data and to really listen to other people with an incredible kind of unforced compassion.

It reminds me a lot of the approach we take to being with people when we do fieldwork. In the field, you have to simultaneously drink all kinds of information in, and at the same time be active in guiding the interaction. There’s this tightrope walk between action and non-action, ego and non-ego. To move back and forth gracefully between these different ways of being requires noticing not just what’s going on around you but what’s going on inside you as well. It’s one of these things that sounds so simple, but really takes practice to be good at.

There are several assumptions we make when we automate functional tests.  This is just a quick list, off the top of my head:

1. We automate when a test is repeatable.

2. We automate when a test is feasible to automate.

3. We sometimes automate when it helps put the application in a desired state.

4. We sometimes automate to express a behavior before that behavior is built.  (That whole “test-driven development” thing.)

5. We automate when all parts of the system are predictable.

It never ceases to amaze me that so many people truly do not understand #5.  You can’t run an automated test if your system fluctuates, if your data changes, if that 3rd party app can’t be counted on, or is updated constantly, etc.  You can certainly try, you’re just not going to be successful.  The time and money spent babysitting your test suite isn’t worth it, not only in real hours, but also opportunity cost.  How much time do you spend analyzing data errors, updating expected results, and re-running until you get it right?  Hint: If it’s more than 1 or 2 seconds, it’s too long.  That’s not what automation is for.

Testers give Developers a hard time, sometimes justifiably, but this is an area where we really could learn from them.  How many unit tests have you ever looked at?  They set the system to a certain, known state, then they run the test.  The data they use is the data they need to test the code they’ve built, and the interfacing systems are mocked out.  If a unit test fails, it more than likely fails for one reason… the code is broken.  That’s the point.

More often than not, when I see automated functional test suites, the idea of a “known state” gets thrown out the window.  Functional tests are… well, they are functional tests.  I don’t want to care how well they run against a “real” production dataset, I don’t want to care about test data much at all.  I care that the function does what it is supposed to do.  If that function is reliant on a certain type of data, let’s put those records into the shared dataset.  (If that data breaks old tests, they’ll need to be fixed, but that’s natural.  The system is growing.)  The idea is simple: If a functional test fails, I, as a tester, should immediately start diagnosing where the code broke.  I shouldn’t have to look in 8 different places.  Was it our expected results?  Was it that vendor app?  Is my test harness bad?

Now, to be clear, I am not advocating that we avoid testing against real-world data.  A production subset (or full snapshot, if available), is absolutely essential for quality assurance.  That’s what manual, exploratory, and performance testing are for.  Ultimately you need a test instance with real data, but you also need one with known data, so you can trust your automated tests.

Honestly, I bet it would be possible and challenging (and somewhat fun) to build a super-intelligent test harness that automatically diagnoses and fixes errors on the fly.  I’m serious.  But if you’ve got the resources to build something like that (without invalidating your test harness with business logic), well… can I borrow some cash?

It seems like I’m never in the states, or unavailable when this is going on.  Would love to attend someday.

GTAC ‘08 - Call for Attendance

Call for Attendance
Google Test Automation Conference 2008
Seattle, WA
October 23 - 24

Google’s Test Automation Conference is built on our participants each bringing their experience, ideas, and insight to the discussions we host. Our participants are all experienced industry professionals working as software development engineers or software testing engineers. For the limited number of spaces available in each year’s conference, we ask each applicant to share what they would bring to the discussions as an active participant in the conference.

To apply for a participant space at the GTAC 2008 conference, you can apply:
http://services.google.com/events/gtac2008

Test Leads/Managers need to check build test metrics constantly (actually, the whole team does), and a painful part of CruiseControl and CruiseControl.rb is checking one build against another.  Or the last 10 builds.  Or maybe all of the builds in the current iteration or release.  That isn’t easy, and sometimes involves Excel charts to show metrics over time.  Not fun.

Recently I’ve been playing around with Ruby and the Google Charts api, to display testing metrics.  So far, all I have is one Ruby script, which parses CC log files for important information.  In this example, it looks for Selenium pass/fail numbers.  Using the Google Charts api (see here:  http://code.google.com/apis/chart), the relevant info is passed to Google.

There are two options with Google Charts.  First, it can take a url, similar to:

http://chart.apis.google.com/chart?cht=p3&chd=t:60,40&chs=250×100&chl=Hello|World

Or, you can create an html img tag, like:

<img src="http://chart.apis.google.com/chart?chs=250x100&chd=t:60,40&cht=p3&chl=Hello|World"/>

The chs, cht, chd, chl are parameters that Google Charts understands, and that is where we pass in the pass/fail metrics.  Here is an example:

This is all done withwith one relatively simple Ruby script.  All it does is iterate over the CruiseControl log files, then searches for the relevant information.  For Selenium RC results, it would look for something like:

Finished in 34.05 seconds

15 tests, 15 assertions, 6 failures, 3 errors

For each log file, if you grab that line and parse the meaningful info, you can save the data off into img tags somewhere and include it in html, like on a wiki, or Mingle.  If you want to learn more about Ruby scripting, I highly recommend Brian Marick’s book:  Everyday Scripting with Ruby: For Teams, Testers, and You.

There are certainly some problems with this solution.  I’m not crazy about the Google Charts api.  Some of the charts are easier to customize than others, and it’s difficult to display actual data in certain graphs.  Google asks that you hit the api “less than 250,000 times a day”, which I doubt would be a problem for most folks.  But you also need an internet connection, which won’t be a problem until you don’t have one.

Also, as you might see, there could be an ethical/legal issue with sending your build metrics to Google.  Not sure where I come down on that one… I’m no conspiracy theorist, but I certainly understand if that concerns people.  Personally, I like the Google Charts solution, because you don’t need to install anything, just as long as you have Ruby.

There are some other options out there.  I’m taking a look at Gruff and Scruffy too, which look a bit easier to implement, but you need RMagick and ImageMagick installed, which can be a pain on OSX.

Another way to test different versions of IE.

IETester is a free WebBrowser that allows you to have the rendering and javascript engines of IE8 beta 1, IE7 IE 6 and IE5.5 on Vista and XP, as well as the installed IE in the same process.

I still prefer Browsershots.

More from Newsweek on the Microsoft weirdos.

Back in Sydney after 5+ weeks of leave in the states.  (I’m still trying to remember what I do for a living.)  My favorite thing since being back… It’s about 18 C in both Chicago and Sydney these days, the Chicagoans are running around in shorts and beach towels, while the Australians have broken out their winter coats.  That always kills me.

Our trip took us Sydney-Melbourne-Chicago-Detroit-Chicago-Dallas-Shreveport-Dallas-Chicago-Bay City-Chicago-Green Bay-Chicago-Sydney-Melbourne-Sydney.  All in all, 42,058 km (26,134 miles).  In the car, 3,577 km (2,223 miles).  In the air, 36,998 km (22,989 miles).  On trains, 1,483 km (922 miles).

I don’t wish that on my worst enemy.  Especially the Chicago-Sydney-Melbourne-Sydney part.  Don’t ask.

Google on Exploratory Testing

Nothing earth-shattering here, but Google’s use of ET around Chat is a good example of functionality that technically could be automated extensively, but is the heavy-lifting around building a test framework worth it?

I also found this part interesting about strategy:

To do this, we start with the definition of a Test Strategy. This is where we outline the approach we are taking to the testing of the product as a whole. It’s not super-detailed — instead it mentions the overarching areas that need to be tested, whether automation can be used to test the area, and what role manual testing needs to play. This information lets developers and PMs know what we think we need to test for the product, and allows them to add unit tests etc to cover more ground.

Google Chat’s Test Strategy helps developers know where to add more unit tests.

GTAC ‘08

Conference is scheduled for October 23-24 in Seattle. Proposals are due on June 6th.

I’d love to attend for many reasons, Seattle is one of my favorite cities in the US. Unfortunately, I doubt we’ll be back from Australia by then. Cliff attended last year, and he enjoyed it. Last year’s conference videos are up on YouTube here.

Sorry, no posts in a while.  Just moved to Sydney, settling in.  Will get back at it soon.