Jade Meskill: Hello. Welcome to another episode of the Agile Weekly Podcast. I’m Jade Meskill.

Roy vandeWater: I’m Roy vandeWater.

Clayton Lengel‑Zigich: I’m Clayton Lengel‑Zigich.

Derek Neighbors: And I’m Derek Neighbors.

[laughter]

Automated Testing Struggling to Go Fast

Jade: Very nice. Derek, you had something you wanted to talk about. What was it?

Derek: Yeah. I’ve been doing the work with a number of teams, and one thing that I’m starting to see emerge as a pattern is the team’s struggle to go fast. They struggle to work particularly within the Scrum framework a lot of times. Not only are they siloed where they got developers and QA that are separate, but where they don’t have a lot of automation. This could be automation around deployment.

This could be automation around testing, so maybe they don’t have an automated testing suite. Or if they’ve got some automated testing suite, there’s still some manual regression happening. Even if they have a fully automated testing suite, they’re not running it automatically.

They’re not having continuous integration, or it takes an enormous amount of long time to build. What I’m finding is it’s very, very hard for them to see what performance looks like, because they can’t get over the concept of their current reality. The current reality is, “Hey, it takes five days to run the test suite. There’s no way we can have a one‑week iteration. That’s just impossible. It takes us one week just to test, once the developer’s done with a feature.”

Have you guys seen instances, where maybe the current reality or the lack of automation makes it so teams just find…I had somebody specifically today, tell me, “That’s really nice that you stand up there and talk about a 10 minute build. Frankly, there’s no way you’ve ever done that before, because I’ve worked for five different companies, and none of them have ever been able to do that. I find what you’re saying impossible to believe.”

Jade: [inaudible 02:06 ] we have a 10 second build?

Roy: Yeah. I agree. I’ve struggled so hard to get a build to take 10 minutes. I don’t believe it’s possible.

Derek: In their current reality, I understand that. If you’re a manual regression tester, and you’ve got a fairly complicated suite, it’s taking two or three days to run a single test plan. I can understand how it feels impossible that you could, with any level of comfort, run a test suite under 10 minutes that made you feel like you weren’t shipping crap.

Roy: We’re working with the team right now, but we’re not working with working microphones.

[laughter]

Letting Technology Get In The Way

Jade: That’s right. We are working with the team right now that a single test takes two weeks of constant computation, a single test.

Clayton: I’ve seen that in a lot of instances where, the technology gets in the way. If you’ve got a java applet that loads a Swf, how do you automate testing of that content? That seems like this impossible thing. I think a lot of people say, “It is what it is, throw my hands up in the air. What else are we going to do but, let those hourly regression people slave away at typing in commands and looking at the screen”?

Roy: Along the idea of trying to test a flash out, I think that’s one of the first mistakes that immediately causes your test suite to get larger, and take longer, regardless of whether it’s manual regression or automated testing. That’s when you write the test case last, after everything is done.

Jade: Yeah.

Test Driven Development Helps Reduce Some Problems

Derek: When you write the test case first, you end up using solutions that are easier to test. You don’t run into those things.

Clayton: That’s one of my favorite things about doing, not even just TDD, but just test first. Where you have to take into account what easy to test and what isn’t. I’m working with a team that has a fairly large fitness test suite. You can, totally, tell the way they went about using fitness to write this acceptance for higher‑level tests, because it was so difficult to test at the unit level, the only thing they could do was test the big black box of spaghetti code. They were all written after the fact.

They were trying to do more of a TDD approach or write unit test. It would’ve been so impossible to write any test whatsoever, if they were actually dedicated to that, they wouldn’t have had to solve that problem in the first place. The reason automation doesn’t get more not popular, it’s because a lot of times team get rewarded for effort. Jade and I were talking about this the other day. If you’re using a lot of effort that probably means you’re dumb. You can be so smart and lazy that you can do a lot of things automatically.

If you work in a system or an environment where you rewarded for staying up till five in the morning working on some stupid thing that should be automated, what incentive do you have to automate things? You’re going to get clapped for, and pat on the back if you put a lot of effort into stupid things. Human nature wise, that seems to make sense. That’s like irrational decision at that point.

Jade: Yup. If you’re putting a lot of effort, it must be important.

Clayton: Yeah. Exactly.

Relying on Backstops Creates Bad Habits

Roy: The part that’s really difficult especially when you go to test after the fact is to justify the expense of writing a test. Because you have this working piece of software that your user could be using right now, why not just release it, and then not worry about the test?

Clayton: I’ve heard regression teams be referred to as backstops and people make a baseball analogy metaphor that’s, “The catcher is supposed to be there to catch the ball and that should be fine, right?”

That’s what I think what the developers think of themselves as, “We’re going to do this thing where we’re going to write something, and I’m going to look at it, and I’m pretty sure it works. That’s kind of the catcher, but in case there’s some wild crazy pitch edge case that gets past me, at least there’s a backstop.”

Pretty soon, they stop trying to catch the ball and everything is just the backstop. I think that’s what happens. Why would you spend the extra time to make sure no bugs or no defects or problems or no nothing ever got to the regression team, because they’re always there?

You know they’re going to be there. If something goes wrong, wouldn’t you rather someone blame them than blame you? That’s always…

Roy: Then you get into that whole QA developer rivalry, too, where you hate them because they’re making you do more work. You don’t get to work on new stuff, because they keep uncovering the crap that you wrote earlier.

Testing vs Checking

Clayton: I really like the way that some people in the QA or testing or whatever community talk about testing versus checking. Those are the things that a computer can do. Making sure that this algorithm works properly in these different cases or whatever.

I really like that idea where testing is more about heuristics and looking at, “How does the system function, and what do I expect to happen? What people perceive and is it consistent with the rest of the thing?”

All the stuff that you, actually, need the human brain for. Those are valuable things that people could be working on, actual people. Everything else really should just be automated. [inaudible 06:56 ] should be automated testing.

The $465 Million Deployment Mistake

Jade: You posted a really awesome article, I think at the beginning of this week, about a case where a company neglected to use automated deployment. In this case, instead of automated testing, but another case where something is done repetitively and there was an opportunity for automation that wasn’t taken. In this case, I think it ended up costing that company $365 million.

Clayton: Some trading company, right?

Jade: Right. We’ll attach the article to the description, but it was pretty crazy. They break down exactly what happened, and it ends up being, “We just didn’t automate something that should have been automated.” Now it’s human error that comes into play.

Business Rules So Complicated They Are Untestable Is a Smell

Derek: I see some funniness in that one of the things that had come up in some the discussions today, too, was, “Well, one of the things that QA is really incredible for is our business rules are so complex that nobody understands them. Literally, nobody actually understands the business rule.”

The great thing is what we do is we have QA, what would happen is you would basically code the story as a developer, and as you code the story as a developer, my fantastic test plan is going to cover every edge case, so that I can actually tell you how you didn’t understand the busi