See this page as a slide show

CT320: Testing

Philosophy of testing

1. Get over it
2. Trust, but verify
3. Test the typical cases
4. Test invalid input
5. Test the limits
6. Automation

Get over it

• Some programmers don’t enjoy testing. They want to create, not test. Too bad.
  • I don’t enjoy doing dishes, but that’s the price I pay for using a plate.
  • Try to think of it as a step in perfecting your creation.
  • Many big companies have QA departments, who do testing. Hooray!
    • It’s fun work.
    • Still, don’t send them trash. Everything runs more smoothly if you manage to do a minimum of testing.
• Nearly all programmers have a mental block against testing. It’s like asking parents to find flaws in their children.
  • This is why industry uses non-author testing.
  • You just have to deal with it.

Trust, but verify

Sure, whoever programmed this is really smart (especially if they’re you). It’s probably correct.

• Faith can be a wonderful thing.
• Data is better.

Test the typical cases

• Testing the typical cases is often more of a social thing than an engineering thing. For non-author testing, if you can present a typical, non-extreme, case that fails, the author will take it seriously. “Dude, it doesn’t even work for this case!”
• If you don’t bother verifying the test cases that your teacher/programmer gave to you, then you’re an idiot.

Test invalid input

Test the limits

• Consider a program that’s a guessing game. You think of a number 1–100, and it has to guess the number. You tell it “too high”, “too low”, etc.
• If this program works for 42, it’ll probably work for 56.
• Test it for 1, 100, 0, and 101. Those are the edge cases. They’re at the boundary between “should work” and “shouldn’t work”. You want to ensure that a < isn’t accidentally a ≤.
• Also, 50 might be an edge case, for a binary search.

Case study

Consider a command, zot, which takes a number of files as arguments, and an option which determines which files should be output. Think of it as a conditional cat.

    zot [-f first-last] file ...

Where first and last are optional inclusive ordinal file indices.

• zot -f 2-3 alpha beta gamma delta is equivalent to cat beta gamma.
• zot iota is equivalent to cat iota.

How do we test this command?

Test cases (valid cases)

    zot alpha
    zot alpha beta gamma delta
    zot -f2-3 alpha beta gamma delta
    zot -f 2-3 alpha beta gamma delta
    zot -f 2-2 alpha beta gamma delta
    zot -f 2-  alpha beta
    zot -f  -2 alpha beta gamma

Test cases (invalid cases)

    zot
    zot -fx alpha
    zot -f3 alpha
    zot -f1 -f2 alpha beta
    zot -f0-1 alpha
    zot -f1-2 alpha
    zot -f2-1 alpha beta gamma
    zot -q alpha
    zot badfile

Automation

• Poor testers test manually. When the code changes, they test again.
  • Well, some of the tests.
  • If they remember what the tests are.
  • And have the energy.
• Good testers write an automated test suite (as simple as a shell script). They run the test suite after every code change.

    #! /bin/bash
    (
        zot -fx alpha
        zot -f3 alpha
        zot -f1 -f2 alpha beta
        …
    ) >&out
    diff out known-good-output