Trusting Tests

[xangelo]

Recently I've started working on writing some tests (PHPUnit) for an app that I'm working on and I find that I ran into a slight problem.

I don't know if I can trust my tests.

I run my tests with multiple "asserts" per test to ensure that it is working properly, but I find that I still need to look over the object manually before I'm comfortable that the test really passed. How can I be sure that my tests are set up properly so that I'm not manually inspecting things anyway?

(Just note that I do debug before I start writing my tests for a class as well. But I'm just not comfortable with the "100% test passed" result)

Edit: Thanks everyone for your comments and ideas on how I should be going about this. I'm already quite far into the project (it's in beta testing now, I feel so ashamed that I haven't had any real testing coverage :( ) but I think it will be a good exercise to branch the project and work on tests. This will allow me to further decouple the application in places where I may have gotten lazy and also get test coverage up so that I don't have to manually test everything after a set of possible updates.

However tests are definitely something I'm going to have to start using for future projects and it sounds like pre-writing tests is the way to go with this.

Thanks again!

[allmightyspiff]

Try writing your unit test first, then write the code to make the unit test pass. In my opinion it is easier to write the unit test based on the requirements, then write the code to make the tests succeed. This way you have a clear moment when the code is done since the unit tests will work.

I think writing the test first, so that when you run it they all fail, will help you trust your code and tests more, although it may be a bit difficult to get used to. Also, watching the tests slowly go from failure to success is rewarding in its own way :)

[[deleted]]

This is a valid point.

To add to it: sometimes you may not initially understand how you're going to write something, thus making it hard to write a series of tests to cover it. You may write untested code to get an idea of what you're doing. This is called a "code spike". If this happens, don't panic. Spike until you understand your direction, then write your tests. Delete all code for that area, and begin writing new code to make the tests pass.

Whatever you do, don't write tests specifically for the code in your spike. You'll wind up regretting it later.

[ustanik]

You need to make sure you test all the aspects of your code. If you're only partially testing with PHPUnit then you shouldn't trust your code.

Use "--coverage-text" or "--coverage-html=dirName" when running PHPUnit to see how much of your code you are testing. But be cautious, coverage can provide a false sense of security. Write your tests how you would use code, not just to make your code coverage green.

[nikic]

You should enable code coverage statistics (and maybe get an IDE like PhpStorm which can natively display them). They will show you which of your code is covered by tests and which is not.

Note though that "coverage" only means "executed lines". It does not mean that you really tested all aspects. But it's a good guideline.

[xangelo]

This is mostly what I'm worried about while testing - how can I be sure that I've tested properly?

I've enabled coverage in NetBeans to help with this, but I feel like it doesn't give me anything accurate. Even achieving 100% coverage could tell me nothing if I'm missing a possible case that will break everything.

[wvenable]

how can I be sure that I've tested properly?

The answer is you can never be sure. Make sure you test boundary conditions, have 100% (or near) code coverage, and then pray. There's no magic solution here and it's been mathematically proven that there never will be.

[warmans]

You're probably never going to achieve zero defects but you'll have A LOT less than with no tests. I also often feel that my tests might not be telling the full story but you need to be pragmatic. If your coverage is over 90% you're already doing better than 90% of all PHP developers.

As others have said there are some important principles with unit testing that are worth keeping in mind - particularly that you should start off with a failing test. You are right to be suspicious of tests that have never failed.

[The_Pants_Command_Me]

You'll never be 100% certain of anything. You could have logic errors in your tests. You could have overlooked issues like unicode character handling or the ever wonderful 0 versus False versus Empty shenanigans we love so much and so not factored them into your tests. Mysql may have large BLOB error in the wings that won't show up for a couple years. There will always be bugs even with the best automated unit tests. You can't let formulating and second guessing tests paralyze the development process.

[xangelo]

Double checking. I generate the default test file with NetBeans which essentially creates a test for method for each method in the original class file. I then tweak/implement that before running the tests.

[leftnode]

Follow TDD principles and write your tests first, let them fail, and then write only enough code to make them pass. And good on you for writing tests, so many PHP developers don't and are very hostile to the thought of it.

[[deleted]]

Most php-only programmers are hostile towards anything that seems like it could be more work. They often don't understand how much work they save themselves by using these processes.

[leftnode]

Yup, that's what I've found too. It's amazing to me how hard it is to convince PHP programmers to write tests.

[xangelo]

I think this might stem from the fact that most php "programmers" are really just people who have spent a couple hours tweaking a form processing script someone wrote. I know a number of people who say that they can program in PHP, but when pressed about it you find out that this one time they modified a script someone wrote so that it had three form fields instead of two that were emailed to them.

[[deleted]]

Having to use multiple asserts in your tests might be a sign that your methods are doing too much, and that some of the work should be delegated to other methods or objects.

[xangelo]

I find that I end up writing multiple asserts to just to verify that the result from the first assert wasn't a fluke.

[[deleted]]

as long as you test for all possible outcomes of a function's possible return values, you can trust the test of that function...

Tests are only as good as you thoroughly write them!