Speed up unit tests #113

Closed
opened 7 years ago by jking · 2 comments
jking commented 7 years ago
Owner

By now, the whole test suite takes a non-trivial amount of time to run. Partly this is because code coverage is always computed (assuming XDebug is installed, which is itself a source of slowdown), but tests which make use of password_hash() and password_verify()---as well as tests which parse feeds---incur significant delays from sha-256 hashing. Tearing down the database schema and re-creating it for each and every database test probably contributes as well (though this is probably offset somewhat by using an in-memory database).

All these problems can be worked around, but would require increasingly esoteric PHP extension dependencies for anyone wishing to hack on the server. It would also require time and effort to profile and actually tune the slower tests.

By now, the whole test suite takes a non-trivial amount of time to run. Partly this is because code coverage is always computed (assuming XDebug is installed, which is itself a source of slowdown), but tests which make use of `password_hash()` and `password_verify()`---as well as tests which parse feeds---incur significant delays from sha-256 hashing. Tearing down the database schema and re-creating it for each and every database test probably contributes as well (though this is probably offset somewhat by using an in-memory database). All these problems can be worked around, but would require increasingly esoteric PHP extension dependencies for anyone wishing to hack on the server. It would also require time and effort to profile and actually tune the slower tests.
jking added this to the Future milestone 7 years ago
jking added the
enhancement
testing
help wanted
labels 7 years ago
Poster
Owner

Another way to speed up unit tests would be to use atoum (which can run tests in parallel) rather than PHPUnit. This would mean rewriting most of our ~500 tests, however.

Another way to speed up unit tests would be to use [atoum](http://atoum.org/) (which can run tests in parallel) rather than PHPUnit. This would mean rewriting most of our ~500 tests, however.
jking referenced this issue from a commit 6 years ago
Poster
Owner

For various reasons, Atoum seems not to be any faster than PHPUnit, and is actually slower by a significant amount in a limited test run.

Thus far testing times have gone from:

  • With coverage: 1:13
  • Without coverage: 0:27
  • With fixed timeout test: 0:23

Disabling Xdebug entirely seems to cut off another 10s, but disabling it for testing and enabling it for coverage may get a bit hairy. It may be worth doing either way, as it would be useful to fully control loaded extensions during testing.

For various reasons, Atoum seems not to be any faster than PHPUnit, and is actually slower by a significant amount in a limited test run. Thus far testing times have gone from: - With coverage: 1:13 - Without coverage: 0:27 - With fixed timeout test: 0:23 Disabling Xdebug entirely seems to cut off another 10s, but disabling it for testing and enabling it for coverage may get a bit hairy. It may be worth doing either way, as it would be useful to fully control loaded extensions during testing.
jking referenced this issue from a commit 6 years ago
jking closed this issue 6 years ago
jking modified the milestone from Future to 0.3.0 6 years ago
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Notifications
Due Date

0000-12-31

Dependencies

This issue currently doesn't have any dependencies.

Loading…
There is no content yet.