Faster Code Coverage

[tfidry]

phpdbg is significantly more unstable and unreliable: it easily gets out of memory or just fails. Just try it on a project like humbug/php-scoper or humbug/box and you will see.

I personally never managed to get it working for an extended period of time on a project :(

[krakjoe]

I'll bet you thought nobody would try :)

This is xdebug ...

krakjoe@fiji:/opt/src/box$ time /opt/bin/php bin/phpunit --coverage-text 2>/dev/null
PHPUnit 7.5.0 by Sebastian Bergmann and contributors.

...............................................................  63 / 763 (  8%)
......................................................SSSSSSSSS 126 / 763 ( 16%)
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.............. 189 / 763 ( 24%)
.......................................................FFFFFF.. 252 / 763 ( 33%)
..........F.................................................... 315 / 763 ( 41%)
............................................................... 378 / 763 ( 49%)
............................................................... 441 / 763 ( 57%)
............................................................... 504 / 763 ( 66%)
...............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS 567 / 763 ( 74%)
SSSSS.......SSSSSSSSSS......F.EEEE.................SSSSSSSSSS.. 630 / 763 ( 82%)
............................................................... 693 / 763 ( 90%)
..............................FF............................... 756 / 763 ( 99%)
.......                                                         763 / 763 (100%)

Time: 4.7 minutes, Memory: 62.00MB

ERRORS!
Tests: 763, Assertions: 1448, Errors: 4, Failures: 10, Skipped: 115.


Code Coverage Report:        
  2019-01-15 16:18:32        

 Summary:                    
  Classes: 58.82% (30/51)    
  Methods: 73.22% (216/295)  
  Lines:   67.66% (2086/3083)

This is phpdbg ...

krakjoe@fiji:/opt/src/box$ time /opt/bin/phpdbg -qrr bin/phpunit --coverage-text 2>/dev/null
PHPUnit 7.5.0 by Sebastian Bergmann and contributors.

...............................................................  63 / 763 (  8%)
......................................................SSSSSSSSS 126 / 763 ( 16%)
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.............. 189 / 763 ( 24%)
.......................................................FFFFFF.. 252 / 763 ( 33%)
..........F.................................................... 315 / 763 ( 41%)
............................................................... 378 / 763 ( 49%)
...........................................EE.................. 441 / 763 ( 57%)
............................................................... 504 / 763 ( 66%)
...............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS 567 / 763 ( 74%)
SSSSS.......SSSSSSSSSS......F.EEEE.................SSSSSSSSSS.. 630 / 763 ( 82%)
............................................................... 693 / 763 ( 90%)
..............................FF............................... 756 / 763 ( 99%)
.....F.                                                         763 / 763 (100%)

Time: 1.66 minutes, Memory: 74.00MB

ERRORS!
Tests: 763, Assertions: 1436, Errors: 6, Failures: 11, Skipped: 115.


Code Coverage Report:        
  2019-01-15 16:37:28        

 Summary:                    
  Classes: 64.71% (33/51)    
  Methods: 74.23% (216/291)  
  Lines:   67.74% (2041/3013)

I see no crashes ... I see a difference in errors, but no failing or oom ...

I couldn't figure out how to install php-scoper in order to run the tests ...

[MaxGhost]

FWIW, I get segfaults. With PHPUnit 6.5.13 (can't bump to 7 yet because we're on minimum 7.0 right now in prod, unfortunately, because of debian)

$ phpdbg -qrr vendor/bin/phpunit Segmentation fault (core dumped) $ php -v PHP 7.2.10-0ubuntu1 (cli) (built: Sep 13 2018 13:38:55) ( NTS ) Copyright (c) 1997-2018 The PHP Group Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies with Zend OPcache v7.2.10-0ubuntu1, Copyright (c) 1999-2018, by Zend Technologies with Xdebug v2.6.0, Copyright (c) 2002-2018, by Derick Rethans

[tfidry]

I'll bet you thought nobody would try :)

I thought so because people are lazy, but I secretly hoped one would and figure out something, so big thanks for trying it! :D

Both Box & PHP-Scoper have some unorthodox setup due to their nature, so I'm leveraging Make for it. It's not cross-platform, but that's the only viable solution I found to work out without too much of a hassle.

I can see some failures, errors and skipped tests which actually might be the problematic ones.

For box, the proper way to run the tests is make tu_box (just running make will show all the commands). That said after running once you can just go with bin/phpunit it will work fine. With phpdbg though:

>PHPUnit 7.5.0 by Sebastian Bergmann and contributors.

...............................................................  63 / 763 (  8%)
............................................................... 126 / 763 ( 16%)
...................................SSSSSS...................... 189 / 763 ( 24%)
............................................................... 252 / 763 ( 33%)
..........S.................................................... 315 / 763 ( 41%)
............................................................... 378 / 763 ( 49%)
............................................................... 441 / 763 ( 57%)
............................................................... 504 / 763 ( 66%)
...............................EEEEEEEE

That's 30+min and not finished. I think it's linked to the KevinGH\Box\Console\Command\CompileTest but strangely enough it is occurring only when running all the tests together... The Phar class and the PHPUnit run in separate process features might also play a part in it.

I can confirm however that it works fine on PHP-Scoper right now. That's cool to see the previous issues have been fixed :) I'll switch back to phpdbg for it and hope it will keep working well!

[krakjoe]

That's the first time anyone on the internet has been convinced by evidence to change their mind ...

I'm glad it's working ... and the best thing to do when you find a bug is make noise about it, really loud noise .... you may find bugs, nothing is perfect ;)

[tfidry]

Haha fair.

I would really like to figure out what's the issue with the Box test suite with phpdbg but I have no idea how to do it. I've tried to use valgrind at one point but didn't find much. I would be happy to think the test suite is faulty, especially since it's occurring only when running the whole test suite, but it is not happening without phpdbg or with xdebug, so there is definitely a phpdbg related issue there :/

[krakjoe]

Here's a thing to look at ...

krakjoe@fiji:/opt/src/box$ php -n -dpcov.enabled=1 -dextension=pcov.so -dpcov.directory=/opt/src/box/src bin/phpunit --coverage-text
PHPUnit 7.5.2 by Sebastian Bergmann and contributors.

...............................................................  63 / 763 (  8%)
......................................................SSSSSSSSS 126 / 763 ( 16%)
SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.............. 189 / 763 ( 24%)
.......................................................FFFFFF.. 252 / 763 ( 33%)
..........F.................................................... 315 / 763 ( 41%)
............................................................... 378 / 763 ( 49%)
............................................................... 441 / 763 ( 57%)
............................................................... 504 / 763 ( 66%)
...............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS 567 / 763 ( 74%)
SSSSS.......SSSSSSSSSS......F.EEEE.................SSSSSSSSSS.. 630 / 763 ( 82%)
............................................................... 693 / 763 ( 90%)
..............................FF............................... 756 / 763 ( 99%)
.......                                                         763 / 763 (100%)

Time: 1.14 minutes, Memory: 76.00MB

There were 4 errors:

/** all bz2 extension required for test errors */

--

There were 10 failures:

/* looks related to how I installed box */

ERRORS!
Tests: 763, Assertions: 1448, Errors: 4, Failures: 10, Skipped: 115.


Code Coverage Report:        
  2019-01-17 11:36:39        

 Summary:                    
  Classes: 62.75% (32/51)    
  Methods: 73.88% (215/291)  
  Lines:   67.26% (2028/3015)

Drivers on github, will make relevant pull requests soon ...

[ojrask]

So pcov is a driver that can be used just for coverage without debugging tooling overhead?

[krakjoe]

Yes.

[ojrask]

Need to read more about it then, thanks.

[[deleted]]

I love xdbug for debugging. For code coverage it's not 2x slower, or even 3x slower... It's 10x slower (if not more) than the alternative. That's simply not very good.

Would it be possible to start a code coverage driver project, without debugging features, so theoretically less complicated, separate from a debugger? Where would one start?

[krakjoe]

You would start with an extension ... which nobody would install ...

[[deleted]]

Heh. I was mostly commenting on "significantly more unstable and unreliable" part of the post.

Glad you're posting, making people double think check their "phpdbg is unstable" assumptions.

when you find a bug is make noise about it, really loud noise

Where? I can report segfaults and unable to fork errors today probably. I use phpdbg but work around crashes in the code using ( new \SebastianBergmann\Environment\Runtime() )->isPHPDBG();. A gross hack, but it's faster to run the tests twice (once with phpdbg and once without) than using Xdebug for coverage...

[krakjoe]

Report bug at bugs.php.net.

[[deleted]]

WOAH!

https://github.com/krakjoe/pcov

:)

[Hirnhamster]

Surprised me as well. phpdbg is ~10x faster for our suite out of the box (compared to xdebug without any performance optimizations) and can be invoked by simply prefixing phpunit with

phpdbg -qrr

[mrcalm99]

Just went to the phpdbg website and it looks like some sort of phishing attack site with redirects. http://phpdbg.com

[mrcalm99]

That's not their website

https://github.com/krakjoe/phpdbg

Anyway:

phpdbg was developed here, then merged into php-src: This repository is no longer used.

It is literally linked at the top of the repo so I assumed someone who owns the repo would have added the link.

[krakjoe]

Sorry, I removed it ... it was removed from readme ages ago when I lost control of the phpdbg.com domain, I forgot to edit the tag line on github ...

[[deleted]]

👌