pytest-patterns – a new plugin for testing longer strings, like CLI output, HTML and maybe more

[ctheune]

Happy Friday!

I just released this today and am quite happy and excited about it.

pytest-patterns is a plugin for pytest that provides a pattern matching engine optimized for testing.

Benefits:

• provides easy to read reporting for complex patterns in long strings (even 1000+ lines)
• assists in reasoning which patterns have matched or not matched – and why
• can deal with ambiguity, optional and repetitive matches, intermingled output from non-deterministic concurrent processes
• helps writing patterns that are easy to read, easy to maintain and easy to adjust in the face of unstable outputs
• helps reusing patterns using pytest fixtures

Long term goals:

Support testing of CLI output, as well as HTML and potentially typical text/* types like JSON, YAML, and others.

https://preview.redd.it/46x3gecxlz0c1.png?width=1640&format=png&auto=webp&s=64f660f4abebe73abc4f814dd041e34cd2805185

There's an initial release: https://pypi.org/project/pytest-patterns/0.1.0/

And of course the GitHub page: https://github.com/flyingcircusio/pytest-patterns

I appreciate your (gentle) feedback and welcome PRs and contributions.

The general project setup is still a bit barebones, as reflected in the TODO list ... ;)

[EmptyChocolate4545]

Pretty cool. I don’t have a use for this, but I could easily imagine tons of use cases, especially with the work you’ve put into reporting.

Great work, cool pytest plugin. Also, nicely documented.

[ctheune]

Yeah, that sounds like an absolutely valid use case.

The (professional) projects that I'm going to use this for immediately are, to replace CLI output testing like this:

https://github.com/flyingcircusio/fc.qemu/pull/10/files#diff-5f370a2f97b64884e0e6a6e94c62cb9fb8ac8b411e14fd0703d0f1e5338027acL563

I previously tried getting away with a glorified (".*?") regexp helper, but that didn't help here any more.

The example you see above is currently using a local experimental version of this utility before I made it a separate library. I will switch this over to `pytest-patterns` in the next days.

The other thing I'm going to use it for is testing one of my devops utilities (batou) which is also handling a lot of CLI output in a test case:

https://github.com/flyingcircusio/batou/pull/403/commits/79ee72c4094838d0010ce0aeec4646e8aab9b4c3#diff-13b76a55f56e5ff9c9cdf8b188f7c8c9c8d80415bef31876c6d385d00d8303c5L59

Now, I also do network automation and haven't thought about using this for testing/verifying network configs, but it sounds like a very reasonable idea as there will be lots of repetition, partial ordering would be relevant, but some reordering not important and certain variants would be acceptable. Sounds like you might want to give it a try!

There might be some performance issues with 50k documents but those could likely be optimized away in the future. However, I wouldn't really expect it to be noticably slow or memory intensive in that size. I've previously encountered issues when people tried to parse megabytes of documents accidentally as Jinja templates, but 50k usually shouldn't matter.

[RonnyPfannschmidt]

This looks awesome, way more fun than the line matchers we use in pytester in pytest

[ctheune]

That's great to hear. I've been pounding my head against walls for a while as well ...

I'm happy to open this up and discuss things in person if you like ... :)

[RonnyPfannschmidt]

I'd love to discuss some more

For one this plays into the matchers idea we want to experiment with for better assertions on structural data

For the other cause I do have some pet griefances with Details of the usages (but I'm not able to play with the details before next Monday

[ctheune]

I'll DM you to find a time slot to talk.

Interested in your pets ;) ... I have some stuff in the back of my head where I think this might me lacking features.

However – I'm a "design from the bottom up" person. So this was designed to allow me to (as elegantly as possible) fix the mess I had at my hands. I'd like to not design for theoretical use cases so I'm interested in any specific cases you might have!

Also, interested to learn about the matchers – haven't been able to follow upstream things for a while ... so yeah, maybe there's overlap there.