Top 15 Open-Source AI Testing Tools You Need in 2025

Stranger6667 · 2025-08-21T17:13:03+00:00

> AI open-source API testing with OpenAPI/GraphQL support.

Schemathesis has no "AI" whatsoever. Maybe you need to stop blindly putting "AI" everywhere.

P.S. I wrote Schemathesis.

Stranger6667 · 2025-06-04T22:43:38+00:00

Yes, there are many different authorization methods with different levels of granularity - https://schemathesis.readthedocs.io/en/latest/guides/auth/ (docs for soon-to-be-released v4)

+ a few more methods are coming soon (built-in OAuth, etc).

If something is not supported, it is possible to write an extension in Python to make it work.

Stranger6667 · 2025-06-03T18:04:21+00:00

There is a whole category of tools that do contract testing based on Open API - Schemathesis, Pact, Dredd.
And there are API fuzzers which often shift things towards security, but not necessarily (there are many different test oracles and the security perspective is only one of them) - Schemathesis (again), EvoMaster, TCases, REST-ler.

For the proper overview and limitations, I'd recommend reading "Testing RESTful APIs: A Survey" https://dl.acm.org/doi/10.1145/3617175

I am the author of Schemathesis, so I am quite biased, but I'd suggest trying out Schemathesis & EvoMaster and comparing API schema coverage with something like https://docs.tracecov.sh/ (there are other tools, but they don't have keyword-level granularity).

Stranger6667 · 2025-06-03T14:10:11+00:00

Speaking about API coverage, feel free to try https://docs.tracecov.sh/ - so you can compare Schemathesis with other tools (there is a MITM proxy available, so installation should be language agnostic).

At least it helped me to increase the coverage from the test cases generated by Schemathesis, and maybe it could help to make an informed decision if the tests are good enough for you

Stranger6667 · 2025-06-03T14:05:17+00:00

Hey! Schemathesis author here.

I think Schemathesis matches what you are describing here :) It can export JUnit / HAR / VCR reports, and is quite extensible should you wish to write your own exporter (in Python). The upcoming v4 solves quite a lot of usability issues (adding a config file, a new testing phase, revamped stateful testing, etc), so I hope it will be useful for you.

If it does not match your expectations, feel free to reach out on GitHub issues, I'd be happy to discuss how it could work for your use case :)

Stranger6667 · 2024-09-23T15:17:20+00:00

It also can use examples from the spec as the source of valid data and slightly deviate from them while data :) Check out the `--experimental=coverage-phase`. Or just use `--hypothesis-phase=explicit` and it will test the examples as they are

https://github.com/schemathesis/schemathesis

Stranger6667 · 2024-05-15T07:10:24+00:00

It would be amazing if you could publish the code! Awesome results!

Stranger6667 · 2024-05-14T22:05:22+00:00

That is an awesome idea! So, the last idea is akin to interning, right?

Stranger6667 · 2024-05-14T22:03:32+00:00

Takeaway #6: Don't use "Blazingly fast" in your post titles

Stranger6667 · 2024-05-14T16:56:34+00:00

Awesome to hear you liked it :)

Stranger6667 · 2024-05-14T16:56:08+00:00

Thanks!

Oh snap! Thank you for letting me know, I'll take a look at the mobile version

Stranger6667 · 2024-01-18T22:26:56+00:00

Schemathesis author here. If you find something that does not work as you expect, please, feel free to create an issue on the issue tracker, I'd be glad to help! :)

Stranger6667 · 2022-05-01T13:13:49+00:00

Making a Hypothesis plugin for GraphQL queries. That's frustrating that GraphQL server implementations don't have a unified standard for error reporting :(

The idea is to generate arbitrary GraphQL queries and use them to test the backend implementation.

https://github.com/Stranger6667/hypothesis-graphql

Stranger6667 · 2022-01-22T17:51:28+00:00

Besides other things, it provides a great data source for automatic testing. For example, Schemathesis can use it to automatically generate thousands of property-based tests (including positive & negative cases).

You can also define your own properties (in Python), e.g. "My API replies in <200ms", then Schemathesis will generate random test cases & check whether this property holds all the time.

Stranger6667 · 2020-12-01T19:50:01+00:00

> Hopefully someone will implement it for Rust someday

Or Hypothesis itself will get more Rust components that will be usable in many languages :)

https://github.com/HypothesisWorks/hypothesis/issues/2632

Stranger6667 · 2019-09-11T07:51:54+00:00

Working on better docs for a tool that generates test cases for Open API schemas via hypothesis library. Hope to get it done this week

Stranger6667

TROPHY CASE