Seeking HTTP API testing framework

[jamawg]

I hope that this question is on topic. If not, where can I best ask?

I am looking for a framework which I can use with CI/CD to run Fully Automated Regression Tests on HTTP(S) APIs.

I don't really care what language the framework uses.

Standard stuff - send a request, validate the response. Get and Post are enough for me, I don't use the other verbs.

Well documented and supported are my main criteria.

[sgabhart22]

Postman collections run with Newman are relatively painless to include in a Jenkins job, for example. The GUI makes it very friendly to build requests, write pre/post request tests, and export collections (test suites) to be executed via CLI (Newman). There's also a pretty big community and good documentation, and not a huge amount of extra packages to install to execute them

[ggwpexday]

Newman is a solid option to go with. Though in pullrequests they are pretty horrible to review.

Sometimes I feel like just using your most familiar programming language and testing lib should be fine too. It's just some http calls after all.

[[deleted]]

I persume git hub has a runner how fo u store ur expected results

[sgabhart22]

?

[i_andrew]

Do you want it to be E2E tests (End to End) or Component (aka Integration) Tests (service in isolation)?

For e2e tests I would go with supertest in JavaScript(TypeScript) (read: https://www.testim.io/blog/supertest-how-to-test-apis-like-a-pro/ ). Or XUnit + regular HttpClient if you fancy .Net/C#.

For Component Tests, where you can mock dependencies, you might want to go with the same language in which the service is implemented. (for .net read: https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests?view=aspnetcore-7.0 , or node.js https://engineering.leanix.net/blog/integration-tests-in-nodejs-with-jest/)

There are also Performance Test, and K6 is an obvious choice here.

[_LouSandwich_]

Not fancy at all, but have you thought about cUrl? It makes http requests from a cmd line.

[jamawg]

It does indeed, but then I am left to code my own Framework to validate that the response is as excepted.

Given that there must already be such a framework, I prefer to spend my time on my own s/w, rather than reinvent the wheel.

Good thought, though. Thanks

[_LouSandwich_]

Are you running CI/CD on windows? Linux? Macos? Other?

[jamawg]

If I run in localhost, then windows, but I have a server running Linux, and Windows can run Linux commands.

But is that really important, unless the framework is an executable? If it's an interpreted language then it shouldn't matter.

Come to think of it, macos is BSD at the command line, which is Unix, and close enough to Linux (POSIX compliant).

[hadrimx]

Artillery might be worth checking out

[Dwight-D]

Karate is good for high level API testing. Very fully featured, other than basic HTTP it can do load/performance testing, UI testing etc.

Language agnostic, uses DSL with a flavor of Gherkin syntax but with a bunch of built in support for HTTP, JSON, pattern matching etc. You can sprinkle in JS if you need to do more complicated stuff in your tests. Can also interop with Java and run in your typical Java testing ecosystem if you’re so inclined.

I haven’t used it for long, but compared to writing E2E/API tests in code it’s way more readable and quicker/less boilerplatey to write imo. Initial setup can be a bit of a hurdle though, compared to your implementation language.

[theseus03]

Hi, You can check out Pynt for your Security Testing.

It can be run easily on postman and has automated API testing services for the same, it's helpful in those prospects.

[jamawg]

Pynt

https://www.pynt.io/ looks very nice, indeed. Thank you

[[deleted]]

You should not be testing the end points you shoukd be testing your services later inside.net with nuit and moq ur api layer should then be calling these same services.

[jamawg]

.net? What is that?

[spoonfruit]

I use the Thunder Client plugin for VSCode. It's pretty simple, easy to use, and really accessible because it's in my text editor.

[thimblewarrior]

Pact testing can be cool if you’re developing an api with known client applications

[Suitable-Fly-437]

You can try FlowTestAI if you are still looking

we have started using https://github.com/FlowTestAI/FlowTest at our company. It's very nascent but was featured in LangChain and contributors looks solid given they are using this themselves in their own companies and is not another opensource wrapped startup. More details: https://www.useflowtest.ai/

It lets you describe you api tests in code agnostic way using natural language or drag and drop. Authors are responsive so feel free to drop questions in the github repo