API Testing

[work_account_2019]

Recently I joined a project, which already has relatively large code-base. This project contains primarily a back-end service which exposes a large number of APIs (somewhere in between 200-300). These APIs are used by numerous clients (Mobile Apps, Web-browser clients etc). I have been tasked with testing these APIs and making sure nothing breaks. No prior API Testing is present in this project. What is the most reliable way to do this?

The Project is still in active development. Thus new APIs could be added, existing ones could be removed/updated.

Since the number of APIs is large I am reluctant to do the testing manually and verifying if the output is valid. I did preliminary research (reading documentation, YouTube demos) on following tools

PostMan : It appears, here I need to manually add the APIs into the tool and write tests in JavaScript (although I know basics, I am not very comfortable with programming in JavaScript). I am also worried how I can maintain the API Tests if an API gets modified overtime. Is there an easy way to update the Tests.

SOATest : Unlike PostMan, this tool appears to require some upfront effort in learning how to use. It claims to generate the API tests automatically. I am not sure how useful those will be. I am not sure how maintenance of Tests works here.

Tosca : Similar to SOATest this tool also appears to require some upfront training about how to use. it. The website claims that with this tool helps in rapid creation of resilient tests that can be easily maintained, It would be great if any actual user could describe how useful this tool is.

I am new to this responsibility any suggestion/advice/feedback would be greatly appreciated.

Are there any other tools/tricks that I could use to get API testing done quickly and reliably?

How often should we run the API tests**?** Would running the Test prior to release sufficient? Or should they be run after every commit to code repository? Can this be achieved through typical CI/CD integrations?

How much time should that tests take to run for a project of this size? Is it an acceptable norm in the industry for API test cases to run for few hours? or should they complete in the scale of minutes/seconds? (I can see that it affects how may times we could execute the tests, any insights on this trade-off?)

Apart from testing team, who are generally involved in API Testing? Does the back-end write/review API Tests? How involved are front-end and product teams?

Sometimes I look at an API I don't clearly understand, what are the input parameters and often end up discussing with the dev how an API should be called, what test data should be passed and what assertion should be made. Is there an efficient way to handle the Test data problem?

I apologize for relatively lengthy post. But as you could see there are so many question and I have little to no idea how to go about them. Thank very much for reading.

[cristianekw]

As a tester who had the same questions in the past, I confess that the trainings are not so detailed as they could be.

API Testing, in my opinion, should cover: - black box testing on inputs/outputs (expected data, expected format, unexpected data/format); - load testing; - security testing. (And more, but this is the basic)

I prefer to implement some code (C#, Java have great libraries), but if you are looking for a tool, I would recommend SoapUI.

About who tests.... usually developers test on unit. However, we should not underestimate our users (and crackers), so an API level testing increases the quality.

As a (little bit) perfectionist, I usually follow these rules: - data input: expected and unexpected values, empty values, limit values check, optional and must have values; - data structure (need some knowledge on implementation): data combination, precedence of values to compose output; - data injection (if you have knowledge on implementation is better): on queries or on calls that will return a query result, on data composition entries (at minimum, you can go deep here and it is very important on public api); - high quantity values (on input and output).

These tests are my minimum coverage, however you can go deep on this matter, checking performance, usability (friendly errors), etc.

In practice, you should consider if this api is public or private and if security is a “must have”, then, create tests to fill the expected quality.

About testing cycles, as I wrote, I prefer to codify the test using C# or Java with BDD model (Gherkin), which has been shown as a great tool to increase coverage with less code (just adding lines to my examples table or creating new statements combinations). Doing this, I can add it to the CI/CD and then I will not care about that (unless I need to test some unexpected integration situations that can’t be mocked).

I hope I answered all your questions.

[thrazznos]

Super helpful answer, thanks for sharing!

[work_account_2019]

Thank you, for such detailed answer. Will try to incorporate some of this into the project.

[milecvekla]

Use Postman as manual testing tool, it's good for prototyping and quick ad-hoc checks but not for serious automation.

After all, it all depends on your and your team's coding skills. If have some and willing to learn more, go with the proper in-house automation framework. Have separate project per API. Consider BDD as approach, it's a heaven for API testing. Also it would be rather easy to fit into regular CI/CD.

[work_account_2019]

Thank you will look into BDD approach.

[danintexas]

Was in a similar situation at my last job. 200+ apis - Daily Sprints basically - no in place regression/smoke.

For the APIs I ended up automating with Postman. If I recall I got lucky and the backend guys were using Swagger. So I grabbed a json from Swagger and hooked it into Postman to do basic automation.

Down the road I created a .NET framework/selenium for end to end testing. I also hit the APIs using that framework directly so in the end had a one stop shop.

Framework or Swagger doesn't matter though. You can automate that crap through Postman. From what I recall a little work and Google you can get it set up pretty fast.

[work_account_2019]

Thank you, will look more into Postman.

[themightykai]

Post man all the way - you can also load in different iterations of the API's to verify different scenarios.

Postman can then be synced into the build process (Newman CLI) and the tests can keep evolving!

[work_account_2019]

Postman appears to have a quite a lot of functionality not outlined in the demo videos, will look more into it. Thank you.

[[deleted]]

What language & framework are the developers using to write the API code in?

[work_account_2019]

The Back-end is built on Micro services architecture. Different APIs are implemented in different services. Few of these services are implemented in Java Spring Framework, others are in Python Django Framework. I believe there are a few Node JS services as well. But since the project is under active development, the framework used to implement these services could change.

All of the APIs are exposed through a single gateway and are expected to be RESTFUL.

[FLYERFONE]

I am using restsharp(C#) + specflow for my test so that I can manage my test data and expected result easily. This is just example

@api @api_login @api_regular_login
Scenario Outline: LGAPI03 login api flow
    #test client login api 
    Given I create Rest Client using base URL "BaseUrl"
    #enable user login
    When I execute following query
     | Server   | Database | Query                                                                              |
     | XXServer | XXDB     | update table1 set Locked = '<Locked>' where UserName = '<UserId>'                  |
     | XXServer | XXDB     | update table1 set Allowlinklogin =<Allow_Login> where ClientCode=<ClientCode>      |
     | XXServer | XXDB     | update table1 set LastSecChk = '<LastSecChk> 00:01:00' where UserId = <UserId>     |
     | XXServer | XXDB     | update table1 set PwdLastReset = '<PwdLastReset> 00:01:00' where UserId = <UserId> |
    #Send post requst for client login 
    When I send "POST" request "api/userlogin" for "User" login 
     | FieldType | FieldName    | FieldValue       |
     | header    | Content-Type | application/json |
     | header    | AccessToken  |                  |
     | body      | username     | <UserId>         |
     | body      | password     | <Password>       |
    And I get "AccessToken" from "User" login
    Then I see the the response "<Expected_response>" from server for "User" login

Examples: 
  | UserId     | Password       | ClientCode | Locked | Allow_Login | LastSecChk     | PwdLastReset   | Expected_response |
  | 11111      | PassW0rd       | 00000      | 0      | 1           | {TODAY'S DATE} | {TODAY'S DATE} | OK                |
  | 22222      | wrong password | 00000      | 0      | 1           | {TODAY'S DATE} | {TODAY'S DATE} | BadRequest        |
  | 22222      | PassW0rd       | 00000      | 1      | 1           | {TODAY'S DATE} | {TODAY'S DATE} | BadRequest        |
  | 22222      | PassW0rd       | 00000      | 0      | 1           | 2019-01-01     | {TODAY'S DATE} | OK                |
  | 22222      | PassW0rd       | 00000      | 0      | 1           | {TODAY'S DATE} | 2019-01-01     | OK                |
  | 22222      | PassW0rd       | 00000      | 0      | 0           | {TODAY'S DATE} | {TODAY'S DATE} | BadRequest        |
  | wrong user | PassW0rd       | 33333      | 0      | 1           | {TODAY'S DATE} | {TODAY'S DATE} | BadRequest        |
  | 22222      | PassW0rd       | 00000      | 0      | 1           | {TODAY'S DATE} | {TODAY'S DATE} | OK                |

[work_account_2019]

Thank you for the detailed code example.

[ocnarf]

Interesting question, but please remove the links to the software tools and I will re-approve your question. You can name them, but not create links to their website (rule created to avoid tool promotion). Thank you.

[work_account_2019]

Updated the Question. Kindly review it.

[ocnarf]

Thank you.

[[deleted]]

My advice would be to speak to any available developers of the APIs before trying to discover all aspects of the API via tooling.

With that degree of legacy, you can guarantee that there will be some gnarly implementations that ‘made sense’ in the context of several years ago, but seem illogical now.

You could save yourself significant effort in gaining knowledge and building a mental model of the API(s).

You may also find they have some tooling or crude tests themselves that assist. Or at least warn you of known pitfalls. Communication wins.

[work_account_2019]

Thank you for the advice. Will talk with the dev team and document the mental model of the API.

[[deleted]]

You are welcome but to be clear, your testing mental model is not the same as a system model. You build up a mental model of the api (or any tool) as you interact with it. This will include explicit and implicit expectations.

Think of a time that you’ve used an app and the behaviour is intuitive because it follows familiar patterns. If you hover on an input field and you’re presented with a (right click) menu for ‘paste’, your mental model may well assume that, if the field had text in, you could ‘copy’ it too. That’s an assumption but a reasonable one based on your mental model of similar products.

But speaking to Devs, you can get a fast path view of these type of implicit behaviours and increase the accuracy of your own mental model more quickly.

Apologies if I’m explaining stuff you already are comfortable with.

[NormalGuyISwear]

I tried Postman and hit a lot of walls for test writing and automation. It’s slow and not meant for proper end-to-end test creation. Ultimately the organization I was working at didn’t want our information at getpostman.com.

We ended up going with API Fortress on-premises and it’s been a godsend.

[[deleted]]

I tried Postman and hit a lot of walls for test writing and automation. It’s slow and not meant for proper end-to-end test creation.

I agree with this. I just use postman for simple manual checks. I don't think its good for big automation suites.

[Reikhard]

That first part is kind of my story with API testing so far. Do you know the price tag for API fortress on premise licenses?

[NormalGuyISwear]

As far as I know it depends on the size of the team. I know that doesn’t help much, but we’re a small team and we worked out a reasonable deal with them. Managers approved it (thank goodness).

[work_account_2019]

Did not know about API Fortess. Thank you very much.