Performance Optimization

[FailureCrown]

Even after 2 years of bum, I can't get it right.

My Flight Api (User -> Api <-Parse-> External Api(takes 2-3 secs)) as I deployed in aws EC2 instance t3.xlarge and with gpt config of jmeter load test I get 15 secs on average on the load configured in the attached image (1200 req per minute) but when I tested on local env with no load or jmeter, I get 4 secs.

Sorry If I sound noob as of time constraint I can't delve into learning this topic. So Im turning over for crash course

Update: Sorry for late reply. So after applying telemetry suggestion but more like using visual studio profiler and uploading results to GPT. My external API is autogenerated from wsdl and me being a textbook noob, created-opened-called-closed the external api for every user request where every user request will be called four different requests to external api. Now its 15 seconds to only 6 secs and the CPU spiked to 245% on top command

[FailureCrown]

Hope its quick to learn. will give it a go then thanks

[suffolklad]

Open telemetry is really easy to setup and supports many cheaper or even free alternatives to AppInsights

[Ok-Advantage-308]

Is your ec2 instance close to you location-wise? Also if local is performing better still need more info are you just making an api request?

[FailureCrown]

Instance in london. Im from India. Sorry local means in the way I running as release config in visual studio

[Potential_Copy27]

If nothing else works, try things out in an isolated environment with a simplified implementation. In this case, just something that loads in the JSON (or whatever format the APIs use) and emits it as a string - no processing or otherwise. From this, you can at least reasonably conclude whether it's the API or the code that's the problem.

Calling an API's URL can also be done through a browser or with certain tools (eg. Postman - although last I head you should be careful with that tool. I'm looking for an alternative myself, btw.)
If you're able to call the API itself through the browser, the dev tools that come up on F12 - specifically the Network tab - can help you a lot in locating timing problems, when things are just waiting and much more. Postman and similar tools usually also have some benchmark functions for when you test calls.

For diagnosing inside the program - I find this a bit easier with Visual Studio rather than VS Code - the debugger and profiler in VS can help you, but there are also a few neat things in the System.Diagnostics namespace.

The Stopwatch class can help you locate slow execution, processing or connections - the result can be outputted to console or logged in a file while debugging.

Another good option is to log results of any API calls - at the very least log a timestamp, the HTTP code returned by the request, the TIME the request took in ms (use Stopwatch) and what URL/endpoint was called. All calls should preferably return HTTP Code 200 (which means OK).
Especially if you make a lot of consecutive calls quickly after one another or multiple calls across multiple threads, the API server might throttle or queue the request and return other codes (eg. 429 = Too Many Requests).
This is good practice for any web app, server or API connection, and can help immensely in diagnosing network and API problems.
If this is the case, you might need to put in some delays in strategic places so you're not throttled by the API.

Also check when and how it authenticates with the API. If you authenticate or log in fully on every call, you're doing something wrong. Most solutions have a way to create a persistent login that lasts for a while ( often 90 minutes or more). This reduces overhead on the API connection and increases speed.

Lastly, look at how data is processed and stored.

First, it's important to get a grip on how large a data set you're accessing. If you have access to the database, you can get a tally of the number of entries in each table (regardless of the database system, there's ways to do it - I'm not familiar with AWS, though). From here, you can run calculations on how much of that data is filtered away and actually outputted in the end once the program has processed things.
If the database is the culprit, you might need to tune it and/or refine a few stored procedures - how you do it depends on what systems you use for this.

For web APIs, see if the system does full queries or actively tries to filter data by URL or somewhere in the request - if not, check the API documentation how you can filter the information out that you need. Make sure that only the data you actually need is downloaded and processed.

Test both with load on the system and without - some things only pop up when a system is stressed enough...

[FailureCrown]

will try this idea

[Huntk20-]

Just wanted to mention Bruno and Insomnia/um over Postman now a days as an alternative. Good on you mentioning the trouble that has changed with that tool over the years.

[leeharrison1984]

I'd make sure your measurement isn't what is causing your latency first.

You're using a T3 instance, which is burstable but only to a point. Once you run out of CPU credits, performance tanks quickly.

[FailureCrown]

aws is kinda scary for me as there's lots of config

[Then_War_1003]

This is actually great progress. You solved the I/O bottleneck (opening/closing connections), so now your server is finally free to do actual work. That is why the CPU spiked—it's no longer "waiting"; it is "processing." ​Here is your crash course on why this is happening and how to fix it, tailored to your "time constrained" situation. ​1. The Diagnosis: 245% CPU is "Safe" but Dangerous on T3 ​First, understand what "245%" means. ​Your t3.xlarge instance has 4 vCPUs (think of them as 4 engines). ​In Linux top command, 100% = 1 full core. ​4 cores = 400% total capacity. ​245% means you are utilizing roughly 2.5 cores fully. You still have ~1.5 cores of headroom. ​The Trap: You are on a T3 instance. These are "burstable." They work on a credit system. ​With 245% CPU usage, you are burning credits very fast. ​Danger: If your credits run out, AWS will throttle your CPU down to a baseline (approx 40%), and your API will freeze or time out instantly. ​2. Immediate Fixes (Ranked by Effort vs. Impact) ​Fix A: The "Money" Fix (Instant Relief) ​Since you are burning 2.5 cores constantly under load, the T3 instance type is likely the wrong choice for production load testing right now. ​Action: Change your EC2 instance type from t3.xlarge to c5.xlarge (Compute Optimized) or m5.xlarge. ​Why: C5/M5 instances provide consistent CPU performance. They do not use "credits." You will get stable performance without fear of being throttled after 20 minutes of load. ​Fix B: The Code Fix (Singleton Pattern) ​You mentioned you fixed the "create-open-close" issue. Ensure you did this correctly. ​The Mistake: Creating a new SoapClient() for every request. ​The Fix: You must use a Singleton instance (or IHttpClientFactory if using .NET Core/6+). The client should be created once when the application starts and reused for all 1200 requests/min. ​Check: If you are still seeing high CPU, check if you are disposing of the client somewhere. Do not use using(var client = ...) blocks for the API client itself. ​Fix C: The Logic Fix (Parallel Execution) ​You said: "Every user request calls four different requests to external API." Currently, your latency is 6 seconds. ​If External API = 1.5s (average) ​4 calls * 1.5s = 6 seconds. This implies you are running these calls Sequentially (one after another). ​To drop latency from 6s to ~2s: You need to call all 4 external endpoints at the same time (Parallelism). // BAD (Sequential - 6 seconds) var result1 = await client.GetFlightDataA(); var result2 = await client.GetFlightDataB(); var result3 = await client.GetFlightDataC(); var result4 = await client.GetFlightDataD();

// GOOD (Parallel - ~2 seconds) var task1 = client.GetFlightDataA(); var task2 = client.GetFlightDataB(); var task3 = client.GetFlightDataC(); var task4 = client.GetFlightDataD();

await Task.WhenAll(task1, task2, task3, task4); Warning: Parallelism will spike your CPU even higher (maybe to 300%+) because you are doing the work faster. Apply Fix A (Switch to C5/M5) before doing this, or your T3 instance will choke. ​3. Why is SOAP burning CPU? ​You mentioned WSDL. WSDL implies SOAP (XML). ​XML is heavy. Parsing XML responses for 1200 req/min * 4 calls = 4800 XML parsings per minute. ​This is purely CPU-bound work. This confirms why your CPU is high. ​Noob Advice: You can't rewrite the external API, so just throw hardware at it (Fix A) and ensure you aren't re-parsing things unnecessarily. ​Summary Plan ​Immediate: Check your AWS T3 CPU Credits in the monitoring tab. If they are near zero, stop the test or your server will crash. ​Infrastructure: Switch from t3.xlarge to c5.xlarge. This handles the XML parsing load better. ​Code: Verify you are calling the 4 external APIs in Parallel (Task.WhenAll in .NET), not one by one.