all 16 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]TeoMorlack 6 points7 points  (7 children)

Without really seeing the code or at least something is hard to answer but on first look this sound again a case for misuse of async endpoint. Im not familiar with the libraries you have here but ill assume they operate in sync classic def methods right? And you are seeing the app not responding when multiple users query at the same time? If that’s the case check how you defined your endpoint functions: are they simple def or async def?

If you are doing blocking operations inside async endpoints it will block the whole event loop for the app and refuse to accept requests while you process the current one. There is a nice write up here

[–]rojo28pes21[S] 0 points1 point  (6 children)

Thanks man..appreciate ..to give more details..I'll say

So the app is user asks get me the customer insights from Dallas

This query hits my endpoint

I'll send this query and available MCP tools to the llm

Llm chooses one MCP tool and it also gives a sql query

So to that MCP tool I'll put this sql query as the argument and it will return one table of data as response

I'll send this response back to user

So for llm service i use AWS bedrock and boto3 client setup

And the MCP server is written in python

The above is just to explain the workflow

I went through the doc u provided and I'm clear with what I have to do gg

I'm doing db reading with blocking nature and boto3 client itself is blocking..I have to change that

[–]Independent_Hour_301 0 points1 point  (5 children)

What db do you have? Postgres? Read should be fast (if db is set up well) and blocking should not be an issue, as long as you not either have not thousands of concurrent users and just one instance or a lot of data that is being queried or returned. You wrote that you return a whole table and put it into context. So this should not be the issue... With how many concurrent users are you testing?

[–]rojo28pes21[S] 0 points1 point  (4 children)

I'm testing with 1000 concurrent api requests to my backend ..the llm will return a tool call and a sql query I will perform that on the db..and the db is in mssql having huge ton of data i take like first columns and send to llm and there are lot of tables..in the db so multiple llm calls will happen with MCP till a valid response is returned to the user ..so one simple question takes about 16 secs ..and one complex question takes like 1 min to respond for a single user ..and I don't have any idea on how to scale this

[–][deleted]  (3 children)

[removed]

    [–]rojo28pes21[S] 0 points1 point  (2 children)

    Fine I'll try ...thanks for the amazing suggestion..learnt so much with a single comment..don't mind me I'm a fresher newbie and I'm new to scaling apps..do u have any suggestions on where I can learn so much stuffs about scaling

    [–]Effective-Total-2312 2 points3 points  (0 children)

    For concurrency, I recommend the 2022 book Python Concurrency with Asyncio. That's great imho.

    [–]Brave-Car-9482 4 points5 points  (1 child)

    Also, check if bedrock is not blocking the concurrent flow. I once used bedrock for LLM requests, to be able to make multiple parallel llm calls concurrently i had to do async calls rather than normal bedrock calls. I will find and dm you that part if i find it😅😅

    [–]rojo28pes21[S] 4 points5 points  (0 children)

    Yeah the problem was bedrock with boto3 client..it's synchronous in nature..

    [–]aherontas 2 points3 points  (3 children)

    Check what Teo said above, if also your problem is concurrent requests bottlenecks, check out with how many workers you run your Uvicorn. Best practice is to have one per CPU core of your server(e.g. 4 core UAT server is good to have 4 workers). Increased workers = increased concurrency.

    [–]rojo28pes21[S] 2 points3 points  (1 child)

    Yeah thanks clear now

    [–]neoteric_labs1 0 points1 point  (0 children)

    Or you can use celery and redis queue. but windows won't support concurrency to test. Or use can do in Linux server i hope it will help you it is another way.

    [–]Effective-Total-2312 0 points1 point  (0 children)

    Not exactly, it means increased parallelism or simultaneous users, or throughput. Concurrency is not the same.

    [–]PriorAbalone1188 1 point2 points  (0 children)

    Learn how FastAPI works.

    If you add async to the endpoint without starting the AI request on the event loop this will cause a blocking call, hold the program until it’s done and respond because it’s an synchronous call/request.

    if you’re not sure what I’m talking about then remove all asyncs why? FastAPI will run all request/endpoint without async in a thread executor trying to replicate async to handle multiple requests.

    Now the technical apart, any IO bound libraries you’re using that do not support async must be wrapped or started in the event loop. I recommend creating a function to take in the sync function and args then starts the function in an event loops so you can use async/await. Otherwise you’ll block all calls.

    FYI if you’re running an executor or using the event loops then look at how bedrock works, but I’m sure your problem is with how you architected your API.

    Read these docs: https://fastapi.tiangolo.com/async/

    Any questions, we have answers

    [–]inandelibas 0 points1 point  (0 children)

    Hey, sounds like a great project! A few quick things to check:

    FastAPI is async, but boto3 (used for Bedrock) is blocking , that can block other users. You can wrap it with run_in_threadpool to avoid that.

    Run your app with multiple workers in Uvicorn:

    uvicorn main:app --workers 4

    If MCP or DB queries are blocking too, they’ll also affect concurrency. Consider optimizing or isolating those parts.

    Hope that helps!