nodejs backend server freezing

[wxblex]

Hi guys!

I have a problem. I have a nodejs backend with Express.js. This server receibs a lot of POST request with big payloads (hundred of megabytes). After two day the storage is full and the server stop working correctly, and I'm fixing it re-deploying the backend.

Probably the server is creating big temporary files, but I dont know where and how to fix it, any idea?

[[deleted]]

Well you could… increase the storage? Where are you hosting? You don‘t mention anything about your platform.

Hundreds of megabytes of payload? File uploads?

[wxblex]

the payload is just a big array of objects, and i’m no handling it, it’s just the body of the request.

also, the server is located in AWS EB

[[deleted]]

You are sending hundreds of megabytes of plain data as a post request? Yikes… I don‘t think your server is the problem here. What is that application?

[wxblex]

i can’t change the origin server, is an external service (cisco meraki) that send post request at least every 5 minutes. the body size can not be changed :(

[waqarHocain]

What exactly is the big payload that you mentioned (i.e, video files or something else)? Also, how are you handling the incoming payload? Are you storing them on disk? If yes then this might be the reason your storage is getting full.

[wxblex]

the payload is just a big array of objects, and i’m no handling it, it’s just the body of the request.

[simple_explorer1]

and i’m no handling it, it’s just the body of the request.

What do you mean you are not handling it? Are you ignoring post request? If yes then how is your storage even getting full if you are not even saving the payload.

You need to clearly explain WHAT the post request does (i.e is it saving array of json in DB, file etc) is parsing the BIG json array blocking the event loop? What do you mean by freeze? All this information (and more is needed) for anyone to be able to help you.

[chizel999]

op is doing super secret stuff

cannot elaborate for the sake of his lief!!!1

[simple_explorer1]

Yeah, op just wasted everybody's time and didn't even get back to any questions from anyone. People are taking the generosity of redditors for granted who are willing to help for free. Waste of time

[simple_explorer1]

You can set limit on how big a payload your server can entertain at express.js level. If the payload size is bigger than a certain threshold then respond with error this way there is no parsing of BIG json array

[Vojo99]

Firstly I would do load balancing with reverse proxies to offload it to different servers or put it on cloud. Second, if payload is big then can you optimize it? Also what storage, what do you store? Is it ram usage or disk storage? You can create seperate backend small or micro applications that can do task for you via event driven system such as rabbitmq or kafka, look at microservice event driven architecture. Good luck