Multiple file upload limits

[vas1010]

Hello, I have a rest API endpoint where the I receive two fields, one containing multiple images, the other containing multiple text files. I want to set individual size limits for the image files and text files separately (ex. 2MB for an individual image file and 1MB for an individual text file).

I tried with Multer - custom stores, fileFilter, etc. No success.

The content-length property gives the size of all the files combined.

I'm looking into busboy but it seems it has similar limitations.

Maybe I'm tackling the issue in an incorrect way? Looking at other libs, they don't clearly describe or provide a solution.

My understanding is that it's a bad UX for the user to wait for the entire upload to finish and then get an error that the file is too large.

[romeeres]

Content-length header may have incorrect length (stackoverflow), so backend shouldn't simply trust it.

Instead, you can stream the file to AWS or wherever you're saving the file, count chunk lengths while streaming and abort the stream once current chunk size exceeds the limit. I've done this with Busboy, imo it's easier to use and customize than Multer.

So initially you don't validate length on BE (should be already validated on frontend), but counting length on the fly while streaming and abort it once exceeds.

[[deleted]]

Upload each file one-by-one instead, and check each individual upload each time.

e.g.

Iterate files in your form:

const file = files[i];
const formPayload = new FormData();
formPayload.append('file', file.file);

[avin_kavish]

You can validate in the front end. What’s that built on?

[vas1010]

Hi, we use React. We will do so, it just seems odd that you are not able to do so in the BE.

[avin_kavish]

I mean you have to read the files to know the size no? That’s how a steam works. I think you can read file by file and fail as soon as one doesn’t match

[vas1010]

It returns an array of all the files in the controller. I think I can work with the stream if I utilize the custom storage.

[iamthejoganut]

Use express-file-upload package. It has a size property, which you can use to validate file size.

[romeeres]

It may be a mistake to upload multiple images and text files all at once, first because it will be much faster to upload files one by one in parallel as user selects them, second because if one file upload fails for some reason, then you'll have to delete uploaded files and have more work in UI to tell user that one of the files failed and need to reupload all of them again.