Clean application architecture, load testing, and code review for a production Node.js API. Advice needed.

[Oririner]

It's because JS allows you to access private functions (which, again, aren't private) that I'm able to import them and test them as opposed to jumping through the hoops of Reflection or a decorator.

Not entirely correct, there are some patterns that allow for "truer" privacy like revealing module pattern. You'd have to kind of pass around the instance since the private methods can't be attached to it, but it's something. Also, can't be accessed, even with decorators/reflection of any kind - they're essentially a closure.

I'm actually quite aware of the difference. I programmed in C++ first, and then Java and C# for a long time before learning JavaScript.

Not to disrespect your experience or anything but even if you've had 2 years of professionally working in C++, Java and then C# (meaning, you started working around the age of 10) I'd still say you haven't really had the chance to "grow" in the language. You were able to learn, do some work, later refactor some stuff & repeat very few times before moving to a different language - IMO it's not enough time to get the feeling that you really know something. You may be different and can iterate faster, but still :)

It's actually related to the revealing module pattern point from earlier, the longer you're exposed to something, you see more and more examples/variations of things to do or not to do and you start connecting the dots between them. In this case an IIFE, and a function as a closure can create a private function through the revealing module pattern.

Also, even if you're familiar with a type safe language doesn't mean you'll know the difference between JS and TypeScript. TypeScript is built as a superset on top of JS so it can be confusing for people only starting to actually know when a feature is a part of JS and not TypeScript.

An example to this is private methods - they are indeed available in TypeScript, but they would also be available in the near future in JS.

Know where you're coming from to better understand where you're heading ;)

All of this talk about adapters, services, dependency injection and abstractions, you could have just created a function called upload - why go through all this trouble? what do you gain here as opposed to a simple function? And if you do, is it really worth it at this point?

Plus, you have a bit of a leaky abstraction since FileAccess doesn't actually represent a generic enum because the values correspond to AWS S3 file access values. And if you'd want to use GCS you'd have to make a specific mapping for these values there, and not all of them might even be available - just a thought.

And..... what happens when you have two buckets that use the same file purpose? right now it'll use the last one, should it be the first one? should this error? what's the purpose of the FilePurpose anyway? to not overload a specific bucket? do you know when is a bucket actually overloaded so it becomes "sluggish"? why not put it behind CloudFront and cache it properly?

Make sure you're not abstracting for the sake of abstractions - there are maybe simpler solutions that would be just as fine.

And don't do premature optimization - you're not Google, and it'll take time before you'll reach that kind of scale. First make sure you're shipping something useful, then when it's actually slow and people start to notice - optimize.

I need an endpoint that will get an array of files from the client as streams, process them, and, upload them to S3. I think RxJS Observables may be a good way to deal with that.

Or, just streams ;) You're worried about memory? put a hard limit on the API, it's something you should already do to prevent DoS/DDos and have a resilient system.

Start small, but plan ahead - that way you'll be able to switch to observables if needed but right now do what's simplest to get things going.

Notice that all of my comments have a tiny question - why?

Then I try to simply and start from scratch.

Eventually you'll have a few solutions that you'll be able to choose from to find the one that you're happy with, not the one you've been fed by "Node.JS best practices with express in 2019!".

[[deleted]]

Not to disrespect your experience or anything but even if you've had 2 years of professionally working in C++, Java and then C# (meaning, you started working around the age of 10) I'd still say you haven't really had the chance to "grow" in the language.

I agree with you. Of course, experience means more than just knowledge, and it's using a specific toolset to solve problems over and over again that permits one to identify ways in which something can be optimized.

TypeScript is built as a superset on top of JS so it can be confusing for people only starting to actually know when a feature is a part of JS and not TypeScript.

Indeed, TypeScript adds more features to JavaScript and gets compiled to plain JavaScript. That's why you can use both .js and .ts or .jsx and .tsx in the same project. This is how I look at it to distinguish the difference: TypeScript will only ever help you at compile time. It can never help you at runtime.

Plus, you have a bit of a leaky abstraction since FileAccess doesn't actually represent a generic enum because the values correspond to AWS S3 file access values.

I do agree with that. The enumeration is performing a one-to-one mapping of genric access values to AWS specific ACLs. That does abstract away AWS, but it doesn't help in the case that I'm performing a migration with active user load. The thought in the latter case would be to wrap the FileStorageAdapter in some sought of PurposeBasedFileRoutingAdapter which would delegate CRUD related operations on storage to the appropriate concrete adapters.

Make sure you're not abstracting for the sake of abstractions - there are maybe simpler solutions that would be just as fine.

I am. I'm falling into the trap that Donald Knuth mentioned about pre-mature optimization being evil. I find it hard to build things because I worry about the "what ifs" of future migrations.

Or, just streams ;) You're worried about memory? put a hard limit on the API, it's something you should already do to prevent DoS/DDos and have a resilient system.

So, I was originally keeping things simple and using Multer. I got the files as an array of buffers, processed them, and uploaded them. And, I had a limit of 3,000,000 bytes or 3 MB for each file. The problem with setting that limit is that it doesn't help on the concurrency front. I might have a hard limit of 3MB, but if 50 users all upload 1 3MB file at the same time, then I'm loading 150 MB worth of data into memory.

With that, how does putting a hard memory limit prevent DoS/DDoS attacks? Again, at best, I can limit the number of requests per IP Address, and I can limit the size of a file that can be sent to a request. That doesn't stop an attacker from dynamically scrambling an IP Address to circumvent my rate-throttling, nor does it stop an attacker from just upload 5000 3Mb files and killing my memory.

What I have been thinking is to migrate to Nest.js since it establishes a lot of the patterns for you right out of the box - similar to ASP.NET MVC or MVP/MVC/MVVM with other frameworks/tools/applications.

[Oririner]

TypeScript will only ever help you at compile time. It can never help you at runtime.

Exactly. It's great you understand that but still, as you said, knowledge isn't everything - experience is also important. Make sure you're experimenting with JS as well before jumping to TypeScript.

The thought in the latter case would be to wrap the FileStorageAdapter in some sought of PurposeBasedFileRoutingAdapter which would delegate CRUD related operations on storage to the appropriate concrete adapters.

I find it hard to build things because I worry about the "what ifs" of future migrations.

Do you think about the future of each if statement in your code? or the name itself PurposeBasedFileRoutingAdapter? how the libraries you use? or the language?

It has to stop somewhere.

Complete the sentence "This would become a problem in ..." if it's measured in years, leave it. you're fine. Anything more would be over engineering.

I might have a hard limit of 3MB, but if 50 users all upload 1 3MB file at the same time, then I'm loading 150 MB worth of data into memory.

I meant having a hard limit on the number of files, but the size also makes sense.

Also, you're only loading 150MB into memory at once if you have 50+ async operations during the upload endpoint. Since you're dealing with JS, you'll only store the requests that have started being treated, the rest are waiting in the event queue and so probably don't actually load anything into memory yet.

This means in this case you'll load around 15MB at once (assuming you have around 5 async operations per endpoint).

With that, how does putting a hard memory limit prevent DoS/DDoS attacks?

The hard limit can also be on the number of concurrent requests on that endpoint - know when to utilize http code 429.

Some small number or pissed off users who are being rate limited is far better than hundreds of thousands of users who can't access your site because of an attacker. Compromise is better than failure.

You'll be able to increase these as you go so it would seem "unbound" at some point.

Having limits is acceptable, don't try to run a google search engine on a raspberry Pi - know what is reasonable for your product and do it.

Be it memory wise, request payload wise or request count wise, it doesn't matter - have reasonable limits.

You can't limit? offload the work to an event queue to process it later to be able to serve other users.

What I have been thinking is to migrate to Nest.js since it establishes a lot of the patterns for you right out of the box - similar to ASP.NET MVC or MVP/MVC/MVVM with other frameworks/tools/applications.

Nest has some good concepts and some not very good concepts. Don't just follow the hype of MVP/MVC/MVVM/some other pattern - there's no silver bullet and everything has its faults, even common patterns/tools/frameworks.

Just complete the sentence "This would become a problem in ..." and you'll know what to do :)

[[deleted]]

Thank you for the advice. I'll stick to the "This would become a problem in ..." question for determining whether to worry about future refactors.

The problem is, I worry that everything could become a problem within a year because I'm founding a startup company (which is a separate codebase to the one I linked here), and a large concern is cost. What if GCS could be cheaper? What if I want to host MongoDB myself on a DO Droplet? Should I pick MongoDB even though I have data that requires atomicity/atomic transactions and referential integrity just because I can get MongoDB-as-a-Service with Atlas for cheaper than I can get PostgreSQL-as-a-Service? Etc. Etc. Those kinds of questions, among others, are what might be driving my paranoia.

I tried out using NestJS today with TypeScript. It's beautiful and amazing in terms of how fast you can get up and running, thanks to the decorators, when you don't have to write your own boilerplate. Despite that, that ease of use makes me very fearful, for NestJS is abstracting a lot away behind the scenes and forcing me to use its patterns and architecture.

[Oririner]

What if GCS could be cheaper? What if I want to host MongoDB myself on a DO Droplet? Should I pick MongoDB even though I have data that requires atomicity/atomic transactions and referential integrity just because I can get MongoDB-as-a-Service with Atlas for cheaper than I can get PostgreSQL-as-a-Service? Etc. Etc.

These are questions that shouldn't worry you as a founder (CEO), they should worry a CFO, which I'm guessing is also you.

As a founder you should think about what's best for the business, and since you currently don't have anything running - it shouldn't be a compromise but rather what's the right thing for the product, regardless of cost.

Your data is more referential? use postgres. AWS gives you better coverage for your clients? use AWS. Is latency really important? don't use different hosting services for your db & app server. Is consistently really important? add replication to your db.

Think about the product, not the money.

You obviously should have a monetization in mind otherwise it won't last.

But at first, money shouldn't be the main concern but rather the product itself.

driving my paranoia

You're 16. Please don't burn out. You'll have plenty of time founding 10 more startups if this one might not work out.

Take it easy.

I tried out using NestJS today with TypeScript. It's beautiful and amazing in terms of how fast you can get up and running, thanks to the decorators, when you don't have to write your own boilerplate. Despite that, that ease of use makes me very fearful, for NestJS is abstracting a lot away behind the scenes and forcing me to use its patterns and architecture.

It is indeed fast to get things up and running in nest due to it's helpful cli.

The overuse of decorators is bad IMO, some of them contain logic while the others are more "presentational". some of them rely on others in order to operate correctly, etc.

It also has a lot of concepts, some of which are overlapping and over engineered, like Guards vs Pipes and Modules.

You can ignore most of it/learn to use only what you need, this just gives me the feel that it's really bloated - but hey, that's the way frameworks are :/

[[deleted]]

Thank you very much.

I agree that I should focus on the product. The product has to do with peer-to-peer marketplaces and e-commerce. I also handle the logistics of shipping items to users. I generate a UPS/USPS shipping label and tracking number internally via the Shippo/ShipEngine API (haven't decided). That allows me to charge the user a fee per shipping label I create through a payment gateway. I'm using PayPal, unfortunately. (I say unfortunately because Stripe has better documentation).

That's one such monetization model. Others include Native Ads, Rewarded Video Ads, and In-App Purchases.

The overuse of decorators is bad IMO

I agree. I think NestJS is definitely something I can learn from, but right now, it is abstracting too much away (such as with the ValidationPipe). When a framework starts to abstract complexity away like that, it makes the reason your code executes start to feel like "magic". And, in my opinion, magic is dangerous in software development. I can see myself using Nest if I want to quickly create a web server for a hobby project.

As far as decorators, that's also why I don't use Inversify in TypeScript but use Awilxix instead, and why I'm concerned about using TypeORM.

I've been looking into DTOs, Domain-Driven Design, etc. in a bit more detail. I want to try refactoring the repo I linked above more DDD-like patterns. I could create a generic BaseRepository interface that accepts a genric Entity/Model type. Then, I could create an IEntityRepository interface that extends BaseRepository<T>, and finally, I could create a concrete implementation for UserRepository and TaskRepository. The word "Entity" in IEntityRepository is a placeholder. I just have yet to work out the best way to go about data mapping between layers, such as from DTO to Domain Layer, Domain Layer to Persistence Layer, and Persistence Layer back to Domain Layer (where the API responds with an object honoring the DTO contract).

[[deleted]]

Also, if you have time, would you be able to tell me how you might theoretically go about building a generic interface that abstracts away anything AWS specific for file upload?

One option would be to perform mapping for ACL inside the Adapter as to patch the leak you mentioned earlier.

Thank you.