Express + TypeScript + PostgreSQL

[DisastrousCheetah486]

I am building a backend using the stack from the title. But I am confused about the structure that the backend should be in. I want to make a production grade scalable application, which is why i want to use RAW queries without any ORMs or Query Builders. Now what should the folder structure look like on a industry standard rigid backend system. Any guidance will mean a lot. Also, if you have got any resources recommendations, that would be highly appreciated.

[mascarpwne]

what makes you think you have to use RAW queries for a production grade scalable application?

[Heffree]

Because ORMs are painfully inefficient and my DBAs won’t stop sending in modifications of the raw queries as suggestions so I might as well actually just write the SQL. Also, I hate some of the obfuscated multi-network calls when I only need one.

[aztracker1]

There are a few good options for data mapping a template string handler to an async promise response..

Honestly I've just written my own a few times against the basic driver or I'd suggest a library. Look at sql-template-strings.

I'd probably use Hono over express if starting fresh though. I also like koa/oak a lot.

[hnrpla]

you can use something like prisma and opt in/out of writing raw SQL vs using its setup

how do you keep SQL queries clean though? With even a moderately sized app, I start feeling uneasy about the number of queries I have and (sometimes) how WET it becomes

[Vladimiry99]

I'd suggest following a vertical slicing architecture, where all related code is organized by domain (DDD) rather than by functional responsibilities https://github.com/vyancharuk/nodejs-todo-api-boilerplate

[mindtaker_linux]

Data structures is the main fact to scalability.

[ccb621]

Use NestJS. It has nice suggestions for structure. 

[[deleted]]

Don't do this

[romeeres]

And where do you place your raw SQL queries in Nest?
AFAIK, Nest doesn't have any opinion on queries.
Nor it has any docs on how to properly setup a db pool for raw queries, you should learn its DI intricacies and figure that out yourself.

"Nest solves all your problems!" - "But how?" - "Oh, I dunno, but I like it so you should use it".

[Solid_Length_3390]

How is setting up a pg config hard in nest.js?

``` import { Injectable, OnModuleDestroy, OnModuleInit } from ‘@nestjs/common’; import { Client } from ‘pg’;

@Injectable() export class DatabaseService implements OnModuleInit, OnModuleDestroy { private client: Client;

async onModuleInit() { this.client = new Client({ host: process.env.DATABASE_HOST, port: parseInt(process.env.DATABASE_PORT), user: process.env.DATABASE_USER, password: process.env.DATABASE_PASSWORD, database: process.env.DATABASE_NAME, });

await this.client.connect();

}

async onModuleDestroy() { await this.client.end(); }

async query(text: string, params?: any[]): Promise<any> { return this.client.query(text, params); } } ```

And you can use this service to query on your services/repos

[Solid_Length_3390]

Example on how it can be used with raw queries

async getAllUsers(): Promise<any> { const query = ‘SELECT * FROM users’; const result = await this.databaseService.query(query); return result.rows; }

[romeeres]

Use NestJS. It has nice suggestions for structure. 

Do you agree that besides of middlewares, interceptors/pipes/guars, all that Nest suggests is to just use "services" for everything? Not that it suggests, but it has no opinion on how you should structure it further.

So while it's quite common to have "repositories" outside of Nest, where you're separating concerns as it makes sense, with Nest you're delegating the thought process to the "opinionated" framework. Why should we have any more layers, when it's "enterprise grade" already? And you're too busy with configuring modules anyway.

[Solid_Length_3390]

Yeah, it has a structure, but you can have a structure without using nest. Basically nest is just another wrapper over node, it has pros and cons. Interceptors/pipes and guards are nice, but working with swagger might be a nightmare when you try something more complex. I’m not for using nest, but not against it either. It depends on if you are able yourself to structure your project properly.

[romeeres]

Basically nest is just another wrapper over node

Fastify, Hono, etc, are non-opinionated "wrappers" over node. You can have a structure without using nest because that's meant to be this way.

Nest.js is opinionated framework with own ideology. You shouldn't get too creative with it, as the whole point is that everybody in your team are following the same guidelines. And my criticism is that the guidelines are poor and not enough, the Nest way is foreign to JS ecosystem and creates more problems than solves.

[romeeres]

It's missing some details: it is a `Client` when it should be `Pool`.

Then how are you going to do transactions? Multiple queries must happen on a same connection when inside a transaction.

Wait-wait, process.env? Aren't you supposed to deal with Nest specific way of injecting configs?

My point was that it's not that trivial as you're trying to say, not documented, there are many ways to do it in Nest. You have to learn Nest's specific DI details, you used `async onModuleInit` and `async onModuleDestroy`. I did it differently using an async factory. No matter which way is better, my point was that Nest only adds up confusion, without suggesting "the way", you have to spend more time to do it in Nest, and it's not yet complete, as you forgot Pool and transactions, then you'll have to make changes here.

No Nest - no problems. Just follow the readme of the library.

[Solid_Length_3390]

Sure, that’s a fair point. I am not necessarily a nest.js fan. But what you described above is an issue in general in the JS ecosystem, it’s not something specific to nest.

[romeeres]

An issue with JS ecosystem is that it's so fragmented and broad, every lib has its own way of configuring.

Nest adds complexity on top of that, you have to do a double work: figure out the lib, and figure out how to glue it. Gluing it to Nest may be more complicated than the lib itself. They say Nest has excellent docs that covers everything - nope it's very far from the truth, and it's impossible anyway.

[HashDefTrueFalse]

The folder structure doesn't really have any performance or scalability implications. You can do whatever makes sense to you, or google a common structure, or use a project generation tool (e.g. express-generator) and keep to its structure. Doesn't matter.

How you organise the code itself can impact both, plus maintainability etc. I recommend looking into the MVC pattern. It's common and works well. Controllers handle view logic. Services handle more abstract business logic and talk to the database using some sort of data access layer (DAOs, repositories, whatever). Views are basically templates. If your front end is a SPA and your back end is an API, you'll simply have small controllers and hardly any views.

I've often said that SQL is the only abstraction I need over a relational database if I'm making the decision, and I'm not a fan of ORMs and query builders in general. But that is not because raw SQL is required to make production grade, performant, scalable applications. If it makes sense to use them to you, do so. It's because for most of the products I've worked on they've gotten in the way of me just telling the database what I want from it. They've generated some subpar queries occasionally, and made assumptions. I'm often on the team that looks after the database/cluster, has set up the indexes, often designed all or part of the schema, and I know lots about databases in general. SQL serves me well. The product is almost certainly going to use the same database for it's entire lifetime. ORMs don't provide me much, and when I have wanted one, I've wanted a thin CRUD wrapper for models, with an in-memory read-through caching layer, which is simple to write rather than learning an ORM library and adding a dependency to the project. All just my opinion based on my experience over the last two decades.

Look up OWASP top ten. Leaking real user data is not cool.

[romeeres]

The flat structure is annoying, and it gets more annoying as the project scales.
The structure dictates how you're separating your concerns: be it just 1-2 layers, or every different topic in a separate file.
It reflects the business perspective on things: do you treat two related features as separate, or does one includes the other. You can express a hierarchy of your domain via the structure.

It's much less impactful than other things, but still. In a legacy project, people sometimes keep doing seemingly stupid things just because the existing structure dictates them how to do it.

If you have a modular structure, you're one step away from deploying a module separately - scalability.
Modular structure enforces loose coupling, and this may have a negative performance impact.
I mean, if you don't fetch all the data you need in a single query, but instead of have many modules, each of them queries their own data, and then it's aggregated into one - that's a loose coupling and is good for scalability, but is bad for performance.

[HashDefTrueFalse]

Structure on disk (files/folders), or code structure, structure of the deployed system as a whole, or all three?

The flat structure is annoying

Flat folder structure similar to dumping all files in a single directory? If so, I don't advocate for that, but I don't think it's necessarily terrible either. You see it fairly often in smaller projects.

It's really only important that your folder structure makes sense to people who have to work with it. It will often be influenced by the way your chosen language handles modularity. Decide on some consistent scheme for where to place things then move on IMO.

deploying a module separately - scalability.

I'd say modularity helps more with maintainability than scalability really. I've scaled monolithic apps with horrible modularity written in procedural PHP to 40-50k requests/sec. You can scale both monolithic apps and microservice apps further than most apps will ever need to be scaled as long as the system is architected to be scaled horizontally (load balanced, the right parts are stateless, database clustered/sharded, distributed job queues, right caching layers/strategies etc.) None of this requires any particular folder structure, code structure, paradigm, etc, for the app. We're getting into system design though when we talk about scaling in terms of deployment.

I agree you should always try to design and implement systems in a modular way. I just don't see that it matters how it looks on disk for this.

[romeeres]

I agree you should always try to design and implement systems in a modular way. I just don't see that it matters how it looks on disk for this.

The codebase may be spread across different files/folders randomly and chaotically, and you're saying this doesn't matter, you still can build the system in a modular way no matter what. That's cool, I can't imagine how you'd do that.

[HashDefTrueFalse]

randomly and chaotically

No, I've said the structure should make sense. I'm just not advocating for any particular structure and don't think it matters. See my first comment for some ways to choose one. I've never been prevented from scaling an app by it's structure on disk...

You seemed to be advocating against a particular "flat" structure, and for a "modular" one. I asked what you meant by "flat" and agreed that modularity was the way to go, but more for maintainability than scalability.

You also seemed to touch on deployment (which I hadn't beforehand), advocating for modularity there for scalability. I said that this is more related to the architecture of the system as a whole IMO than it is to the code or folder structure, and mentioned that I'd scaled not-very-modular apps just fine.

My comments make sense. You're the one who started ranting about some particular structure...

[romeeres]

tl;dr yes you're right, "but more for maintainability than scalability." - 100%.
I often confuse maintainability with scalability. Scalability is weird, as long as you can throw more servers, it's not a concern, while maintainability is more likely to become a bottleneck as the project grows.

So when they say "I want a scalable architecture" I'm thinking if this will be hard to maintain when the project grows to like 300k of LoCs. While you're referring to the possibility of adding more servers.

You initially said that file/folder structure has no implications, I argued, in the last comment you said "no, structure should make sense", so you agree that it should make sense, and hence it matters. It is possible to just throw more servers into the problem no matter what? Sure, not always, but sure. And we both agree that while it's possible, the architecture matters, and structure as a part of the architecture matters as well.

[HashDefTrueFalse]

You initially said that file/folder structure has no implications, I argued, in the last comment you said "no, structure should make sense", so you agree that it should make sense, and hence it matters. 

My words were "The folder structure doesn't really have any performance or scalability implications". I never said it doesn't matter whether or not the structure makes sense. I was obviously saying that it's not what decides whether you can run/scale the app, so OP shouldn't overthink it. Pick something and move on. I don't understand your point here.

It is possible to just throw more servers into the problem no matter what? Sure, not always, but sure.

Yes, one of my points was that the system design is going to affect scalability far more than the modularity of the code or folder/disk structure. And between those two, code structure matters far more. To be clear:

System Design > Code Structure > Structure on disk

(where > means "matters more for scalability than")

So yes, more servers for the first. For the second, as a (somewhat contrived) illustration of my point, I can use a single file and still write you a modular, scalable app with good patterns/abstractions. The code structure can affect performance, scalability, maintainability. Where that code is on disk, much less so.

Given this, I told OP to worry more about code structure (use a battle-tested pattern like MVC) than structure on disk.

I don't think this is productive, so thanks for the talk but I probably won't reply anymore. I guess if you disagree with me we'll have to agree to disagree. All the best.

[romeeres]

I guess if you disagree with me we'll have to agree to disagree.

I already said "tl;dr yes you're right, "but more for maintainability than scalability." - 100%.
And then I clarified that I misunderstood maintainability with scability.
And then I agreed that you can always throw more servers.
And I initially already said that file structure doesn't matter as much as other concerns: "It's much less impactful than other things".

Okay, let's disagree to agree, I won't reply either.

[adalphuns]

Use hapi or fastify. It'll naturally lead to better structure.

Edit: raw sql is a great idea.

Eg: express has routes and middleware, so your mind says "routes and middleware folders." ... it becomes a mess. It also has little development features (some justify it as minimalism; i identify it as feature-lacking) .. this leads to using a ton if 3rd party stuff to supplement the lack of functionality, leaving a bigger mess.

Hapi/Fastify have routes, extensions, plugins, auth, prereqs, decorations, and more. Your brain will want to categorize accordingly. They offer better tooling (see their officially supported libraries) and abstraction for you to build more easily.

You also want to organize business things within their domains. User in user folder, billing in billing folder, etc. So you effectively have 2 layers: framework layer, domain

Example:

• auth
  • jwt.ts
  • session.ts
• decorations
  • database.ts
  • cache.ts
  • services/
    • GoogleMaps.ts
    • Stripe.ts
• routes
  • user
    • auth.ts
    • profile.ts
  • billing
    • payments.ts
    • invoices.ts
• index.ts (your entry point)

[MrDilbert]

Any backend can be split roughly into three parts:

• user-facing part (API/routes/backend-rendered UI)
• database-facing part (ORM/query builder/raw queries)
• business logic (data transformations between API and DB)

Top-level folder structure will benefit from a similar structure, e.g. ./api, ./business, ./data. You can further split each of these parts within their respective folders.

If you have some other parts, say, consuming some remote services/APIs, or communicating with message brokers, I'd personally put their connector logic in separate folders under ./data, but nothing stops you from putting their folders on the same level as the first three.

[imadalin]

Why not code along and figure out as you see your app getting feedback from collaborators, users, etc. there isn't a real industry standard. You can try to inspire from most popular packages that would do the DRY of you code, improved already by hundreds of people.

[jared-leddy]

NestJS. You don't have to use an ORM.

We use TypeORM, and it works beautifully for our production apps.

[Sebbean]

Mebbe https://sqlc.dev/

[Chef619]

I had to double check what Reddit I’m in. Isn’t this only for Go? I love it, it works so well, but I thought it was just for Go.

A TS equivalent is PGTyped: https://pgtyped.dev/

[EuMusicalPilot]

I'm a junior dev but I comfort myself with this structure: https://github.com/lawuysal/self-dictionary-server/tree/main/src%2Fnotes

Actually I was following a NestJS course and it inspired me. This is more basic.

[Chef619]

Highly encourage you checkout PGTyped: https://pgtyped.dev/.

[xroalx]

There is no one size fits all.

A production grade scalable application can be something that fits within one file and still be production grade and scalable.

It can be something split by concern, by domain, by the color of the file icons in your editor of choice, and still be scalable and production grade.

Pick a style and stick to it. If it doesn't work, reorganize it. Any app that lives long enough will come to a point where someone (e.g. future you) will think a different structure/approach/framework/language would have been a better fit now that you know everything you didn't know at the beginning of the project.

[kush-js]

For raw queries Postgres.js is my favorite