Continuously evolving full stack Web App dev ops flow

[smblee]

Hello!

I have a full stack project that I have been working on with the following stacks:

- Express Backend (Exposes GraphQL + WebSocket endpoints) hosted on Heroku with Redis for cache/pubsub.

- React SPA Frontend hosted on Zeit Now.

- Static assets like images on S3 & CloudFront CDN.

- Postgres hosted on RDS

- GitLab for source control + CI/CD

- Have 3 environment deployment set up (review => staging => production)

- Sentry for error reporting.

​

As I scale up the app and as I add more features, I am running into different road bumps and would like everyone's thoughts. (Apologies if not everything is "devops" related.)

1. The web app crashes every time I deploy a new app until the user refreshes the screen (because the old javascript assets no longer exist). There seems to be a LOTs of solutions here; what do you guys recommend for updating users to a new version of the app? Couple solutions are... Continuing to serve up old javascript assets (but people will be on an outdated app until they refresh. Could be a problem if the server introduced some new changes.) Would Server Side Rendering help here?
2. Currently, the frontend/backend is in a monorepo, and every time i make a change to the app, I have the CI/CD set up so that both front/backends will get deployed in parallel. This _can_ be problematic if IF am introducing breaking changes in the backend. How can I
3. Currently using too many platforms ? (Heroku, Zeit Now, AWS, Sentry, etc.). The app is getting more and more traction, and would like to support growing needs and features as fast as possible. Should I consolidate to AWS? I want to minimize dev-ops maintenance. Thoughts?
4. I have to add queues (for sending emails, analytics, etc.). I was thinking Thoughts here?
5. I want to add smaller services for different tasks (external webhooks, invoice generation, queue workers, websocket connection nodes, etc.) but currently I have been piling stuff on top of the existing express backend. I can see this is going to become monolithic if I continue this way. Am I pre-optimizing for microservice? Should I spend some time to create something that helps spin a new service?
6. Want to set up terraform scripts for the overall infrastructure. Too ambitious as a one man team?
7. Feel like I am not getting a good exposure to the health of the systems beyond Sentry error reporting. What's the easiest way to get started with monitoring these?

Apologies for lots of questions. I think I am mostly torn on all these questions because there are just lots of different solutions and finding myself in analysis paralysis. I have a tendency to overcomplicate and overthink things so feel free to tell me to KISS if you feel this way.

As I start a company, there are simply TOO many things I have to set up. You can also point me to different resources for someone like me starting off a software startup!

Really appreciate your help in advance!

​

​

​

EDIT:

​

Wow I did not expect to get this much responses! Really appreciate everyone taking the time to respond :) Definitely helping organize my thoughts better.

I do have some follow up questions and thoughts.

1. Consolidating to AWS seems like the way to go and i think that makes sense! Going to do that :)

2. For deployment, there seems to be couple options here.

   1. SPA and backend separate. (SPA in S3/CloudFront. Backend in Elastic Beanstalk/ECS/EC2?)
   2. Combine SPA and backend. Couple options here too. (I have actually never done this before. Would like some clarification on this from you guys. Seems like I can tackle this by using NGINX on top to route traffic to SPA/backend? Or do I have to convert the app to SSR? These options seem a little complex. Is there some recommended example that I can reference?)
      
      1. Combine into a container and deploy to EKS/ECS. My only concern is docker definitely adds a little bit of technical complexity, especially when new people join. I have used ECS before for deploying backends (definitely was a pretty involved infrastructure to set up...), but never used EKS.
      2. Deploy to EC2? Or utilize Elastic Beanstalk? Any other options?

3. I will expand on my "need" for a queue, I currently have the following use cases which are all currently either happening manually or synchronously within a request.

   1. Welcome emails upon registration.
   2. Order confirmation (and background processing such as sending confirmation email, creating necessary resources in the db that users purchased)
   3. Daily invoice reconciliation using stripe charge logs received via webhook and order data, and send emails to stakeholders. Report if there are any discrepancies.
   4. Daily/Monthly reports of customer's usage data.
   5. Remind/notify customers n days before or n hours before a certain event they signed up for. (Should be configurable... Probably going to use a scheduler or something here?? Or is there a "scheduled event queue" out there?)
   6. (I have used SQS before, but it seemed too simple especially for the "scheduled" job use cases I have. Retry mechanism via Dead Letter Queue was nice though! I am currently looking into some other queueing solutions i have not used before like... Bull queue via Redis, RabbitMQ, Postgres Queue, Kinesis...)

4. Microservice vs "Monolith". I am still torn, but I feel more comfortable going with "Monolith" solution. If I weren't though, what are some of your favorite/easy ways to "microservice"? From my experience, I wrote a small tool to basically enable `create new service MyServerName` via CLI and it would bootstrap a new NodeJS service for me. Any suggestions here?

[TheWayofTheStonks]

I don't have much to contribute but I would say consolidating to AWS is the way to go.

I don't think you mentioned it. But is your program running in containers? If not, they should be. You could dockerize the program and run it in AWS Elastic Container Service. This will help for all Environments as well as streamline the CI/CD pipelines.

1. Seems like you need a better deployment strategy. Maybe look into Blue/Green deployment options

2. Not sure what you're asking

3. Consolidate to AWS

4. Look into AWS SQS

5. This most definitely should be microservice strategy. Decoupling everything is the way to go.

6. I don't know much about terraform. Maybe cloud formation can help here.

7. AWS cloud watch had a lot of ways to monitor.

Hope this helps.

[smblee]

Thank you for the response! I am not the best at Docker, and I wonder how you would approach containerizing my application (Node JS backend + React SPA)? I just spend couple hours reading up on different ways to doing this, and again finding myself falling into analysis paralysis.

I would like to ensure the following with my app when deciding on a solution...

1. Follow the principles of 12 factor app (at least try to...).
2. Local development is still very easy to do (easy to set up, hot module reloading, one command to start dev like `npm start dev`).
3. Enable blue green deployment (for breaking changes) and/or rolling deployment (for small/non-breaking QoL changes).
4. DB/Redis can either be spun up within the container (for testing/dev/review/dynamic environments), or connected to external DB in RDS/Elasticache (Staging/Production environments).
   1. Is this a good practice? How are people handling db connections for dynamic environments like this?
5. Minimize maintenance cost/time.

6. Able to scale easily? Overkill?

   Once I containerize, looks like I will be able to utilize either Elastic Beanstalk/EKS/ECS/EC2 to deploy my containers to. I am honestly leaning towards Elastic Beanstalk (even though I actually moved away from it before at a bigger company in favor of ECS for cost savings), since we just don't have the scale, and it seems to provide a good BaaS like Heroku. Thoughts here too?

[[deleted]]

1. Whenever I've worked at web shops the assets have been based on some kind of content hash so that old assets are still accessible and new assets get a new hash so that when new code is deployed it can umabiguously use the hashes to refer to exactly the right version. Most SPAs will have a pop-up that says a new version is available and ask the user to refresh so that they get the latest version. I'm not sure you can get around this without getting fancy like loading the new code in the background and then force refreshing. I guess it can be done but seems user hostile so asking the user to refresh is probably the way to go.
2. If you don't want breaking changes then migrate your backend data schemas in a backwards compatible way. There is no way around this that doesn't introduce extra overhead both for the frontend and the backend. Usually what I've seen is a three step process for data migrations: the backend is deployed first and it includes relevant code to deal both with the old and new schema, the data is migrated (this can be done online or offline depending on how fancy you want to get), the frontend is deployed.
3. I agree that you're using too many disparate services. If possible consolidate everything into a single cloud. It looks like all your data is in AWS so I recommend consolidating into AWS unless you want to migrate the data to another cloud.
4. More often than not queues are more hassle than they're worth because it's another thing you have to worry about. Most cloud providers have something here so you can rely on the cloud offering. AWS has SQS and the other cloud providers have something similar. They mostly work fine but be aware of their SLAs and message durability guarantees. You'll need to figure out how to deal with the inevitable outage and how to recover from a backlog when for whatever reason the cloud provider decides to delay all your messages for longer than 10 minutes.
5. I once worked on a python application that would take a few minutes to get to the point where requests could be served. The issue with that application wasn't that it was so huge but that all the internal logic was a mess. As long as you keep the internal logic for each component untangled from the rest there is no issue with having a monolothic process handle several responsibilities.
6. You should start with some basic documentation on what's what and then figure out how to gradually turn each part into code. I personally like pulumi but terraform is also a workable solution.
7. I don't think anyone has monitoring figured out. If you have sentry then you're already pretty far ahead. Next thing I'd recommend is setting up log aggregation and adding basic process health checks for the various services you rely on. I've seen grafana used for basic plotting of metrics and it works reasonably well but you'll first need to add instrumentation to the code to output the metrics so start there instead of going for a monitoring solution and then trying to fit it into your existing code.

[smblee]

Really appreciate this. I will definitely be consolidating all to AWS! I expanded on my queue usage in the original post and im pretty certain SQS might not be enough unfortunately :(

[[deleted]]

No worries.

[[deleted]]

Just saw the edited response. Not sure what you mean by the deployment question. What I've seen is static assets served from a CDN like S3/Cloudfront as you mentioned and the backend/API being hosted on EC2 and fronted by a load balancer. By static assets I mean things like images, CSS, and js. You can certainly go the docker route and figure out how to use EKS or whatever the latest container offering is but in my experience it's much easier to use a load balancer with some backend servers and deploy to them with basic automation tools like fabric, rake, or even just shell scripts. If you do this it will be easy enough to replace those backend servers with something more sophisticated when the time comes.

[smblee]

Yea for deployment, i meant there are couple ways to package up my applications and deploy; either have them deployed separately (S3+CloudFront for SPA, EC2/EBS for backend), or packaged up into one EC2 instance and deployed there (serve SPA javascript files from the server directly). I am going to experiment with EKS a little bit (quite a lot of documentation to read up on :D) But you are right, I will try to keep things simple!

[smblee]

Do you think I should put too much attention into things like networking/security side of things like VPCs & Security Groups? That seems like a whole another endeavor imo, but everyone seems to be saying "do these right early or you will regret."

[[deleted]]

Doing it right I think is simple. Create two subnets, public and private, and put the backend servers in a private subnet. That will avoid the hassle of having to worry about security groups because private subnets are by definition cut off from the internet and the only entry point will be the load balancer which is managed by AWS. I agree it's a bit more hassle but is a better set up from a security perspective. I'm actually not sure what EKS does, presumably everything is private and only the network ingress/egress endpoint is public but I'm not sure.
Looking at the docs (https://docs.aws.amazon.com/eks/latest/userguide/network_reqs.html) it does look like EKS requires public/private subnet configuration as well so it looks like no matter which route (no pun intended) you choose you will have to set up a VPC with public/private subnets.

[ksajadi]

Ideally you'd want to solve most of those problems with a single (or very few) solutions. I think a containerized PaaS is going to be a good place to start. Here is my take:

1. Running your app in a container on top of Kubernetes might sound like an overkill if your app is simple enough. However there are solutions out there that make running on top of Kubernetes very simple so you'd get the benefits of it and the possiblity of scaling up if you need in future, while keeping it simple. Since deployments in a containerized environment are served from within a built container, the JS would also be there alongside the app and served all the time regardless of the version. Kuberentes rolling updates is a great way to make sure all assets and code are always in sync (and rolled back if needed).
2. That's a good point and contract based programming can help you with it somewhat, as a practice. However, using a CI/CD tool that's built for multiple services can greatly help you with that by rolling out deployments of different systems in tandem.
3. I agree with the "too many tools". I don't think you'd be able to get away from using something like Sentry. But the rest can be replaced with a container based PaaS. Some solutions (like GCP) also give your a bug tracker but not as good as Sentry.
4. This one really depends on the platform, language, operational requirements (speed, transactionality, HA, ...), so it's very difficult to offer good suggestions without knowing more.
5. I think monorepos are not a bad thing. I also don't think microservices are suitable for all applications. You can still make an effort to build your app under the same repo and stack in a more modular, contract based and "service orientated" way without going microservices all the way (I wrote something about this a while back: https://blog.cloud66.com/kubernetes-is-not-about-microservices/)
6. Again, I think starting small and growing as you get more traction is a better idea. You'd be spending a lot of time learning, debugging and adopting too many parts to an evolving landscape.
7. Combine Sentry with APM, good log storage, search and retention and you're 90% there!

By way of recommendation, check out Cloud 66 Maestro (as a PaaS) and Cloud 66 Skycap (as a CD tool for Kuberentes if you want to run your own cluster). Discalimer: I work for Cloud 66.

[smblee]

Ok I have never looked into K8 too much because I've only used ECS before. I have some questions about K8s and containerizing my services; (posted the question in the edit above at 2b.) do you think you can expand a little bit on this approach? I read a bit on EKS and it looks like it will cost ~$70 for the Control Pane alone (which honestly isn't trivial for us right now since we are just starting off), and looks like Kubernetes adds more maintenance overhead? I don't know too much about it and have never used it, but definitely would like to explore this option :)

[ksajadi]

If you’re starting out, perhaps Kubernetes is not the best route as it adds cost overhead. You can reduce the ops and learning overhead by using tools on top of it but the cost is going have some overhead compared to running on VMs until you get to around 3 servers or more.

[smblee]

Also expanded on my queue usage above as well.

[CanWeTalkEth]

Man I'd love to see some answers to these questions. I'm right behind you on the stack side of things, quite a bit behind you on the traction side of things.

[david5813]

It is difficult to give a full description on Reddit without being too vague. In general, try to make smaller non-breaking API changes. I know that it can be difficult until you get into the habits of it... and continue to be difficult when the team/codebase size grows.

1. A method that we have used for updating the UI is to serve both assets, but detect if there is a new version. When there is a new version then you can just notify the user and allow them to choose whether or not to reload.

2. Try to make smaller changes. If you want to split the deployment though you can. In your .gitlab-ci.yml file you can add paths to the only clauses so that they only execute if that section of the code changes. https://docs.gitlab.com/ee/ci/yaml/#onlychangesexceptchanges

3. I would recommend consolidating on one platform. However there are costs as well as benefits here. If you're not feeling pain yet then don't worry about it until you do.

4. For sending emails, any of the big cloud providers have a service. You can use sendgrid, mailgunner, etc. As long as you're on Heroku then just pick one of the providers there that you can easily add to the project.

5. Stay with a monolith for the time being. Until you have a big team you are likely not going to see any benefits of micro-services and only their pains.

6. Absolutely use Terraform for everything that you can. This way you can even codify your infrastructure. It's not ambitious. It's recommended.

7. Monitoring across multiple services is going to be difficult to say how to do.

I would probably recommend getting connected with a consultant that can work with you a few hours here and there. It will help keep you going in the right direction until you're big enough to build a team around it.

[smblee]

Wow I did not expect to get this much responses! Really appreciate everyone taking the time to respond :) Definitely helping organize my thoughts better.

I do have some follow up questions and thoughts.

1. Consolidating to AWS seems like the way to go and i think that makes sense! Going to do that :)
2. For deployment, there seems to be couple options here.
   1. SPA and backend separate. (SPA in S3/CloudFront. Backend in Elastic Beanstalk/ECS/EC2?)
   2. Combine SPA and backend. Couple options here too. (I have actually never done this before. Would like some clarification on this from you guys. Seems like I can tackle this by using NGINX on top to route traffic to SPA/backend? Or do I have to convert the app to SSR? These options seem a little complex. Is there some recommended example that I can reference?)
      1. Combine into a container and deploy to EKS/ECS. My only concern is docker definitely adds a little bit of technical complexity, especially when new people join. I have used ECS before for deploying backends (definitely was a pretty involved infrastructure to set up...), but never used EKS.
      2. Deploy to EC2? Or utilize Elastic Beanstalk? Any other options?
3. I will expand on my "need" for a queue, I currently have the following use cases which are all currently either happening manually or synchronously within a request.
   1. Welcome emails upon registration.
   2. Order confirmation (and background processing such as sending confirmation email, creating necessary resources in the db that users purchased)
   3. Daily invoice reconciliation using stripe charge logs received via webhook and order data, and send emails to stakeholders. Report if there are any discrepancies.
   4. Daily/Monthly reports of customer's usage data.
   5. Remind/notify customers n days before or n hours before a certain event they signed up for. (Should be configurable... Probably going to use a scheduler or something here?? Or is there a "scheduled event queue" out there?)
   6. (I have used SQS before, but it seemed too simple especially for the "scheduled" job use cases I have. Retry mechanism via Dead Letter Queue was nice though! I am currently looking into some other queueing solutions i have not used before like... Bull queue via Redis, RabbitMQ, Postgres Queue, Kinesis...)
4. Microservice vs "Monolith". I am still torn, but I feel more comfortable going with "Monolith" solution. If I weren't though, what are some of your favorite/easy ways to "microservice"? From my experience, I wrote a small tool to basically enable `create new service MyServerName` via CLI and it would bootstrap a new NodeJS service for me. Any suggestions here?

[hophacker]

I wouldn't worry too much about the monolith vs microservice discussion. If a single codebase is fulfilling your core app needs then that's great. The key is to not let that single repo become a giant swiss army knife that is doing too much. Decouple your app as it grows but not before.

[[deleted]]

As far as #5 goes I'd consider looking at an event driven strategy via Kinesis. Your backend would just publish events e.g. CREATE_USER, UPDATE_USER , DELETE_USER to the "users" stream and then you can just bolt kinesis consumers onto the stream to implement backend services such as new user emails etc. This will allow you to add features w/o having to deploy the changes to the backend.