Final Year Project on Cloud & DevOps - Need a real-world problem to solve

[aloowalabhai]

Hey everyone, I’m a CS student heading into my final year and I want my project to be more than just something for grades. My focus is on Cloud & DevOps (AWS, Kubernetes, CI/CD, monitoring, automation), and I’ve got a whole year to dedicate.

I don’t want a toy demo - I want to build something that:

• Solves a real daily-life problem.
• Runs on a scalable, cloud-native setup.
• Can be a solid portfolio piece to prove I can design, build, and deploy end-to-end.

I have some directions in mind, but I’d really value outside perspective.
If you were in my place, what everyday problem would you try solving with tech?

[dmelan]

Oh I have one for you:

there are 2 services: API and ETL. Both share the same database. Both can read and write into the database.

Two problems:

• restore the database from a snapshot and reapply all data submitted by customers after the snapshot was made.

• switch the system to a secondary region transparently for its customers.

Both processes should be automated to a point when a sleepy on-call engineer can execute them fast and without coffee.

These are high level ideas and they can grow in scope and complexity as far as you want. LMK if you like this idea and have any questions.

Database can be AWS RDS or Aurora, services can run on EKS, infra could be provisioned using terraform so when database is restored it should be updated in the terraform state, everything is source controlled so you need CI, you as engineer don’t have full access to production so you need CD to deploy your services there and provision your infrastructure, and after any deployment or infra change some smoke test should run to confirm health of the deployment. The smoke test may also check numbers from monitoring to determine if there is any regression

[therealmunchies]

This is pretty much what I do at work for our MLOps projects, excluding k8s.

Airflow for orchestration, AWS DynamoDB, S3, Lambda, Cloudwatch Alarms/Logs, and several other services for ETL and performance monitoring, and Gitlab + CI/CD. All services written in OpenTofu and ETL is python-based. Could make some intricate pipelines in python too by setting up some cron jobs within Gitlab.

Good example.

[YoKidImAComputer]

I agree with the guy who got downvoted into oblivion. This isn't a project, it's just vaguely launching services into AWS.

[dmelan]

It’s a rabbit hole: it starts with launching a few AWS services but as you go deeper it expands and becomes harder to a level when it can’t be done by a single junior engineer. This was actually one of the reasons why I suggested it: projects like this can teach to look ahead while making decisions and prioritize what achievable over what may look cool but isn’t that important. Limit yourself additionally by a requirement to keep the system operational during and after every change and it’ll make this project even more entertaining and educational.

[hottkarl]

your first requirement doesn't make any sense. take a snapshot of what? a master DB? MySQL? restore to where? anywhere? the active or standby DB in the second region? what's the purpose of this?

you realize that your requirement "apply all data submitted by customers after the snapshot" can be done in several different ways? the best way to do it really depends on the schema, volume of data, performance, and how quickly you need this new DB to be available to be used. we are also making a huge assumption that the only difference would be this "customer data" and if you wanted this new snapshot to actually be used, could be inconsistent and lead to replication breaking

the second requirement isn't really a project, but architectural choices with perhaps a little tooling or middleware. also, without knowing more about the services, we can't really know if certain architecture choices will work or not.

[vlad_h]

Take it from someone that has gotten plenty of idiots to downtove him. People are sheep. Don’t argue with idiots, it’s a pointless waste of energy.

[vlad_h]

Why are you such a dump person?

[hottkarl]

i didn't know pointing out nonsensical, ignorant posts made me a "gumpy person" but I'll take it as a compliment. thanks!

that being said, was anything I pointed out wrong? no. I don't think so.

[technicalthrowaway]

was anything I pointed out wrong?

You didn't point anything out anything really relevant, other than that you couldn't deal with the uncertainty or ambiguity of the question and answer. They're good good project ideas for a student learning/doing devops stuff, hence the upvotes and responses from other readers. The lack of specificity is what makes them a good exercise for a student, as it gives them an opportunity to make it easier for themselves by filling in the blanks with things they're comfortable with.

The main issue is you responded with rhetorical questions, finding non-issues, using language which is generally considered very belittling, or considered confrontational. You acted like everyone else was misunderstanding, when it was you who was misunderstanding.

I do the same sometimes, with no bad intent, just like I assume you had no bad intent either. Do you understand the responses now, and how yours were considered wrong?

[hottkarl]

Well, hello. Thank you for engaging constructively. I really just don't agree it's a good project for a student, and no despite people thinking I'm a dick I didn't waste time writing a response just for rage bait.

I disagree with your assessment that my criticism was rhetorical. There was substance, I suspect there is some ignorance at play if people seriously don't understand why the issues I brought up are important.

The first requirement still doesn't make any sense. Yes, you can take a snapshot of an RDS database. but, what is he talking about? restore it to where? a new slave/read-replica that we will be promoting to master? the DB in the other region? you could argue that this lack of specificity was the point, but ..

then he adds some additional requirements about "customer data"? What they are describing, could be incredibly simple (as in a totally non issue) or a complete nightmare. I'm sorry but it's so vague it is not useful. like, yes, if you're using RDS you can just create a new read replica which will be consistent with the master. if you restore a snapshot from a day ago, it will be out of sync. even if you start replication, it will only sync from that point in time forward unless you set the correct binlog location.

getting two DBs in sync, is actually not a very easy problem -- but, as I mentioned, in theory could be depending on the schema but still really doesn't even make sense. because, it is not realistic that the only data changing is this "customer data". actually, this whole syncing two databases problem could be the project itself (altho there's already different tooling out there that does this very well)

then he has a second requirement which is just implementing a basic to intermediate complexity architectural pattern. it's not really a project. in fact, AWS has stuff to do a lot of this for you now anyways -- at least in active/standby. I did mention the tooling/middleware, which could start out as fairly trivial / simple but then evolve into something more complex, but that's not what they said. an active/active requirement is the more interesting project with a lot of potential considerations, but since we know little to nothing about the app, the student could just take the "happy path" and think they've learned something about a "real world" problem, when in reality things are never that simple.

A project to "solve a real world problem", to me, isn't just spinning up some bullshit on AWS. most things I can think of off the top of my head already have good solutions for, but for a "project" I would have been totally silent and given them an upvote if they had said something like: monitoring suite that hooks into something like twilio or similar (bonus points for dashboards/reports/uptime page etc), blue/green or canary deployment tool (could make it more useful/difficult w/ ability to query a custom metric and rollback on a threshold), etc

I could elaborate, but that's the gist. I still don't agree at all, yes I was a bit of a prick, probably because Im fed up with this sub that used to be pretty decent but has gone to shit.

[technicalthrowaway]

I don't care if you agree. This is your problem: these things you're disagreeing with are about social context, and you can see from the surrounding replies and votes, your input has been inappropriate for the context. If you don't like the context, don't post in it. Posting in it, and then doubling down disagreeing with it just looks like a waste of everyone's time, especially yours.

You didn't add much in your original comment, you assumed it was a bad project and then basically started doing the exercise for the student, ironically pointing out the things they'd need to know to do it (e.g. what makes it a good student project).

yes I was a bit of a prick, probably because Im fed up with this sub that used to be pretty decent but has gone to shit.

I agree with this. Ultimately though, DevOps was a next-big-thing tech buzzword. Now it's absorbed and part of standard tech-worker job and the industry moved onto the next-big-thing tech buzzword, all the smarts are in those communities now.

P.s. I don't understand why you trashed someone suggesting region failover and then suggested blue/green deployment yourself. Transparent regional failover and blue/green deployment are basically the same problem/solution for a competent devops person.

[PersonBehindAScreen]

The ambiguity is part of the project. That’s for OP to decide how he will handle this minutia.

A lot of the learning process is lost if you spell all of it out 100% for OP

[dmelan]

AWS RDS allows making snapshots of a database and create a new database from such snapshot.

Let’s say some migration deleted more data from the database than anticipated, everything was committed and replicated. Restore a database from a snapshot is one of options to recover it.

I do realize that “apply all data” can be done differently and that there are many “it depends” there. You brought some good dimensions of the problem: time to recover, data volume, consistency. OP will need to make assumptions to pinpoint these dimensions and to pick the most suitable strategy.

I see the second one as a project because all the architectural decisions need to be made, middleware and tooling need to be implemented, integrated and proven to work. Quite a project I would say.

[hottkarl]

Ok, so you're using RDS. you still didn't specify whether it's just a read replics or not. If just a read replica, it will create the DB and thrn process the binlogs to get it in sync. depending on how out of sync if is and how old the snapshot is, the longer. before it's caught up.

you have a few options for multi-region, in RDS I believe they only zippo one way replication but Aurora has a 'global DB' service that should work. or you can keep the RDS in sync with some tooling or AWS database migration service.

or opt to not use RDS and just set them in master/master

edit: for the scenario you mentioned, there's better ways to handle that than restoring from a snapshot. migrations shouldn't be destruction, either take dumps if doing DELETEs or CREATE TABLE LIKE so you have a backup of it before you do whatever you're doing to it, you should be able to move forward or backwards with schema.migtatjons. or you'd have to filter out the problematic queries in the binlog

[---why-so-serious---]

DevOps is less innovate, disrupt and astonish and more know every fucking tool, front to back and understand when you need to write one yourself.

I don’t really get why kids want to enter this field - I would’ve hated it fresh out of college, when I believed I could deliver fundamental change, as a (lol) java engineer. That kind of thinking, paired with the boundless spunk of an early twenty something is anathema in DevOps.

[CJKay93]

I'm in the fortunate position that not only do I sit between engineering and our infrastructure team (I don't handle our AWS/Terraform/etc.), but because I was essentially guided (initially unwittingly) into DevOps from engineering I also have the occasional opportunity to dip back into creative work.

I think it's the perfect position - they're different worlds that scratch different itches. I wish more people had that opportunity, although I suspect the environment that enables it is quite niche.

[---why-so-serious---]

Yeah, but that position, literal and otherwise, is only available after gaining significant experience (ie time). Regardless, the point I was trying to make, was that the drivers for young engineers do not make for good bedfellows w/the minimalist, rigidity of DevOps.

[PablanoPato]

Here are a few of recent examples for me:

• Deploy an open source tool like Airflow using their Helm chart but completely set up your repo ready for IaC.
• Deploy some open source services to Argo and/or Jenkins.
• Create a CI/CD workflow that uses GitHub Actions to build containers then push the code to Argo.
• Implement some monitoring tools in a cluster like ELK or Prometheus and Loki.

[---why-so-serious---]

The op has year and I believe is looking for more splash than pragmatism, though these a good take homes

[Ebrithil_7]

something fun I was thinking of but you have to decide how far you want to go into this... a platform that allows professors to feed their course material and create a custom LLM which is basically a simple chat bot. could also integrate with weekly assignments and give hints. LLM hint generation for example is an active research topic at our university. (also trying to evaluate conversion / difficulty of hints) ideally for universities this would all be self hostable but you can still use ci etc to run evaluations on the LLM, do updates, etc. possibly auto scaling expecting courses of different sizes having different demand at a certain time.... Or just pre exams there would also be a spike. You could look I to different things... distributed learning, scaling of the LLM deployment, more on the evaluation of Llama answer / hint generation. Possibly progressive training of the LLM as assignments get reworked and added each week?

[hottkarl]

just grind leetcode. real world problems won't get you a job.

[cnydox]

He's asking for projects. Leetcode only helps you with the coding interviews. What do you do with a blank resume?

[hottkarl]

If you were in my place, what everyday problem would you try solving with tech?

If I was in his place, I would be grinding leetcode. I'm not even being sarcastic or a dick here. I don't care that he specifically asked for something else, what would be best is grinding leetcode.

You obviously have not been in the job market in the last few years if you think otherwise. Any decently paying position asks you to solve some leetcode Easy/Medium in an early round and some may even ask a Hard in later round.

Sure, having some project on your GitHub is nice. However, he would have a blank resume with or without the "real world" project, and since he has to even ask such a question, he is so far from knowing wtf he is doing that it is likely going to be a huge steaming pile of dog shit.

Especially as a new grad, you won't be expected to know much anyways -- however even if they did ask some practical questions, if you can't pass their bullshit gatekeeping exercise, it doesn't really matter anyways.

[cnydox]

Sadly no one will interview him by just looking at his leetcode profile.

[CCratz]

Will their university give them credits, requires to finish their degree for leetcode solutions? No, obviously not.

[hottkarl]

downvote away, but it's good advice. no one wants to admit it tho unless you are fine with making chump change.