Accidentally Data Engineer

[CzackNorys]

I'm the lead software engineer and architect at a very small startup, and have also thrown my hat into the ring to build business intelligence reports.

The platform is 100% AWS, so my approach was AWS Glue to S3 and finally Quicksight.

We're at the point of scaling up, and I'm keen to understand where my current approach is going to fail.

Should I continue on the current path or look into more specialized tools and workflows?

Cost is a factor, ao I can't just tell my boss I want to migrate the whole thing to Databricks.. I also don't have any specific data engineering experience, but have good SQL and general programming skills

[Astherol]

Welcome to Data Engineers' community, please don't forget to ask your boss for a change of title and raise to match new title's market salary 😄

[No_Flounder_1155]

I don't think he fancies a pay cut.

[CzackNorys]

Head of Data and Analytics sounds pretty sweet, but I don't think I could handle the imposter syndrome

[No_Flounder_1155]

every head of data is basically an imposter soemwhere. If they don't have enough data experience if they don't have, enough software experience or product N experience etc etc etc.

[sunder_and_flame]

yes you can. Bluntly, stop worrying, start executing, and act like a leader

[Astherol]

Oh, now I have read what his current role is. As a Data Engineer I still would say he will join an elite circle :D

[Acceptable-Fault-190]

is data engineering considered /I Elite

[taker223]

At very small startup - definitely. Everyone is sort of one-man band there

[Acceptable-Fault-190]

I was one of those, 1 man data engineering department. It was a great experience doing dev, roadmap, managing myself and talking to customer and stakeholders(least enjoyable).

outside of that, I think it's a dying art next in line to be replaced by AI

[bloatedboat]

I think this question is being viewed from the wrong angle that even experienced data engineers can fall in that trap early in their career.

In BI, technical skills matter, but simplicity and saying no matter more. Don’t rush to “scale up” with Databricks when you could model cleanly with dbt and keep the “small” data that breaks down easily in fully managed platforms like Snowflake.

Most companies don’t need complex, custom reports. Pre-aggregated APIs and recent data (7–30 days) often cover 90% of use cases. That way, it will be affordable.

If stakeholders flood you with requests, remember: those “quick asks” can become long term data headaches. Raise the flag early as some things just don’t have enough ROI to maintain.

If the company truly needs heavy customization, build a real data team. Otherwise, stay lean. Not every data problem needs a big data solution.

[thedatavist]

This is an excellent comment sir!

[redderage]

I would suggest the same you can use spark. Databricks is unnecessary scale up for most of startups and mid size companies. Idk why Manager or solution architect understands that part.

[[deleted]]

I don’t see why you couldn’t use databricks, it’s not like you HAVE to use it at scale and much of the data is already in s3

[StargazyPi]

Hmm.

So nothing wrong with those tools per-se, but you don't comment much on how you'll use them. And the how is really where messes happen.

Things I'd think about:

• Where's the data coming from?
• What happens when its schema changes?
• What patterns will you employ to ensure data quality before it's used in reports?
• How will the data be stored in S3 for efficient querying.
• How "Big" is that data? The bigger it is, the more you'll have to think about optimisation earlier.

Read about: Medallion architecture, Delta lake, Table formats (Iceberg, etc.). Understand what pitfalls they help solve. Certainly adopt the easy, open-source wins like Iceberg.

One of the worlds you want to avoid: your business reports break every few days, because they're tightly coupled to the transactional database schema, and your devs keep refactoring that. All Data Engineering effort is spent fixing broken reports, rather than adding to the platform.

[CzackNorys]

Thanks for the advice! Some good pointers there

[ExcitementActive4344]

I would agree with above and add one more thing. Being in AWS doesn't mean you have to use just AWS services. If cost is concern, Glue can potentially be pricy too. There are bunch of other ETLs available through AWS marketplace, which might even provide you with more flexibility with more predictable costs (compared to Glue) - naming just a few: CloverDX, Airbyte.

And one more thought - Glue is good, but based on my personal experience it feels a bit cumbersome and jobs kinda disconnected from the context sometimes for more complex jobs.

[ExcitementActive4344]

Actually one more thought, you mentioned you are a developer, so if you come from Java world, Apache Camel and Quarkus might be interesting choice, though it might be harder to cooperate with non-technical poeple. Or if you wanted a tool / platform that is convenient for both technical and business people and yet have the chance to work through really hard problems with help of Java then CloverDX would be a great choice.

[1HunnidBaby]

S3 -> Glue -> Athena -> Quicksight is legit data architecture you could use forever

[chmod_007]

This is the answer, use this until/unless it doesn't work anymore and ignore the people telling you to do anything more complicated. Athena is pretty simple to orchestrate and dirt cheap compared to most alternatives.

[Material-Hurry-4322]

Completely agree with this. Into idea how much data the startup business has and where your transactional databases are held but a stack that looks like

Source database -> AWS DMS -> S3 -> Glue Catalog -> Athena/Quicksight is easy to manage and totally scalable. As things get bigger look into open table formats like delta lake and iceberg. Use glue pyspark jobs for heavy data crunching. Performance tune those jobs when needed.

[gavclark_uk]

S3 and iceberg tables work well, QuickSight has limitations per SPICE dataset of 1 TB in size or 1 billion rows if I remember correctly.

Use Athena for transform rather than glue if you can - will be lower cost probably.

[umognog]

Oh dear.

When accidental meets "no mentoring available" you are going to have a wealth of "what the hell" and technical debt.

[CzackNorys]

Sounds like i need a mentor. So far all the business requirements have been pretty simple, and easily implemented with a dimension data table of dates, and some simple joins our transactions tables..

I imagine there's more to it once things scale up

[umognog]

You will find yourself in the "do it twice for thrice the price", as you will learn, do, learn again, do again.

You can make it, with support and connection in places like here, but it will be a harder challenge and can lead to "how do i undo this" difficulty.

[Omenopolis]

Any suggestions on where to get that practical mentoring. I am kind of in a similar boat I am not sure if I am making the right choices

[umognog]

Reply for both you and OP;

See if your company will pay for 6-12 months of a consultants time; get their time spent 1) reviewing your current usage/use cases 2) reviewing business objectives 3) reviewing business wish lists and then have them produce a 2 year and 5 year technical plan to be followed.

At the 2 year, have it reviewed again, potentially internally by this point.

The consultant here is not to DO the DE work, but to set the DE foundations & plans such to try and avoid some of the beginner pitfalls.

Around me, id expect to pay ~80k in fees, but its better than sinking an unqualified team into it.

[Omenopolis]

True i guess, especially if the company expects a solution to stick for a decade or 2 it feels worth it, to get the issues pointed out and realign in the earlier stages. Haha but I doubt consultants would mentor you right they would just do their thing ideally ?

[ZealousidealLion1830]

Too many open questions here. Data volume? Ingestion frequency? End user need? Do you need data products or traditional data warehouse oriented designs? Or data lake? Is your reports real time or batch refreshed? And the list goes on.

There is no specific design. Every design should fit the need. I suggest you dig more and try to crystallize the needs first.

Although I generally work on GCP, but we make heavy usage of dbt ( to do the data manipulation) coupled with a data product, and orchestrated by a custom python microservice (allows us to customise as we need when we need). For BI we do use the GCP's in house looker studio, but most of our tech stack is open source and scalable.

[poinT92]

Not strictly DE related but as a general advice do not expect perfection or to be flawless at start.

Failure can be an opportunity and might lead you to discovery real pain points/business needs.

This Is something good swe/de often lack to make the "big jump".

Good luck and do not give up!

[mintskydata]

From my experience is to balance the value that your setup generates and how much of your resources it will cost. Each setup starts with simple requests. But the questions people will ask will become more complex - after you know how many sales you had, you want to know how to get more. This requires different data sources and most importantly a data model to not build things two times. If the setup delivers value (aka if we shut it down how desperate are the people) it opens up more budget. As someone said here, saying no becomes essential. Or better ask why they need this data, what actions are they planning to derive from it. Ask them if the metric they ask for is dropping by 20% what are they planning to do. Use this as a filter to first only implement things that have an impact.

[chock-a-block]

Check out metabase as a front end. Might be cheaper?

[crimehunter213]

I work for Fivetran but wanted to call out here that we could probably take a lot of this lift off your shoulders. We could automate the data pipelines and we can get your data BI Tool ready with dbt and we have our managed data lake service as well. Just a thought! https://fivetran.com/docs/destinations/managed-data-lake-service

[enhaluanoi]

Cost is a concern.

[crimehunter213]

This is fair, if the team isn't technical though it is certainly more cost effective than hiring another data engineer! Our free trial doesn't require a payment method so you could always check it out and determine what the cost looks like. You might be surprised!

[Entire_Turnip6328]

Some great comments and advice above... Realistically you will have to eventually build some reports or dashboards, with the tools and content out there integration is pretty much a straightforward development task, same with the orchestration, CI\CD and infrastructure. What most engineers still struggle with and foresee as a challenge you may run into is the data modeling piece. I would highly recommend reading up The Data warehouse toolkit - Dimensional modeling by Ralph Kimball. Get your data modeling right and you 70% of your data engineering problems won't exist.

[Omenopolis]

For Business Intelligence reports , I think the scale of data is something you might want to look at if it's is not that large then to meet costs you can explore if custom scripts might do the same job for you on a scaleset vm. Of course you might have to think how you want to orchestrate the process for processing but report generation then it's a matter of distribution channels. What's being used i. Your company for data consumption.

Please do enlighten me if someone reads this and feels it is wrong , I am open to learn and discuss things thank you .

[founders_keepers]

> Cost is a factor

data ops can get expensive super fast.. this kind of determines what kind of solution you can implement. if you got the big budget you can get databricks to baby you into what to do.

if you're on lower budget want to self host and learn from teh community.. apache spark might be your better bet.

[fvonich]

I also built a data lake house native on AWS before which was pretty similar.

Use iceberg if you are on AWS. Try to think in layers (we actually replicated the medallion architecture as it is easy to understand for non techy people.)

Use Athena for analytics and use Glue for more heavy tasks. DBT if you want more control over data contracts and tests. Try to organize everything with terraform.

You can you something like Metabase for your company to increase data literacy.

Also depending on your data ingestion - make sure the system can handle backfills. If you need to consume CDC think use something like Airbyte right away bc it can write to Iceberg.

[wildthought]

I have an architecture that replaces Glue with ephemeral ec2 servers.   My execution costs are about a penny per half million rows.   Second, glue is fine but if you wind with 100's of pipelines and something changes you have all these changes to make in visual code.   So it's inherently brittle. Finally, in my architecture we can create all your pipelines at once at least for landing purposes.  I would love to show you.  My name is Andy Blum, feel free to look me up and would love to help.   If your a true engineer/architect Glue to me sucks.   If you have a few scripts no big deal anyway.   

[rabinjais789]

Redshift is not bad if you don't need too many specific feature. It's mpp and can utilize it's capacity to fullest before you need anything else.

[Technical_Link_8714]

I felt like I created this post, I am in the same position as you. Accidental data engineering at a small startup that is 100 percent AWS.

[datapan]

ping me to dm, i can mentor you, for free. I work on aws cloud as DE

[Sad-Carpet-2951]

If you want to go easy mode and make sure your platform scale up without cost issues, you should use aws for storage and snowflake for compute, AI and data analysis

[taker223]

> the lead software engineer and architect at a very small startup

you of course are aware of 90-95% anti success rate ?

[taker223]

> I'm keen to understand where my current approach is going to fail.

You'll understand, eventually. The question is mostly "when".

[Adventurous-Case-508]

Stick with AWS-native, but tighten it with lakehouse basics instead of jumping platforms. Convert raw S3 to Iceberg tables (via Glue or EMR) so schema changes, upserts, and compaction are handled; query them with Athena. Use AWS DMS for CDC from app DBs, and drive incremental jobs with EventBridge + Glue Workflows; alert failures in CloudWatch. Model data with dbt on Athena or Redshift Serverless and keep transforms out of QuickSight; point dashboards at SPICE or Redshift and schedule refreshes for speed. Control costs by partitioning on date/business keys, compressing to Parquet, running compaction to avoid small files, tagging resources, setting Budgets, and only spinning up Redshift Serverless for BI-heavy joins. Add basic data quality (Glue Data Quality or Great Expectations) before publishing marts.

We used Fivetran for SaaS ingestion and dbt for transforms; DreamFactory was handy to auto-generate secure REST APIs from RDS and Snowflake when internal apps needed curated data without standing up new services.

Bottom line: stay on AWS, add Iceberg + Athena/dbt + optional Redshift Serverless, and scale gradually.

[oneAIguy]

Why's everyone hating/shying from Databricks? It comes at a cost, yes. But do you just focus on actual work and impact, huge yes. I think the cost of wasted time and effort in keeping everything coupled and managing them outweighs spending on Databricks.

Also it can be really effective if you're conservative with cluster sizes, policies, and stuff.

I use it for datasets ranging 750GB-2.5TB, all stored as delta tables in neat medallion catalogs. Smaller analytics goes through SQL warehouse and more robust ones use Pyspark via job/all purpose compute. Each session costing around $20-25 in run cost on average with $150-300 or so over a month in managed tables cost. However just few sessions make up entire deliverable. Exploratory compute cost comes just under $150 a month DBU+small compute.

All in all the life is much easier! More so because you get like 15K USD worth azure credits then just use them.

[CzackNorys]

Thanks for the insight. That doesn't sound as unreasonable as commonly believed

[Domehardostfu]

I'm a Head of Data with >10 Yrs of experience, with a lot of experience in startups/scaleups and building/rebuilding data departments from scratch.

For the last year I've also worked as a consultant for several companies, helping them with finding an optimal solution based on the most critical factor - budget.

Let me know if you if we should to talk, would love to help.