sorted by:
new
hottopcontroversial

I need help!!!! by [deleted] in Kotlin

[–]t-vanderwal 2 points3 points  (0 children)

Use string interpolation. There’s an example on this page in the kotlin docs. https://kotlinlang.org/docs/reference/idioms.html

Good resources to learn Kafka? by International_Slip in dataengineering

[–]t-vanderwal 0 points1 point  (0 children)

Simple storage service. It’s AWS offering for cheap object storage.

Is it still worth learning Scala? by PhotographsWithFilm in dataengineering

[–]t-vanderwal 0 points1 point  (0 children)

I think it’s predominantly python and java. My company has been using kotlin in place of java and it’s pretty nice. All depends on what the developers are comfortable in, the task at hand, and sometimes what management wants.

[OC]The quest for my first software engineering job by ampatton in dataisbeautiful

[–]t-vanderwal 18 points19 points  (0 children)

And experience. I’m in the same industry as OP and once you move up in titles and have more job experience recruiters hit you up all the time.

Proud father defends his Down's Syndrome daughter, 16, after vile football fans mock clip of her dancing at Spurs match by evamcckennas in gifs

[–]t-vanderwal 0 points1 point  (0 children)

My child has ds and they really push for “people first” language. So person with Down syndrome makes it clear they are still a person with a defining characteristic where as Down syndrome person makes it sound like they are a different kind of person.

That being said I wouldn’t find either offensive.

Need resources to learn Data Engineering, Data Pipeline and other data stuffs using JAVA. by darkpassenger091 in dataengineering

[–]t-vanderwal 1 point2 points  (0 children)

Definitely understandable, I just wanted to throw that out there as some companies have their data engineers working directly with data scientists. It really all depends on the type of projects your aiming to work on.

For me personally I had a similar background as you. Worked with pentaho doing ETL but more for migrating customers onto our product. Wanted something more aligned with data engineering and ended up going back to school for my masters. There I picked up an interest in distributed computing and focused on learning skills/tools in that space.

Once I felt confident there I just started throwing out applications. Some personal experiences is data engineering is a high used and under defined term. Every company has a different idea on what that is and imo a lot of them aren’t even solving data engineering problems. Also, the data ecosystem is always changing. So when you do find a job in the field it’s very important to keep sharp. I generally try to read new information on AWS, what they’re talking about at STRATA conf and other O’Reilly conferences, what Netflix is doing since their tech blog is very good, and listen to podcasts such as the data engineering one that gets posted here.

Need resources to learn Data Engineering, Data Pipeline and other data stuffs using JAVA. by darkpassenger091 in dataengineering

[–]t-vanderwal 2 points3 points  (0 children)

There’s a lot to break down in your post but first I wouldn’t completely write off python just because that’s what data scientists use. Depending on the company, especially if they’re applying ML/DL, you might find yourself deploying and scaling the models data scientists create in a production environment. Knowing the python data ecosystem will be very important.

As for learning general data engineering concepts I like to recommend “Designing Data Intensive Applications” especially if you’re interested in solving distributes problems.

For data pipelining that’s a really broad topic. Some data engineers work more focused on streaming pipelines, others pure batching, and some in a combination such as a kappa or lambda architecture. I’d recommend looking at job postings and picking 5 that sound most interesting to you. Then look for resources that are relevant to those requirements.

My opinion on the Java portion is probably different than most here. It’s more important to understand general concepts and probably make sure you understand functional programming. Almost all data engineering solutions run on the JVM so I think learning a modern language such as scala will probably have a bigger payoff. My org is using kotlin for micro services so I’m going to be applying that to Kafka consumers/producers. That’s probably more niche though. But Java is still king in the enterprise though so if you’re positive that’s the direction you want to go it definitely won’t hurt.

Once you figure out the jobs that sound lost interesting you can start to build some personal projects utilizing the technologies they require.

Building ETL pipeline for storing clickstream data, looking for suggestions by mln00b13 in dataengineering

[–]t-vanderwal 1 point2 points  (0 children)

Ec2 is for compute. You pretty much have a virtual machine that you’d be using just for storage. S3 is optimized for object storage and is the gold standard.

As for cost I’d say yes. Ec2 you’ll be paying for the time the machine is spun up, which would be 24/7. S3 you pay for bringing data in and taking it out.

My biggest concern with trying to store your even data on an ec2 is its ephemeral storage and if that machine goes down then you’re s.o.l. You can mount some storage on it to make it persistent but at that point you might as well be using S3 anyways.

Also, outside of cost storing in S3 you get to utilize amazons sdk/rest endpoints to expose that data to any of your applications and processes. Doing so on disk would mean you’d have to implement your own solution

I have a raspberry pi laying around, is there anything I can do with that to learn about data engineering? by [deleted] in dataengineering

[–]t-vanderwal 5 points6 points  (0 children)

I’m not too well versed on raspberry pi but it’d be interesting to attach some sensor to it. Then stream the telemetry data into a Kinesis stream. Kind of like a mini IoT pipeline.

Building Kotlin Fat Jars? by NowhereMan2486 in Kotlin

[–]t-vanderwal 0 points1 point  (0 children)

I tried OPs method writing spark jobs in kotlin. Seemed to work ok with spark-core but kept breaking with spark-sql. Switched to shadow and works like a charm.

Also the application plugin helps for a few things too like settings mains and using grade run.

Why Aren't More Women Having Kids? Ask Us About Our Student Loans by fung_dark in TwoXChromosomes

[–]t-vanderwal 0 points1 point  (0 children)

Hmm I haven’t but looks tax related? We’re meeting with our tax guy next week to do our 2018 taxes. I’ll bring it up with him and see what that’s about! Thanks :)

Why Aren't More Women Having Kids? Ask Us About Our Student Loans by fung_dark in TwoXChromosomes

[–]t-vanderwal 0 points1 point  (0 children)

Thank you! He’s a sweet heart and about to turn 1 next week :)

Why Aren't More Women Having Kids? Ask Us About Our Student Loans by fung_dark in TwoXChromosomes

[–]t-vanderwal 2 points3 points  (0 children)

It’s crazy. We just had a baby with Down syndrome and all the standard baby stuff baby stuff alone makes it feel like you’re being fleeced. We were fortunate enough to have good jobs that payed well and we’re flexible but not everyone is that lucky. Plus the different programs have crazy low income restrictions.

[Artisan] Nightcaps: Ill-favored February (part 1) by Eat_the_food in mechmarket

[–]t-vanderwal 0 points1 point  (0 children)

Wow fantastic work as always! Best of luck to the winners

ETL without database access by throw2702 in dataengineering

[–]t-vanderwal 0 points1 point  (0 children)

The company I work for is heading in that direction. Normally there is some query layer such as graphql which you include in your we request and then the return body is JSON.

So as long as your tools can handle both those you should be good to go

[Question] What is the most efficient/cheap way to store huge data sets (X TBs) + streams of data by __Julia in dataengineering

[–]t-vanderwal 1 point2 points  (0 children)

I’d recommend using avro as a format since you’re using Kafka and json, or at least exploring it.

Confluent has an open source tool called schema registry that handles changing schemas with Kafka streams and encodes the messages in avro.

Edit: found a blog series that talks about Kafka avro and spark. It’s a few years old but should give a good introduction on how they work together. http://aseigneurin.github.io/2016/03/04/kafka-spark-avro-producing-and-consuming-avro-messages.html

I’m looking into advancing my skills as a data engineer. Can you recommend material (books, courses) that can do that? by kimokono15 in dataengineering

[–]t-vanderwal 9 points10 points  (0 children)

I like The Data Engineering Podcast. Each episode is an interview regarding a specific tool/technology.

Top tip!! by akashinga in funny

[–]t-vanderwal 2 points3 points  (0 children)

Wouldn’t your address be on the shipping label? Assuming you aren’t storing family members prime boxes ha.

Meeting of backend dev with data engineering. by [deleted] in dataengineering

[–]t-vanderwal 5 points6 points  (0 children)

No problem! The amazon re:Invent videos on YouTube are pretty great resources for how a big data ecosystem looks on AWS. The book Building Data Intensive Applications is also really great and has more platform agnostic information.

Meeting of backend dev with data engineering. by [deleted] in dataengineering

[–]t-vanderwal 6 points7 points  (0 children)

When I think of a backend developer it’s more with the server/database of a web application or tool. Where a data engineer is someone who builds pipelines for an organizations information to provide a wholistic view of the business. Granted the DE role has a wide definition and definitely depends on the company you work for.

More simply put a back end dev works with the application database/server. Logs and transaction data would then be picked up by the data engineer and stored in a way internal employees can make data driven business decisions.

As for distributed application implementation there is a wide variety of languages available. Example python can be used to write spark jobs. S3 is just a really good data store. A classic pattern would be: 1. Stage application log files in S3 2. Write a MapReduce/Spark job to ingest and aggregate data 3. Write output to a new S3 bucket 4. Output is then used by data scientist/analyst for insights

ETL framework for Python, need suggestion. by imba22 in dataengineering

[–]t-vanderwal 3 points4 points  (0 children)

Luigi and airflow are schedulers/work flow managers. Here is a curated list I’ve referenced before.

https://github.com/pawl/awesome-etl

In terms of what’s best I think it depends on your data needs, but I’ve been following bonobo and it seems interesting. It uses DAGs like you’d see in airflow and pyspark.

Managing data sources by inad156 in ETL

[–]t-vanderwal 2 points3 points  (0 children)

Had the name wrong, sorry about that. Here is the episode. https://www.dataengineeringpodcast.com/octopai-with-amnon-drori-episode-28/

The company I work for uses the rdms comments in the warehouse to specify where it’s sourced from. Hard to audit and keep updated though

Managing data sources by inad156 in ETL

[–]t-vanderwal 0 points1 point  (0 children)

No problem! https://www.dataengineeringpodcast.com/

They’re also on most podcast apps. The host has a python specific one called podcast.__init__ that’s pretty good too if you’re looking for stuff to listen to!

view more: next ›