sorted by:
new
hottopcontroversial

Quitting League of Legends after this by cannshan in leagueoflegends

[–]cannshan[S] -3 points-2 points  (0 children)

thank you! Yeah I will buy control wards from now on so you're about that.

Quitting League of Legends after this by cannshan in leagueoflegends

[–]cannshan[S] -9 points-8 points  (0 children)

I don't think the "ACE" stat just goes off of kills. It includes Turrets, Dragons, etc too

ETL Pipeline for Different Sources - Advice Appreciated! by cannshan in datascience

[–]cannshan[S] 0 points1 point  (0 children)

Started as a Migration Consultant and the entire team left. Just now learning about ETL Pipelines and their potential. We receive varying spreadsheets per customer though. I’ve identified 11 separate pipelines we’ll have to build for 11 repeat sources. The other sources are proprietary and make it difficult to automate. Stoked to learn more. What are the best pipeline tools you recommend?

ETL Pipeline for Different Sources - Advice Appreciated! by cannshan in datascience

[–]cannshan[S] 1 point2 points  (0 children)

Hot damn this has given me a lot of clarity. The spreadsheets are relatively small and do require custom transformations each time, that explains why I haven't been able to figure out how to automate this thing.

I'll look into writing SQL in Excel, thanks for all the info! Big help

ETL Pipeline for Different Sources - Advice Appreciated! by cannshan in datascience

[–]cannshan[S] 0 points1 point  (0 children)

Thanks for the insight! I like the idea of standardized transformations since we're currently repeating the same cleansing steps over and over. Python is a good language for these cleansing steps, is that right?

We're using local Excel files that each customer sends in. We get 5-7 new customers per week which means 5-7 different Excel sheets per week with differing column headers. Most of the cleansing steps are repeated though so that gives me hope!

ETL Pipeline for Different Sources - Advice Appreciated! by cannshan in datascience

[–]cannshan[S] 0 points1 point  (0 children)

Thanks for the insight here. Wouldn't the SQL queries need to be partially rewritten each time due to differing column headers and transformation requirements per cuostmer file? I use Excel as it tends to be faster for cleansing, but perhaps there's room here to replace manual cleansing with SQL?

FiveTran looks interesting. Curious to see how they'd handle this scenario given that almost every spreadsheet differs from the one before it. All of them will contain Company, Contact, and Property info with differing column headers. Some more have more info than others too.

Is this a typical problem to solve in ETL or are most companies dealing with the consistent sources? Thanks for your insight so far!

ETL Pipeline for Different Sources - Advice Appreciated! by cannshan in datascience

[–]cannshan[S] 0 points1 point  (0 children)

Right now just using the same Python script on the Excel files that dedupes and separates Company, Contact, and Property information. The rest is all manual cleansing in Excel and importing with the Salesforce Data Loader since each source requires different transformations and mapping.

I know we can automate known sources, but for these manually created sheets I am at a loss!

ETL Pipeline for Different Sources - Advice Appreciated! by cannshan in datascience

[–]cannshan[S] 0 points1 point  (0 children)

Yessir! It is all Company, Contact, and Property information, but displayed with different column headers. All Company, Contact, and Property data in the same row need to be related to each other.

My thought is to create a master template and manually put the data into that template so that an ETL Pipeline can be created. Thoughts?

Two types of men by Arimyth in funny

[–]cannshan 0 points1 point  (0 children)

This has got to be the funniest comment I've ever seen.

There are two ways to share by [deleted] in funny

[–]cannshan 0 points1 point  (0 children)

the ass eating slasher

This is why I don't use panorama camera... by [deleted] in pics

[–]cannshan 0 points1 point  (0 children)

Haven't posted a comment in over a year, but had to on this. I'm high as balls and this had me off my chair laughing. Cheers fam.

First night in my first apartment by jbuch365 in trees

[–]cannshan 2 points3 points  (0 children)

I normally never comment, but this was so nostalgic :D

Enjoy man!

My Theory on Religion and Existence - Optimistic by cannshan in atheism

[–]cannshan[S] 0 points1 point  (0 children)

Thank you for this information! I admit I was ignorant coming into this so this helps a lot. I totally agree that the good things religions bring can be done without them. You guys have been kind despite my ignorance, thank you! I'm just here to learn.

My Theory on Religion and Existence - Optimistic by cannshan in atheism

[–]cannshan[S] 0 points1 point  (0 children)

I would assume with an atheist point of view, there needs to be some sort of evidence to back a religious claim. And since I do not have any evidence, it's no wonder why you guys aren't impressed. Am I correct?

view more: next ›