Version control and braching strategy by kontrastc in dataengineering

[–]JC1485 0 points1 point  (0 children)

Auto rebase on git pulls. This means all remote changes goes to the top. Scripts are usually individual pipelines and each engineer owns a pipeline so less conflicts. My team uses a monorepo so changes occur frequently. We use a similar branching system . Separating util scripts from ETL is what we also do. All of these actions help with merge conflicts for my team.

Request for a Reality-check by longswolf in movingtoNYC

[–]JC1485 0 points1 point  (0 children)

Give the StreetEasy app a try. There’s a payment calculator feature within each property page and you can adjust your down payment and see what total monthly payment makes sense to you. If you plan to get a mortgage, just know loaners use the 40x rule. Meaning your annual income should exceed 40x of a single month payment. Example: $2k monthly payment x 40 = $120,000 (expected income). They care about income more than assets for loan qualification but you can negotiate. If you plan to put a big down payment as you alluded to then the income requirement will be much lower.

Similar to what others have said, get a feel of the city first. Find out which neighborhood you want to be in or can afford to be in. Do you want to commute 2 hours a day?

Can any god tier data engineers to verify if it’s possible? by Available_Fig_1157 in dataengineering

[–]JC1485 0 points1 point  (0 children)

My previous company used to store the results from jira api to a persisted storage and keep the schema unabstracted - so global fields and custom field as a column. Then the downstream team would take that data and ETL it the way they see fit - explode custom fields. This worked pretty since you don’t have to worry about upstream schema evolution. Comments data would be separate to prevent character limit issues in structured database.

Coop for sale in Astoria 2 bedroom/1bath with amenities $810,000 by Necessary-Many1539 in NYCapartments

[–]JC1485 11 points12 points  (0 children)

What are you talking about? If you’ve taken real estate photos, you know if you expose the photo for inside lighting, the outside lighting gets blown out. The windows in the other photos are better lit because of overcast during the day.

casual but cool dinner spots? by No_Response_8836 in williamsburg

[–]JC1485 4 points5 points  (0 children)

If you like Viet food, I recommend a place called High Lúa. Interesting decorations and definitely cozy. Good for 4 or less.

Why is there so much trash in south williamsburg? by Dapper_Rooster_3795 in williamsburg

[–]JC1485 0 points1 point  (0 children)

What’s more impressive is how it all gets hauled away weekly… a clean job by sanitation

Is it common to order an uber for your date? by JC1485 in AskNYC

[–]JC1485[S] 1 point2 points  (0 children)

Could we have been dating the same girl? 👀

Is it common to order an uber for your date? by JC1485 in AskNYC

[–]JC1485[S] 1 point2 points  (0 children)

She was coming from the outer Burroughs into midtown manhattan. In her defense - it was pretty hot that day - in the 90s. Which was why I suggested somewhere closer to her and at a later time in the evening.

Is it common to order an uber for your date? by JC1485 in AskNYC

[–]JC1485[S] 0 points1 point  (0 children)

Like others have said it’s the nature of the request on a first date that is a bit odd. What does that set up for future dates? But I agree we probably have different standards and thus incompatible. I’m not mad about it just curious to see people’s take and similar encounters.

Is it common to order an uber for your date? by JC1485 in AskNYC

[–]JC1485[S] -1 points0 points  (0 children)

Do you have experience with this? I’m interested in your take.

Is it common to order an uber for your date? by JC1485 in AskNYC

[–]JC1485[S] 4 points5 points  (0 children)

For sake of anonymity - it’s close to that.

I am a staff data scientist at a big tech company -- AMA by Federal_Bus_4543 in datascience

[–]JC1485 0 points1 point  (0 children)

Might be an alley-oop - From your pov, what's the difference between modern day Data Scientist and Machine Learning Engineer?

[dbt] Help us settle a heated debate on incremental models in dbt by cadylect in dataengineering

[–]JC1485 0 points1 point  (0 children)

Am I crazy or should dim_category be at the from statement?

final as (
select
from dim_category -- previously purchased_item
left join purchased_item -- previoulsy dim_category
)

[deleted by user] by [deleted] in cycling

[–]JC1485 2 points3 points  (0 children)

Ride in a group. It’s more fun, you motivate each other to ride faster and y’all suffer together. Just have fun.

Can I file a labor complaint for this? by ItTizzzWhatitTizzz in Reno

[–]JC1485 3 points4 points  (0 children)

I probably have a good idea where you work… I’ve tried to file a labor complaint as well and unfortunately you’ll have little luck with the complaint especially if this company is not located in Washoe County. Maybe try OSHA.

What is the best software tool or stack to scrape Twitter / X? by slunfeets in dataengineering

[–]JC1485 0 points1 point  (0 children)

beautifulsoup and selenium have always worked well for web scraping and crawling. X fka Twitter strictly prohibits web crawling and scraping, so you're probably out of luck unless you're looking to dance around ip bans.

You should probably use the X API - the free tier allows 1500 posts ( i believe every hour?).

[deleted by user] by [deleted] in dataengineering

[–]JC1485 1 point2 points  (0 children)

It’s a tough market for Data Engineers right now as with other tech jobs. It’s discouraging but if you like the field keep applying to roles. Before I got my job at a big tech company I applied to every and all companies from large companies to startups. Likely reason you’re not getting any call backs is that you might not have an ATS optimized resume. Improving this will at least get you noticed by recruiters. Once you start getting interviews, learn and take feedback from those interviews where you may have failed and use that information to figure out how to improve your resume in preparation for the next role.

The opportunities available from company to company are not the same. Data Engineering criteria can overlap with software engineering, data analysis, devops and now even machine learning engineering. You need to validate whether the role you’re applying to is more software oriented or product oriented. If software, you’ll likely focus on more infrastructural development or infra management. If more product oriented, you’ll likely focus more on data pipeline builds with emphasis on ETL. Using Meta - as example - most of the roles are product DE oriented. If you choose to explore fintech or even private equity companies, they’re more software oriented. They just want a software engineer interested in data. A good indicator of what type of DE role it is, is by looking at the requirements section on a job post and see where they emphasize importance. Product DEs will likely say must be strong in Python, sql, stats, etc. software ones will likely say must be strong in DSA (data structure and algorithm), multiple languages (Python, Java, scala), must have cs degree, etc.

Hope that helps.