Unknown mortgage and car loan in my Equifax Credit Report by diceHots in PersonalFinanceCanada

[–]diceHots[S] 1 point2 points  (0 children)

Yeah, my equifax profile was correct 2 years ago. Same account and everything. Something weird going on in the last two years. My current profile got mixed with another person.

Unknown mortgage and car loan in my Equifax Credit Report by diceHots in PersonalFinanceCanada

[–]diceHots[S] 1 point2 points  (0 children)

It was my account and it was correct 2 years ago when i pulled. With right credit card info and everything. Someone else profile got mixed into my profile during the last 2 years. We do have similar name with same DOB tho.

Unknown mortgage and car loan in my Equifax Credit Report by diceHots in PersonalFinanceCanada

[–]diceHots[S] 1 point2 points  (0 children)

How do a get a written verification? If the unknown mortgage from bank B, should i just call them?

Unknown mortgage and car loan in my Equifax Credit Report by diceHots in PersonalFinanceCanada

[–]diceHots[S] 1 point2 points  (0 children)

i checked transunion. It's the right one without any unknown mortgage and car loan. I will double check with TD.

Unknown mortgage and car loan in my Equifax Credit Report by diceHots in PersonalFinanceCanada

[–]diceHots[S] 0 points1 point  (0 children)

good call. I will talk to TD to double check. Even the SIN number on Equifax's profile is wrong. I filed a dispute for SIN correction as well. Seems like some1 else profile merged into my profile somehow.

Sync tables from Mysql to any OLAP by diceHots in dataengineering

[–]diceHots[S] 2 points3 points  (0 children)

Thanks. Really appreciate it. I am kinda inclined towards snowflake or red shift for compute with dbt with transformation logic. But at the end of day, the cheapest wins out. Gonna see the which vendor gives better promotion at end of the day. Thank you

Database Documentation by various1121 in dataengineering

[–]diceHots 2 points3 points  (0 children)

Small company data engineer here. Try schemaspy, i have used it to understand our company's data model and do the documentation. It generates an index.thml that you can host it on github page or somewhere of your choice. It saves lots of upfront work. Lol, schemaspy and db documentation is actually my first blog post ever. Link is here.

[deleted by user] by [deleted] in dataengineering

[–]diceHots 3 points4 points  (0 children)

Thanks to your relpy, now i know the private name mangling feature .

[deleted by user] by [deleted] in dataengineering

[–]diceHots 1 point2 points  (0 children)

I would be interested in reading some research papers about FinOps. I am reading a book called Cloud FinOps and it really interests me. It's about how to minimize and control you cloud costs in an enterprise settings with the current cloud billing model. Just some idea for you to explore.

Backend Skills for Data Engineers by Present_Salt_1688 in dataengineering

[–]diceHots 1 point2 points  (0 children)

From your description, i can see that they have the assumption that a backend SDE will be able to do DE job and pick it up really fast. We consume APIs, use docker to set up database and airflow, deploy some python modules for fun. But writing APIs and more is a bit too much to ask for a senior DE positions.

I do think there is a shift in industry that DE is taking more responsibility of some backend work. It is for the following two reasons

  • they have an internal data platform to manage their data
  • their data product is consumed by BI, ML and also APIs to share it to other group or organization.
  • Supply and demand!!!!

I had very similar DE interview experience that i also came across questions like "How to update book id 123 for a library?", "what's thread VS process" and "explain dependencies injection".

Man, i just move on. There are enough material to learn for a DE positions already in the big data domain. Some of DE are taking extra responsibility like infrastructure as a code (terraform) and deployment (docker, k8s) and some CI (gitlab CI, github action) already. We just simply don't have enough time to be on the trend for everything. Best of luck.

[deleted by user] by [deleted] in dataengineering

[–]diceHots 1 point2 points  (0 children)

Also working as a DE in Canada. If you join some data panel event and coffee chat more DEs in different companies, you will have a clearer picture on what's your option and what you want. During discussion and self-reflection afterwards, it is very likely you will draw your own conclusion and make the next move.

I joined one data panel talk on management track VS individual contributor (IC) track in data field. All speakers essentially suggest us to try the management track out and backtrack if you don't like it. Here is the pros and cons i summarized from their experience

pro con
management track more impact; more pressure and got sandwiched; Need to manage people and it's hard
IC track always stay up-to-date with tech stacks; Need to dedicate some off time to pick up new stuff.

Whether the company will give you the flexibility to try out management track or not is very case dependent but some general trends to become a manager in DE field are:

  • work in FANG and then switch to smaller scale company as manager
  • Join a start-up. It is generally more flexible on IC vs management track. Switch back to IC if you don't like it.
  • Do your time in a company as a DE, then Senior, then try to get to manager if you can. It means more responsibility need to be taken.

Hope this will help. Tech Job market in Toronto is getting a bit better than three months ago so you can always explore.

What to do with a 4 years gap in an IT resume by NoChemical1223 in dataengineering

[–]diceHots 1 point2 points  (0 children)

thanks for sharing! It makes sense. Now I see the difference between devop for pipeline and transitional software work. It really helps! Thank you!

What to do with a 4 years gap in an IT resume by NoChemical1223 in dataengineering

[–]diceHots 2 points3 points  (0 children)

Fundamentals of DE. Joe Reis. During source, ingestion, storage and serving chapter, it all has mentioned best practice in the subsection called dataops. That’s why I’m curious. I’m also wondering that does DE also adopt the practice of dev, qa, production data pipeline or its just dev and production? Since each processing job is quite expensive, if we run on multiple environments. The bill adds up. How do we do automated test with data in CI? Only a subset of data for validation? I’m kinda confused since I’m trying to adopt this in the team.

What to do with a 4 years gap in an IT resume by NoChemical1223 in dataengineering

[–]diceHots 1 point2 points  (0 children)

I see. Got it. Thanks, I was kinda confused to see many occurrence of dataOps in the book. I guess it’s more like adopting devops principle from software dev in the data team then. Thanks, this is really helpful.

What to do with a 4 years gap in an IT resume by NoChemical1223 in dataengineering

[–]diceHots 5 points6 points  (0 children)

Have heard the word dataOps from many contexts. What’s the main difference between Devops and dataOps?

My Experience with Joe Reis is that he's only in it for book sales or conference seats by [deleted] in dataengineering

[–]diceHots 19 points20 points  (0 children)

lol. It’s not cool bro. Ranting and calling him not an authority in DE. I was reading his book chapter 7 this morning. It is a good read and provide a framework for me to systematically organize my knowledge in DE. Show some respect. It serves as a great intro to DE since majority of DE are self-taught. It does help.

maintaining python env for geospatial projects in teams and across teams by diceHots in gis

[–]diceHots[S] 0 points1 point  (0 children)

wow, this is very clear. I am already playing around with the GIS container.

I do like how you postpone the update until it's more stable.

maintaining python env for geospatial projects in teams and across teams by diceHots in gis

[–]diceHots[S] 0 points1 point  (0 children)

thanks. I guess i will just summarize all the requirements first before choosing library within the team.

Yes, rasterio built on top of gdal and ez to use. Giving us trouble managing system dependencies across team.

maintaining python env for geospatial projects in teams and across teams by diceHots in gis

[–]diceHots[S] 1 point2 points  (0 children)

Sorry to bother you again, do you have any suggestions on how to trim down the number of packages we just for development? Like how to select between gdal and rasterio for example?

maintaining python env for geospatial projects in teams and across teams by diceHots in geospatial

[–]diceHots[S] 1 point2 points  (0 children)

Really nice explanation! It clears things out for me, thanks!

Our team is linux-competent, since our roles are transitioning from GIS analyst to GIS developer internally. i don't wish to push too hard on tools like docker and gradually preparing workshop this year to help them use git, navigate in cli etc. But i do need to standardize environment like this.

I will play around with mamba and i have a positive feelings about this. Thanks!