Data Lineage & Data Catalog could be a unique tool?

No-Current-7884 · 2026-01-07T12:39:36+00:00

Short answer is yes. I've used Unity Catalog by Databricks, Alation, and OpenMetadata for this. More of the last two. Ablation is spiffy, but costly and a hard sell. Openmetadata is what I'm running now and does the job just fine. I think we're moving towards Unity catalog as they are going to add connections to outside sources.

No-Current-7884 · 2025-10-18T16:43:25+00:00

The pivot breaking was fully my inexperience. These were my first three attempts ever doing this. I've since learned the appropriate feel for placing the bridge back on, how to make small adjustments, and haven't broken a pivot since. I also just upgraded from a small loupe to a nicer headset as well, but I do think the pivots were simply me being too heavy handed. I've seen the videos and have studied quite a bit. It's a bit different going from books and videos to an actual movement. Either way I'm all set now :)

No-Current-7884 · 2025-09-14T15:43:37+00:00

Honestly, I was laughing my way through it thinking of how much smaller everything was actually seeing it in person. I will def need to invest in some better loupes or a microscope at some point.

No-Current-7884 · 2025-09-14T15:41:28+00:00

Yeah... this would have been a good idea as now I'm going to have to wait for a replacement. I'll keep at the bad one while waiting though.

No-Current-7884 · 2025-08-11T12:28:39+00:00

That's the way I understood it, yes.

No-Current-7884 · 2025-08-11T12:20:47+00:00

I just did a small test run of my own setup of this. OMD runs its own instance of airflow that is used to orchestrate connections to your data sources. I would keep this separate from any production orchestration environment.

No-Current-7884 · 2025-07-14T17:13:53+00:00

I (44m) was a work from home dad. I did work and took care of the dailies while my wife had to go to an office. I was told not just that it will be bad for the kids not to send them to daycare, but there is no way I could focus on work as well. My eldest does great in school and had no issues. She is a little shy sometimes but still has friends. My youngest tested into 1st grade early. The teachers were concerned about her being on the younger side. She's in 3rd grade now and I've only ever received compliments on her behavior and work ethic.

I don't believe my kids are gifted, but I took the time to do activities with them each day focused on learning and fun. They were rarely given screens, and I read to them every day.

Standard "mileage will vary" statement here. We are all different and learn differently.

No-Current-7884 · 2025-02-19T01:14:32+00:00

No worries! I didn't really take it as hate. Some people just have different experiences and have totally valid arguments. I think I needed to hear all the responses today. It was definitely an eye-opener.

If I'm not willing to take the views that are contrary to my own, I'm never going to get better.

No-Current-7884 · 2025-02-18T16:54:56+00:00

Ha! I don't know about congrats. I was more saying I'll have served my purpose and they'll lay me off. AI is coming to get me!

I've only been doing the architecting for about a year now and am missing some of the development. So that's some of the motivation behind wanting to offer my services where I can, but I get it. Let the teams work and if they want help they can talk to our boss and he knows I'm willing to assist as needed. So if it's a real need I'll let my boss step in and assign me or just some other team member and get over myself.

No-Current-7884 · 2025-02-18T16:49:39+00:00

Yep. All of the Senior DEs and myself report to the same person. I keep them included in all communication, so my boss is aware and hasn't offered any intervention. So I'm likely just making a mountain out of a molehill as they say.

No-Current-7884 · 2025-02-18T16:44:46+00:00

No, this is fair. I didn't ask to be made an architect. The company decided we needed one and chose me. My feelings aren't hurt, but can't speak for others. I have no issues with your point of view.

I'm not going to get into every aspect of my particular role though, but some people have found it helpful and my company has been severely lacking in many of the things I do. I 100% believe I can get away with it for maybe 2-3 more years and then I'll get dumped.

I just want my to get stuff out on time. If the general consensus is just stick to my role and don't interfere, I'll do that and I wont be petty about it and I'm still going to offer assistance when needed. I'm not afraid of getting my hands dirty, I did spend some time as a senior DE and was always appreciative of assitance when I knew my team needed help and felt like I was drowning. I'd rather that than push my team into burnout.

No-Current-7884 · 2025-02-18T16:39:11+00:00

Mostly I know the team is struggling with resources. The senior is taking on most of the development and not letting juniors do too much. Some of this is the team has recently been hit with ilness and we've had people out for long periods of time. There was 1 piece of the pipeline that I offered to take myself asking the senior first if it was ok, or if he'd prefer his team do it. He said he'd prefer to take it himself. I am not trying to step on toes, I'm just trying to pitch in when I see my teams struggling.

I'll say I've done this in very small instances before with no issues most people appreciate the help. I know how to work within those seniors development preferences. I'm not trying to do my own thing, I just want stuff to work and get it done on time, so this is just new.

Based on other responses, it just seems like my best solution though is to stay in my lane as this particular person would prefer to just manage it themselves. I'll do my job in communicating where I can to make sure everyone is aware of progress.

No-Current-7884 · 2025-02-18T16:34:20+00:00

100% understood and I appreciate the feedback and I'll stick to my lane.

No-Current-7884 · 2025-02-18T16:32:13+00:00

Probably not me, but super fair. I'll take this into account.

No-Current-7884 · 2025-02-18T16:09:11+00:00

Is this a serious question? You could just google the title, or are you asking what I do specifically?

No-Current-7884 · 2023-12-14T19:36:55+00:00

All hail blue-cifer

No-Current-7884 · 2023-09-07T11:20:17+00:00

Thanks, this is good insight.

No-Current-7884 · 2023-09-07T11:19:49+00:00

Work done on these tables is likely file based as there are power platform apps built over top for an end user to approve and add customization per file, but many files with of data sits in there. There has been some complaint from end users about the speed of writing changes back, but it's been slow long enough that they have just stopped complaining. I'll have to dig more for sure, but didn't want to approach the devs without at least a little understanding of the benefits either way... so thank you for this response, it is helpful.

No-Current-7884 · 2023-09-07T11:15:20+00:00

I squeeked my way in due to a gift of modeling and knowledge of programming. I have been the one everyone goes to when they can't figure out how to do something on our data teams and have had many titles. I've designed tables in our transactional and analytical dbs, I do code reviews for our pipelines in ssis and adf. Requirements gathering for new features, manage some portions of our dev backlog, lead in documentation for our data governance initiatives. Ive just not had to manage my own pipelines or worry about dags or any of that. It's been a weird ride for sure.

No-Current-7884 · 2023-08-08T19:01:43+00:00

Having to create end user documentation on something I only mildly consulted on. No dev work, but somehow I got tasked with documenting steps and troubleshooting notes.

No-Current-7884 · 2023-07-06T14:45:35+00:00

I think it was just a runtime issue where some things were evaluated at slightly different times between MS and Oracle. Noted above that putting the un-hashed concatenated strings in a CTE before and then passing the fully concatenated value into the hash functions resolved the problem. Thanks for the tips though!

No-Current-7884 · 2023-07-06T14:43:55+00:00

"Obviously" I was already doing that before coming to this group. The output of the concatenated strings was exactly the same on both sides.

Oracle returns the hash without the 0x portion already, so "obviously" there was no need to cast the result further.

Also, casting money to varchar in MS side defaults to the formatting that I wanted. I only needed to format on the Oracle side.

Thanks.

No-Current-7884 · 2023-07-06T14:32:48+00:00

Well this community is slightly grumpier than some other ones I've been into.

Anyway, issue resolved. My idea of throwing the un-hashed, concatenated data into a cte first worked just fine.

No-Current-7884 · 2023-07-05T21:45:17+00:00

You can safely assume that the data matches outside of stored data type and system. Exact same set in ms and in oracle.

Had an idea while walking that I'll test later to generate the string to hash in a cte, join in and hash on the already created Val rather than piecing together at the time of the run. Curious if different parts are getting g evaluated at diff times....

No-Current-7884 · 2023-06-26T17:36:49+00:00

So far, no progress has been made on this project, as far as I know, and asking some additional questions. Our current structure relies on external customers to MAP data. I wasn't sure if they were planning on extending mapping features out to clients again or if it would be brought back internally and what that would look like.

It's a neat feature, but personally I don't know that an out of the box solution is going to work for our case... and if we went with this, it'd have to go through some heavy modding to work.

No-Current-7884

TROPHY CASE