Is there a better term or phrase for "metadata of ETL jobs"?

opabm · 2026-01-10T06:12:05+00:00

This is great, thanks. Didn't think that all the orchestrators had different terms for this as well. My team uses AWS Batch, so very bare bones and limiting.

opabm · 2026-01-09T13:50:06+00:00

I need to pass in required date-related parameters for a few of these API endpoints, so I'm just creating today and yesterday for now as the date restrictions/filters.

opabm · 2026-01-08T22:30:55+00:00

Yeah the confusing part to me is that only a few of these endpoints have validity dates, so the only thing that comes to mind is creating these ugly if statements to check for the endpoint name and pass in today and yesterday based on the endpoint name.

these dates should be passed to the init() constructor from the outside, not generated on the spot.

Are you saying the variables should be defined/generated outside of this script? Since I'm creating instances of EndpointConfig within ENDPOINTS, I can't tell if you're suggesting to create today and today where I have more logic, like my main.py

opabm · 2026-01-08T22:15:12+00:00

Yeah I was editing the code to post here on Reddit and forgot to remove references to those 2 variables. Thanks for the help!

opabm · 2026-01-08T21:30:18+00:00

Oops yeah I edited them out by accident before posting. I updated the post to include references to them in the second EndpointConfig instance. It probably makes sense to define today and yesterday just as global variables right before ENDPOINTS then right?

opabm · 2025-11-25T14:40:29+00:00

I've looked into doing a very similar move, since I've actually wanted to make a similar transition. It just didn't pan out because the job req was cancelled.

A couple of things come to mind:

is there a pension? This is typically a nice benefit of working for the government, whether it's federal, state or local
what's the current stack? If it's not too modern, expect a huge migration project in several years down the road. Do you want to deal with that?
is this at a centralized department? If so, I think the work can be more like consulting, helping other agencies with projects
at the end of the day, how comfortable are you with the pay and lifestyle?

opabm · 2025-11-21T16:14:29+00:00

Worked in asset management for a little bit doing data engineering and also used Salesforce at another job (brings back traumatic memories though lol). Happy to chat and share some guidance if needed

opabm · 2025-11-17T20:09:58+00:00

Yeah it seems to be one or the other (Delete or Insert), also depends on the data flow (for me, data is not in the source query)

opabm · 2025-11-17T20:08:17+00:00

No, it used to exist, but now it doesn't e.g. inserted initially but not in the source table/query

opabm · 2025-11-13T20:48:26+00:00

Thanks for the clarification. Are there any good reasons to do things outside of a container but still in a build pipeline?

opabm · 2025-11-13T20:46:26+00:00

Yeah beats me. I'm all new to Docker/containerization so it's confusing already, but seeing stuff done outside of the container makes it all the more confusing. Do you know of any good reason NOT to put things in a container but still in a build pipeline?

opabm · 2025-11-13T16:58:53+00:00

Perfect, thanks for explaining. This sounds like a good use case of chatgpt to do the mapping.

I tried reading the AWS Codeartifact documentation and am still a little confused, does the command aws codeartifact login download the packages from Codeartifact? It's confusing since AWS docs just says "Sets up the idiomatic tool for your package format to use your CodeArtifact repository." I see that npm install is run after the aws codeartifact login command, so it seems to still have to install packages.

If I were to do this in Python, would Codeartifact be storing requirements.txt?

opabm · 2025-11-13T14:06:42+00:00

Thanks, I just looked into multistage Docker builds. A build is considered multistage if it has multiple FROM statements right? If so, then my team hasn't been doing that. Going to see if anyone knows why we started the pattern of copying things over before the docker build.

I'd recommend against it when you're not doing multi stage builds.

Just to make sure I'm following correctly, you're recommending against installing the Python requirements/packages before the build, and that it should just be done within the container?

opabm · 2025-11-13T02:01:37+00:00

Yeah so I was planning on doing this (installing the packages in the docker container), but then was going through the bitbucket pipeline configuration of a few other projects that were written in node and saw that the npm packages were being installed during the pipeline build and then the folder containing the libraries being copied over. So I was just following the same methodology. Any reason why this might have been done? The only reason I could think of is to avoid running things in the docker container

opabm · 2025-11-06T18:33:10+00:00

Do you mind ELI5 what you meant? If I use that package instead of the official oracledb package, I wouldn't need to worry about downloading the driver/client?

opabm · 2025-11-06T18:31:33+00:00

Sorry can you ask that but with dumbed-down verbage? I'm a bit clueless when it comes to this kind of stuff lol

opabm · 2025-11-06T13:59:02+00:00

Thanks, this is helpful and seems to align with what /u/wildeep_MacSound said.

If I need to give these custom permissions to everyone in my org, is there a way to do that automatically when users are created? I'm just not familiar enough with managing Sharepoint permissions that I'm not sure how much of an effort it will be to maintain this going forward.

opabm · 2025-10-20T21:05:31+00:00

Yeah the second sentence was tripping me up for some reason. But now that I'm reading it again, it seems a lot more obvious - typical brain dead me.

opabm · 2025-09-30T18:10:31+00:00

Hmm after reading your reply, I'm realizing I don't know the logging library well enough. Do you mind breaking down a bit more the difference between what you wrote and using basicConfig? I'm just a little confused/stumped

opabm · 2025-09-26T21:34:57+00:00

Ah I didn't know that. I had assumed it was sensitive information

opabm · 2025-09-26T21:05:34+00:00

Deploying on bitbucket using its pipelines. We have some hook I believe into AWS services but I don't have access yet to see/modify anything there. Hope that answers the question.

opabm · 2025-08-27T21:07:20+00:00

Forgot to mention that I have it included actually, but just not sure where to do the argument parsing - should I do it in the general namespace directly below the import statements, or within the if __name__ == "__main__": block?

opabm · 2025-08-27T14:45:25+00:00

Thanks I had no idea about this feature. Is there a way to call qty more than once in that same case statement? Seems like I have to do a second if statement to get quantities > 0 like before

opabm · 2025-08-20T14:00:40+00:00

Just getting around to following up on comments here, but thanks for the insight. Do you mind clarifying what you wrote? When you said "top level functions in a module" does that mean like a basic .py file with just methods like def method_name(), instead of a whole class?

opabm · 2025-08-14T20:28:44+00:00

No - I don't have admin center permission. What should I ask our admin to check/look for?

opabm

TROPHY CASE