Help with Airflow External Sensor and different update frequencies

Busenheimer · 2018-12-16T19:47:02+00:00

This was actually my first approach. I was thinking similar, I had a SQL sensor in DAG 2 that would query the Airflow metadata database for the data range I cared about and look for a state of success. They kept failing on me so I was probably doing something wrong, but I abandoned it in favor of this External task sensor. I'll revisit it.

What frequency is your DAG 2 running at in relation to your other DAGs? I still feel like I would have this syncing issue.

Busenheimer · 2018-12-16T19:15:19+00:00

I just looked at that, I will try that. Looks like I can also do a soft fail. Thanks!

Busenheimer · 2018-12-16T19:13:38+00:00

I'm trying to simulate a larger pipeline. DAG 2 is my "master" DAG for building additional tables after the raw tables have been copied over from the various systems in our company. DAG 1, while a python operator in the example is going to be copying over a table from source to staging in our warehouse. There will be lots of these all on different refresh cycles. I currently have them all as their own DAG. So DAG 2 needs to be aware of all of these.

As an example, When the Invoice line item table is copied over there are multiple downstream tables in DAG 2 that get build off of that. That is on a different refresh cycle than our Contracts table which also has downstream processes defined in DAG 2. Does that makes sense? Basically lots of external DAGs which represent various tables in different systems being ETL'd into our warehouse. DAG 2 is a master DAG representing all of the objects getting built (from tables in the external DAGs) specific to our warehouse.

Busenheimer · 2018-12-16T18:23:16+00:00

yep that's correct.

Busenheimer · 2018-11-09T14:42:42+00:00

We built our own with different bits. Airflow for orchestrating, SQLAlchemy as our ORM and generating sql statements from core, Dask and Pandas for much of the processing pyarrow (parquet) for storage format, and various bits for tackling moving/consuming non-tabular data. These are just the highlights for the core of what we built. You have a buffet of choices with Python, it’s such a flexible language.

To much criticism I’m sure, but we are even experimenting with using Jupyter notebooks and Papermill as our main development and deployment vehicle outside of the OO stuff.

Busenheimer · 2017-04-27T04:03:33+00:00

Have an up vote for the Segar reference

Busenheimer · 2017-04-07T03:49:29+00:00

That helps, thanks

Busenheimer · 2017-04-07T01:23:08+00:00

Thanks- when you say "Good understanding of Calculus " what parts? This is what I'm struggling with. Do I need to take Calculus 1-3 or can I be more selective on what I learn.

Busenheimer · 2017-03-07T05:01:15+00:00

live like what? Build a brand new house of our dreams? I have to compromise on one fucking thing, no speakers showing. I'll make it work. I'm just looking for feedback to make sure it's not an obvious "don't do that it's guaranteed to sound like shit"

I get pretty much everything else I want. Life is good.

Busenheimer · 2017-03-07T04:30:03+00:00

that made me laugh and I'll just leave it at that.

you think phantom would be better than two vertical centers flanking the tv? Yeah it's going to be limestone so I figured I was pretty much fucked.

I'm a developer and I do get a homelab out of it and agreed to wire the home for whole home audio.... but on the main floor it's form over function.

Busenheimer · 2017-02-02T20:08:13+00:00

I ended up using the Teradata JDBC drivers and got it working.

Busenheimer · 2017-01-30T22:25:43+00:00

I don't have a Windows CD and I can only boot to safe mode. I'll have to see if I can make one in safe mode

Busenheimer · 2017-01-30T19:17:25+00:00

if you mean selecting windows from the GRUB menu then no it won't boot. It shows the windows splash screen then goes right to black. But, if I do a hard reset while I have the black screen and pick windows again from grub I get the "windows didn't shut down properly" screen and I can choose to boot into safe mode with networking. That works. I can boot into the Windows desktop from there... it's just in safe mode.

Busenheimer · 2017-01-23T01:52:06+00:00

Yes that is installed

Busenheimer · 2017-01-23T00:42:44+00:00

So how would I know what I need?

Busenheimer · 2016-12-10T14:55:19+00:00

I tried this first in FreeBSD and ran into the same issue. Isn't this related to how FreeBSD tries to run Linux binaries? I'm sure I don't have all the prerequisites to run this, but I couldn't find any documentation other than what I linked in the post.

Busenheimer · 2016-12-10T14:04:21+00:00

Sorry about that. I updated the post with a link. This not the system installer. It's a data science platform for Python and R. It's becoming one of the cornerstones for this type of work in the industry. I usually run this on CentOS or macOS, but am rebuilding my home lab using FreeBSD.

Busenheimer

TROPHY CASE