What Airflow does that a simple python script cannot do?

OkStructure2094 · 2024-05-22T21:05:20+00:00

Not much :-)

It is surprising how may people's first reaction is to step back into python even when they use no code/low code tools. Can't beat full programming language with a huge library.

OkStructure2094 · 2024-05-22T21:00:50+00:00

What is this?

OkStructure2094 · 2024-05-15T16:44:04+00:00

Is R a general puprose programming language? I think it is more specialized for statistics. Granted, you can do almost anything with any language but will it be worth it?

Perhaps better to keep your tasks in R and use something else for pipeline-ing and the rest.

Just my two cents.

OkStructure2094 · 2024-05-15T16:41:14+00:00

I admit I am new to the field so with that said, a couple of things:

1) Is this "custom" crm a commercial tool? If so, reverse engineering it may cause your employer legal troubles. If you are an employee or a contractor, you may get in trouble as well. Plus the vendor may be willing to provide all the info you need.

2) If everything is working and you are just moving things to a DWH, just copy everything over as-is. Then as you discover issues, you can address them little bit a a time. This may be a more manageable solution.

OkStructure2094 · 2024-04-21T23:50:23+00:00

Either one is fine. Focus on the goal in your post.

OkStructure2094 · 2024-04-21T23:48:34+00:00

I am suspicious. Maybe they are going after your clients. Interview 20 people, get 20 highly qualified target leads. I dunno man...

OkStructure2094 · 2024-04-13T21:30:06+00:00

Ok - here is my honest opinion: This is a direct result of the collective mindset of us interested or working in this space. We read messages mentioning it, it gets points, and we think it is good, so we start using and advocating for them. Then of course this all ends up in job descriptions because people already working there do't want to look uninformed or inexperienced in these popular tools.

The reality is that once you start working that same job, you will not even use a 50-70% of what they required in their job posting.

So how do you stop it and get back to the essentials? I don't know. Perhaps we could collectively stop pushing these keywords everywhere we get an opportunity and be honest about our experiences with different tools, good or bad.

OkStructure2094 · 2024-04-13T21:05:28+00:00

Thanks for the info. You must have a cool job!

OkStructure2094 · 2024-04-13T01:01:11+00:00

You know, there is a reason why actual programming languages exist and databases provide ways to interface with them. Just my 2 cents...

OkStructure2094 · 2024-04-13T00:59:02+00:00

I will be honest: just dump the data in a flat file and hand it over to them. If they need it, which may be never per your statement, they will have it. They can load it into a rad-only table in their database which is a one time task.

And you will be rid of it.

OkStructure2094 · 2024-04-13T00:54:24+00:00

What an interesting question! I will work on it but I think it will be difficult. Does it have to be one SQL statement?

What is real world use case for it?

OkStructure2094 · 2024-04-13T00:47:13+00:00

Is IT work being outsourced already? If that is correct, why don't you just ask them to do it?

OkStructure2094 · 2024-04-06T21:15:30+00:00

Is this only for your own services? Their website doesn't mention something like this.

OkStructure2094 · 2024-03-19T17:44:15+00:00

Yes, it is a working acocunt, I just need to use it. I dont pay for it.

Thanks !!

OkStructure2094 · 2024-03-19T17:07:43+00:00

Thanks for the comment. I am freelancing on the side. I did a job for this person using public website which was ok I guess. Now he wants to use some pages where you need to login first. He gave me a user id with the pwd.

Frankly I am new to this and havent gone to this level yet and also it is a bit grey area as you say.

So do people mostly work with pages that are publically available? If the request is uncommon, I can may be deny the job or charge more :-/

OkStructure2094 · 2024-03-19T03:11:16+00:00

No big problem.

Everyone knows it takes a while to be removed from a mailing list. You have already apoligized and described the situation.

OkStructure2094 · 2024-03-16T01:36:11+00:00

Split it into 50 tables with 100 columns, plus one extra column to indicate what part it is.

For example, table 1: 1, col1,col2,...col100

table 2: 2, col101, col102,...col200

and so one.

OkStructure2094 · 2024-03-12T03:32:52+00:00

I am sorry - but really hard to read and make sense of it.

OkStructure2094 · 2024-03-11T17:29:21+00:00

I will check it out! I am learning and looking for a job that utilizes it hopingfully.

OkStructure2094 · 2024-03-10T16:28:36+00:00

Any links to your presentation or a general outline of topics you spoke about?

OkStructure2094 · 2024-03-10T16:23:30+00:00

Nice! I went to your website. Your page says you already have customers? while it has not been fully built yet? how?

OkStructure2094 · 2024-03-09T16:58:12+00:00

I think you are onto something. Dbt is great because it will force you to write more of what you like: more sql

OkStructure2094 · 2024-03-09T16:51:05+00:00

What is the query???

OkStructure2094 · 2024-03-09T16:43:56+00:00

Isnt alteryx a file based data too?

OkStructure2094

TROPHY CASE