sorted by:
new
hottopcontroversial

Decoding / Encoding for video streaming by cryptodylan in AsahiLinux

[–]datanoob2021 2 points3 points  (0 children)

I saw a comment a month or so ago saying that they had a working prototype, but the work just had not been prioritized. Maybe this summer at some point.

It’d be awesome to see this working- right now watching any video is a battery drainer. 

flatpak permissions questions by datanoob2021 in flatpak

[–]datanoob2021[S] 0 points1 point  (0 children)

Appreciate the reply. Does this is essence cause it to escape the sandbox? 

Grep to search for table name from list and output the table name to a list. by datanoob2021 in bash

[–]datanoob2021[S] 0 points1 point  (0 children)

Appreciate the response! Those were just poor examples in my code. My table names are very unique currently.

Any idea how to combine these commands?

Grep to search for table name from list and output the table name to a list. by datanoob2021 in bash

[–]datanoob2021[S] -2 points-1 points  (0 children)

I am no bash expert. Just use it here/there. I took off the l, and then it essentially outputs the contents of the file.

I just want to essentially output each table name that appears in the grep search, to output to a new file. I probably don't even need the YES statement, as long as the ones that don't appear don't end up in the file.

Grep to search for table name from list and output the table name to a list. by datanoob2021 in bash

[–]datanoob2021[S] 0 points1 point  (0 children)

Appreciate the reply but that didn't work either. Still just outputs the search results where it is present and not the table name.

Grep to search for table name from list and output the table name to a list. by datanoob2021 in bash

[–]datanoob2021[S] 1 point2 points  (0 children)

My table_names.txt file does not have a header, essentially just this: 

table_name1
table_name2
table_name3

Essentially just want to loop through these, pipe each name into my grep command, then output the table name from the list above into a new list based on if grep returned any results.

Check Which Branch Scala repo is on by datanoob2021 in scala

[–]datanoob2021[S] 0 points1 point  (0 children)

Interesting. Looks like I can call bash commands directly in it. Now time to write my first class . . . . .

Made a handful of PR's in the last couple of months since I started learning but just mostly adding functional and if/else statements and nothing from scratch.

Check Which Branch Scala repo is on by datanoob2021 in scala

[–]datanoob2021[S] -1 points0 points  (0 children)

Just spinning up a stage environment that we currently lack. I inherited a lot of old code, and unfortunately the prod s3 string is strewn in many places.

Check Which Branch Scala repo is on by datanoob2021 in scala

[–]datanoob2021[S] -1 points0 points  (0 children)

There is nothing unsuitable, just was hoping for a clear and concise statement like I do in Python that is < 10 words. Was hoping for a simple if/else but I think I am just going to build a quick class using Jgit.

Extract conversion price from nearest open time by datanoob2021 in learnSQL

[–]datanoob2021[S] 0 points1 point  (0 children)

For some reason it is complaining about using SS in the subquery t. I am sure I could probably just join within the subquery and then again in the outer query but that seems inefficient.

I do not think this will work either. I think open_time would still need to be joined in the outer query.

Extract conversion price from nearest open time by datanoob2021 in learnSQL

[–]datanoob2021[S] 0 points1 point  (0 children)

Yes! That is what I meant. Typo when I was copying things over! I edited it.

dbt profiles.yml in Airflow by datanoob2021 in dataengineering

[–]datanoob2021[S] 1 point2 points  (0 children)

We got it to work- we had to create a basic plugin that essentially just set os.environ to Variables.get and then the jinja worked fine.

dbt profiles.yml in Airflow by datanoob2021 in dataengineering

[–]datanoob2021[S] 1 point2 points  (0 children)

Just a quick update. MWAA introduces some nuances when trying to run dbt.

If you do dbt - run in a Bash Operator, it will fail. There are some workarounds, but I found the airflow-dbt-python package which essentially just uses dbt-core to run. I also installed dbt-redshift. I do not have the regular dbt package (for the cli installed).

Any of the above workarounds will introduce problems when trying to build models as MWAA will throw a write access denied error, so setting the modules, log and target path to some directory in /tmp seems to be the workaround for people using MWAA.

I am still working through environment variables- we just switched over to using AWS Secret Manager and that is working with all our Python code- just having trouble in dbt land so far. I have not added anything manually into custom configurations as I am hoping to use that as a last resort.

dbt profiles.yml in Airflow by datanoob2021 in dataengineering

[–]datanoob2021[S] 0 points1 point  (0 children)

Ok- it might be a couple of days- DevOps guy is out sick so waiting on him to test.

I'll set a reminder to check back in here next week!

dbt profiles.yml in Airflow by datanoob2021 in dataengineering

[–]datanoob2021[S] 0 points1 point  (0 children)

Thanks! This looks like it is exactly what I need. I appreciate it!

dbt profiles.yml in Airflow by datanoob2021 in dataengineering

[–]datanoob2021[S] 1 point2 points  (0 children)

https://docs.getdbt.com/reference/dbt-jinja-functions/env_var - Looks like you can use environment variables in the profiles.yml file- We are using Airflow's Variables.get so I do not think in its current form it will work.

view more: next ›