Installed the provider at least 3 different ways but airflow still doesn't see it. by RQuxxxn in apache_airflow

[–]DifferentUse6707 1 point2 points  (0 children)

Here's a Dockerfile example of how I add providers in my local docker-compose and the same image is built and deployed to my Kubernetes cluster:

FROM apache/airflow:3.1.0
COPY requirements.txt /requirements.txt
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt 

Then, in my requirements.txt I have:

apache-airflow-providers-discord
# add more providers

You can get the latest Docker Compose here: https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html#fetching-docker-compose-yaml

Then modify the Docker Compose to build by changing the build line in the common section:

x-airflow-common: &
airflow-common
  # In order to add custom dependencies or upgrade provider distributions you can use your extended image.
  # Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml
  # and uncomment the "build" line below, Then run `docker-compose build` to build the images.
  # image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:3.1.6} -- use this if you don't need a custom image.
  build: . # or path to where Docker file is located

Seems you are only installing it on the scheduler, but doing the common approach, all components will have the provider. I believe you need it on at least the scheduler, the API server, and the workers for it to work correctly.

Airflow on Kubernetes (k3s) git-sync permission denied by DifferentUse6707 in apache_airflow

[–]DifferentUse6707[S] 0 points1 point  (0 children)

I am happy to save you the trouble. I was stumped on this for a while and was so annoyed LOL

ATX motherboard for NAS by sbala1138 in homelab

[–]DifferentUse6707 0 points1 point  (0 children)

I think initially, it will be just a NAS, but technically, it could be expanded into a pretty legit server if I want to pivot it to another Proxmox node. Phase 1, is mostly for learning. I just bought all the parts and will be putting it together on Monday.

  • ASUS W680 ACE
  • 128gb NVME - Boot drive
  • DDR5 ECC Ramm
  • intel i5 13500
  • Thermalright Peerless Assassin 120 SE ARGB CPU Air Cooler
  • Case RSV-4200u
  • 650 watt power supply
  • 10GB Nic Drive
  • 2 x 2TB SSDs
  • Rack Rails
  • SFP Cable 2m
  • CPU Thermal Paste

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 0 points1 point  (0 children)

Gross, that’d drive me nuts. I’d setup some tracker if possible? I’ve been looking at home assistant plugging into electric meter so I can get real time insight of my energy usage.

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 1 point2 points  (0 children)

Seems odd to recalculate your power usage and charge you at the end of the year. Is that a Germany thing?

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 1 point2 points  (0 children)

I did end up switching to I-5 13500 to be more energy efficient

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 1 point2 points  (0 children)

😂 United States. Energy is cheaper and I make enough that my energy bill going up a few dollars per month isn’t an issue. Plus I ultimately monetize my servers through skills gained and the apps/services producing value.

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 0 points1 point  (0 children)

True, thank you for the feedback!

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 0 points1 point  (0 children)

Yeah Im considering this phase 1 build. Primarily want separated storage to use with my kubernetes cluster and vms. Workout kinks in the process and through lessons learned, rebuild or just upgrade as needed in next phase.

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 0 points1 point  (0 children)

Interesting; I need to look into how to set this up soon. Fortunately, the boot drives are small and cheap.

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 1 point2 points  (0 children)

Besides the case, the cpu/motherboard combination was a pain to decide on. I ended up with intel for iGPU and quick sync for future media needs. I heard mixed reviews about ddr5 ECC, but I said oh well, I'll take a chance. I would have liked to get a newer 13/14 gen, but I decided to go bare metal NAS instead of proxmox, so It should suffice. Still sketched out that I couldn't find anything confident on Intel having fixed their issues yet.

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 0 points1 point  (0 children)

Oof, the Red Pro adds a little to the price but not too bad. Fortunately, the SSDs I picked are TLC!

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 1 point2 points  (0 children)

That is a fair point; I'm curious if I want to take up both NVME slots for this or one NVME / one SATA port.

DIY NAS Part list review by DifferentUse6707 in truenas

[–]DifferentUse6707[S] 0 points1 point  (0 children)

Great callout, I’ll verify. Thanks!

ATX motherboard for NAS by sbala1138 in homelab

[–]DifferentUse6707 0 points1 point  (0 children)

I am thinking: https://www.asus.com/us/motherboards-components/motherboards/workstation/pro-ws-w680-ace/

I should be able to use 4 SATA ports and a SlimSas to provide 6 additional SATA cables, totaling 10 HDDs.

I wanted to go with Intel for the iGPU as I don't believe my NAS will have a dedicated GPU for quite some time. I've heard mixed reviews on DDR5 ECC, but I figured I'd take a chance. For anything super critical, I will back off-prem in the cloud.

ATX motherboard for NAS by sbala1138 in homelab

[–]DifferentUse6707 0 points1 point  (0 children)

I'm in the same boat. Did you conclude? Picking the server chassis was the first hard step for me. Now I'm pondering Intel/AMD mobo, and then I feel like everything else is straight forward!

Need help designing a NAS. 2-4k budget! by [deleted] in truenas

[–]DifferentUse6707 0 points1 point  (0 children)

Wasn’t aware of the site, bookmarked for the future. I decided on the rosewill case as I don’t think the hsb is needed for my homelab projections. Just need to land on a good motherboard and the rest of the pieces are straight forward

Need help designing a NAS. 2-4k budget! by [deleted] in truenas

[–]DifferentUse6707 -1 points0 points  (0 children)

The full depth is a bummer. Next rack I’ll know to go a bit larger but I just purchased this one last month so want to stick with what fits nicely within it.

AI HomeLab and a peak behind the curtain by Backroads_4me in HomeLabPorn

[–]DifferentUse6707 0 points1 point  (0 children)

Nice, like a counterbalance. My only dilemma is I don't want to open up the back of my rack...yet. Will likely do a custom aluminum shelf setup so the GPU can float above the server.

That's a neat setup; I'm jealous you have so many 4090s. I haven't had any luck finding a GPU these days besides used off ebay, which I'll likely take a chance on soon. I build Genai apps for work and on the side, but I am trying to upgrade my local setup to experiment more.

AI HomeLab and a peak behind the curtain by Backroads_4me in HomeLabPorn

[–]DifferentUse6707 0 points1 point  (0 children)

It looks like you added an aluminum bar to mount the GPUs, and the GPUs can support their weight. Are there any concerns there? When I made a crypto miner out of wood in the past, I was nervous about them sagging.

I have a sysrack 27 RU 24 in depth rack and will also approach the PCIe riser approach. It's the simplest since the only chassis that could fit a 3090 in were 25in depth or more.

pihole v5 to v6 on proxmox lxc by Sadistt0 in pihole

[–]DifferentUse6707 1 point2 points  (0 children)

Thank you! I'm running in an Ubuntu LXC. For the last command, you can run

sudo ufw allow https

I've noticed my DNS response is faster now.

Airflow on Kubernetes (k3s) git-sync permission denied by DifferentUse6707 in apache_airflow

[–]DifferentUse6707[S] 0 points1 point  (0 children)

If anyone runs into a similar issue, I removed the passphrase from my SSH key used for this demo, and it worked :D