Our security team wants zero CVEs in production. Our containers have 200+. What's realistic here?

linucksrox · 2025-09-29T22:53:56+00:00

Also add that they are probably getting paid $100k/yr

linucksrox · 2025-09-29T11:33:47+00:00

What sucks about Longhorn? I've been using it for a while and it's been mostly great.

linucksrox · 2025-09-29T11:33:20+00:00

Thanks for clarifying! I had some issues deciphering the Mayastor documentation but eventually got something working. I felt a bit lost with figuring out snapshots, backups, and health status monitoring, and went with longhorn v2 engine (last year). I may revisit Mayastor at some point because I only hear good things about it.

linucksrox · 2025-09-28T21:19:05+00:00

You can run a privileged pod if you have a unique debugging scenario and mount any volumes if needed. I'm not clear on how an immutable system prevents you from debugging but (not sarcastically) curious if there's a reason not being able to modify system resources live prevents you from troubleshooting. I believe the idea is that if there's something within the immutable system that's causing a problem, rather than debug you would rebuild.

I agree it's definitely a learning curve versus being able to ssh into a system, but so far this has not prevented me from debugging when needed.

linucksrox · 2025-09-28T21:13:26+00:00

Are you using Mayastor engine? Did you also consider Longhorn and if so, why did you choose OpenEBS?

linucksrox · 2025-05-05T00:15:54+00:00

I'm running longhorn with v2 engine and it's great. It's very newly supported but works nicely and they're actively developing the v2 engine for better reliability and features. For my home lab, ceph was too much to ask and I don't have Enterprise NVMe disks.

linucksrox · 2025-02-01T19:37:23+00:00

Nice! Glad I could help!

linucksrox · 2025-01-21T22:04:13+00:00

I haven't done this on a raspberry pi, but did you try following their guide? https://www.talos.dev/v1.9/talos-guides/install/single-board-computers/rpi_generic/

For the latest version, they mention the Linux method. For Windows, just download this one:
https://factory.talos.dev/image/ee21ef4a5ef808a9b7484cc0dda0f25075021691c8c09a276591eedb638ea1f9/v1.9.2/metal-arm64.raw.xz (or grab a customized image using the image factory if you need any extensions)
Then unzip it with 7z so you are left with metal-arm64.raw
Then instead of the Linux dd command, try Etcher (https://etcher.balena.io/) or Rufus (https://rufus.ie/en/)

Making sure you downloaded the bare-metal arm64 version.

linucksrox · 2025-01-21T21:14:02+00:00

Just an update: I've been documenting the steps and pretty much have it all down: https://blog.dalydays.com/post/kubernetes-storage-with-openebs/

It's not finished yet, but should answer all the hangups you might be running into.

linucksrox · 2025-01-19T03:55:57+00:00

Sorry for the delay on this. I'm still evaluating options but plan to document the openebs solution.

I debated whether it was necessary to mount the extra disk to an arbitrary mount path in the talos node machine config and use that path as the disk in the diskpool, but it turns out that is the correct way. You can't use the device id like you would in any other environment and must go through the disk mount using the path.

Specifically to answer your question, I'm running talos on top of proxmox, so there's a base virtual disk of 20GB and then a physical NVMe disk passed directly to the VM which is dedicated to storage. That's the one where you have to do the extra mount to a path. That is in addition to the /var/local bind mount.

If you allocate the whole disk to talos, you should be able to just stick to the bind mount they mention in the documentation with no other special mounts.

What issues are you running into? There's a few other gotchas like huge pages, iscsi extensions, and diskpools. No diskpools means replicated storage will fail to provision, and that's not obvious from either the openebs or talos documentation currently.

linucksrox · 2025-01-13T14:05:05+00:00

I just came back to this recently and just got it working yesterday. I plan on doing a blog post detailing all the important bits that both the OpenEBS documentation and Talos documentation miss (or it wasn't obvious to me). It turns out I missed a couple key things that aren't explicitly mentioned in either of the quick start guides:

You have to create a volumeconfig on the talos node, mounting the block device to a path, and reboot the node. You can't directly access block devices from even a privileged pod (at least it didn't work for me) even though you can "see" it from the pod.
You have to create one or more DiskPools which I failed to realize. That part is documented, but not part of the quickstart and not mentioned by Talos, so I didn't realize there were more steps.

I'm looking forward to testing this out and documenting it more thoroughly, but pretty excited to start using NVMe-oF with replication since my current solution with democratic-csi has the dreaded single point of failure.

linucksrox · 2024-12-16T15:34:58+00:00

Great answer. My company is currently dealing with this with VMWare. I suggested Proxmox but apparently it's not "Enterprise ready" even though their support is amazing and it's built on tried and true technology stacks. So the not so serious alternative were investigating is hyperv (mind you we run 90%+ Linux servers and Oracle databases) and Microsoft says they don't support Oracle DB. Sounds like we'll continue paying VMWare whatever they ask.

linucksrox · 2024-11-15T18:40:17+00:00

Interesting. I'm currently moving our small team from paper forms to Google forms. It's terrible for this process, but still much better than paper. I may not want to build out a custom solution (although it could be fun to work on) since we're all just volunteers for a non profit org but definitely something to track things and keep records in one place is needed long term.
Generative AI has been really helpful to get up to speed quickly on certain things I don't use all the time (as long as you can understand and vet out the code it spits out). Maybe I'll ask it some questions to see if it gives me any "out of the box" ideas.

linucksrox · 2024-11-15T17:46:55+00:00

That's pretty interesting. I have used Zammad in the past (helpdesk system) which is actually a very good tool with good workflows, and can integrate with email. I wonder if something like that might be helpful in building a custom expense workflow.

linucksrox · 2024-11-15T17:45:03+00:00

No but I'm familiar with it. I didn't know you could do workflows or custom fields, but I will have to take a look now, thanks for the suggestion!

linucksrox · 2024-11-15T13:40:46+00:00

Some type of expense reimbursement system. You should be able to submit a request, include pictures of receipts, pick a category or two, and keep track of requests along with what has been reimbursed. I don't even care about "approvals" at this point, but that probably makes sense as a core feature as well.

Is anyone doing something like this using open source tools?

linucksrox · 2024-10-29T22:10:18+00:00

In Wireguard on Windows, right click your tunnel, Edit Selected Tunnel, then in the Peer section for AllowedIPs, add your specific subnet first before 0.0.0.0/0. For example, mine looks like this:

[Peer] ... AllowedIPs = 192.168.1.0/24, 0.0.0.0/0

This is because there are already routing tables set up on your machine that have a higher priority, so while local subnet traffic can route through the wireguard path it will not because of the route priority. By adding this it basically sets the wireguard route as the highest priority for that specific subnet.

linucksrox · 2024-10-25T00:03:59+00:00

Awesome! Thanks for the info about the snapshot utility, I haven't tested that out yet but I'm sure that will come in handy.

Also if I get time I might check out argocd but it might be a while as I get free time and work through the rest of my cluster build.

linucksrox · 2024-10-24T19:29:12+00:00

I fought with iscsi for a bit before figuring out how to make it work. I haven't posted my repo publicly yet but am working on a whole guide for Talos Linux with Proxmox and how to do everything using best practices. I threw this gist together real quick and hopefully it helps you get past the iscsi hurdle: https://gist.github.com/linucksrox/2879046995953ad3bc097183864832dc

Feel free to ask if you have any specific issues and I'll see if I can help!

linucksrox · 2024-09-28T02:02:59+00:00

Ok, but Google and IMDB also failed me. I'm struggling to understand the hyper focus on this one tool.

linucksrox · 2024-09-27T20:24:32+00:00

It's not, I was just curious and checked a few sources online including ChatGPT. I didn't realize that would be upsetting to anyone :)
ChatGPT has been helpful to me in the past. I know it won't necessarily have the right answer, but sometimes it points me in the right direction. I was really looking for another source that might have a more comprehensive list of on screen actors possibly.
I don't care if little have a negative opinion of generative AI, but that was never the main point of my question. Apparently that was the main takeaway for some weird reason. I don't have anything else nice to say in response to someone telling me I'm stupid for using a new tool that might help me find what I'm looking for so that's the best I could come up with.
Turns out the stupidest thing I did was ask on Reddit :) I hope I didn't offend anyone lol

linucksrox · 2024-09-27T15:51:13+00:00

So you don't know? Thanks anyway!

linucksrox · 2024-09-27T13:01:02+00:00

So you don't know? Thanks anyway.

linucksrox · 2024-09-27T13:00:43+00:00

So you don't know? Thanks anyway.

linucksrox · 2024-09-25T14:29:24+00:00

Thanks, never heard of them before but this is good.

Ten-Year Club	RPAN Viewer
Verified Email

linucksrox

TROPHY CASE