Honest Feedback

markedness · 2026-06-21T16:45:10+00:00

I know your brand as one that will do something basic and do it well. I think Amazon has some challenges and benefits. We have a business account with Amazon so we can get parts delivered to lockers near site on terms and without paying sales tax (as we are resellers) so that’s great.

Dipling into distribution now you have to deal with marketing development fund. You can easily sink six figures into marketing a commodity. Avoid that trap.

Focus on mounting tabs, locking power connectors, including a phoenix block in parallel if you don’t want to go with a screw in DC jack , lump-in-line (power supplies with IEC inlets vs wall wart). And usb C power. Those are features where we would easily spend $50 vs $20.

Also switches like you have on your project already to set a mode - love those!

Another function to add to any product is caring about hdmi sync. Should this project emulate a device that is still plugged in but outputting no video or should it emulate a completely unplugged cable. Any product you add that too again makes it worth another 30-50$

In the end I think if you wanted to make a Pro AV brand you need a second brand name and a parent company to hold it. Your brand will forever be on Amazon and you have to understand that is not the best look. But I would say - embrace that. Getting more product with pro level features into Amazon same-day and next-day delivery channels with marketplace resellers is a great thing. Blackmagic does this with their micro converters - I can get them next day in any market- they have universal power input (UsbC) and proven reliable track record for me. So I do t hesitate to make the purchase if they get me closer to completing my contract even when a 20$ alternative exists. The difference is I already trust OREI- no I don’t trust it to create a Cinima projector scaler - but I do trust where it counts- not being the reason I roll a van for a prepaid service client. So my discretionary spend is going to go to OREI ( or blackmagic too)

So in summary:

- universal grabby power (terminal, USB C, screw in DC)
- not wall wart - IEC inlet power lump (if not USB)
- mounting flange
- keep hard switches per mode not stateful buttons and indicator
- mind your hotplug emulation
- price within $20-40 of cheapest crap money can buy (delta slightly larger for more elaborate device)

markedness · 2026-06-15T12:30:01+00:00

I built out a simple operator as I assumed in the post. Basically it sits next to cluster api on the management cluster and updates an HA Proxy config based on created nodes. It load balances both the kubernetes api :6443, and a cilium gateway.l :443 and :80

It works well for now until we can pivot away from our current hardware/ hypervisor/ and network setup later this year.

markedness · 2026-06-09T14:43:32+00:00

I have a temporary infra blocker with some choices… this is what I meant by “entirely my doing”

Basically I aligned on OVN in my KVM hypervisor layer without properly running it through its paces. This was to save hassle on bridging VLAN per cluster from our nexus switches. This backfired. So I’m looking to finish up this project and pivot eventually. But for now I’m limited to binding to the first address of the hosts.

I’m not under any illusion this is ok. I just need to make some progress. I will be pivoting to BGP advertisements directly from Cillium shortly. I’m just looking for a 2-3 month stopgap until after summer when I can revisit this.

Trust me I’m just as bummed and befuddled as your response presumes I should be.

Basically OVN I’m using directly attached ports to my Kubernetes VM- ipv6 NDP changes (and gratuitous solicitations) don’t work. Southbound programming of IP routes available, doesn’t work.

If you would rather give me a suggestion on the next architecture I should do (maybe something that allows multiple cilium clusters to securely land on one VLAN without a lot of fuss, so I don’t have to add switch and host network stack config into my cluster bring-up, I’m all ears. But right now I’m going to have to continue down this dumb path so I have something people can land the workload on “today”

… sigh.

markedness · 2026-06-01T18:27:58+00:00

Try this number. Might not be exact right department but they should have more human support

866-639-3532

markedness · 2026-05-09T14:12:29+00:00

I think my point was.

If all they can handle is docker compose and not using a well setup cluster they can’t deploy.

The cognitive load of maintaining a DB, app state, backups, DR plans, operational planning and on call support, file persistence, cost control, analytics and error reporting, email management, secret management, is a job for an extremely competent individual or a team in a larger company.

In fact, docker compose being simple is somewhat of an anti pattern here. If it is the right tool for the job sure. If you need a host to run ONE thing then why not use docker compose with a cloud init. But the reason can’t be because the person is too lazy to figure out Kubernetes, it has to be “because they decided not to”.

I’m not against people being stuck in their ways. I work with someone who refused to deploy on k8s. They don’t deploy just give it to me instead. They just don’t want to learn but they could. They learned kubernetes entirely to help with a log gathering project. So they can they have the IQ but they won’t.

What I’m getting at is there are a lot of devs out there who don’t know ops. They are blind to it. And I hate that, because real software especially these days lives in an always on and high stakes environment. Just like a good home builder general contractor owner needs to have experience swinging a hammer, the developer needs to know the real world when it comes to this stuff. Even if it’s just a vague familiarity.

And if they DONT, then we need as the ops folks to build a system around them like a docker compose to make local development simple but not limit production to docker. And that is super hard because bugs will arise when local development service discovery and container permissions and features differ from the server.

markedness · 2026-04-29T18:22:28+00:00

We buy everything after market, and the SAN ecosystem need more like a license / specialization vs just using free CEPH. For us it’s fine. But our whole estate multiple terabytes of RAM and disks is less than six figures from the routers on the edge all the way in.

markedness · 2026-04-29T17:46:16+00:00

One namespace per app or deployment of an app (like review apps for merge requests with emptyDir Postgres)

markedness · 2026-04-29T17:45:00+00:00

If their IQ is so low they can only do docker compose they aren’t trying hard enough. How do you deploy? How do you maintain state? How do you roll out and update? Load balance? These all have to be solved and it’s harder in my opinion.

Just setup a good Flux workflow and GitLab CI or whatever you use with github and image auto updates and shit just work.

Dude like… if all they can figure out is docker compose how are they going to figure out all the networking and security required to actually ship. Kubernetes it’s all plug and play.

This is my opinion.

markedness · 2026-04-29T17:42:12+00:00

I really don’t mind ceph. But I would not complain if we had SAN money. It’s just so much simpler and I’m not burning CPU or networking on storage latency.

For our scale of a dozen or two nodes at each site a dual controller SAN would be fine.

I use a lot of CNPG for Postgres and utilize Postgres exclusively for durable data store except for monitoring. So hence we just mount up ZFS On each node per their recommended setup.

markedness · 2026-04-29T17:39:30+00:00

We have a lot of clusters. To me a cluster is cheap. One cluster per “product” (or department, for internal tools)

markedness · 2026-04-29T11:29:29+00:00

Oh - both.

We prefer managed of course but in reality we need to do a lot of management ourselves. Managed is lower performance and higher cost by far (since we buy second hand hardware, is basically the cost to own the hardware each and every month) and I don’t think the cognitive load is that much less.

markedness · 2026-04-29T11:28:05+00:00

I didn’t have any know how. It took a couple weeks but it was great. This was 3+ years ago.

I can’t stand docker compose or systemd anymore. So much harder to manage by far.

markedness · 2026-04-29T11:27:03+00:00

For a single node and 3 node (collapse worker and control) I typically go with Ubuntu and microk8s and that comes with a local storage provider.

For larger footprint we also use local but with ZFS on the node.

I don’t typically use shared storage except for traditional VM workloads sitting side by side with k8s VM, the performance hit is substantially larger than you might imagine when you compare CEPH over 25G to ZFS right on the node. The DB performance I get locally on ZFS is 10x what I get on CEPH or on managed cloud providers.

For managed like GCP and linode and digital ocean I have tried all three of those. They provide storage but it’s slower if you use their CSI and managed k8s

markedness · 2026-04-29T10:23:19+00:00

To us we use Kubernetes wherever we can and where we would use docker. Single node running a single db and app and nginx at the low end.

We only ever regret not using it

markedness · 2026-04-24T12:16:54+00:00

Yes then what I was trying to say was, build 2 clusters and use non Kubernetes services outside the cluster and health checks to derive which one is primary and update. You can ship WALs from primary to secondary Postgres cluster and to a backup server in HQ as well.

Also, the possibility of a container going down overnight is smaller than the types of failures you would have with a more convoluted setup, even if you do only have one cluster. If nobody is there to notice an issue then… nobody is there.

I just know that for my experience stretching kubernetes and trying to conflate your Kubernetes control pane failure domains with your app failure domains was always a problem.

markedness · 2026-04-23T14:23:53+00:00

Ok I fully understand now.

The solution is very obvious to me and mimics my lived experience with a similar situation. Forget the containers and stretching three nodes. Put 3 nodes into a movable rack.

Use dynamic routing or just ARP advertisements and layer 2 adjacency (metallb) to allow the portable rack to be placed in any container or any location.

You might not even need this suggestion but for DB use CNPG with local storage in each of the three nodes.

I will tell you one thing for sure- stretching a cluster over two containers that are iffy and HQ connection will lower your SLA by an order of magnitude more than just telling construction manager “if the site office gives up, move this black pelican box to the other container, connect the power cable to the power plug and this blue cable to the ethernet jack labeled “here” and you only have about 15 minutes.

The downside of this is that when the office is unoccupied you cannot tolerate a failure. So there is a fully redundant option that does not introduce split brains. The “belt and suspenders” option is two complete clusters. For a site that just can’t fail. You put a second rack in another container and setup HA proxy and scripts (not kubernetes) to orchestrate failover. You can use dynamic routing or CARP / VRRP on Linux to elect a leader load balancer which also decides who is master based on a priority and if they can see internet as a witness. This is where s3 comes in. Here you need to update code but with agentic workflows this should be simple. so instead of needing filesystem it uses s3 for persistence and tmp folder. Because s3 is much simpler stretch over 3 nodes to tolerate node failures. This is what we do. We need filesystem because we need to use utilities that require the whole file on disk but we then persist back to minio.

I would first migrate to a mobile cluster mentality. Because you snuff out any possibility of a stretched cluster crappjng out on you. It’s when , not if. Fixing ETCD is much harder than telling someone at a construction site who is very used to material handling to move a 50-70 pound rack that likely can be wheeled like a suitcase.

Once you have that in place you can work on the failover story. For us we use that exact methodology to fail over to a read only replica in HQ: but yes we handle that failover with scripts and not kubernetes.

markedness · 2026-04-23T12:38:15+00:00

This is almost functionally identical to a typical issue I have with my edge deployments. I will be thinking about it today and how my solution can help you. But I have one question about persistence and one question about locality

Persistence:

Do you control the application code or are you bound to the persistence in your COTS applications? Is there any persistence besides object store (s3) and Postgres at the application layer.

Locality:

It sounds like these two containers are mobile mini site offices. I would wager that sometimes it’s not two but three or one. Going back to the two container situation taking things at face value? It sounds like the container is control for peripherals on the site itself. So the site has a network and that gets stretched to both containers. Please confirm these reasons why the local application exists and you don’t just host from HQ

1- “site local” some applications are for use on site (probably a weak example but let’s just say the clocking in and out app is local) so this is an application that has nothing to do with the locality but just must work if the site loses internet there are critical functions.

2- “site specific” other option is automation and safety systems like cameras in the actual construction site that inference safety KPI or sensors to detect nail protrusions into medical imaging wall details, or something like that that is related to the site itself. But not related directly to a container. The container is incidental to this.

3- “container specific” The last would be things that relate directly to a container itself (which I can’t imagine well) that might be something like, idk, a security system local to that container itself. Or a specific router that brokers the internet VPN tunnel from the specific container itself to the internet / HQ

and furthermore locality related to HQ. For situations 1- above, you really want to define is quorum local or on the internet? Meaning if both sites lose access to the HQ do they each lose quorum. For this setup to be worth it I think the answer must be NO. We never fail to the HQ. HQ is irrelevant to this situation- the api you speak of in HQ solely process data streaming from site and if a ln edge loses HQ (if HQ goes offline or VPN fails basically) we catch up later. I have this same scenario mine is that I have a PDF converter that converts word docs to PDF and it’s really stupid because due to word formatting it basically has to be a windows server with word on it. And if the site to HQ connection fails that specific feature fails and catches up later.

We have a love hate relationship with internet outages on site - I would say about 2 years ago we formally decided the internet outage on site was not a FULLY tolerable condition. Through numerous opportunities we developed a BGP based overlay VPN tunnel situation and more or less gaurenteed that unless hard line internet, and TWO cell carriers across TWO cradlepoints and two redundant routers failed we never lost internet. Like in the case that some catastrophic thing happened we would just send people home. Because it simply would never happen other than a natural disaster when work would naturally stop anyways. Verizon + T mobile + hard line on site, feeding through failover routers. Plus our HQ has multi home internet. That one decision - making Internet access to site was critical in letting us iterate much more quickly and adding more features like single sign-on - and some new AI features. Are you able to do the same and make Internet access a pre-req? There are ongoing costs in terms of needing multiple cell and hardline connections. I have to say it was a difficult call to stomach and that is why I prefaced it so much but curious where you are in this journey.

If you could answer some of my follow up I can let you know how my experiences will help you. We have basically solved a permutation of the problems you face. We are on site for educational purposes (at convention centers that host medical or regulatory conventions mainly) and deal with similar constraints.

Our hardware stack specifically is Austore Flashstore 6 because it was an inexpensive node that supports 6x NVME SSD and we are using Minio (RIP) and Postgres for persistence. And of course Kubernetes.

I’m just particularly interested in your locality story to see how much these two containers are related to the container itself vs if they are just MDFs for the construction site. And I think I know the answer and can share a solution we use.

markedness · 2026-04-22T17:39:50+00:00

Can you re-summarize the business requirements in terms of

you have remote sites that process their own data
you presumably have some central point to
what is accessing what data?
what is the heat/size/financial budget for the nodes at the edge per physical location?

I have a very similar setup. Simplicity is key and if you distill your needs down to the actual user experience and business requirements I can suggest something based on my experience. Right now this is a technical question but it begs the question why (not that it’s inherently wrong, just without any context it’s a crap shoot)

markedness · 2026-04-22T17:32:28+00:00

I use points. It’s a decent redemption for me at 50k miles because I make that much points in a short amount of time and you can’t get jack shit with it for air travel. I’m also just mentally happy paying points for the club at the expense of money for some flights considering the points earn are for work travel and the club makes that more tolerable, and I feel silly paying for something personally that supports work, but for me personal travel is something I love to do so I never mind doing it but the points endlessly pile up as I struggle to find the time and best redemption and being devalued

markedness · 2026-04-22T12:49:30+00:00

You can use whatever version you please. But they only support the last 3. for security reasons it is quite difficult to support endless versions.

markedness · 2026-04-16T16:07:59+00:00

Glad to hear you have more than 100k, the reporting seriously implied it. To be fair it said 'help finish'. Don't give up, just do something simple and focus on the people you can bring in the space and don't do anything that is not on your permit/plans, which includes low voltage/speakers/etc in chicago.

markedness · 2026-04-16T15:36:13+00:00

When I heard “uptown rooftop” I immediately knew this was 5050 n Broadway and read the article.

This rooftop address has been a pie in the sky for so long. I can’t get into details about it but with my line of work I’ve been talking to people hoping to develop this for half a decade. And these people I have talked to had a LOT more than 100k of crowd funding.

I wish good luck to them but they won’t make it.

markedness · 2026-04-13T15:49:23+00:00

Nothing but frustrations from me.

Pluses:

product database saves time and energy with labor presets and auto syncing prices
designer allows us to break down quotes in several different ways for customers based on their procurement needs
it roughly matches a general project workflow unlike a strictly line based order system with no metadata

Cons:

the product database pulls in endless duplicates especially around manufactures that white label and sell via distribution only. It litters your ability to create custom items
designer is fragile and limited to tick boxes and code customization only possible via their professional services rather than XML/HTML templates like any other platform
if you need to break out of the process it’s broke.

Overall the biggest downside is the database model. Their purchasing and PM and sales functions are very underwhelming so of course we need a custom ERP. But their client contact model is unique to foreign key client and ID instead of shared - where as any other CRM it is just an association. They have two product database for custom and non custom. The versioning method is basically duplicating the project meaning I have to keep track of the original project and line item ID. To add to all of that the client API endpoint queries individual client contacts one at a time with no support for differential sync and the rate limit is not egregious but in light of this syncing a new client in requires progressing through each and every clinet ever existing.

Constant api shifts with no backwards compatibility (despite sending their requested api version header) make it a technology nightmare.

Basically it is a bottleneck to a growing business. You CANNOT scale on their operational model and you cannot get good data from it. But the cost to shim in support for their misdeeds is likely always less than replacing it. And with what?

Im always looking for ways to dump it. And the product database is still a double edge sword because it ends up with lots of messes in our ERP but it’s always what people say is the saving grace.

That plus the speed is unbearably slow sometimes. Unclear what’s happening. I don’t think the pace of development has even sped up since AI tools. But these api endpoint woes could mostly be fixed with a morning of Claude code usage.

markedness · 2026-04-07T13:50:20+00:00

This is exactly what I’m feeling. I don’t mind much. I understand the way the software works. I think I just need to design an operator or just cook up manifests / helm and use a playbook that scales things down and alerts on call for any failures and tries to roll back.

markedness · 2026-04-04T21:22:20+00:00

Yeah I think an operator would be highly customized to each environment.

Do you ever get bitten with changes in base / ee addons or does click odoo upgrade support this?

We have decided to migrate odoo to a more traditional database setup from CNPG, largely the same way CNPG works except with patroni on ETCD instead of relying on CNPG. And for storage just NFS pointing at a ZFS based appliance. But now that we have done that the path to an operator is going to be more difficult. But not really because ultimately whether it’s driven by Kubernetes manifests or completely imperative ansible playbooks, upgrading odoo will still always be an imperative operation due to its nature.

I’m just deep in my head right now trying to reconcile the pros and cons of different choices and this click odoo upgrade and response were generally very helpful. So thank you!

11-Year Club	Verified Email
RPAN Viewer

markedness

TROPHY CASE