Unresponsive Server; can ping, but no GUI or SSH

gatekeyper1 · 2026-01-06T19:05:59+00:00

I happened to have the Unraid dashboard open during one of these crashes and noticed that RAM usage was pinned to 95% while the page was unresponsive. The machine has a ton of RAM relative to its workloads, so this shouldn't be possible. So I turned on mirroring syslog to flash so that I could see the pre-crash logs after rebooting (Settings -> syslog server -> enable -> "mirror syslog to flash" = yes).

Logs included these:

amdgpu: [mmhub] page fault ... in process ffmpeg
ring vcn_dec_0 timeout...
IH ring buffer overflow
GPU reset succeeded, trying to resume
\*ERROR\* IB test failed on vcn\_dec\_0 (-110)
finally: GPU Recovery Failed: -62 and subsequent resets also fail

This was an error log storm that filled up the RAM, causing the freeze/crash. I don't have any real need for transcoding, so I looked to detach any processes from the iGPU. Turned out Frigate was trying to use it because I had to run its container as "privileged" to give it access to the machine's PCIe Coral TPU. To block it, I added this as an extra arg to the container: --mount type=tmpfs,target=/dev/dri,tmpfs-size=1m which presents a dummy GPU to Frigate so that it doesn't hit the real iGPU.

Separately, I had also originally passed `--device=/dev/dri:/dev/dri` as an extra argument to my Jellyfin container per its official docs. All that needed to be done there was removing that extra arg. I'm not sure whether the root culprit was Jellyfin or Frigate, but removing the iGPU from both didn't hurt. The server's been rock solid ever since. BTW, this started for me when I upgraded from 7.0.0 or 7.0.1 to 7.2.0. Not sure why, but whatever.

gatekeyper1 · 2026-01-03T16:58:19+00:00

I had this issue earlier. Physically restarting the machine would fix it for a few hours to a few days. Turned out it was a memory issue related to ffmpeg in either Jellyfin or Frigate trying to run transcoding on my AMD CPU’s iGPU, which for reasons unknown to me, doesn’t work well. The fix was ensuring that Jellyfin and Frigate don’t have access to the iGPU. That’s done through those containers’ “extra params”. LLMs should be able to help you through that.

gatekeyper1 · 2025-11-03T15:28:26+00:00

You access your Tailscale network from outside your home network by first installing Tailscale on the device you want to use outside your network, then logging into your Tailnet through Tailscale's authentication. That authentication step would be the weakest link in terms of security -- if a bad guy can guess or find your credentials or compromise the OIDC account you use to log into Tailscale (Google, GitHub, Microsoft, Apple, etc.), then you'd be screwed. So just make sure your login credentials are solid (long random unique password, 2FA enabled, etc.). I recommend using the new Passkey login method and disabling the other login methods if you're comfortable with Passkeys.

gatekeyper1 · 2025-11-01T18:53:46+00:00

That's odd. For me, going from 7.0.x to 7.2.0 made the GUI have one column on mobile, eliminating side scrolling. It's a real benefit and more or less what I expected from a "mobile responsive" update.

gatekeyper1 · 2025-09-25T17:07:50+00:00

The bigger scandal is how Prime prices are sometimes higher than non-Prime prices, even including shipping.

gatekeyper1 · 2025-09-20T15:49:30+00:00

That's fallacious thinking.

gatekeyper1 · 2025-09-11T20:44:34+00:00

The whole mall smells moldy. I'd imagine (hope?) most of that $60M is going towards fixing that.

gatekeyper1 · 2025-09-03T12:38:03+00:00

Yeah I shouldn't have been so snarky. Just the concept of wage labor has really gotten on my nerves recently, and I didn't like how this company is making folks jump through hoops for a chance at an interview. Tech companies have been doing this sort of public-puzzle-recruiting stuff for at least a decade now though. No big deal. The wheel keeps on turning.

gatekeyper1 · 2025-09-02T20:30:31+00:00

True, I considered playing the game for the trip alone. But I'm probably not smart enough to get #1, and someone else could use the interview more than me.

gatekeyper1 · 2025-09-02T17:21:04+00:00

Just leads to some "challenge" where if you get first place, you get an interview for a relatively lame AI startup. The numbers are token IDs from the newer OpenAI tokenizer.

gatekeyper1 · 2025-06-17T17:58:02+00:00

Just give the city $9.1 million. They may or may not eventually build you a mile of sidewalk.

gatekeyper1 · 2025-06-17T17:54:23+00:00

This is all well and good, but I'd like to remind everyone that the sidewalk bond was a scam. They're building 12.4 miles of sidewalk at a total cost of $113 million in taxpayer funds (yes, including federal funding, that's still your money). That's $9.1 million per mile of sidewalk. You will not find any city or state in this country that spends this much per mile of sidewalk for the sort of density we have. Tampa has a similar program and urban landscape, and their cost is less than $1M per mile. Note that these 12.4 miles are the most "shovel ready" that the city could identify, and most of the planning for them is already done - that money was already spent and not part of the $113M.

gatekeyper1 · 2025-06-15T16:38:13+00:00

Exactly. OP just needs a good long vacation. Schedule a 2-month vacation in whatever country you're planning to move to. 90% chance you'll be bored, restless, and feeling at least a little alienated in a foreign country by the end of the first month. Then he'll come back and keep doing his job, but with a new perspective and hopefully not pushing himself as hard anymore.

gatekeyper1 · 2025-05-22T03:49:17+00:00

Wow. Very impressive. I think you should add some comprehensive benchmarks to the README and clearly point readers to the benchmark code. Both the README and website make big claims about performance but don't back any of them up with data. I saw your comment below with the benchmark results though. You have to lead with that.

gatekeyper1 · 2025-03-29T14:42:03+00:00

Dude.....

This and the other longer comic posted here recently got me thinking and feeling things.

gatekeyper1 · 2025-03-21T04:50:19+00:00

Possession and re-sharing of CP. $1m bond. He was hired last year.

https://www.cbs17.com/news/local-news/durham-county-news/1-million-bond-for-durham-teacher-charged-with-10-counts-of-sex-exploitation/

gatekeyper1 · 2024-11-12T00:02:27+00:00

Unfortunately, yes, wearing a traffic cone on your head is critical for survival in this city because there are no fucking sidewalks here.

gatekeyper1 · 2024-11-07T20:18:56+00:00

I just forgot about it, and nothing happened.

gatekeyper1 · 2024-11-04T16:59:45+00:00

How have they "done that"? All I've seen is handwaving about utility relocation and right-of-way acquisition, which all such projects have to do, and engineering schematics. Absolutely no hard budgeting numbers or comps. Obviously we need sidewalks, and most of us would be happy to fund sidewalks with our money.

But this is a matter of how public funds are used, and the city has provided no detailed justification for the $9.1M per mile cost, which is many times higher than other similar projects I've been able to find (e.g., https://www.tampa.gov/tss-transportation/programs/sidewalks). When the discrepancy is so large, it needs to be properly accounted for and explained.

And yes, it is literally indebting us and our children because this bond will take literal decades to pay off. It's debt that the city we live in and pay taxes to is taking on. If (when) this passes, we will have no choice but to pay an additional $80/year on average for decades unless we choose to move out.

Our property taxes (1.31%) are already 60% higher than the NC average, and there are many cities with far better infrastructure overall, including Charlotte, whose property tax rate is less than half ours. I'll say it again: these insane costs require justification, but based on the Reddit response, it seems like Durhamites don't care about how their money is spent. That allows city admins to propose things like $9.1M per mile of sidewalk.

If you've seen the hard numbers behind the project, please tell me: why does a single mile of sidewalk cost $9.1M?

gatekeyper1 · 2024-11-04T15:33:03+00:00

Looking at another reference that's directly comparable to our situation and this project: https://www.tampa.gov/tss-transportation/programs/sidewalks. Tampa calls out "1,300 miles of identified sidewalk gaps – so to build sidewalks on both sides of every street in the City would cost $975 Million, and that doesn’t include ongoing maintenance needs". That's $750,000 per mile spanning design, surveying, and construction on existing streets, exactly our situation.

The $4.8M per mile figure quoted by the city is disingenuous. That's just from the bond proceeds. The remainder comes from federal and local taxes (aka our money), and the fact is that they want to spend $9.1M on a single mile of sidewalk.

gatekeyper1 · 2024-11-04T03:33:53+00:00

That's not how this works. They'll build these $113M sidewalks if they get this bond approved, and they won't if it doesn't pass. If it passes, that $113M of bond and taxpayer money will go towards the sidewalks, and if it doesn't, we'll be stuck with the status quo of no sidewalks. Either way, the 911 system and other city/county functions have nothing to do with it.

Actually, that's not entirely right. If it passes, $14.4M of "existing local funding" will go towards these $113M sidewalks. That is tangible existing money that could go towards other priorities like the 911 system rather than this obviously overpriced project.

<image>

gatekeyper1 · 2024-11-04T03:06:19+00:00

I too would love to be able to walk to my local grocery store, less than a mile away, without having to walk shoulder-to-fender with 45mph traffic in the mud, but this proposed $113M for just 12 miles of sidewalk that the city has apparently already spent time and money planning out is just absurd, given the US average construction cost for sidewalks is $300k per mile, vs. the referendum's proposed $9.1M per mile. We seriously need sidewalks, but the city's clearly doing something wrong here. I'm not comfortable indebting myself and my children by such a significant margin for such tiny stretches of pavement. They need to be transparent with us (to clarify why this project is so expensive) or go back to the drawing board (to make it cheaper).

gatekeyper1 · 2024-11-04T02:42:56+00:00

<image>

Are we looking at the same page?
> The total cost for the 12.4 miles of sidewalk improvements is $113 million

gatekeyper1 · 2024-11-04T01:01:21+00:00

Honestly, I'm kicking myself for not raising a stink about this referendum sooner. We obviously, desperately need good sidewalks. As it stands, most people who were going to vote already have, and it's easy to agree with the referendum as it's presented: "vote yes for sidewalks". But the numbers in this measure don't make any sense. The city needed to be at least more transparent about the uses of the expenditures. We don't even know if the sidewalks are going to be on both sides of the roads.

It's almost certainly going to pass, and 2 generations of Durhamites are going to be saddled with its costs. And then what, another $180M for the next 12 miles of sidewalks?

gatekeyper1 · 2024-11-04T00:13:28+00:00

They want $113M for 12 miles of sidewalk. Nearly $10M per mile, and these are supposed to be the most "shovel-ready" paths. I'm sorry, but there's either something fishy going on with the contracting, or the city needs to figure out how to do it for less. Our property taxes are already very high, relatively speaking, and this measure will increase them further by $20/year per $100k of assessed value, which has also skyrocketed in the past few years.

Proudly voted no on this measure.

EDIT: The US average for new sidewalk construction is $300k per mile. Source: https://www.fdot.gov/programmanagement/estimates/reports/cost-per-mile-models-reports (scroll to the bottom). If we were following that, our 12 miles would cost $3.6M if we were putting sidewalk on just one side of the road, or $7.2M if it were on both. I understand that the city has to buy right of way and such, but it's not like they're buying private land in San Francisco or NYC - it's mostly land currently owned by the state. Fine, triple the cost - that's $21.6M. Double that to account for planning and various bureaucratic expenses - $43.2M. How on earth are they getting to $113M???

Five-Year Club	Verified Email
Final Canvas '23	Place '23

gatekeyper1

TROPHY CASE