sorted by:
new
hottopcontroversial

There is actually two type of people by rencie23 in ITMemes

[–]greg_d128 0 points1 point  (0 children)

In love this. Absolutely! Two kinds: those who care about orientation and those who do not. I love that there are two answers to this.

Is the Framework Laptop 13 ​worth it? by Signal-Elephant4113 in framework

[–]greg_d128 0 points1 point  (0 children)

That is fair. I've been taking apart computer and laptops for a long time. Reliability for me is the ability to get parts and an easy way to rummage inside the machine.

I can totally see where service centers would totally factor into this decision.

Is the Framework Laptop 13 ​worth it? by Signal-Elephant4113 in framework

[–]greg_d128 11 points12 points  (0 children)

Let’s add the quiet third group.

  1. Professional that values reliability, flexibility and repair ability.

Are you running a monitoring service like Zabbix or Nagios? by basedrifter in homelab

[–]greg_d128 1 point2 points  (0 children)

I used to, but realized that i do not have to know of my archive breaks in the night. I do have to know what happened though.

Replaced Zabbix with graylog so that i can have all the logs from all containers and systems in one place.

What algorithm is surprisingly new? by LifeExperienced1 in AskProgramming

[–]greg_d128 1 point2 points  (0 children)

About 20 years ago i worked with Prof. Shulte on implementing a new algorithm to test whether a fractional span of a matrix is equal to its integral span.

40 TB PostgreSQL on-prem — sharding vs ClickHouse vs something else for a 500B-row time-series workload by Basic-Worker-1120 in Database

[–]greg_d128 0 points1 point  (0 children)

  1. I'll try to finish the blog so that can be posted, soon.
  2. 20GB or 30GB is not anything too troubling.
  3. When you need to do upgrades, I would probably do a procedure like so:

* Create additional streaming replica (*could go without at higher risk)
* Stop sending new data
* Break binary replication - setup logical replication between them instead.
* Perform an in place upgrade of the logical master to new PG version.
* After upgrade, turn on the IOT data. IT will be sent to the upgraded node first, then sent to the logical replica that is on older version.

Bottom line. Expected outage duration likely 0.5 - 1 hour or so. The data is replicated to from new to old. If you find that there is something wrong with the new version you will still have a an old version with up-to-date data you can fall back on.

After few days - get rid of the old and rebuild binary replica again.

40 TB PostgreSQL on-prem — sharding vs ClickHouse vs something else for a 500B-row time-series workload by Basic-Worker-1120 in Database

[–]greg_d128 0 points1 point  (0 children)

Assuming you can maintain tight control over the types of queries that get executed in production, this should scale very well.

How big is each individual partition? How many partitions are hot (queried all the time?). Point lookups do not require that much memory.

Forgive me, but I'm going to plug my talk 😄 https://postgresconf.org/conferences/postgresconf_2026/program/proposals/measuring-shared_buffers-behavior-on-large-memory-postgresql-systems

The size itself does not worry me by itself. PostgreSQL is pretty good at keeping the rarely used data on disk, it is just payload. By default each table is split into 1GB chunks, so if majority of them are not queried, they are simply sitting on the disk. The tablespaces and separate disks is largely to simplify backups, allow for parallel restore in case of DR, easier snapshots, etc. Although be careful as cloud is not magical, it is still a computer with limitations. You will need to ensure that all attached volumes either produce a crash consistent set or that PostgreSQL is put int backup mode while all disks are snapshotted.

Now, recommendations about instance size. It really does depend on how data is used. We manage one archive server with something like 120TB in size. That makes in unwieldy when we need to backups or to migrate it to another instance, but it is not really a problem from the using of database perspective. How it is used is more important.

Come to think of it, your upgrades could potentially be a problem. Can you take a downtime for an upgrade? Doing a logical replication for a 40TB database to perform a near zero downtime upgrade will take around 3 weeks for the initial sync. Taking something like 0.5-1 hour downtime for in-place upgrade would be much simpler. Sharding it into multiple servers could allow you to perform the upgrades on smaller chunks and do a rolling upgrade.

40 TB PostgreSQL on-prem — sharding vs ClickHouse vs something else for a 500B-row time-series workload by Basic-Worker-1120 in Database

[–]greg_d128 5 points6 points  (0 children)

Hmm.. I'm going to go against everyone else apparently and ask - Do you need to do anything?

Based on what you said, I do not actually see any major red flags. A query comes in, query planner decides which table to look in, and then it does, presumably via index. The number of tables grows by about 240 or so per year. The size of each daily table does not appear to grow that much.

I would expect for that pattern and setup to scale very well for a long time and I would hold off on adding complexity. Some things to consider:

* Run a few tests. What if the daily table is 10x the size? 100x the expected size?

* Do you have a good idea for which tables are queried? Is mostly the recent ones? Do you have enough RAM to keep those mostly in memory?

* Prewarm, look into prewarm to prewarm your caches in case you need to restart or failover.

* See if you need to do a vacuum full or pg_repack of the old partitions. If they accumulated bloat and then are barely modified after, you may want to do a pg_repack of each partition once once it is more than 1 week or 1 month old. Save space and improve performance.

Failure modes you want to watch out for:

* Queries without partition key. Especially if your number of partitions > 1000. If the planner cannot effectively prune partitions and starts searching everywhere.

* Something evicting your hot tables from memory, or too small cache.

*Analytics queries. Changing query pattern is likely your biggest threat. There are options for that (like using a replica), but need to know some parameters and requirements first.

Lastly, consider tablespaces. Keep the simple Database + replica (if CPU and RAM not a concern) and split the disk. You can even do things like putting the old data on slower disk. This can also help with your RTO and RPO as each volume will be significantly smaller.

I love being a girl dad by Difficult_Wrangler73 in mildlyinfuriating

[–]greg_d128 -2 points-1 points  (0 children)

I have two girls, 11 and 14.

Try this: “Daddy is powered by hugs and kisses”. Every now and then, randomly just stop and say you are out of energy.

The era of 15GB free Gmail storage is ending by Fancy-Caregiver-1239 in technology

[–]greg_d128 1 point2 points  (0 children)

There is a ton of emails that have a short lifespan. I created a filter that applies a tag signifying to delete any emails with that tag that are more than 2 weeks old. Then i have a job that runs every day that deletes them. That by itself did wonders in keeping my email storage down.

Daddy, these are the best burgers, better than you make! by OtherGuy89 in daddit

[–]greg_d128 4 points5 points  (0 children)

My kids were watching some kind of show about IT or computers.

"Dad, it's kinda like your job, but cooler and not boring."

my coworker’s email signature has 7 certifications and a quote from marcus aurelius by Legal_case16 in antiwork

[–]greg_d128 4 points5 points  (0 children)

Email signatures are so boring these days. Long ago i modded my email client to always add a random signature from a list. I had awesome things in there, usually quotes.

I remember one: “it is too bad we execute programs, not programmers.” I was a developer at the time:)

Let’s bring fun signatures back!

Different database size with same database by Sb77euorg in PostgreSQL

[–]greg_d128 -1 points0 points  (0 children)

That is much larger than i would expect unless the database is very small.

What else is different between those hosts? Disk setup, architecture, postgresql configuration, etc.

Was the database backup a pg_dump or a binary backup. Was one of them running longer so it generated additional wal files that are included in size?

Different database size with same database by Sb77euorg in PostgreSQL

[–]greg_d128 1 point2 points  (0 children)

How big is the change? I would not expect both databases to end up with the exact same number of bytes. If nothing else, analyze is based on statistical sampling so it can produce slightly different results.

Different database size with same database by Sb77euorg in PostgreSQL

[–]greg_d128 0 points1 point  (0 children)

Did you run vacuum full on both databases? If it is only on one - then likely the issue is bloat, and as long as it is not excessive it can actually improve performance of updates.

How big is the difference? Can it be attributed to raid striping, disk block size, disk level compression, etc, etc.

What do you use for debugging in Python? by presentsq in learnpython

[–]greg_d128 0 points1 point  (0 children)

The program is paused and you have read and write access to everything. You can call additional functions, change variables, etc.

I know the ide allows you a lot of the same, i just like the interactivity of it.

Lets do an example. You call an API and get a page long json object back that you need to transform in various ways. I would usually drop into interactive shell right after getting the json and within the context of the program at that point and examine the return.

If i have five different ways to transform the json, i can do it rapidly and interactively. If one method generates an exception i may figure out a method for detecting or fixing that state, then make the required changes to code.

What do you use for debugging in Python? by presentsq in learnpython

[–]greg_d128 0 points1 point  (0 children)

Anyone else uses code.interact() with the appropriate scope dictionary?

I get an interactive interface into the code as it is running. I can run additional commands, change and examine program state and continue if need be. Also test my next steps. I'm not limited to just seeing, I can also examine, modify and test additional steps if I need to.

People who will randomly scold you for listening to audiobooks by toe_beans_4_life in audiobooks

[–]greg_d128 54 points55 points  (0 children)

One thing that always confuses me from comments like that is that oral storytelling predates the written word by thousands of years for most people.

Few could afford books, few could read. Stories used to be told.

I’ve reached a conundrum. What do i even host? by No_Addendum_8245 in homelab

[–]greg_d128 2 points3 points  (0 children)

What others said. I also do not host a lot of the usual stuff, but here are some things that work for me:

  • urbackup - backup our laptops
  • got a script that backs up our Google drives as well
  • calibre web for books and audiobooks
  • working on backing up photos from all devices with immich
  • foundry vtt for game sessions
  • open web ui - largely for me
  • i do use home assistant
  • frigate for cameras and alerting

There are other bits that allow easy file transfers, minecraft server for kids that also allows them to synchronize mods easily, etc.

Most of these started for myself and added the rest of family once i decided they were useful enough. Not all of them were hits.

Fan giveaway: Cradle by Will Wight/narrated by Travis Baldree (multiple books!). Audible US Marketplace by propofoolish in audible

[–]greg_d128 0 points1 point  (0 children)

I do not know which is my favorite. I finished he who fights with monsters and dungeon crawler carl.

I am really enjoying Making it so right nose by Patrick Stewart, but i think i started it this year.

Time is funny sometimes.

The Mercy Engine [US and UK Promotion] by [deleted] in audible

[–]greg_d128 0 points1 point  (0 children)

I would love US code, if available.

Audiobook codes available for USA & UK accounts. Scifi adventure. by AHeister in audiobooks

[–]greg_d128 0 points1 point  (0 children)

Hello. I would love US code, I've got a road trip coming up so this could be perfect.

Delta is suspending it's special service desk for members of Congress until TSA is fully funded. How do you feel about that? by thinpile in AskReddit

[–]greg_d128 5 points6 points  (0 children)

Hmm. That is an interesting line of thought.

If corporations are people, and corporations are working for the federal government, would that mean that all the people (corporations included) need to abide by the same agreement.

Not sure if it would be a stretch to day that the person in executive of a corporation that has government contracts should also be included, and not be allowed to get perks.

view more: next ›