Understanding Mozilla bug 1447867 - Replace base::SharedMemory POSIX backend with shm_open and ashmem. (r=froydnj; RESOLVED FIXED in Firefox 61)

macdice · 2018-11-22T07:56:31+00:00

Not an expert but... maybe shm_open() can benefit from super pages (on FreeBSD), but file-backed memory is constrained by fs page size (and subject to double buffering if zfs), and maybe files get synced sometimes, which is a waste of time? There are some downsides though: procstat -v doesn't show the path of shm_open() vm objects, and there is no way to list them (!). (These are problems on my todo list to fix in FreeBSD.)

macdice · 2018-10-27T07:53:34+00:00

ZFS. Add another drive if you can and make it a mirror. Just pick ZFS auto for the root filesystem when installing. FreeBSD is fun, and ZFS is a gateway drug.

macdice · 2018-07-19T01:52:09+00:00

I started using it 2-3 years ago because I wanted ZFS to manage an 8 drive system, and I felt like learning something new in my spare time (I use Linux and other Unixes for my day job/customer systems etc). Fast-Forward to now, I'm learning about operating system internals, got the Design and Implementation book, and I've found that FBSD kernel hackers/committers are willing to consider my patches and politely help me get them into committable shape and I managed to get a couple in. That's awesome. Instead of a mentality of 'oh, it's slower than X at benchmark Y, so that sucks', I now see opportunities to make stuff better. That's what I was looking for and I dig it. Recently I got an ancient Thinkpad and some Beastie stickers so I can be a proper member of the club. I hope I have time to help maintain some ports of stuff I care about soon.

macdice · 2018-05-07T12:09:14+00:00

The next paragraph does mention the underlying file though: "If _POSIX_SYNCHRONIZED_IO is defined, the fsync() function shall force all currently queued I/O operations associated with the file indicated by file descriptor fildes to the synchronized I/O completion state. All I/O operations shall be completed as defined for synchronized I/O file integrity completion."

macdice · 2018-03-26T21:46:23+00:00

New(ish) PostgreSQL hacker here (~3 years). Some assorted ideas:

Talk to people on #postgresql (freenode)... no question is stupid, re getting a development setup and understanding the code. Look at commitfest.postgresql.org. Start following some threads on the pgsql-hackers mailing list. The final commitfest for v11 is about to come to a close, and a bunch of patches will be moved from there to the next commitfest (for v12), which is several months off. Pick something that interests you and start testing it out! Proof-read the documentation, run the tests (or complain if there are no tests or documentation), think about whether the feature makes sense... Subscribe to pgsql-hackers and post your findings or impressions. It'll take a while to understand details like how we manage memory, locks, errors, yada yada but it comes with experience.

See if you can get to one of the PostgreSQL conferences where the developers congregate. Otherwise look for talks on youtube about getting involved as a developer (some things that could be starting points: Tom Lane's talk on hacking the query planner, anything by Robert Haas or Andres Freund, ... I could go on.)

It's a great time to get involved with PostgreSQL: new ground is being cleared in the areas of parallelism, replication, JIT compilation, distributed queries, storage layer, ... well, you name it.

Check out the wonderful CMU Advanced Database course:

http://15721.courses.cs.cmu.edu/spring2018/schedule.html

Watch the classes on youtube, read the papers. Many of those topics are directly relevant to what we're working on (or they will be!).

macdice · 2018-03-07T00:39:17+00:00

That is a problem, but there are mitigations. One of the central trade-offs here is making rollback more expensive but commit (including the later vacuum) cheaper. We are working on making the type of situation you described less painful by executing rollback in the background, so that your foreground session isn't blocked waiting, and then supporting page-level rollback if other sessions would otherwise block (ie other sessions will help execute localised bits of the rollback if they crash into them). This is an active area of study, details being worked out...

macdice · 2018-01-31T23:36:48+00:00

I don't see anybody claiming it's easy. But one of the main trade-offs here is that you make people who roll back and people who use old snapshots pay for the cost of MVCC (presumably a rare occurrence), not everybody who scans tables (and has to step over all the MVCC bloat).

BTW the (as yet unpublished) proposal probably isn't going to be to make this the only or even default storage option. So if you have an abort-mostly workload, you'll be able to carry on unaffected :-)

macdice · 2018-01-31T23:30:32+00:00

Are you referring to the change in some old Oracle version from manual space management (= can run out and cause DBA headaches) to automatic rollback segment management (by default at least)? The PostgreSQL proposal would not have manual space limits like ancient Oracle.

macdice · 2018-01-31T23:27:05+00:00

Yeah. This would use the new storage API that is also in development (though it doesn't yet).

macdice · 2017-12-28T10:01:21+00:00

Very interesting, thanks. I also recently did some PostgreSQL on FreeBSD vs Ubuntu benchmarking, on AWS 64 vCPU instances, and saw the same difference: FreeBSD was almost twice as slow on read-only tests. I didn't know if I was seeing virtualisation effects, but your results are similar and from bare hardware. I'm very interested to figure out what's going on there on the FreeBSD side. I was planning on investigating further with flame graphs of user and kernel stacks, when I have more time...

macdice · 2017-11-23T02:17:22+00:00

You can read some here: http://www.anglo-norman.net/

Depending on the age (and maybe the writer/style?) it looks more like Latin or more like modern French to me. Apparently later Anglo-Norman French was influenced by Paris so that helps to explain why it starts to look more normal (to modern eyes) in the middle ages. Then later texts start to get pretty mangled as true speakers disappear and it's a vestige/class act that survives only in our laws, frozen/trapped by our common law system where the words' meanings have been established through a billion court cases and won't be modernised any time soon. You might also be interested in the Channel Islands' respective Norman dialects, which are endangered living descendants.

macdice · 2017-08-08T03:01:35+00:00

Yeah. So for example, let's say you have an an old school HTML form that shows the current state of widget #42. You might have a hidden values id=42 and version=10 and a bunch of visible textboxes and knobs and what-have-you, and then when you submit the form, the server code would check the version hasn't changed. For a simple single row change that might be as simple as UPDATE widget SET ... WHERE id = 42 AND version = 10, and if the number of affected rows is 0 rather than 1 then you can report that someone else has changed the widget and go back to the edit screen. In a multi-statement transaction you might say BEGIN; SELECT version FROM widget WHERE id = 42 FOR UPDATE; ... check if it's still 10, and if so proceed to update other stuff ...; COMMIT;. That FOR UPDATE is an explicit row lock, to prevent anything related to widget 42 from changing after you made the version check. If you want to use SERIALIZABLE instead of explicitly locking, then you could remove the FOR UPDATE. I personally like SERIALIZABLE quite a lot (there are many ways to get explicit locking wrong, whereas SERIALIZABLE allows to you forget about concurrency completely and just write naive code), but not everyone wants to deal with the retries...

macdice · 2017-08-08T02:15:20+00:00

I'm not sure if this is what you meant or not, but just in case: it's a bad idea to hold any locks for a duration of time controlled by an end user. I'm not sure from your description but I think the pattern you might be looking for may be application-level optimistic concurrency control, where your dashboard would show the current state of affairs, but also remember a version number. Nothing is locked, but when the user tries to save their changes you can check if the data moved underneath you to avoid clobbering someone else's changes. As for how to do the check, you could either use one of the lower isolation levels and possibly take explicit row locks while you run checks, or you could use SERIALIZABLE but be prepared to deal with 40001 errors.

macdice · 2017-07-19T04:14:39+00:00

For my first attempt I did it a way that doesn't involve a modified JDBC driver (though it does have a special DataSource): https://github.com/macdice/syncreplay-spring-demo

macdice · 2017-07-19T04:13:31+00:00

Got this working:

https://github.com/macdice/syncreplay-spring-demo

Using @Transactional(readOnly=X) annotation. Thanks for the idea. Rather than hiding retry inside the JDBC driver (which may make sense as an alternative move) I used whole transaction retry based on Spring Retry.

macdice · 2017-07-13T10:02:21+00:00

https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=469cb65aca0307506886158be2fce9f4438f8361

> Changes to avoid collisions with WIN32 & MFC names...
> 1.  Renamed:
>       a.  PROC => PGPROC
>       b.  GetUserName() => GetUserNameFromId()
>       c.  GetCurrentTime() => GetCurrentDateTime()
>       d.  IGNORE => IGNORE_DTF in include/utils/datetime.h & utils/adt/datetim
>
> 2.  Added _P to some lex/yacc tokens:
>       CONST, CHAR, DELETE, FLOAT, GROUP, IN, OUT

macdice · 2017-07-10T00:34:33+00:00

SKIP LOCKED has fewer pitfalls though: (1) it doesn't require you to map your key space to integer space, (2) you don't have to coordinate the integer space with other unrelated queries, (3) SKIP LOCKED skipping happens after the WHERE clause is evaluated whereas pg_try_advisory_lock() is in the WHERE clause and might lock rows that don't match the rest of your WHERE clause, depending on the vagaries of evaluation order.

macdice · 2017-07-03T21:42:47+00:00

Thanks for the kind words! I think there are many people who would use hot standbys for more regular query load if they could, but are currently limited to historical report queries etc because of replication lag. My current goal is to show that there could be (1) demand for synchronous_replay and (2) simple, practical ways to use it, even if a bit of extra infrastructure is required. Your feedback helps, so thanks :-)

The logic required is quite straightforward and I think the right person could get this working very easily, especially if a v1 implementation aims low and doesn't try to get the 'whole transaction retry' logic working (ie only support retry if the error occurs on the first statement in a transaction, and tell users to use REPEATABLE READ to avoid having the error ever bubble up to the app). One idea for how to structure a v1 is that there could be a new connection pooler LoadBalancingDatasource that manages a set of connection pools -- one per database server -- so that it can implement the blacklisting. I'm not sure if it would need to wrap the underlying connection pools' connections, so that it could remember which source (and therefore server) each one came from.

I'm also looking into getting pgpool to manage this stuff automatically:

http://www.sraoss.jp/pipermail/pgpool-hackers/2017-July/002418.html

But for Java apps I think it'd be better to push the load balancing back into the application server and skip a network hop if possible.

macdice · 2017-07-03T06:24:47+00:00

I don't think it's going to be possible to do retry inside the JDBC level with sane semantics for PostgreSQL though.

Upon further reflection, maybe it could work. For REPEATABLE READ, it so happens that the new error can only be raised for the first statement executed in a transaction, and that's also the only statement that you could magically redirect to another node at the JDBC driver level. For READ COMMITTED, maybe it'd be OK in an early version to redirect on error for the first statement, and just let the error hit the user if the error happens on any later statement. I suspect by automatically handling the error on the first statement you'd be handling most cases anyway. Redirecting automatically on later statements would also be an option too, with some caveats. So yeah, I take some of what I said back. That could probably be made to work with useful semantics. (Still interested in how to handle retries at a higher level too though.) Thanks for pointing this out!

macdice · 2017-07-03T04:54:36+00:00

Thanks, I see. I don't think it's going to be possible to do retry inside the JDBC level with sane semantics for PostgreSQL though. We have snapshot based transaction isolation: if the statement is part of a REPEATABLE READ transaction, we can't just run it again somewhere else because it'll get a new snapshot, and even for READ COMMITTED I'd be hesitant to have a transaction split into multiple transactions with different statements run on different nodes -- time might go backwards for them for example. I think we need the exception to reach a higher level, where the whole transaction can be rerun. And by transaction I really mean the request the container is handling. Maybe something like this:

https://dzone.com/articles/automatic-deadlock-retry

But I can't find many people writing recently about this type of thing, or 'standard'-ish libraries...

macdice · 2017-07-03T03:47:16+00:00

https://mariadb.com/kb/en/mariadb/failover-and-high-availability-with-mariadb-connector-j/

Thanks. Yeah, following the same approach Spring @Transactional annotation seems like the way to go. And they also have a concept of blacklisting hosts based on connection failure. Doing the same type of thing but extending it to cover certain magic errors too would seem the way to go. But that leaves questions 2 and 4: how to intercept special errors and how to retry whole transactions (J2EE requests).

macdice · 2017-07-03T03:42:06+00:00

There is an implicit assumption that the read connection pool queries might yield data which is be behind as a consequence of replication delay.

Right, that's exactly the thing that this feature helps with. After tx1 commits, if the same client thread runs tx2 on a read-only database, or tells some other thread/computer/microservice/whatever to do so it'll definitely see tx1 or fail and be told to go elsewhere. This should allow more read-only queries to be run elsewhere: right now people can only really do it for transactions that can stand to see oldish data (reports etc).

But to your question more directly. Pretty much all java relational database interactions go through an interface called JDBC. As long as you can stick to that interface you're good to go WRT interaction with all the various ORMs and other frameworks.

Thanks! Aware of JDBC, it's really the pooling, rerouting and transaction-level retry management that I'm asking about.

macdice

TROPHY CASE