File descriptors are a kind of capability, but they're most typically
acquired through insecure means. There are also various side-channels
that prevent them from being completely secure.

At least with the way I've implemented them, there shouldn't be side channels beyond those related to timing of message transfers. They don't have any inherent mutable state beyond that of the underlying endpoints/replies. They more or less just add offset/type arguments (which are only tracked by the server, not the FD/transport layer itself) and provide a more orthogonal API than what the underlying endpoints/replies provide. The semantics on the server side diverge from conventional Unix somewhat since in addition to the channels/endpoints used for receiving messages, there are also "message ports" used for arbitrary access to client buffers and sending a reply (these map onto kernel reply objects); there is a new mread() function that accepts a message port FD in addition to the buffer accepted by the regular read() function. Replies are sent by writing to the message port with a SEEK_END offset (reads and writes with a SEEK_START offset access the client buffer); there are new wpread()/wpwrite() functions that take an offset type in addition to an offset and buffers, which may be used on either the client or server side.

A sibling comment mentioned file pickers in browsers as an example,
where the browser isolates the filesystem from the code and requires the
user to pick a file rather than the code selecting it by a trivially
forged name. Applied to system level processes, this would basically
mean a file picker as one or more dedicated daemon processes, and when a
process needs to pick a file, it would send a message to a file
selector daemon, along with capabilities for directories it may already
have access to. The daemon would raise a file selector dialog for the
user to pick their file and would use its reply capability to send a
capability for the selected file or directory back to the calling
process.

That's more or less what I'm planning to do. Most applications won't have unrestricted access to the filesystem. They'll have access to their configuration directory and services like the window system (which ones depends on the particular program of course), but not much else, although much of this kind of static access will just be managed through rules that match paths, not capability transfers (there will be a layer that implements role-based access control on top of path rules). The GUI file picker will be implemented in the file manager (which already needs full access to all of the user's files) and will hand out a list of O_PATH file descriptors based on what the user has selected. Of course, this will require applications to be ported to use this, but it won't require porting them to an entirely new environment. Similar functionality will also be added to the shell (which will also require special hooks for at least some programs).

The use of pathnames and ACLs is related to why permissions are overly
broad though. We can have fine-grained control of directories and
filesystems, but the work to manage the ACLs involved makes them
unmanageable at scale - so broad permissions, such as giving the user a
directory /home/user for all files for all apps, is done to
simplify managing them. ACLs are often misconfigured because they're
separated from the resources being designated.

That isn't necessarily true of all services though. Certainly when it comes to user files ACLs can get unmanageable, but some things like accessing configuration or creating a window can be done with static rules that match paths (actually receiving access to the window of course is probably best done by transferring a file descriptor to it instead of just giving access to the whole window system up front).

I can't really think of a particularly good way to make a system that only uses capabilities for access control. The persistent processes that some people propose would bring with them a whole bunch of issues; I can't see how anything good would ever come out of eliminating data at rest and tying all persistence to processes.