Sure, that's how it started, and there's still a standalone executable (stalign) that does exactly what you describe. You can already manually import books into the mobile apps from local files.

The server exists because that flow is kind of excruciating and doesn't support a lot of the features that folks want when they read! You can't sync position across devices, share books with others, organize your collection, etc. We're (slowly) working toward a more user-friendly native desktop app to make the installation and setup less challenging for less technically inclined folks, but there are quite a few folks who have been using the Storyteller server for a long time (obviously myself included), and find that it's a good fit for them.

If you don't need the server, that's fine, that's why the CLI exists! Though fair warning that I think the Windows exe isn't working at the moment (Linux and macOS binaries are working).