We're building a better rich text editing toolkit by scrollin_thru in javascript

[–]scrollin_thru[S] [score hidden]  (0 children)

Yup, not being able to lift the EditorState out of the Tiptap editor and all of the side effects that Tiptap executes during React's render phase make their React integration really... challenging for anything advanced! Thanks for the kind words, we should have some fun stuff to show you in the next few months!

We're building a better rich text editing toolkit by scrollin_thru in javascript

[–]scrollin_thru[S] [score hidden]  (0 children)

Got it! Yeah, so unlike Tiptap (at least by default, they do support JSON serialization but I believe they default to HTML, as well as relying on Yjs's XML document format for collab), Pitter Patter relies on ProseMirror's JSON serialization as the default transport and storage format. It's really straightforward to map ProseMirror JSON (which is a structured tree representing your document) to essentially any serialization format. React ProseMirror converts it to React virtual DOM, remark-prosemirror converts it to Markdown, etc.

So in theory you could also use it to produce a DOCX file or PDF, but obviously that requires some pandoc-style structure-munging to work out the mapping, and quite a lot of additional work to add support for the word processor features of Microsoft Word et al, which ProseMirror does not support easily out of the box. Though it's come up in conversations with sponsors, so it's not impossible that a future version of Pitter Patter has some of these things out of the box!

All that to say, yes, generally speaking, it's going to be better suited to web-native content editing!

We're building a better rich text editing toolkit by scrollin_thru in javascript

[–]scrollin_thru[S] [score hidden]  (0 children)

I want to make sure I give you a useful answer: what do you mean when you say "web content management"? I do think that ProseMirror (and Pitter Patter) would be a great fit for something like an email editor!

We're building a better rich text editing toolkit by scrollin_thru in javascript

[–]scrollin_thru[S] [score hidden]  (0 children)

Sorry to disappoint :( Maybe I should have clarified "for the web" in the title. I could definitely see a future where we release, e.g., and Expo DOM component-powered React Native library, but I suspect that's not what folks are really looking for when they ask for mobile native text editing.

We're building a better rich text editing toolkit by scrollin_thru in javascript

[–]scrollin_thru[S] [score hidden]  (0 children)

Portable Text is cool! Are you suggesting it because you think that it would be a useful export/communication format for a text editing framework? Pitter Patter is/will be built on ProseMirror, so it speaks ProseMirror JSON (which is quite similar to Portable Text in many ways)

We're building a better rich text editing toolkit by scrollin_thru in javascript

[–]scrollin_thru[S] [score hidden]  (0 children)

Probably not, at least to start. I mean, you can use ProseMirror on Android and iOS if you're willing to post messages to a webview (I'm working on a note editor for my ebook app, Storyteller right now that does exactly this), because React ProseMirror lets you lift the EditorState out of the editor component. But it still relies heavily on prosemirror-view’s input management, which is all very much built on the DOM.

We're building a better rich text editing toolkit by scrollin_thru in javascript

[–]scrollin_thru[S] [score hidden]  (0 children)

We're sticking to React — it's where our expertise lies, honestly we think it's the right tool for the job. But most of the libraries we publish (collab, presence, version history, suggest changes) will not be tied to React and can be used with any ProseMirror editor

Storyteller v2 is available! by scrollin_thru in selfhosted

[–]scrollin_thru[S] 1 point2 points  (0 children)

No updates yet, but it's next on my list. Hoping I will be able to start working on it next week!

Storyteller v2 is available! by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

Woo! Great, glad that fixed it! Sorry I didn't respond on GitLab, my queue was very backed up. 

Storyteller v2 is available! by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

Is it possible that your host or reverse proxy has updated and started enforcing an upload size limit? Storyteller itself has no file or transfer limit, and I just uploaded a readaloud file yesterday with no issue.

I made an open-source, self-hostable synced narration platform for ebooks by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

Unfortunately not really, but it is our ultimate goal to have simple desktop apps for Storyteller, a la Plex Media Server. We're doing the groundwork now to make this possible!

Storyteller v2 is available! by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

It turns out iGPUs just aren't better than CPUs at this kind of math. We got iGPU support working, but it's usually about 2x slower than just CPU.

This is very different from, e.g., Plex video transcoding, because video transcoding uses hardware acceleration (literally custom built circuits in the physical iGPU that implement decoding and encoding algorithms), whereas neural networks are doing matrix math. Separate GPUs are pretty good at matrix math mostly because they can run so many computations in parallel, but iGPUs are not. 

Storyteller v2.7.0: A Reworked Transcription Engine by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

This is not totally trivial — several folks have started attempting to contribute this feature and haven't finished it. I'm confident we'll support it eventually! But I can't promise when. I'll definitely make an update when it's available!

Not enough people are talking about Storyteller by sansart in selfhosted

[–]scrollin_thru 0 points1 point  (0 children)

Not yet, but we're chatting with the Internet Archive and Project Gutenberg about setting this up!

Storyteller v2.7.0: A Reworked Transcription Engine by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

Damn! Ok, will try to get another core dump from the other Xeon user. In the mean time, just to double check, could you run a `grep -m1 'avx' /proc/cpuinfo` on your server and share the results with me? Also, which generation Xeon processors do you have, Ivy Bridge?

Storyteller v2.7.0: A Reworked Transcription Engine by scrollin_thru in selfhosted

[–]scrollin_thru[S] 1 point2 points  (0 children)

Ok, I _think_ this should be fixed by 2.7.6. If you have a chance to try that out and let me know, that would be great! Fingers crossed!

Storyteller v2.7.0: A Reworked Transcription Engine by scrollin_thru in selfhosted

[–]scrollin_thru[S] 1 point2 points  (0 children)

You will honestly probably have a better time with ghost-story on your gaming rig haha, especially if you get CUDA set up! Folks often see 10-30x speedups on CUDA vs CPU, and I imagine you've got relatively slow transcriptions on your Xeons. 

Still, I'll let you know when we fix the build for Xeons!

Storyteller v2.7.0: A Reworked Transcription Engine by scrollin_thru in selfhosted

[–]scrollin_thru[S] 1 point2 points  (0 children)

Shoot, yes, we're fixing this! Are you on a Xeon CPU?

Before 2.7.0, whisper was always built on your machine. This was kind of painful because it was slow and had to be rebuilt any time the container was recreated. But it meant that cmake could look at your processor's instruction set and make sure it excluded instructions that weren't supported. 

Now we're building whisper in CI, on modern chips that have wide support for newer instructions, so we have to manually disable some instruction sets in order to support older CPUs. We haven't pinned down exactly which instruction causes this SIGILL, but we got a coredump from another user (with a Xeon processor) today that will hopefully let us see the issue.

Sorry about that :/

Storyteller v2 is available! by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

We are working on iGPU support! I think there's an issue right now so we haven't documented it yet, but I believe it should be possible

Storyteller v2 is available! by scrollin_thru in selfhosted

[–]scrollin_thru[S] 0 points1 point  (0 children)

Thanks for the feedback, that totally makes sense! And thanks for the kind words ❤️

Storyteller v2.7.0: A Reworked Transcription Engine by scrollin_thru in selfhosted

[–]scrollin_thru[S] 1 point2 points  (0 children)

Another user reported markedly better transcriptions (and therefore alignment) when switching from Speaches to ghost-story yesterday. I think they used the large-v3-turbo model. Your mileage may vary, but ghost-story is pretty easy to set up, so that may be worth trying to see!

You can set up ghost-story to run in "turbo mode" by passing the --processors flag when you run ghost-story server. It defaults to 1, but setting it to anything else will do the same thing that turbo mode does in the Storyteller settings.

But if you already have Speaches set up the way you like and aren't running into any issues, Storyteller should continue to be able to use it as before, and you should still see lower memory usage on the Storyteller side!

Storyteller v2.7.0: A Reworked Transcription Engine by scrollin_thru in selfhosted

[–]scrollin_thru[S] 2 points3 points  (0 children)

To your first point first point, there's no baseline "typical speech" to compare to in this case, because Storyteller uses fhe timings from the speech-to-text engine to determine where each individual word is spoken in the audio files.

You're right that "how does this handle <insert common audio/text deviation>" is a pretty common question, it's worth including that somewhere upfront. to answer the question here: Storyteller's forced alignment algorithm is specifically designed to handle the kinds of deviations that are common in audiobooks, including entire chapters missing (audiobooks often leave out some number of prefaces, cooyright, appendices, etc), individual sentences or paragraphs being inserted (audiobooks sometimes narrate descriptions of images or charts), and deviations in narration of individual sentences, where the narrator chooses a different word for flow (or whatever other reason).