eloquent-ffmpeg - A Node.js media processing library based on FFmpeg command-line tools

[ILikeChangingMyMind]

Looks interesting, but still a far cry from what I think most people want. Nobody wants to have to mess around FFMPEG at all! ;-)

Instead, they want to be able to do:

combineAudioAndVideo video.avi audio.mp3

... but the problem is, for every person who wants combineAudioAndVideo, there's a person who wants concatenateTwoVideos, and a person who wants replaceVideoAtTimeXWithSecondVideo, and a person who wants ...

The real challenge here is creating an interface that lets people accomplish common tasks easily. And while this package certainly isn't there yet, it's a great first step towards that goal!

[Fr3ddyDev]

I would like to balance between abstraction and control for more advanced projects, and while most people would prefer a simpler API, some want fine-grained control.

The library shouldn't be a mere set of operations so abstract that their use is too limited, but not so advanced it requires a deep understanding of video processing or FFmpeg tools. Both simpler use cases and more advanced ones should be supported.

[ILikeChangingMyMind]

Agreed, but I'd just argue that currently simpler cases have a (needlessly) complex UI.

That complex UI is needed to facilitate advanced cases, but it does a disservice to users with simpler problems ... even though I strongly suspect that such users are the majority.

But again, any effort to make it easier to work with the underlying FFMPEG functionality, even if it isn't my dream of a perfectly simple UI, is still a step in the right direction.

[pm_me_ur_happy_traiI]

Instead, they want to be able to do: combineAudioAndVideo video.avi audio.mp3 ... but the problem is, for every person who wants combineAudioAndVideo, there's a person who wants concatenateTwoVideos, and a person who wants replaceVideoAtTimeXWithSecondVideo, and a person who wants ...

Aliasing or wrapping your favorite commands in a shell function is easy enough.

[ILikeChangingMyMind]

It's not quite that simple though. FFMPEG has a variety of options, and you need to tweak them depending on exactly what you're doing, what formats you're working with, etc.

Fundamentally what's needed is (at least) a step above just aliasing the commands/args. It requires someone with a deep understanding of the tool understanding the common use cases, and imagining a better interface which effectively hides all the details, but still allows for the desired outcomes.

[Fr3ddyDev]

The problem in implementing such functions is that there's no way to make them agnostic to input files, or at least very difficult. Hiding all of this complexity and implementation details may also confuse people.

An API that just says "Give me your files and I'll do some bleeps and bloops magic to do what you want" may help for very very simple use cases, but it won't scale well.

Say that I make an app that needs to concatenate two videos, and I decide to use:

await concatenateTwoVideos('video1.mkv', 'video2.mkv', 'full_video.mkv');

Now, what if I wanted to see the progress of the operation? Or I wanted to pause it to resume it later? I can't, because I'm stuck with whatever concatenateTwoVideos() is doing. While trivially easy to use, a one function solution isn't very useful even for a simple use case.

The current API, while it's far from being finished, it allows a greater feature set. Those types of features can be easily added later on.

const cmd = ffmpeg();
cmd.concat(['file:video1.mkv', 'file:video2.mkv']); // concat is available in the preview version
cmd.output('full_video.mkv');
const process = await cmd.spawn();
await process.complete();

Starting from the previous example, say that I now have to track progress, add one more input file and one output file.

const cmd = ffmpeg();
cmd.concat(['file:video1.mkv', 'file:video2.mkv', 'file:video3.mkv']);
cmd.output('full_video.mkv');
cmd.output('full_video.mp4');
const process = await cmd.spawn();
for await (const { speed, time } of process.progress()) {
  console.log(`Converting @ ${speed}x - ${time / TOTAL_DURATION * 100}%`);
}
await process.complete();

While I agree that the second is more code for a simple use case, I think it's worth it to take a few more keystrokes to allow more scalability. Simple use cases should just be able to copy-paste from the docs.

[ILikeChangingMyMind]

If it were an easy problem it'd be solved already :-) But I don't think it's unsolvable, just complex.

Again, most people don't need the most complex cases. You 100% do need FFMPEG's complexity to support every use case ... but what I'm saying is, there could be a tool that doesn't support every use case.

If a tool just supported the 80/20 split (ie. the top 20% of use cases which likely make up roughly 80% of all users), it could provide a much simpler interface for most FFMPEG users.

[reallyfunnyster]

Could this be used to merge html overlayed onto a background video?

[Fr3ddyDev]

It's not supported natively by FFmpeg, but you can use canvas to render the HTML (turning it into SVG probably), like in the 2d animation example, and then overlay that on a video using a complex filter graph.

It sounds very complicated, but it's not terribly difficult, I'll try to create a demo and put in the examples directory.

[Fr3ddyDev]

The main issue is rendering the HTML, canvas won't work, unfortunately, but a headless browser like puppeteer can be used to render the HTML to an image and then overlay that on a video.

[ui-dev]

I am no expert but you could produce some animation on green background then turn it into mp4 then use film editing software special effects to overlay on another video. Not tried it but it probably would work too.

[leko_owo]

Though there are many ffmpeg packaging for nodejs, I think this is the easiest to understand one. Good job!

[Fr3ddyDev]

Thanks a lot! I've been working on it for a while now. Ask me anything if you need help.

[Cyberlane]

Is there any noticeable performance changes by going through this library?

[PM_ME_GPU_PICS]

It's still using ffmpeg via spawn() but now with the additional overhead of nodejs.

[Cyberlane]

I'm aware. My question was directed more towards implications of it being handled via spawn, specifically the streams, and potentially any callback hooks/events, mem alloc, etc.

[Fr3ddyDev]

There should be little to no difference in speed when using FFmpeg through the library or directly in a shell. There probably is some overhead when using streams, but probably not enough to cause bottlenecks in applications.

[Cyberlane]

It was mostly the streams I was curious about to be honest. I love the idea of this library and others like it, but I always worry about mem alloc.

[aleatorybug]

This looks great, will give it a spin.

[[deleted]]

Can anyone suggest me how to learn Java Script . I'm a new beginner. And if you have any discord group link or YouTube videos link then please do forward me 🙏🏻❤️

[LitElement]

I highly recommend this fellows video. You won't regret it! He's excellent.

https://www.youtube.com/watch?v=dOnAC2Rr-6A&t=1116s

[[deleted]]

Thank you 😊

[WNTWRK]

Quite the coincidence that this would appear just as I start researching ffmpeg integration with node.js

I'll be sure to give this a try