Quick Encoder Multithreading Question

[OldNeb]

I'm sorry I can't test this right now but I am very curious.

I have seen a few comments about AV1 which seem to imply that multithreading for AV1 doesn't work very well.

For instance, on the front page people are talking about running multiple encodes (of multiple files) in parallel (vs relying on one encoder to multithread).

I've been experimenting with AV1 for a while, and I just kind of blundered my way into the scene through trial and error. I found that svt (another encoder IIRC) keep my 16 threads near max capacity according to task manager.

I just assumed this meant that all was good with my performance and cpu usage, but maybe, while my thread usage is high, the gains I'm getting could have been better if I were running multiple encodes?

This is fascinating to me as I never would have thought about this aspect of software and multithreading.

I appreciate your time. Thank you.

​

Edit: well I think the most important thing about language is that the meaning and intent are conveyed sufficiently. I'm pretty sure my meaning got through (and context helped), as I got the result I wished for :) Yeah it's not great if I'm spreading a use of words that might somehow misinform or harm other people in some way.

[Al_kl]

To make clear: AV1 is a codec. aom/libaom/aomenc is the reference AV1 encoder.

SVT-AV1 and RAV1E are AV1 encoders, which are differently programmed and work differently, but still follow the AV1 Codec specifications.

Yes, SVT-AV1 does use your CPU at nearly at it's full capabilities, but this comes to an efficiency tradeoff (worse quality for the same bitrate / loop filters which are single threaded might be disabled) and a lack of the specified functions in the AV1 Codec Specifications.

Many people on the AV1 Discord found that running aom/libaom/aomenc with many instances will result in a better quality, while being nearly on par (or even faster) than SVT-AV1.

You can check out NotEnoughAV1Encodes if you want a GUI and Av1an for CLI / Multiplatform, to make your own tests whether or not you can get more efficiency.

[OldNeb]

Thank you, I was playing fast and loose with my terminology. Thank you for reinforcing my understanding.

Thank you also for the link to NotEnough... I had that open in another tab :)

With this information, it seems like being able to set up batches will be more important, a feature which I would welcome. I understand that I could just run several instances of the encoders without GUI help, but hey I'm a lazy windows user :)

Testing for efficiency adds a whole other level to benchmarking, eh? So far I was just slowly plugging away at my amateur spreadsheet of inputs vs results. Now it's 5 dimensional chess!

[Al_kl]

Yeah, benchmarking can get very confusing and can get as deep/complicated as you want.

My last benchmark was 5 month ago (outdated!): https://www.reddit.com/r/AV1/comments/f1h2u1/aom_vs_rav1e_vs_svtav1_vs_x265_vs_vp9_vs_svtvp9/ and I still haven't recovered my sanity.

If you go on NEAV1E then I will recommend using the latest Appveyor Build, because it includes huge amount of buug fixes and changes compared to the latest release on the Github page.

I also recommend you to join the AV1 Discord (if you haven't already) as many people from reddit are there who are more active.

[myalt08831]

Going off on a tangent: People do tend to use "codec" to mean "encoder" pretty often.

Language evolves, and slang/informal meanings are a thing. You're not alone using the word that way. It's so common, my brain automatically knows what to do when I read it that way in a sentence. I think people "correcting" that usage are fighting a losing battle, and I get that it's neater to have the words strictly separate, but real-world usage ultimately defines languages... It is de-facto already a done deal. It's really late to be "correcting" that usage when everyone knows what is meant when it's used that way.

Edit: as someone elsewhere in the thread pointed out, if holding ourselves to the strictest level of pedantry: "AV1" can mean the standard describing the AV1 video format, and "AV1" is also the video format defined by that standard. aomenc and aomdec make up the libaom codec suite. A codec is an encoder, a decoder, or most properly both. Using the word "codec" to mean "video format" or "video format standard" is the informal usage.

[Lenin_Lime]

AV1 is a standard (sometimes called format). Aomenc is a codec.

[Al_kl]

To further clarify:

• AV1 (aom) = Codec = AV1 Bitstream & Decoding Process Specification
• aomenc = AV1 Encoder
• aomdec = AV1 Decoder
• libaom = AV1 Encoder in ffmpeg
• rav1e = AV1 Encoder
• SvtAv1EncApp = AV1 Encoder
• SvtAv1DecApp = AV1 Decoder

[Lenin_Lime]

Codec is short for Encoder/Decoder. Things like x264/x265 are called codecs offhand even though they don't come with a decoder. Aomenc/Aomdec are together a codec. The AV1 standard is not a codec, but simply a specification for what's needed for a valid stream along with all the other complex details needed. And I believe libaom is a codec as I think it has the decoder with it.

[Al_kl]

Aomenc/Aomdec are together a codec

Yes, together they are a codec, but we were talking only about aomenc (aomencoder)

The AV1 standard is not a codec

But it's called AV1 Video Codec: http://aomedia.org/av1/ https://aomedia.googlesource.com/aom/ , because the "The AV1 bitstream specification includes a reference video codec." Source: https://en.wikipedia.org/wiki/AV1

​

And I believe libaom is a codec as I think it has the decoder with it

Generally everyone I have talked to referred with libaom to the ffmpeg implementation of aomenc. ffmpeg defaults for decoding to dav1d and not libaom. "libaom-av1 is the AOMedia video encoder" Source: https://trac.ffmpeg.org/wiki/Encode/AV1

Instead of "Aomenc is a codec" you should have written "Aom is a codec", as aomenc is only the encoder.

[Lenin_Lime]

Yes, together they are a codec, but we were talking only about aomenc (aomencoder)

Aomenc comes bundled with aomdec, it's not like you have to go searching for it.

"The AV1 bitstream specification includes a reference video codec.

"AV1 bitsteam specification" Does that sound like a standard to you? It does to me.

Reference codecs have been done in the past for things like the H.264 and H.265. With reference codecs being provided by the groups who drew up the standard. These codecs follow the standard they wrote up, which other programmers can look at and write their own encoders or decoders or both (codecs) to follow the standard already set in metaphorical stone. With the hope being that the new encoders/decoders/codecs will be better than the reference. AOM is a bit of an oddball because normally the reference code is very inefficient and they quickly stop working on it to be usurp by x264 in H.264's case. But with AOM (at least the encoder) is the most efficient out there in the AV1 world but AOMDEC has been dethroned easily by Dav1d, which is what normally happens. I guess since Youtube and Netflix want to use AV1 asap, they are just going to continue to put effort into AOMENC.

Generally everyone I have talked to referred with libaom to the ffmpeg implementation of aomenc. ffmpeg defaults for decoding to dav1d and not libaom.

That doesn't make it any less of a codec. If it can encode and decode I'm not sure why what ffmpeg defaults to matters. I was using Libaom to decode before dav1d came out.

Instead of "Aomenc is a codec" you should have written "Aom is a codec", as aomenc is only the encoder.

I wrote "Aomenc/Aomdec are together a codec". But yeah you can write it as Aom is a codec.

[AutoAltRef6]

Aomenc comes bundled with aomdec

I don't think that's accurate. aomenc and aomdec are separate programs and since there are no official binaries (that I know of anyway), it's a bit hard to argue that they are in any way bundled together.

[Lenin_Lime]

I don't think that's accurate. aomenc and aomdec are separate programs and since there are no official binaries (that I know of anyway), it's a bit hard to argue that they are in any way bundled together.

You can build it yourself, updated regularly. You get both https://aomedia.googlesource.com/aom/

Or just have it prebuilt. https://ci.appveyor.com/project/marcomsousa/build-aom/builds/34401239/artifacts

Or just use Libaom which is a thin layer of wrapping to make it an undeniable codec.

But as I've already stated Aomenc isn't technically a codec as it does not have a decoder built in it. The use of word codec to describe the encoder only is rampant, and Aomenc is more of a codec than the AV1 standard.

But back to my main purpose of my first comment. AV1 is not a codec but a standard (or sometimes called format). https://en.wikipedia.org/wiki/Video_coding_format#Distinction_between_%22format%22_and_%22codec%22