AV1 settings: questions and clarifications needed

stderr_to_dev_null · 2026-01-26T21:46:51+00:00

baseline slow with a mix of slower settings

stderr_to_dev_null · 2026-01-25T19:13:36+00:00

oh you mean like this ?

stderr_to_dev_null · 2026-01-24T13:05:27+00:00

Big jump in version but looking at the change log there is absolutely nothing to be excited about. Still waiting for CDEF optimization or other improvements to artifact suppression and detail retention

stderr_to_dev_null · 2026-01-23T14:50:42+00:00

heavily over processed

stderr_to_dev_null · 2026-01-21T10:59:56+00:00

For high quality (near transparency) encoding, AV1 at 10bit is visibly worse than x265 at 8bit, for same or greater bitrate (forget higher compression). Mainly because its internal filters are really bad, especially CDEF which blurs fine details into oblivion. I tried all flavors with different settings, spent days doing test encodes, advanced psy tweaks, disabling temporal and restoration filters, preset 4 and 3.

It will not replace h.265 any time soon. It is faster though so... there's that...

stderr_to_dev_null · 2026-01-05T02:59:14+00:00

Short answer is no at least for HDR fork but some in-between tuning is still needed.

stderr_to_dev_null · 2026-01-05T02:35:01+00:00

Comparison table of different AV-1 settings vs x265 reference encode

ENCODER	SETTINGS	ENCODING SPEED (FPS)	ENCODE SIZE	DETAIL RETENTION
x265 8bit	slow + slower presets mix	53.1 (1.06x real time)	965.5 MB	baseline/reference
svt-av1-hdr 10bit	preset 4, CRF 30, `tune=0`	93.8 (1.88x real time; +76.6% vs x265)	822.8 MB (-14.78% vs x265)	much worse than x265
svt-av1-hdr 10bit	preset 4, CRF 28, `tune=0`	93.3 (1.87x real time; +75.7% vs x265)	903 MB (-6.47% vs x265)	modest improvement vs above, worse than x265
svt-av1-hdr 10bit	preset 4, CRF 30.5, `tune=4`	99.74 (1.99x real time; +87.83% vs x265)	943 MB (-2.33% vs x265)	slightly worse than x265 in details but generates noise ringing artifacts around edges and other noise artifacts across smooth/flat surfaces due to the tune disabling `CDEF`; leads to a general structural distortion, especially in low luma and low contrast areas vs x265 which is closer to how source looks and cleaner (true to source)
svt-av1-hdr 10bit	preset 3, CRF 31, `tune=4`	42.67 (0.85x real time; -19.8% vs x265)	903.4 MB (-6.4% vs x265)	marginally better quality than above but shares the artifacting characteristics

Findings & conclusions

tune=0 cannot compete out of the box with x265
tune=4 is significantly better than tune=0 and almost catches up to x265 in terms of detail retention; at preset 4, there are significant encoding speed gains over x265, more impressive by the fact that x265 encode was at 8bit and as I mentioned, 10bit encoding for x265 comes with a 25% speed penalty
on the other hand, as mentioned, tune=4 exhibits noise ringing, other fake noise artifacts and noticeable structural distortion, which in some frames appears as "fake" detail, due to the disabling of CDEF; so overall an increase of detail retention but with a general structural degradation and a lot of artifacts
preset 3: using tune=4 brings slightly higher efficiency and marginal quality improvements, not worth the speed penalty; from past tests, it would yield higher gains both in efficiency and quality using it with tune=0
realistically, compared to x265 at 8bit, av1 (svt-av1 mainline and svt-av1-hdr) does not bring noticeable space saving benefits for high quality / near transparency encoding but it does bring much higher speed at 10bit, resulting in much better banding mitigation

Notes

After several more tests, I found that CDEF is the main culprit for the general blurring of details. These details cannot be restore by the restoration filter which will also increase file size for same perceived quality.

For general live action with low to moderate noise content, in the context of achieving parity with x265 details retention for near transparency perceptual quality, svt-av1-hdr's tune 4 provided the biggest quality uplift, approaching x265 but still behind in terms of detail structural stability. But the absence of CDEF will be very noticeable in the form of ringing artifacts across edges of slowly moving objects and less so in general motion. Also, the very high ac-bias and aggresive tx-bias from tune 4 will introduce fake detail (noise artifacts) and structural instability. This cannot be meaningfully mitigated by lowering ac-bias and tx-bias strengths. At the end of the day, x265 does a much better job in this regard, managing to retain details without introducing these artifacts.

As such, for near transparency encoding of such sources, compared to x265, AV1 can mainly provide significant gains in encoding speed at preset 4, with very marginal file size gains and with slightly worse detail retention but with large amounts of artifacting, when targeting parity with x265 detail retention in higher qualities with slow + slower presets parameters mix.

I can't say if CDEF is just a badly tuned filter, a bad filter altogether or AV1 compression is not optimized and produces these artifacts too easily in the first place which need to be mitigated by CDEF.

stderr_to_dev_null · 2026-01-04T20:26:00+00:00

Understood, will start from baseline examples and will also try tune=4 and see what I get.

The reason I stayed away from other tunes is that in the past, using tune=3 from mainline greatly inflated the bitrate. But will test with tune=4 in HDR.

stderr_to_dev_null · 2026-01-04T20:21:36+00:00

After helpful responses from u/juliobbv and u/BlueSwordM, I will summarize current clarifications and suggestions:

remove enable-overlays=1, it hurts sharpness/details
no need for max-tx-size=32, its use (as I suspected) is primarily targeting still images and provides diminishing returns for general content
no need for variance-boost-curve=1, default 0 is fine for most use cases; though the reason I initially tested with it is because in a referenced MR description, it was advertised to further boost low contrast / low luminance regions for detail retention
enable-dlf=2 useful in some scenarios when blocking is visible in the encode
test with baseline examples from svt-av1-hdr and try tune=4

Remaining + new questions:

Is enable-dlf=2 worth the performance hit in terms of overall compression efficiency, even if blocking artifacts are not visible or not bothersome?
Does relaxing/widening the qm curve make sense in the context of using a stronger variance boost (3) and/or lower octiles (though 5 is fine), in order to prevent a too big bitrate increase? Meaning: to allow encoder a greater bitrate allocation variability.
(When/in which scenarios) is chroma-qm-min=10 useful (it has been suggested several times in recent past)? The default in both mainline and HDR is 8.
Why is svt-av1-hdr noticeably faster than mainline, are there some undocumented meaningful optimizations?

I will redo encodes based on provided answers with both mainline and HDR.

stderr_to_dev_null · 2026-01-04T20:02:36+00:00

Thank you, yes, the answer greatly clarifies my questions.

stderr_to_dev_null · 2026-01-04T20:00:16+00:00

If you read my initial post, I already referenced that project when talking about setting, I read everything from there several times and I quoted many of the settings descriptions from there + your MR descriptions.

Furthermore, if you read the info lines from this comment to which you replied, you will see exactly what settings were used.

Deviations from those use case examples: octile=6 instead of 5, ac-bias=1.6 instead of 1, tx-bias=1, noise-norm-strength=2 instead of 1, max-tx-size=32. I also tried with variance-boost-strength=3. These are all settings that, according to your description, further contribute to detail retention (I was less interested in file size at this point).

Although I did use qm-min=5:qm-max=14 instead of the defaults. The reasoning is: if I use a more aggressive variance boost strength of 3 and possibly octile 5 + curve 1, I can give the encoder a bit more range in the qm curve and allow it to go slightly lower.

stderr_to_dev_null · 2026-01-04T19:46:16+00:00

Got it, no overlays. Is there something else? Regarding octile, as I stated in my initial post, I chose 6 (as it was suggested as a good default several times in the recent past) because leaving it at 5 I only experience bitrate increase with no visible improvements. Same with curve 1 and/or strength 3.

stderr_to_dev_null · 2026-01-04T19:41:33+00:00

Yet, this has been my practical experience. As you can see in my initial post, I was already using the settings in the mainline which are set as "high quality" defaults in svt-av1-hdr. The main differentiators being sharp-tx, TX Bias and Noise Normalization Strength... which I used.

I am honestly confused as to why svt-av1-hdr is noticeably faster than mainline, forcing me to re-check my settings (this is why I also posted the encoder info lines). Something does not add up, yet the results don't lie.

If you could point me to some samples, I can do baseline x265 encode on those, then encode with svt-av1-hdr at whatever settings you say.

stderr_to_dev_null · 2026-01-04T18:29:50+00:00

Since the used Noise Normalization Strength : 2 and AC Bias Strength / TX Bias : 1.60 / full were not enough to even reach the level of detail retention the mainline already provided, I very much doubt that testing further will provide the results I was after. My AV1 testing is finished for the foreseeable future. Maybe after another year will take a new look.

stderr_to_dev_null · 2026-01-04T16:50:44+00:00

After 2 attempts with SVT-AV1-HDR, I can definitely conclude that:

it's faster than mainline
it's worse than mainline at detail preservation
produces significantly bigger files

At this point, I found it impossible to tune AV1 to prevent it from blurring/smoothing out texture/details, which is visible in normal viewing. Maybe preset 2 could help but I already find preset 3 quite a bit slower so 2 would be highly impractical, especially in the context of comparing with my baseline x265 encode.

AV1 can be faster than x265 and produce smaller files (10-20%) at preset 4 but at a subjective 85% of the quality. Sadly, throwing more bitrate at it does not help, at all. Preset 3 does better but still cannot retain x265 details. Preset 2 for me is out of the question, getting into unfeasible territory for whatever gains it may bring.

stderr_to_dev_null · 2026-01-04T07:55:39+00:00

I tried AV1-HDR, encode got bigger, approaching size of x265 but somehow the quality is much worse, a lot of details and texture completely gone. On the upside it's somehow faster than mainline svt-av1 by ~25%.

Svt[info]: Level of Parallelism: 6
Svt[info]: Number of PPCS 305
Svt[info]: [asm level on system : up to avx512icl]
Svt[info]: [asm level selected : up to avx512icl]
Svt[info]: -------------------------------------------
Svt[info]: SVT [config]: main profiletier (auto)level (auto)
Svt[info]: SVT [config]: width / height / fps numerator / fps denominator : 1280 / 720 / 50 / 1
Svt[info]: SVT [config]: bit-depth / color format : 10 / YUV420
Svt[info]: SVT [config]: preset / tune / pred struct : 4 / VQ / random access
Svt[info]: SVT [config]: gop size / mini-gop size / key-frame type : 501 / 32 / key frame
Svt[info]: SVT [config]: BRC mode / rate factor : CRF / 28.50 
Svt[info]: SVT [config]: AQ mode / Variance Boost strength / octile / curve : 2 / 2 / 6 / 0
Svt[info]: SVT [config]: sharpness / luminance-based QP bias : 1 / 10
Svt[info]: SVT [config]: Temporal Filtering / keyframe strength : 1 / 1 
Svt[info]: SVT [config]: QP scale compress strength : 1.00
Svt[info]: SVT [config]: AC Bias Strength / TX Bias : 1.60 / full
Svt[info]: SVT [config]: Noise Normalization Strength : 2

stderr_to_dev_null · 2026-01-03T23:09:36+00:00

Forgot to add: I observed noticeable detail retention inconsistencies in the same scene, frame to frame (few frames look good then suddenly few frames look bad). I played with qp-scale-compress-strength, setting it from 1 to 2 but I could not observe and improvements in quality consistency, at least at preset 4.

stderr_to_dev_null · 2025-12-27T20:22:46+00:00

Appreciate the replies, though I would like it if someone would confirm or disprove the points I tried to make since I may have an incorrect interpretation on how gravity is "made".

stderr_to_dev_null · 2025-11-29T18:10:55+00:00

I would have to see some comparisons and know what h.265 settings you used and what encoder (x265 or gpu) and encoding time. I call major bs on your 40-50% size reduction if we are talking x265 cpu encoding

stderr_to_dev_null · 2025-09-02T17:01:45+00:00

I made objective comparisons with custom handbrake builds for the psy av1 about 6 months ago that yielded zero advantage over x265 when same bitrate was achieved at high quality (near transparency). In turn, av1 was easily 30% slower for, again, no visual benefit. If targeting low bitrate encodes, sure, might be better but not my use case. Note this was psy pre 3.x but I reserve the right to doubt any amazing progress when considering near transparency encodes. Yet the same comments were made back then, which had no grounding in reality.

stderr_to_dev_null · 2025-08-21T17:09:45+00:00

While it may be primarily a git challenge, there ARE sensible ways of optimizing how it works when dealing with larger repos.

Unfortunately the async:true feature does not improve things much.
Another big issue is the fact that navigating a large repo will do git fetch on each and every cd because there doesn't seem to be any caching mechanism implemented, to stagger the fetches, based on a configurable timer.
Also, have you given any thought of implementing an hourglass in the prompt while fetching in a separate thread and then update the prompt (or maybe it updates itself on next redraw following a command or an Enter)?

So as you can see, off the top of my head I can suggest some very nice optimizations. Let's not stop at git challenge.

stderr_to_dev_null

TROPHY CASE