I've reached a point where I think most of the work is solid, but I'd love some second opinions:

• The whole process is implemented as a separate step with an independent ParallelManager
• Optimization is disabled by default, with CLI interfaces closely resembles the relevant parameters in KTX2
• The diff function for trial blocks are routed to existing specialized ones for different planes & partitions, with a new one added for the constant symbolic block case
• HDR, swizzle, different weights should be working but I haven't looked much into it yet
• By default the extra texels in edge blocks are just value-clamped at image borders and treated as normal blocks, which may unnecessarily increase the errors since trial blocks may differ in those 'non-existing' texels. But for simplicity I kept this behavior, otherwise will have to change all 3 compute_symbolic_block_difference functions just to handle this case, adding more involved masking operations (see compute_block_mse).

Have you looked at image quality to bitrate trade offs, and how this compares to just using a larger ASTC block size to reduce bitrate?

Have you looked at image quality to bitrate trade offs, and how this compares to just using a larger ASTC block size to reduce bitrate?

I'm not sure I understand the question. RDO is all about findign a good balance between quality (distortion) and (bit)rate. The intent is to reduce entropy between blocks to reduce the size after Deflate-style lossless compression. I wonder if there is a point at which a larger block size produces a smaller result than RDO+Deflate. It would be good to know. But keep in mind that the purpose of RDO+Deflate is to reduce file and transmission size. It does not affect the GPU memory used.

the purpose of RDO+Deflate is to reduce file and transmission size.

Yes, but as you noted ASTC can do the same thing natively (just use a larger block size) which gives file, transmission, AND memory savings. There isn't much point adding this if it ends up the wrong side of the pareto frontier vs just using a lower ASTC bitrate in the compression.

Ignoring that, I'd still like the PR to include some analysis of the new feature and quality/bitrate which can be added to the documentation so we can explain when it should be used, and how to use it effectively.

pareto frontier

What is this?

I'd still like the PR to include some analysis of the new feature and quality/bitrate which can be added to the documentation so we can explain when it should be used, and how to use it effectively.

Absolutely. From that we can determine if it is useful. We need to compare the zstd deflated file with and without RDO to determine its effectiveness.

pareto frontier
What is this?

Scatter plot whatever pair of metrics you care about for a compressor (quality-performance for lossy compressors, quality-size for RDO, size-performance for lossless compressors). The pareto frontier is the line made through the best compressor/configuration for a given quality or performance setting.

For a new optimisation to be useful it needs to be above the frontier - e.g. giving better performance for the same quality, or visa versa. If you're behind the frontier, then there isn't any point - better options already exist.

Okay finally had some time to get to this... Although I wasn't able to make a full performance report, some simple test can still provide useful insights:

For a 2048x2048 test image, the following script is run:

./astcenc-avx2.exe -tl test.jpg 6x6.png 6x6 -thorough -dimage
./astcenc-avx2.exe -tl test.jpg 6x6r.png 6x6 -thorough -dimage -rdo
7z a 6x6.7z 6x6.astc
7z a 6x6r.7z 6x6r.astc

./astcenc-avx2.exe -tl test.jpg 8x6.png 8x6 -thorough -dimage
./astcenc-avx2.exe -tl test.jpg 8x6r.png 8x6 -thorough -dimage -rdo
7z a 8x6.7z 8x6.astc
7z a 8x6r.7z 8x6r.astc

./astcenc-avx2.exe -tl test.jpg 8x8.png 8x8 -thorough -dimage
./astcenc-avx2.exe -tl test.jpg 8x8r.png 8x8 -thorough -dimage -rdo
7z a 8x8.7z 8x8.astc
7z a 8x8r.7z 8x8r.astc

./astcenc-avx2.exe -tl test.jpg 10x10.png 10x10 -thorough -dimage
./astcenc-avx2.exe -tl test.jpg 10x10r.png 10x10 -thorough -dimage -rdo
7z a 10x10.7z 10x10.astc
7z a 10x10r.7z 10x10r.astc

./astcenc-avx2.exe -tl test.jpg 12x12.png 12x12 -thorough -dimage
./astcenc-avx2.exe -tl test.jpg 12x12r.png 12x12 -thorough -dimage -rdo
7z a 12x12.7z 12x12.astc
7z a 12x12r.7z 12x12r.astc

The key results are as follows:

So I guess it may not be some ground-breaking optimization:

• Like @solidpixel questioned, 8x6 seems like a better option than 6x6-rdo considering RDO optimizes on transmission size only and is usually a time-consuming process.

But in many cases it still may provide interesting trade offs:

• RDO fills the gap between neighboring block sizes (e.g. 8x6-rdo metrics sits comfortably right between 8x6 & 8x8 and adjusting compression quality per-se wouldn't make this much of a difference)
• We can still push some boundaries on top of 12x12
• In our own use cases, comparing outputs with near-identical PSNRs (e.g. 8x6 vs. 6x6-rdo), we tend to feel the RDO ones almost always yield better results (because of the smaller block size, perhaps), but of course this is a subjective matter.

Super, thanks. I'll try and take a look next week.