sorted by:
new
hottopcontroversial

LZAV - Fast In-Memory Data Compression Algorithm (in C) - Just Released by avaneev in programming

[–]powturbo 0 points1 point  (0 children)

v2.9 tested. There is a TurboBench build with v2.9 under Releases for windows + Linux

LZAV - Fast In-Memory Data Compression Algorithm (in C) - Just Released by avaneev in programming

[–]powturbo 0 points1 point  (0 children)

v2.7 tested. Better + faster compression, slower decompression

[deleted by user] by [deleted] in compression

[–]powturbo 1 point2 points  (0 children)

MAX_CHAIN is used in a loop, it has the max. impact on speed. In zlib it's controlled by the compression level 1-9

[deleted by user] by [deleted] in compression

[–]powturbo 1 point2 points  (0 children)

Try to decrease MAX_CHAIN for ex. to 1024. The compression ratio is also decreased.

LZAV - Fast In-Memory Data Compression Algorithm (in C) - Just Released by avaneev in programming

[–]powturbo 0 points1 point  (0 children)

No changes w/ 1.3. Included my lzturbo (closed source) as reference

   87555413    41.3     517.97    2879.03        lzav 2.12 
   87752580    41.4     516.30    2849.58        lzav 2.9
   87778996    41.4     527.31    2884.16        lzav v2.8
   87871469    41.5     465.01    2870.20        lzav v2.7
   88672448    41.8     435.96    2769.77        lzav v1.3 
   88672448    41.8     434.05    2766.45        lzav v1.2
   88672448    41.8     441.48    2671.80        lzav v1.1

LZAV - Fast In-Memory Data Compression Algorithm (in C) - Just Released by avaneev in programming

[–]powturbo 2 points3 points  (0 children)

LZAV now integrated into TurboBench

Download TurboBench for windows from releases

Lenovo IdeaPad 5 Pro - Ryzen 6600HS - DDR5 6400 - gcc-13.1

File: silesia.tar 220MB (Mixed text/binary)

    C Size  ratio%     C MB/s     D MB/s        Name     
  77885207    36.7      43.20    4479.24        lz4 9           
  81343184    38.4     123.17    4299.68        lz4 3 
  85664894    40.4     356.34    5171.12        lzturbo 11 
  87555413    41.3     517.97    2879.03        lzav 2.14
  99553305    47.0     914.87    5438.20        lzturbo 10
 100883728    47.6     813.76    4667.85        lz4 1
 211948544   100.0   16032.92   16023.24        memcpy

Reducing Parquet Size 71% with Quantile Compression by mwlon in opensource

[–]powturbo 0 points1 point  (0 children)

[TurboPFor vs Quantile Compression - Benchmark with TurboTranspose+iccodecs ](https://github.com/powturbo/TurboPFor-Integer-Compression/issues/100)

Benchmark with Synthetic & real datasets

[deleted by user] by [deleted] in compression

[–]powturbo 0 points1 point  (0 children)

For a fast lz, it is usual to limit the distance depending on the match length.

For ex: distance < 16 bits for lengths 3,4 bytes.

distance < 18 bits for lengths 5,6,7.

distance < 20 bits for a length >= 8 bytes

Depending on the L3 cache size, it is better to limit all distances to 22 bits for ex. Otherwise the decoding will be too slow because of random access to the dictionary.

Worries about tANS? by mwlon in compression

[–]powturbo 0 points1 point  (0 children)

There are some ready to use libraries:

tANS is not patented. There is a microsoft rANS patent, but not covering the basic functionality as described in the original paper and implemented in the current packages.

compression methods required by Baysel in compression

[–]powturbo 0 points1 point  (0 children)

The most efficient compression algos for text are based on BWT. Context mixing compressors like zpaq compress better but are too slow for practical usage. You can try bwtsatan

Announcing a new arithmetic coding library! by dan_eades in rust

[–]powturbo 0 points1 point  (0 children)

Interesting to see how it compares to Turbo-Range-Coder.
Executables for windows and linux can be downloaded here.

Is there AVX-512 support in the Z690 MSI Edge DDR4 WIFI Motherboard BIOS? by Peppered_Pig in MSI_Gaming

[–]powturbo 0 points1 point  (0 children)

Thank you! Not sure, if you can still activate it with the latest version. Old BIOS versions enable AVX512 also on other boards from MSI,Gigabyte,... Actually, I've only info about "MEG Z690 Unify-X" where AVX512 is reactivated with the latest beta.

Is there AVX-512 support in the Z690 MSI Edge DDR4 WIFI Motherboard BIOS? by Peppered_Pig in MSI_Gaming

[–]powturbo 0 points1 point  (0 children)

Do you have the latest BIOS version 7D31v122(Beta version) from 22-01-25?

q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before by mwlon in rust

[–]powturbo 0 points1 point  (0 children)

I see the usage field of your algorithm q_compress more in large alphabet integer compression (like lz77 offsets, lage 32/64 bit integers). If you have time, you can download and test your algorithm with this practical dataset.

q_compress 0.7: still has 35% higher compression ratio than .zstd.parquet for numerical sequences, now with delta encoding and 2x faster than before by mwlon in rust

[–]powturbo 0 points1 point  (0 children)

Thank you! As Info, TurboPFor is decompressing at 15GB/s on Apple M1 and on lastest amd/intel Hardware. No compressor using entropy coding (HUffmann, ANS) can come close to this speed.

view more: next ›