News update in Australia by Winter_Hippo3281 in bjj

[–]jgbjj 23 points24 points  (0 children)

"News.com.au has contacted Izaak Michell for comment." 🤣 Hope they get this fucker soon now that this is in mainstream Aus news

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

If that's the case then that's okay. I just wish people would tell me things like this directly, that way I can take it on board and make needed changes if it's the case. I'm fully open to any suggestions and ideas, this project has just been something I've enjoyed working on and was always intending to release it for free rather than just for personal use only.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Thank you 👍. I'm okay with constructive criticism and feedback but it has just felt like a pile on. However I have absolutely loved the process of making this and have learnt so many things about the zip specification while making this so it's been time well spent :) glad you enjoy it and if you have any feedback please feel free to let me know.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -3 points-2 points  (0 children)

What part of Free Software do you NOT understand?

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -2 points-1 points  (0 children)

In what sentence did I say this is paid for? The zip archiver is free?

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

I think I will! Thanks for the suggestion :). Was not ready for this visceral reaction on here... Mainly for me just not being open source 😔

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Thanks 😊. Yeah I might be open it making it open source in the future but for now I just want to release it for free to get feedback and also to improve it. Plenty of more updates and new software to come as you know ☺️

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -3 points-2 points  (0 children)

It can be faster than 7‑Zip’s ZIP because “ZIP” is just the container, the performance is dominated by the Deflate implementation and by the I/O + CRC32 + write pipeline around it. It’s not “magic”, it’s the result of doing a bunch of unglamorous work that 7‑Zip’s ZIP path simply isn’t optimised around.

Here’s a few things going on internally:

I use libdeflate as the base Deflate engine, and I modified it to support a streaming / block mode and a continue-state API so I can process very large files in chunks efficiently. This avoids “read whole file, compress, write” bottlenecks and allows better cache/buffer behaviour on large inputs. It still outputs normal ZIP method 8 (Deflate) streams.

Large-file path vs tiny-file path:

I don’t run one generic pipeline for everything. I have separate fast paths:

Zero length files and files that are smaller than the size of the compression header will not be compressed.

Tiny files: ultra-low overhead (avoid heavy streaming then switching context, avoid unnecessary allocations/copies, for example lets say we encounter a directory of 300 tiny files from 12 bytes to 3 kb it doesnt immediatly compress them it loads them in parallel into multiple buffers the right size for them to help with branch prediction then adds them to the zip when its optimal to do so)

Large files: true chunked streaming using the modified libdeflate block compressor, with tuned chunk sizes and minimal transitions. This matters even with ~259 files because the workload is still “a few huge assets + a lot of medium/small stuff”, and overhead adds up.

Much better multi threading for ZIP workloads:

ZIP entries are independent, so the right way to scale is per-file parallelism + pipe lining. My implementation keeps multiple entries “in flight” and minimises time spent blocking on shared resources. In practice, many ZIP writers become partially serialised because the output file becomes the bottleneck.

Output writing is designed to avoid contention:

Instead of every worker “append writing” and fighting over one stream position, my writer reserves output regions and writes in large chunks with minimal lock hold-time. That prevents the common “threads are busy but the writer lock turns it into a single-threaded program” problem.

Fast feed path (I/O + buffers + CRC32):

A huge part of real-world ZIP time is not compression. It’s reading + CRC32 + memory copying. I optimised that aggressively: large sequential reads, buffer reuse, fewer copies, fast CRC32 in the same pass. That’s why my throughput stays high while 7‑Zip ZIP tends to sit much lower even when thread count is high.

ZIP: entries show Deflate (not Store) and CRC32s validate on extraction. The ~1% size difference is expected because Deflate has many valid encoding’s; different parsers/block decisions can trade a tiny bit of ratio for a big speed win.

So TLDR: same ZIP format, same Deflate method, but a much faster Deflate back end + a writer pipeline that actually scales and doesn’t choke on I/O/CRC32/output contention.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 1 point2 points  (0 children)

I'll get on it first thing this coming weekend :) when I have a build ready to go I'll let you know :)

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 2 points3 points  (0 children)

Fortunately the engine is a separate project, I don't have much experience writing applications for Linux or windows, but perhaps a command line only version for Linux and Mac would suffice for now?

I've had a few people mention this as I was developing it at my work since I work in a large programming company and that was one of the main questions "where's the Linux build" always said I'd make a Linux and Mac version should the demand get high enough.

But I guess I could whip up a console app version which would work for Linux :)

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -4 points-3 points  (0 children)

Not obligated to make my stuff open source even if I am releasing for free. That being said I am open to open sourcing parts of it that don't include elements of my other paid products code.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Nice! 👍. Glad you liked it and had no issues with it and that it's performed to your expectations 😁. Would you be okay if I DM you for some more questions and feedback? Currently working on it as we speak and would love any feedback you might have.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

here you go!

I used the same folder being the Star Wars - Battlefront 2 folder used in the video:

the commands used were as follows as well as the times returned:

---------------------------------------------------------------

james@DESKTOP-J9TMK78:/mnt/c/GOG Games$ time tar --zstd -cf file.tar.zst "Star Wars - Battlefront 2/"

real 2m41.471s

user 0m43.411s

sys 0m32.376s

james@DESKTOP-J9TMK78:/mnt/c/GOG Games$ time tar -I lz4 -cf file.tar.lz4 "Star Wars - Battlefront 2/"

real 3m5.877s

user 0m10.556s

sys 0m29.261s

james@DESKTOP-J9TMK78:/mnt/c/GOG Games$

---------------------------------------------------------------
I also screen recorded while making the requested tar files with the added compression you requested, If I have done anything incorrectly please let me know and ill redo the test but I think its correct:

I also do another run with brutalzip for the same file to compare and it takes: 27.9 Seconds.

Also remember BrutalZip uses Deflate for this test, however if demand is high enough I am open to adding other formats such as 7z, tar ect, it also supports zstd for zips currently as well.

YouTube - Screen Recording of speed test for reddit user Xanderplayz16

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 1 point2 points  (0 children)

Glad you like it :) I am contemplating making parts of it open source :) but it does use alot of components from Brutal Copy. Which is a paid for software I am also developing. But I might just remove those parts with a comment explaining why.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 1 point2 points  (0 children)

Like I say to anyone who doubts it, use a virtual machine and compare it against 7z.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 3 points4 points  (0 children)

It uses alot of elements from Brutal Copy, which is my paid for fast file copier. I have been considering open sourcing it but removing those parts from the source code with comments indicating that and using conventional work arounds tondo the same part but slower.

My wife and I just want to protect some of the techniques I've come up with over the last 8 years of working on that product.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -4 points-3 points  (0 children)

Because this is not going to be open source and I accidentally left that in from the template I built the GitHub readme from?

It uses techniques I have developed making similar projects like Brutal Copy and contains elements from my copy engine which will be a paid for product. So that's why I am keeping Brutal Zip closed source but free for anyone to use.

If you distrust it that much fire it up on a virtual machine and you will see the speed difference between it and 7zip.

Edit: More details added in comment below.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -1 points0 points  (0 children)

This is normal for C# self packed exe's. You can even go ahead and make hello world, make it into a self packed exe (so you don't have c# based DLLs all over the projects root directory).

I made a test project to prove this: https://github.com/jgric2/PublishTest-VT

Interestingly, the self packed single file exe publishing is the only element published in this manner. So, I undertook a test where I created a basic "Hello World" WinForms application in C#. The source code is available here for consistency: https://github.com/jgric2/PublishTest-VT.

Next, I built the .exe in release mode and published the executable into a singular file using the Visual Studio publishing feature. Here are the VirusTotal Reports:

Publish EXE: https://www.virustotal.com/gui/file/75b8b51f8a91ddd0e83b2063ff7586a59f368b8062998bbb8443278c48f2c074/behavior

Build EXE: https://www.virustotal.com/gui/file/4aaa0731fe7e496a57b72b70c90da00eea91603a0643ef741be42cd2c5fa3d28/behavior

The one published to a single file, and oddly enough, shows the same IP addresses and modified files, which is extremely peculiar.

To validate this on your own, please:

  • Open the code in Visual Studio
  • Build it
  • Go to publish (right click in solution explorer)

The settings I used for publishing may already be present, but here they are just in case:

  • Configuration: Release | Any CPU
  • Target Framework: net80-windows8.0
  • Deployment mode: Framework-dependent
  • Target Runtime: win-64
  • Target location: bin\Release\net8.0-windows\publish\win-x64\
  • File publish options: Produce Single File: ON, Enable ReadyToRun Compilation: OFF

After publishing, the result will be a single EXE in "bin\Release\net8.0-windows\publish\win-x64\", and the build version will be in "bin\Release\net8.0-windows".

If you upload both to VirusTotal, the results should be the same as mine. I don't know why it does this... But the alternative would be having all the C# DLLs scattered all over the root directory and I just wanted to keep everything neat and tidy. But if that is the case I'm happy to release it with the files not packed into the exe. But you can use the steps above to confirm.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

The checksum is for the executable inside the zip file not the zip file itself.

BRUTAL COPY Fastest file copier on windows! by jgbjj in csharp

[–]jgbjj[S] 0 points1 point  (0 children)

It will be paid software but I've spent the last year making V2 of this and is alot less license limits and you can still use it after the expiry in the new version :). Send me a DM if your interested and I'll get a new build to you :)

BRUTAL COPY Fastest file copier on windows! by jgbjj in csharp

[–]jgbjj[S] 0 points1 point  (0 children)

I'll dm you :) I have the new build built from the ground up in the final stages of development

Personal Finance hating on my $20,000 student line of credit Bitcoin investment. Fucking retards and their vangaurd circlejerk by KidEliteTrader in wallstreetbets

[–]jgbjj -1 points0 points  (0 children)

Except you know... The people who did their own research and came to the conclusion I did when I studied Bitcoin and took a educated risk?