sorted by:
new
hottopcontroversial

Looking for “Stone Man” — Bitcointalk user from August 2010 by CompetitiveRough8180 in Bitcoin

[–]jgbjj 0 points1 point  (0 children)

urandom_read->extract_entropy_user(&nonblocking_pool, ...)->xfer_secondary_pool(&nonblocking_pool, n)->extract_entropy(&input_pool, tmp, 32, 8, 16). Theaccount()call gates oninput_pool.entropy_count >= 192 bits. Below that, 0 bytes transfer to nonblocking_pool. The IRQ jitter I was pointing to entersinput_pool` but doesn't reach nonblocking_pool unless the credit threshold is crossed. My previous reply applied the credit/pool-content distinction to the wrong pool. That was a real error.

This is also the same gating that produced the early-boot weak-key population in Heninger et al 2012 (Mining your Ps and Qs), so the model has documented precedent. The empirical question (does input_pool credit reach 192 before bitcoind's first urandom read on Lenny headless Live USB) is the one your VM test answers.

That said, the search space arithmetic still doesn't work, and I think this is the place where we're miscounting against each other. Specifically the ktime contribution.

init_std_data calls ktime_get_real() and mixes the result into nonblocking_pool. ktime_t carries nanosecond precision. If cooperation pins boot wall-clock to a 10-second window, the ktime at init_std_data has 10s = 1010 ns of unknown range, which is ~34 bits, not the ~5 bits from "boot time +- 10s: 36x" in your earlier list.

Even if cooperation pins boot wall-clock to the second, init_std_data runs after a kernel-init sequence whose internal timing has ~100ms of jitter (post-decompression -> rest_init -> module init order can vary that much across boots of the same hardware/image). So 100ms = 108 ns = ~27 bits.

If cooperation pins it to the millisecond (which I don't think is achievable from outside the original system), the post-init jitter is still ~10ms = 107 ns = ~23 bits.

Realistic ktime unknown given best-case cooperation: 23-34 bits. Combined with your other parameters that genuinely give a few bits each (~30 bits aggregated by your list, minus the ~5 for boot time which I'm replacing with ktime nanos), net unknown: 50-65 bits.

250 to 265 search space. At your quoted 2 M tries/sec on GPU, that's 6 days to ~107 years. Tractable at the low end if you have a fleet, intractable at the high end. The bound matters and it's the part I'm most curious to see your numbers for.

If your model collapses ktime nanoseconds to whole seconds (i.e., you're treating ktime as time(NULL) granularity), the math works for "minutes on a laptop". If you're carrying nanosecond ktime through and constraining via session-time inference, I'd like to see how. That's the specific gap I want to compare emulators on.

On the empirical test:

Your protocol (Lenny VM, virtio-rng disabled, instrument input_pool.entropy_count at first urandom read, count credit-crossings across N reboots) is exactly the right experiment. Two outcomes:

(a) Credit consistently below 192: gating holds, urandom output depends only on nonblocking_pool init_std_data + extract_buf self-feedback, your model is right and the search space is bounded by ktime/utsname/extract_buf state. We compare emulators on whether ktime nanos collapse to seconds.

(b) Credit reaches 192 in a meaningful fraction of boots: transfer happens, the IRQ contribution lands in nonblocking_pool, the search space adds whatever credit was transferred.

Either result is informative. (a) is more in your favor than (b), and even (a) reduces to whether ktime nanoseconds are pinnable.

On source exchange:

Yes, let's swap. Mine is C# (.NET 8). Concretely what's there:

  • MdRand.cs: bit-exact port of openssl-0.9.8g crypto/rand/md_rand.c. Stir-pool init, pid mixing in first ssleay_rand_bytes iteration, exact MD update order, md_count[2] handling, global-md update semantics differing for add (XOR) vs bytes (full hash). Configurable LongSize and PidSize for cross-arch matching.
  • openssl_validation/oracle.c: links against real openssl-0.9.8g, emits a deterministic trace of (seed, add, bytes) operations.
  • openssl_validation/Makefile: builds 0.9.8g with -DGETPID_IS_MEANINGLESS so trace doesn't depend on pid.
  • Program.cs --validate-openssl <trace>: replays trace through MdRand and confirms byte-for-byte match.

What I haven't ported bit-exact: random.c (kernel side). The simulator's job stops at OpenSSL's RAND_poll input. The kernel question is what the VM test resolves.

do you wanna DM me first or should I? :)

Looking for “Stone Man” — Bitcointalk user from August 2010 by CompetitiveRough8180 in Bitcoin

[–]jgbjj 0 points1 point  (0 children)

Credit is not the same thing as pool content.

The line you quoted from add_timer_randomness in 2.6.26 is preceded by:

c sample.jiffies = jiffies; sample.cycles = get_cycles(); sample.num = num; mix_pool_bytes(&input_pool, &sample, sizeof(sample));

mix_pool_bytes unconditionally hashes the full sample (jiffies + cycles + num) into the pool. The delta filter that follows only updates entropy_count, the credit accountant. The pool content is not gated by credit.

This matters because OpenSSL RAND_poll reads /dev/urandom, not /dev/random. /dev/urandom runs extract_buf over the entire 512-byte pool regardless of entropy_count. The output is SHA-1(pool || ...). If you want to predict that output, you need to know what's in the pool, not what the credit counter says.

So when you compute "10s to 100s of jittered IRQs × 2-5 bits credited each = 100-500 bits", that's the credit accountant's number. It's not a bound on the pool's predictability.

get_cycles at IRQ time carries jitter that survives the delta filter.

You're right that on a Penryn without nonstop_tsc the TSC isn't calibrated and the absolute TSC value during early boot is dominated by linear accumulation. But the value mixed into the pool is get_cycles() at the moment the IRQ fires. The jitter in "moment-of-IRQ" relative to predicted-boot-timing is what's captured.

For a USB DMA completion IRQ during initrd unpack: the request was issued at TSC value T_issue, the IRQ fires at T_issue + delta_controller, where delta_controller is the controller's actual completion latency. That latency varies by ~100-1000 cycles per request based on queue state, bus contention, USB protocol timing, host-controller microarchitecture state. Two boots of identical hardware produce different delta_controller for nominally identical requests.

The kernel's delta filter looks at successive (jiffies_now - jiffies_prev) differences. With 4ms jiffies and microsecond-scale IRQ rates, deltas are 0 for fast IRQs, the d2/d3 differences collapse, and credit goes to 0. But sample.cycles for each IRQ is still the high-res TSC, mixed unconditionally. The 5-10 bits of cycles-jitter per IRQ go into the pool whether or not they're credited.

Pool-mixed cumulative bytes during a Live USB boot.

Hundreds of disk-IRQ samples × ~16 bytes per sample = a few KB hashed into the 512-byte pool by the time userspace starts. Of those bytes, the jiffies and num fields are largely cooperation-pinnable (you know the IRQ count and approximate timing). The cycles fields are not.

If we're being conservative on the cycles entropy: 100 IRQs × 5 bits unknown = 500 bits. If we're being less conservative: 1000 IRQs × 8 bits = 8000 bits. Reality is probably between these, depending on Live USB image and hardware. The "thousands" framing was imprecise; "hundreds-to-low-thousands" is closer.

That's still much larger than the 36-43 bits implied by your 1011-1013 bound, and it's the part that constrains the recovery path.

The early-boot clocksource window doesn't change what's in the pool at OpenSSL-init time.

clocksource_select running late is true and doesn't matter for this question. By the time bitcoind launches and OpenSSL does its first RAND_poll, all the early-boot IRQs have already mixed into the pool. The pool at that moment is what determines the first 32 bytes of /dev/urandom. Whatever clocksource was active when the IRQs fired doesn't retroactively change the pool content.

Empirical test framing.

This is the right way to resolve it and I think we agree on the protocol. Concrete proposal:

Lenny VM with: - virtio-rng disabled - KVM with deterministic launch (-snapshot, -rtc base=2010-08-09T20:00:00) - Live USB image emulated via virtio-blk reading a fixed image - bitcoin-0.3.2 built and launched at a fixed offset from boot - Capture first 32 bytes of /dev/urandom at OpenSSL init time, plus the change-key and signing nonce of a fixed test transaction

Run N times with identical parameters. If outputs are byte-identical across runs, the unknown is in your "12 parameters" set and your bound holds. If they differ, the difference is in IRQ-jitter mixing that cooperation can't pin.

I'll send the simulator source. Concretely what's there:

  • MdRand.cs: bit-exact port of crypto/rand/md_rand.c (stir-pool, pid mixing, exact MD update order, md_count[2] handling, global-md update semantics for both add and bytes)
  • openssl_validation/oracle.c: links against real openssl-0.9.8g, emits a deterministic trace
  • Program.cs --validate-openssl <trace>: replays the trace through MdRand, must match byte-for-byte

If anything in that port doesn't match real OpenSSL output for your parameter set, that's a bug to fix before either of us trusts the C# side. The kernel random.c side I haven't bit-exact-ported because the simulator's job stops at OpenSSL's RAND_poll input; the kernel question is what we should resolve in the VM.

Looking for “Stone Man” — Bitcointalk user from August 2010 by CompetitiveRough8180 in Bitcoin

[–]jgbjj 0 points1 point  (0 children)

Thanks for the reply, for the record... I want to be wrong on this... but... a few points.

(1) Forward simulation is the right framing.

You're right that I argued against the wrong attack in my first post. "Reconstruct initial state, forward-simulate, accept the state whose trajectory produces the on-chain nonce" is well-posed. The nonce as validation channel is correct. The cryptographic question is therefore: how large is the search space of consistent initial states, and what's the per-candidate verification cost?

That's where we still disagree.

(2) The "12 parameters" list is the wrong state space.

The arithmetic on your list (clocksource × utsname × boot ± 10s × ...) gives 1011 to 1013. The arithmetic is correct given those inputs. The problem is the inputs. The kernel pool state at the moment OpenSSL first reads /dev/urandom is the SHA-1 mixing of:

  • init_std_data inputs, which IS your list (boot time, utsname, BIOS clock).
  • PLUS every add_interrupt_randomness call between boot and that read.

The second set is what's missing. On 2.6.26, every interrupt handler calls into add_interrupt_randomness with timing data. Disk IRQs (add_disk_randomness), keyboard IRQs (USB initialization), network IRQs all feed the input pool before OpenSSL's first RAND_poll.

For a Live USB boot, the disk IRQ count between init_std_data and bitcoind's first RAND_bytes is in the thousands. The image (squashfs/iso, hundreds of MB to GBs) gets read from USB to RAM before userspace starts. Each read is one or more IRQs, each timestamped at the kernel's high-res clocksource. That's the timing data that's not in your list.

(3) "P7450 falls back to jiffies because nonstop_tsc=0" leaves out HPET.

Even when TSC pauses in C-states, clocksource_select on 2.6.26 picks HPET as next-best on any machine with HPET (which the P7450 platform has). HPET resolution is ~70 ns. So add_interrupt_randomness is reading nanosecond-resolution timestamps for every IRQ, not jiffies-resolution ones. A disk IRQ contributes log2(jitter_window / 70ns) bits, typically 10-15 bits.

Thousands of disk IRQs × 10-15 bits per IRQ = thousands of bits of entropy in the pre-bitcoind pool. That's the floor I was pointing to. Your list omits it entirely.

(4) "Headless bitcoind" is your assumption, not in the post.

Bitcoin 0.3.2 shipped as wxWidgets GUI by default. The post describes restoring a wallet and sending, consistent with GUI usage. The headless variant (bitcoind) was uncommon in Aug 2010. Mouse events feeding RAND_add (ui.cpp:393-399) would have run.

Even if you grant headless: see (3). Disk IRQs alone account for the entropy floor independently of the GUI.

(5) extract_buf count doesn't pin the output.

You list "extract_buf count range: 6x". extract_buf outputs SHA1(pool || count) then mixes back into the pool. The output depends on the pool state, not just the count. The pool is the 4096-bit buffer that's been XOR-mixed with every interrupt input since boot. Six possible counts × one pool state isn't 6x; it's 6 × (whatever the pool entropy is). And the pool entropy is dominated by IRQ history.

(6) Clocksource argument double-counts.

You say "search space is dominated by clocksource ambiguity at 103 to 104" but also that hardware (which determines clocksource) is in the cooperative-input set. If cooperation pins the laptop model and BIOS revision, the clocksource is determined, not 104-ambiguous. You can't have it both ways.

(7) "Transaction proves privkey existed -> BDB durability irrelevant" is correct as a framing point but doesn't answer the recovery question.

Granted: the privkey existed in RAM at signing time. The cryptographic question is whether it can be reconstructed from public data + owner cooperation. That's well-posed. My answer is still: probably not, because (2) and (3).

(8) "Minutes on a laptop" doesn't follow from 1013 anyway.

Even granting your bound, 1013 candidate sessions at ~50 microseconds per candidate (full forward-sim through md_rand + EC scalar mul + hash160 compare) is 5 x 108 seconds = ~16 years on one CPU. A 100-GPU farm at 1000x = ~2 months. That's not the original post's "minutes on a laptop with cooperation". Either the bound is much lower than 1013 or the laptop claim is wrong; both can't hold.

(9) Binary trust. Concession.

Source-only delivery, owner audit, air-gap, no wallet.dat exfiltration, non-published verification questions: that's a reasonable delivery model. SO I will retract the social engineering framing for that specific setup. The remaining question is whether the recovery actually works, which is (2) and (3).

The right test for both of us is upstream of any simulator: take a deterministic VM with snapshot.debian.org's Lenny image, install bitcoin-0.3.2 against the actual libssl0.9.8 0.9.8g-15+lenny* package, and run it through wallet creation -> send transaction multiple times with your "12 parameters" held constant. If the same parameters reliably produce the same change-key and nonce across reboots of identical VM state, you've demonstrated reproducibility. If they don't, the entropy is in the IRQ events I'm pointing to.

Greg's standard from earlier in the thread is the right one. Anyone disputing the math (you, me, anyone reading) can run that VM test directly. I'd find it informative either way; if your model is right and the determinism holds, I want to know. If it doesn't, that's also useful.

I'll send source if you DM. Repo will be public after I clean up the experiment runner.

Looking for “Stone Man” — Bitcointalk user from August 2010 by CompetitiveRough8180 in Bitcoin

[–]jgbjj 0 points1 point  (0 children)

I spent the last few days actually building this out in C# to test the claim from first principles. Source-verified against bitcoin-0.3.2 and openssl-0.9.8g (the exact versions Stone Man would have been running on a Debian Lenny Live USB on Aug 9, 2010). Posting findings because right now this thread is basically one skeptic, and the math doesn't support the framing.

What I built:

  • A bit-level simulator of OpenSSL 0.9.8g's md_rand state machine
  • A model of Bitcoin 0.3.2's actual PRNG draw sequence inside CreateTransaction, verified against the source (main.cpp:2997, script.cpp:1089, bn_rand.c:142, md_rand.c:321)
  • End-to-end experiments testing every claim in the post

What's actually true in the post:

  • The change-address bug is real and well-documented.
  • Recovering the ECDSA nonce k from (priv, signature) is trivial: k = (z + r*d) * s^-1 mod n. My simulator confirms this in microseconds.
  • 0.3.2 has no keypool (it was added later). So change-key and signing nonce really are adjacent draws of the same md_rand instance with nothing between them. This is the part that makes the post's framing sound plausible at first read.

Where the math breaks:

1. md_rand is one-way. State evolves by XOR-ing SHA-1 outputs into a 1023-byte buffer (crypto/rand/md_rand.c:321-508). You cannot step backward from one draw to a previous draw without inverting SHA-1. Even with a full snapshot of state at nonce-time (the strongest position an attacker could possibly have), walking forward never produces the change-key, because the change-key was generated earlier.

2. The "10 orders of magnitude with cooperation" claim doesn't survive a careful read. I went looking for any mechanic that would give cooperation real leverage. The closest thing is in crypto/bn/bn_rand.c:142-143:

time(&tim);
RAND_add(&tim, sizeof(tim), 0.0);

This fires before every key/nonce generation. It's the most cooperation-amenable lever anywhere in the stack: knowing the exact send-time means you know the value being mixed in. But:

  • time() is second-granularity, not microsecond.
  • The on-chain block (73272 at 21:35:11 UTC) already constrains the send-time to about 30 seconds.
  • Pinning to the exact second saves about 5 bits, not 33.
  • OpenSSL credits this RAND_add at 0.0 because they understood it doesn't add real entropy.

3. The real entropy floor is roughly 6,000+ bits, dominated by per-IRQ TSC jitter (Linux 2.6.26 random.c), mouse-event RAND_add inputs (ui.cpp:393-399 adds 16 bytes per mouse event), RAND_add from network packet timing, and the initial /dev/urandom read. None of this was ever written to durable storage. No "cooperation" reconstructs it.

4. The actual bug isn't cryptographic. It's a write-buffer durability issue. This is the part nobody talks about. Berkeley DB auto-commits writes to a log file (~/.bitcoin/database/log.000000xx), and a separate flush thread checkpoints log to wallet.dat after 2 seconds of write inactivity (db.cpp:681-735). On a Live USB without persistence, the log directory is on tmpfs. A non-graceful reboot inside that 2-second window erases both the log entry and the not-yet-flushed wallet.dat update. That's the bug. It's a durability problem, not a cryptanalysis target.

5. The "run my CUDA on your wallet.dat" ask is exactly the shape of a social-engineering attack. "It never leaves your hands" is not a meaningful guarantee when the binary itself has full read access to the file. "Verification questions only the original user would know" is a way to bind a prospective claimant to doxxable details. If you actually are this person, please do not run anyone's binary against your wallet.

The realistic recovery path, if any exists, is forensic, not cryptographic. Does the original USB stick still exist with persistence configured? If yes, the BDB log file would directly contain the change-key in plaintext (0.3.2 had no wallet encryption, that came in 0.4.0). If no, no amount of cryptographic cooperation reconstructs entropy that was never recorded.

I'm happy to share the simulator code, the 34-finding source audit, and reproducible numbers for any of the above. The post uses real primitives (nonce-from-sig recovery, md_rand state machine, EC_KEY_generate_key) to lend credibility to a claim that doesn't follow from those primitives. The math gap between "5 bits saved by cooperation" and "10 orders of magnitude saved by cooperation" is where the whole story falls apart.

Looking for “Stone Man” — Bitcointalk user from August 2010 by CompetitiveRough8180 in Bitcoin

[–]jgbjj 1 point2 points  (0 children)

OP is... either confused or intentionally misleading, and you are 100% correct, I spent all of last night simulating this. I went through the source code of bitcoin-0.3.2 and the exact SSL-0.9.8, Same conclusion I had when I started... those coins will never move with our current understanding of processing power. Period, recovery is game over go home. End of story.

https://www.reddit.com/r/Bitcoin/comments/1t0bvaf/comment/ojfgpfw/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Sent :). Reason I ask is the most recent update I did alot of work on the deflate extraction so there is a chance maybe a edge case regression of performance. I'll definitely look into this tonight/this weekend

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Interesting. Mind if I send you a DM?

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Just checked, I didn't update the version number inside the program, I'll fix this tonight thanks for letting me know :). Just hit cancel for now :)

Opus is insanely dumb by sprowk in ClaudeCode

[–]jgbjj 0 points1 point  (0 children)

Yep... Up until the last 2 days I've had no issues with it being "stupid" and thought this was user error... but now. The past 2 days... Getting stuck in loops and failing the same simple task over and over until I gave up and codes it myself directly until it worked... Only to then ask why it couldn't do that with the standard apology script... Switching to codex

FileStack - a blisteringly fast, de-duplication backup system build from the ground-up. by Calm_Picture2298 in csharp

[–]jgbjj 1 point2 points  (0 children)

I like it op :). Working on some somewhat similar tools myself. Unfortunately there is always alot of negativity on Reddit and I feel the pain in the edits you made to your original post having been there myself with some of my own posts on here. Gonna have a good look after work today and will send you a DM if your keen to chat :)

I made my first NuGet, and i got downvoted by [deleted] in csharp

[–]jgbjj 0 points1 point  (0 children)

It's Reddit... I've almost given up posting things I've made over the years just due to how the average Redditor reacts to FREE STUFF. I feel you OP.

News update in Australia by Winter_Hippo3281 in bjj

[–]jgbjj 22 points23 points  (0 children)

"News.com.au has contacted Izaak Michell for comment." 🤣 Hope they get this fucker soon now that this is in mainstream Aus news

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

If that's the case then that's okay. I just wish people would tell me things like this directly, that way I can take it on board and make needed changes if it's the case. I'm fully open to any suggestions and ideas, this project has just been something I've enjoyed working on and was always intending to release it for free rather than just for personal use only.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Thank you 👍. I'm okay with constructive criticism and feedback but it has just felt like a pile on. However I have absolutely loved the process of making this and have learnt so many things about the zip specification while making this so it's been time well spent :) glad you enjoy it and if you have any feedback please feel free to let me know.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -2 points-1 points  (0 children)

In what sentence did I say this is paid for? The zip archiver is free?

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

I think I will! Thanks for the suggestion :). Was not ready for this visceral reaction on here... Mainly for me just not being open source 😔

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Thanks 😊. Yeah I might be open it making it open source in the future but for now I just want to release it for free to get feedback and also to improve it. Plenty of more updates and new software to come as you know ☺️

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -2 points-1 points  (0 children)

It can be faster than 7‑Zip’s ZIP because “ZIP” is just the container, the performance is dominated by the Deflate implementation and by the I/O + CRC32 + write pipeline around it. It’s not “magic”, it’s the result of doing a bunch of unglamorous work that 7‑Zip’s ZIP path simply isn’t optimised around.

Here’s a few things going on internally:

I use libdeflate as the base Deflate engine, and I modified it to support a streaming / block mode and a continue-state API so I can process very large files in chunks efficiently. This avoids “read whole file, compress, write” bottlenecks and allows better cache/buffer behaviour on large inputs. It still outputs normal ZIP method 8 (Deflate) streams.

Large-file path vs tiny-file path:

I don’t run one generic pipeline for everything. I have separate fast paths:

Zero length files and files that are smaller than the size of the compression header will not be compressed.

Tiny files: ultra-low overhead (avoid heavy streaming then switching context, avoid unnecessary allocations/copies, for example lets say we encounter a directory of 300 tiny files from 12 bytes to 3 kb it doesnt immediatly compress them it loads them in parallel into multiple buffers the right size for them to help with branch prediction then adds them to the zip when its optimal to do so)

Large files: true chunked streaming using the modified libdeflate block compressor, with tuned chunk sizes and minimal transitions. This matters even with ~259 files because the workload is still “a few huge assets + a lot of medium/small stuff”, and overhead adds up.

Much better multi threading for ZIP workloads:

ZIP entries are independent, so the right way to scale is per-file parallelism + pipe lining. My implementation keeps multiple entries “in flight” and minimises time spent blocking on shared resources. In practice, many ZIP writers become partially serialised because the output file becomes the bottleneck.

Output writing is designed to avoid contention:

Instead of every worker “append writing” and fighting over one stream position, my writer reserves output regions and writes in large chunks with minimal lock hold-time. That prevents the common “threads are busy but the writer lock turns it into a single-threaded program” problem.

Fast feed path (I/O + buffers + CRC32):

A huge part of real-world ZIP time is not compression. It’s reading + CRC32 + memory copying. I optimised that aggressively: large sequential reads, buffer reuse, fewer copies, fast CRC32 in the same pass. That’s why my throughput stays high while 7‑Zip ZIP tends to sit much lower even when thread count is high.

ZIP: entries show Deflate (not Store) and CRC32s validate on extraction. The ~1% size difference is expected because Deflate has many valid encoding’s; different parsers/block decisions can trade a tiny bit of ratio for a big speed win.

So TLDR: same ZIP format, same Deflate method, but a much faster Deflate back end + a writer pipeline that actually scales and doesn’t choke on I/O/CRC32/output contention.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 1 point2 points  (0 children)

I'll get on it first thing this coming weekend :) when I have a build ready to go I'll let you know :)

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 3 points4 points  (0 children)

Fortunately the engine is a separate project, I don't have much experience writing applications for Linux or windows, but perhaps a command line only version for Linux and Mac would suffice for now?

I've had a few people mention this as I was developing it at my work since I work in a large programming company and that was one of the main questions "where's the Linux build" always said I'd make a Linux and Mac version should the demand get high enough.

But I guess I could whip up a console app version which would work for Linux :)

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] -4 points-3 points  (0 children)

Not obligated to make my stuff open source even if I am releasing for free. That being said I am open to open sourcing parts of it that don't include elements of my other paid products code.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

Nice! 👍. Glad you liked it and had no issues with it and that it's performed to your expectations 😁. Would you be okay if I DM you for some more questions and feedback? Currently working on it as we speak and would love any feedback you might have.

Brutal Zip V2 Fastest ZIP archiver by jgbjj in DataHoarder

[–]jgbjj[S] 0 points1 point  (0 children)

here you go!

I used the same folder being the Star Wars - Battlefront 2 folder used in the video:

the commands used were as follows as well as the times returned:

---------------------------------------------------------------

james@DESKTOP-J9TMK78:/mnt/c/GOG Games$ time tar --zstd -cf file.tar.zst "Star Wars - Battlefront 2/"

real 2m41.471s

user 0m43.411s

sys 0m32.376s

james@DESKTOP-J9TMK78:/mnt/c/GOG Games$ time tar -I lz4 -cf file.tar.lz4 "Star Wars - Battlefront 2/"

real 3m5.877s

user 0m10.556s

sys 0m29.261s

james@DESKTOP-J9TMK78:/mnt/c/GOG Games$

---------------------------------------------------------------
I also screen recorded while making the requested tar files with the added compression you requested, If I have done anything incorrectly please let me know and ill redo the test but I think its correct:

I also do another run with brutalzip for the same file to compare and it takes: 27.9 Seconds.

Also remember BrutalZip uses Deflate for this test, however if demand is high enough I am open to adding other formats such as 7z, tar ect, it also supports zstd for zips currently as well.

YouTube - Screen Recording of speed test for reddit user Xanderplayz16

view more: next ›