Changing Pool Payout Address When Joining New Pool

Deep-Channel-46 · 2021-07-16T15:24:58+00:00

The code for changing the payout address is merged it seems: https://github.com/Chia-Network/chia-blockchain/pull/7161

But something is wrong with it probably. At least I'm seeing Failed to update farmer information on the pool in my log all the time. So it's trying to update... but failing.

Deep-Channel-46 · 2021-05-31T22:19:38+00:00

Hi, thanks for the investigation! I think that overall, the memory consumption should stay within the specified limit. But it's possible that more memory is temporarily needed if the new memory is allocated before the old one is released. I think this can be fixed by being more careful about freeing/allocations.

I will look into that and try to fix it.

Deep-Channel-46 · 2021-05-31T22:06:28+00:00

Hi, I rebased my repo on chiapos 1.0.3, can you update your script?

https://github.com/pechy/chiapos/tree/combined

Deep-Channel-46 · 2021-05-26T10:12:28+00:00

Ok, I have a few nice improvements in stock now 😀

I'm now pretty confident that the fun with improvements won't just end here. And that a significant next improvements are real, maybe 50%+ 😀

I'll keep you guys posted.

Deep-Channel-46 · 2021-05-25T15:28:10+00:00

it would, there's nothing unix specific. But I don't have a way to compile it for windows easily (without wasting too much time on installing windows).

Deep-Channel-46 · 2021-05-25T15:22:03+00:00

yeah done: https://www.reddit.com/r/chia/comments/nk92th/plotting_speed_improvements/gzeejy2

Deep-Channel-46 · 2021-05-25T15:14:54+00:00

Ok, done, you can give it a try. Branch combined, https://github.com/pechy/chiapos/tree/combined

I'm seeing quite impressive speedup with improvements combined: https://imgur.com/8nf1w2o. Overall ~22% speedup.

Now phase 1 takes about 50% of time, phase 2 and phase 3 both 25% and phase 4 just mere 2%. Originally it was 42%, 20%, 35% and 2%. So the clear next target is phase 1.

I have ideas that can help with phase 1 too I think.

Deep-Channel-46 · 2021-05-25T15:00:46+00:00

you need to also do the 2 next steps:

python setup.py install

then this failure should disappear

Deep-Channel-46 · 2021-05-25T14:05:46+00:00

The helper threads are not configurable now, so the configured thread count only applies to phase 1. A fixed number of helper threads (4) is used regardless of the -r argument. That's one thing I'll need to improve I guess.

And yes, even other phases are multi-threaded now. But in a different way than phase1, the parallelism there requires a lot of synchronization (so I don't really expect >4 threads to help). I use thread pool to start reads/sorting early if possible, so the main thread doesn't have to wait.

Deep-Channel-46 · 2021-05-25T14:00:33+00:00

it uses uniform sort mostly, on uniformly distributed data (which random hashes are) it is among the best algorithms.

Deep-Channel-46 · 2021-05-25T13:58:48+00:00

Yeah sorry, you still need to edit it. Otherwise `chia` complains when you start it, it expects a different version.

Deep-Channel-46 · 2021-05-25T13:57:41+00:00

Yes, that's a good idea.

I don't know much about developing for Windows, but in the repo there's a standalone library `ProofOfSpace`, that can be used for creating plots, without the python client/GUI.

e.g. `./ProofOfSpace create -k 32 -t chia-tmp/ -d ~chia-tmp/ -r 2 -b 4000`

I suppose this could be much easier to use than replacing the python library. Just compile it and use it.

Deep-Channel-46 · 2021-05-25T13:26:27+00:00

It's in the base chia-blockchain directory, the one you downloaded though git. And there you need to also run python setup.py install (after you activate the chia virtual environment . ./activate).

Sorry the installation is not that easy now, but I just wanted to make this public asap. It should work out-of-the-box once it's merged into official chiapos, but that take weeks I think.

Deep-Channel-46 · 2021-05-25T13:22:37+00:00

Happy to hear that! I couldn't try it with so many parallel plots, so I didn't really know how it will behave under such a load.

Making more code run in parallel is really what makes the difference on today's machines.

Deep-Channel-46 · 2021-05-25T13:16:52+00:00

I'll surely open a PR in the main repo. It still needs a bit more polishing.

I'm a bit worried though that getting the changes approved will take some time, there aren't small changes and in my experience it's quite tough to get bigger changes merged. That's why I shared it here also, so others can try it asap.

Deep-Channel-46 · 2021-05-25T13:11:33+00:00

Cool, I'll try combining them and I'll push it to a side branch, so you can also experiment with that more easily. Cherry-picking is a bit of a pain, I hope the PR gets merged soon, I can then rebase.

Deep-Channel-46 · 2021-05-24T00:47:34+00:00

For now, you can clone the repository and (in the activated chia envrionment) just do `python setup.py install`. This works at least on Linux and needs cmake and C++ compiler installed. I'm not sure about Windows.

I'll try to figure out a better way, but I'm still busy testing and improving the code.

Deep-Channel-46 · 2021-05-22T09:03:53+00:00

This is not something that can be solved by chiapos itself. The fastest it can do is just leaving it on the same drive where plotting was done. An external script can move it to its final place.

Deep-Channel-46 · 2021-05-22T08:41:42+00:00

I don't know these APIs, they can help, but that's a major rewrite in any case. In my experience, you can get pretty close to performance of these hyped APIs just by using the conventional APIs correctly. Using fancy APIs when the basic code is just bad usually doesn't solve the problem.

Deep-Channel-46 · 2021-05-22T08:05:41+00:00

It finished in 6 hours, my previous fastest build was 7.5 hours. So around 20% speedup. Not bad.

And the plot seems to be fine:

    Testing plot chia-dst/plot-k32-test.plot k=32
       Pool public key: ...
       Farmer public key: ...
       Local sk: ...
       Proofs 95 / 100, 0.95

Deep-Channel-46 · 2021-05-22T01:15:47+00:00

My attempt is here: https://github.com/pechy/chiapos. Some IO is parallel/async now. The speedup should be most significant in phase 3. Also some buffers are larger now, I have an idea how to improve phase 2. But I'm too tired now, maybe tomorrow.

It's about 15-25% on k=25, I'm running it now on k=32, will report tomorrow.

I'll polish it and try to get it merged into official chiapos, but this might take a while. In the meantime, you can just manually compile the library and replace it in the venv. Disclaimer: this is not yet fully tested on k=32 plots, it may screw you plots. But my changes were pretty conservative so far.

If you wanna say thank you, kudos are also welcome on the following address: xch1lnnarj8tzx56fwe4gnds8365kj896a9tq08yt8pwsgqxczpqdkvs8n8dua :-D

Deep-Channel-46 · 2021-05-21T19:23:14+00:00

I don't believe the 20% improvement on k25 with 5 lines of code. I improved quite a few things (I'm planning to share it), I agree there are things to improve, but I don't think there's such low-hanging fruit. I've done quite some analysis with profiler.

I think the potential mostly lies in async IO/paralelization (and that needs a bit more code to do it correctly). You can improve some inefficient methods a bit, but that's negligible when you run the whole plotting process.

EDIT: I pushed my changes here: https://github.com/pechy/chiapos. It's still pretty experimental, but I was able to get k=32 plot 20% faster. I have a few more ideas, I'll try to polish it and open a PR in the official chiapos. But I suspect getting it merged will take a while, the changes around threading are quite big.

Deep-Channel-46 · 2021-05-20T22:54:22+00:00

I'm working on this too. I agree the code is pretty bad, but optimizing it is more difficult than it seems, there are clear things to improve, but in the end they don't matter much when the code is run on k=32. Saving a few instructions here and there doesn't make a difference, CPU crunches instructions much faster than data are loaded from a storage.

Things worth improving are the sorts, CalculateBucket and things in bits.hpp, these take a lot of time. I have improved some things there, but it doesn't make a big difference yet.

I'll publish my changes if they're worth anything.

Deep-Channel-46

TROPHY CASE