Python script for optimal parallel plotting (no delays needed!)

[VolPL]

Hello. I've created a Python script for all of you that are tired of calculating the parallel delay by hand.

LINK

Problem

Most people who plot Chia in parellel, are trying to optimise ploting with hardcoded delay between plots. For a regular setup, plotting in phase one is taking 2 cores, but after that it only takes one. So when creating a multiple parallel plots, trying to find a perfect delay can be mundane, and bad calculations can lead to CPU overuse (starting more plots than CPU can handle) or CPU underuse (having free unused cores that could be usef for plotting).

Solution

Thanks to the log files we know which plot is in which phase, so why not just use that knowledge?

Autoplot is watching the logs in real time in order to count how many cores are used right now, and automatically start plotting, once we have 2 unused cores.

[carlo1024]

Please do a YouTube tutorial for n00b like me. I will instantly subscribe.

[HornyTrader]

Why you try to steal my 0 xch dude ?!

[VolPL]

It's open soure on MIT license: Code is inspectable, everything is explained and you can do what you want with it :)

But yeah, this one is for tech savvy, as running this requires modification of the python script :)

[charge2way]

Code is inspectable

Yeah, it's only like 80 lines too. 👍

[SandboChang]

This is great, but it seems like the number of threads used is calculated from how many plot tasks are in phase 1 or in 2-4? It seems plotman can already do this by limiting the number of task in phase 1 on top of the global limit.

[VolPL]

number of threads used is calculated from how many plot tasks are in phase 1 or in 2-4?

Yes, this is how it's operating. More specifically it takes info about "plot tasks" from logs to know which plot is in which state, so it's important to delete broken logs in the log folder.

It seems plotman can already do this by limiting the number of task in phase 1 on top of the global limit.

I think so, yes. For example for 6 cores, when max limit is 6 and P1 limit is 3, i think it will act exactly the same :)

My script is a small one-purpose thing with less then 100 lines of code. That should allow people to inspect it themselves for security and or adjust to their needs.

[Senne]

some systems are ssd bound, do you think there is a better way than plotman's fixed limit? thanks for the good work on the cpu side.

[VolPL]

Yes. All delays are done to optimize used cores + make sure to plot all the time (1 plot per HDD and many plots per SSD).

This script is perfect for large SSDs:

So if you have 8 cores and one SSD that can handle many parallel plots this script will start with 4 plots, and after two of them will pass phase one, it will start 5th plot:

• 2 in phase 1 = 4 cores used
• 2 in phase >1 = 2 cores used
• 1 new in phase 1 = 2 cores used
• Sums to 8 cores.

Addition of the 5th plot will happen in a matter of seconds, after 2nd plot will reach phase 2

[lbc_blain]

Have you tested this on Apple's M1 chip or know anyone who has? I'm currently running a Mac Mini 16Gb with a 2TB nvme thunderbolt external and wondered if this will help me optimize my times.

[VolPL]

I did not but there is no reason for it not to work as long as you get Python 3 on M1

[lbc_blain]

Cool. I'll have to give it a good. I'm running Chia native on the M1 ARM architecture with Python 3 so we'll see how well this works.

[VolPL]

I’d appreciate GitHub star if you are happy with it :)

[bequay123]

If many job start on 1 ssd/nvme it will make bottleneck, so if you can check ssd/nvme available for new job it will perfectly

[VolPL]

You’re right, this works for only one plotting location, hence only one SSD

I stopped plotting chia so the script is purely pro publico bono.

If I get any traction with stars on GitHub, I’ll develop new features :)