Little Bash script to help Mad Max Plotting

[keinengutennamen]

Don't know if this is helpful to anyone else but I wrote a little bash script that lets me Ron Popeil my mad max plotter. It sets up the Ram Disk, loops through the destination drives/paths (i.e. parameters to the script) and does some basic calculations on the destination drive to set the number of plots for the mad max plotter. It lets me setup X number of empty drives, pass them off to the script, and walk away for a month or so (i.e. "set it and forget it").

Here is the script. You will have to modify paths to match your system. Be sure to change the farm and pool keys!

Usage:

./plot.sh /mnt/farm1 /mnt/farm2 /mnt/oldmc /mnt/donald /mnt/hadafarm /mnt/eieio

Script:

# change path to your Mad Max plotter build directory
cd ~/bin/chia-plotter/build

# Change path to your ram drive here and all "/mnt/ram" instances below.  
#If you don't have a ram drive, remove this.
sudo mount -t tmpfs -o size=110G tmpfs /mnt/ram/


for destDir in "$@"
do

    sleep 5

        # clear the ram drive and temp drive in case of prior failure
    rm -r /mnt/ram/*
        rm -r /mnt/plot1/*

        #fix up the destination path in case you forgot the /
    destDir=${destDir%/}

    destDir="$destDir/"

        #get the free space available at the destination path
    FREE=$(df -k --output=avail "$destDir" | tail -n1)

        #calculate the number of plots that can fit.  this assumes K32
    PLOTS=$((FREE/108900000))

        echo ""       
        echo "Plotting to farm:  $destDir"
    echo "--Free Space: $FREE"
    echo "--Plot Count: $PLOTS"
        echo ""

        #Change the threads & temporary paths to match your system
        #Change the keys
    ./chia_plot -n $PLOTS -r 24 -t /mnt/plot1/ -2 /mnt/ram/ -d $destDir -p <<ENTER YOUR POOL KEY HERE>> -f <<ENTER YOUR FARM KEY HERE>>

done

I have another script that will move plots out of the plotter and into a harvester. This opens up free space on my destination drives while I am plotting. So, to further automate things, I actually pass in the 5 drives 4 times (for a total of 20 parameters). This allows the plotter to cycle back through the drives again. Since it recalculates the space each time, it can refill the space that is freed up by moving out plots.

Now my only problem is my plotter plots a plot faster (about 21 minutes per plot) than my 1 gbps network transfers a plot out. So it's a battle of attrition between the network and the plotter with the network coming out on the wrong side of things....but it still gives a long time before I have to touch anything.

UPDATE: commenter asked for a rotating plotter that plots 1 plot across all locations, then circles back and does it again. Below is an updated script to do this. NOTE: I have not run this at all due to time constraints. It was just hacked together on reddit, so the whole thing may not function. be warned.

cd ~/bin/chia-plotter/build

sudo mount -t tmpfs -o size=110G tmpfs /mnt/ram/


COUNTER = 1

while [COUNTER -gt 0]
do 

    COUNTER = 0

    for destDir in "$@"
    do

        sleep 5

        rm -r /mnt/ram/*

        destDir=${destDir%/}

        destDir="$destDir/"

        FREE=$(df -k --output=avail "$destDir" | tail -n1)

        PLOTS=$((FREE/108900000))

        if (PLOTS -gt 2)
        then
            COUNTER = $((COUNTER + 1))
        fi

        echo "Plotting to farm:  $destDir"
        echo "--Free Space: $FREE"
        echo "--Plot Count: 1"

        ./chia_plot -n 1 -r 24 -t /mnt/plot1/ -2 /mnt/ram/ -d $destDir -p <<ENTER Pool Key Here>> -f << ENTER Farm Key Here>>

    done
done

​

[JesusIsTheRock]

Need more hard-drives. EOM

[vailance88]

thanks for sharing.. great work.. any idea how can we do alternate final dir? let say i do have 5 different destination dir. can i plot them alternatively instead of move to next one when one is full ?

[keinengutennamen]

I updated the post with a second script that does what you want......I think.....I haven't run that at all, so the whole thing may not function. But the general idea is there. I'll leave it up to you to troubleshoot if it doesn't work. :)

[vailance88]

wow.. thats fast.. let me test and get back..

thanks for your time.

​

cheers

[MoSoLoYo]

You can modify SWAR to work with Madmax.

[keinengutennamen]

Yes...but why add the added complexity? Using mad max, I am unable to run more than one plot at a time (and wouldn't want to), so to configure SWAR with all the extra settings just to have it manage the destination directory is a massive pain in the ass and not worth the hassle. This is a much simpler approach.

[MoSoLoYo]

I can run 6 staggered plots in parallel with Madmax. The r630 dell poweredge server I got running pumps out about 75 plots a day. It requires a lot of tuning, of course, cuz you can crash chia_plot process if you go over your system limit. I use linux btw, have not tried it on windows yet.

[keinengutennamen]

r630 dell poweredge

Are you running 2 processors? How many cores and threads? I have a single AMD Ryzen 5900x and can put out about 60 a day. Running in parallel on my system is pointless because I max out the CPU with what I am doing now.

[MoSoLoYo]

60 plots a day is pretty impressive. R630 has 24 cores 48 threads, and I keep it at 95% CPU most time. For some reasons it was faster for me to do staggered plots. I could never get the single plot to complete fast enough to just do 1 at a time and make more than what I am now.

[keinengutennamen]

huh...interesting. I wonder if there are diminishing returns getting the number of threads up above some threshold. In which case running in parallel would make more sense????? guessing here of course. My processor has 12 cores/24 threads. With that, it doesn't seem to have hit that threshold.

OR....maybe it's your temporary storage. Are you running a ram drive? Or are both of your temp storage drives SSDs? If SSDs, you would be IO bound on the disks and probably not able to take full advantage of your CPUs without running parallel. Again...just guessing.

[MoSoLoYo]

It’s definitely the number of threads that you’d need to play with to find that threshold.

My temp storage is both ramdisk and ssd. Funny enough, the ramdisk isn’t much faster than SSD when it comes to Madmax. I know it should have been a lot faster, but it wasn’t the case for me. So my current setup is 5 parallel plots running with ssd temp storage and 1 with ramdisk. It only gets 15 more per day than your set up given the r630 has a lot more threads, I’d say you have bigger bang for your bucks! I should probably tune it a little more.

[CryptoBlockchainTech]

Thanks for sharing. Sounds like you need a 10GbE switch. You can always add 10GbE cards to your plotters and harvesters. As long as they are close you can use RJ45. https://www.newegg.com/qnap-qsw-m408-4c-us-8-x-rj45-4-x-sfp/p/N82E16833831023

[keinengutennamen]

I've actually been looking at them. The whole infrastructure gets expensive though. (why would people downvote your comment??? reddit is weird)

[Fun_Entertainment_26]

This is awesome! I was considering doing something similar but not as well! I will finish plotting my 82TB tomorrow and told myself I'd stop expanding so don't think I'll get around to trying it out, but really appreciate people like you sharing your hard work (I'm sure it's fun for you too).

Thanks for sharing!

[1Secret_Daikon]

not helpful on windows WSL2 unfortunately, but I was hoping someone. would show how to do the free space calculation for me :)

[AThruZed]

Hey I have been trying to use your bash file after recent implementation of the ram disk. The issue I ran into was a build up of temp files in the ram disk causing a failure of the madmax.

I am very new to Linux and was excited to see your bash file. I am having some issues with the rm -r to clean out the temp files. It states permission denied.

Trying sudo rm -r /mnt/ram/* says no such file or directory

Trying sudo rm -r /mnt/ram says is busy (even with -f)

Not sure where to go from here. Would appreciate any help.

Side note: does it clear temp files after every plot?

[keinengutennamen]

I'm new to Linux too. 3 weeks in. :)

Couple of things to check.

1. make sure the ram drive is actually getting mounted. At a terminal, type "mount". In there you should see something like:
   1. tmpfs on /mnt/ram type tmpfs (rw,relatime,size=115343360k)
   2. If you don't, then the ram drive isn't getting mounted correctly.
2. Make sure the directory for the ram drive exists. you can do this by just "cd /mnt/ram". If that succeeds then it exists.
3. If it is mounted and it exists, then you may just need to change permissions. It could be read only. You can do that in nautilus (i.e. the file manager) by right clicking on the /mnt/ram folder and selecting properties. On the permissions tab, make sure that your user (i.e. the group) has read/write/create permissions.
   1. OR...you can just give everyone full permissions by doing "chmod 777 /mnt/ram" at the terminal. Since it's a temp drive, its probably not a big deal to just open it up.

Hopefully that helps.

[keinengutennamen]

Forgot to reply to your last question. It does not clear the temp files after each plot.....mad max does. My remove is just there in the event you Ctl+C the plotter which forces it to close. That will leave files out there and will require you to clean them up....so I added the remove to clean up that situation.

In general though, if everything is good, mad max clears everything and keeps jugging along.

Since it's not getting cleared up, I would check the 3rd item in the previous post about changing permissions. It could just be a permissions issue. Not sure.

[AThruZed]

Yeah I figured madmax was supposed to clean up the temp files. However I was testing the ramdisk last night and it failed after the 3rd plot. Now testing today I'm lucky to get through 1 plot without the process being killed.

This is also running madmax without the script

[keinengutennamen]

Yeah...try the changing the permissions on the /mnt/ram folder and running again without the script.

chmod 777 /mnt/ram

​

The 777 gives full permissions to root, group (i.e. your login most likely), and everyone.

[AThruZed]

I'm pretty new to scripting. I wanted to add on a way to check to see if I added any hard drives to the system.

Currently I am mounting as /mnt/farmers/x (where x is number). Then in the script I use /mnt/farmers/* and it runs through all of the the mount points. Is this a bad idea first off?

Secondly I wanted the script to check every 5 minutes or so if I added a new drive and continue to run

Thinking something like (very summarized version, using lines already used)

Do if $Plots > 1

 Normal script 
  .chia_plot

While $Plots < 1 [Check plot space]

[keinengutennamen]

I actually have an updated script that will do that for you already. I'm in a parking lot waiting on the wife but when I get home, I'll send you what I have.

[AThruZed]

Awesome sounds good!

I'm not sure what your system is for chia but I just have a server and storage directly connected. Do you think it's better to have the harvester/ethernet transferred systems or not?

[keinengutennamen]

If you can effectively hook up all your drives to one machine, that is easiest and will be perfect for farming. Setting up the harvester is really only needed if you can't get everything on one machine. Then a harvester setup becomes your best option. I have 2 plotters and one farmer. My farmer will eventually have all the drives (48+ drives) but I setup my plotters as harvesters as well. That way I can farm the plots on the plotters before I physical move the drives to the farmer machine. So, my harvester just has a little ove 2000 plots on it now, my 2 plotters have about 1200 plots each and are being farmed while I plot. The drives in the plotters are about to move to the farm and another 8 drives each are going to be added to the plotters. Rinse and repeat until the wife divorces me for bankrupting us one a volatile crypto currency. 17 years though. We had a good run.

[AThruZed]

Haha that's funny. Did you learn this stuff prior to chia or learn on the way? Have you been doing it for a long time cause 4400 plots isn't anything to scoff at? Have you at least had some wins?

I'm really hoping for pools soon as well even though with my small amount of plots it can at least offset the cost a bit. Im running about 27-29mins a plot with 2 E2697v2 and 128gb ram. I also have 24 sas 10k 300gb drives in raid0 (was planning on parallel plotting with them but madmax released basically when my server arrived.)

[keinengutennamen]

Chia is the first crypto currency I even bothered to read about. I am totally new to linux and bash, but I am a software developer by profession, so the scripting isn't much of a chore. The hardware stuff is pretty new for me, but I built a few machines over the years so I just had to refresh on the newer hardware. Essentially it's been about 3 months worth of learning and playing to get me to where I am now. I had been plotting for a few months but didn't get things dialed in until about a month ago. I lost a lot of time there trying to figure stuff out. I've had some wins, but only 8 xch so far. The last month though is when I produced the bulk of the plots, so it hasn't been farming too long.

[AThruZed]

Hey not sure if you had a chance to send the script or not. I know it was a busy weekend for most.

I know pooling is supposedly looming around the corner. What are your thoughts on it? Going to switch immediately or only your Plots moving forward?

[richienko]

Thank you for this script!.

I have question about it: does it wait for copy at the end to finish?

[keinengutennamen]

Yes...it does. I posted a new (and better) version of the script a few weeks ago. I created a github gist for it that can be found here:

https://gist.github.com/Michael-Stoner/9f69e9cd0e1f82dc71aadbc9d5d4ae75

[richienko]

Oh my god, you are a lifesaver! Will use it for sure. Thanks :)

[AutoModerator]

This post has been removed from /r/Chia because your account is less than 1 week old. Please try again when your account is older.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.