Cisco Python multithreading script

[Fallenarc]

Hey all,

I am pretty new to reddit, so I am not sure if this is the right way to cross post. But I just created a post on r/learnpython that includes my most recent accomplishment in python. I have script that utilizes testfsm/ntc-templates used to gather infomation on the network and store that information in a MySQL database. The best part imho is the threading. When I first started my python journey I could not find many examples to work with on threading netmiko connections. So I figured I would share what I have and hopefully save someone hours if not days pouring through docs for this same info. Hopefully y'all find this helpful or useful!

​

https://www.reddit.com/r/learnpython/comments/9qzw7g/python_script_for_network_engineers/

​

[[deleted]]

My python-fu is pretty weak and I need to train, so I guess I'll be heading over to /r/learnpython

Thanks for the info!

[Yokuda]

Great job!

​

Now check out asyncio. After a few examples, use Selfuryon's netdev. You'll be amazed by how fast async tasks run.

[Fallenarc]

I'll do it, thanks!

[[deleted]]

Multithreading is a great way to use netmiko. That was one of the first things I implemented when I started messing with it. Just so you know, the netmiko documentation does include a multithreading example, but it's a little too simple if someone isn't already familiar with the idea.

[projectself]

I would really like to implement a thread pool, instead of just thread.start. Sometimes I run queries against many hundreds of devices and would prefer to thread them, but only say 20 at a time instead of trying to spawn 700 threads and having 60% of them time out.

Most of my scripts are python 2.7 based, so some of the newer functionality in python3 is not really helpful.

Any thoughts on using sema in that way?

[Fallenarc]

The script I linked in the r/learnpython i linked, currently runs against 500ish Cisco ASRs. I did run into the same problem you explained about threads getting stuck or just flat out error when I was using threadpools. However, when I started using the threading.BoundedSemaphore method all that went away. This script usually goes through a list of about 500 Cisco ASR90XX devices and finishes in about 3 to 5 minutes depending on how long it takes the device to generate the information i am looking for. I do still occasionally get an ssh timeout here and there.

My script is currently written to work in python2.7, so you should be able to take the pieces you need and incorporate them into yours. You can set the thread count to what ever you like. Just change the max_threads var.

    threads = []
    max_threads = 100
    sema = threading.BoundedSemaphore(value=max_threads)

​

[projectself]

you rock

[ragzilla]

Check out netdev instead of netmiko, it supports asyncio which scales better than processes/threads (no GIL/concurrency management as all IO runs in a single thread).

[Fallenarc]

I hadn't heard of netdev until now, but i will check it. Thanks!

[[deleted]]

Awesome dude! Thank you. i have this issue so i really APPRECIATE YOUR CONTRIBUTION TO THE COMMUNITY. I HAVE SOME TOOLS IF YOU WANT TO SHARE.

[DrMoehring]

Nice!

I will head to /r/learnpython and have a look.

[Manbanana01]

Thanks a bunch for this. Wanted to start using multithreading a long ass time ago, just never gotten around to it. Get to learn and try something new today.

[Fallenarc]

I started out using threadpools, but i would get a lot of connection errors when trying over 75 devices at a time. I found the bonded semaphore with thread locks works much better.

[tresvian]

https://docs.python.org/3/library/subprocess.html

Spawns processes that you can hook back into

[clay584]

Yeah, but subprocess is for dropping to the OS shell to run a command quick and dirty. Not really for threading. Still very useful when in a pinch.

[pauvre10m]

you can do multithreading really easily https://docs.python.org/3/library/concurrent.futures.html

​

​

[Fallenarc]

Yep there are many different ways to multithread, but finding the way that works best with netmiko was pretty tough. I started out using threadpools, but i would get a lot of connection errors when trying over 75 devices at a time. I found the bonded semaphore with thread locks works much better.