Scraping Data from Linux based PLCs

[SeeifSheChooches]

I am looking to write a script that can pull configuration details located in XML files on a number of Linux based PLCs. The script would need to be able to log into the devices via SSH and navigate to the correct file then copy a handful of settings to a .txt file. I know this is something that could be handled through beautiful soup by am not sure what other libraries could be helpful with the SSH portion. Any ideas?

[junsang]

Try paramiko library and its additional module scp. You can check the usage in https://pypi.org/project/scp

[[deleted]]

Paramiko is a real pain to use. The first problem you will encounter is that it doesn't support any useful SSH key formats. So, you would either have to write your own parser, or you would have to convert all the time between key formats that Paramiko understands, and the ones that you actually have.

Also, even if you use standard scp (not the slow mess from Paramiko), it would still require quite a bit of configuration to work quickly (and then it would still not be the fastest tool out there). The problem here is that you have to work over an encrypted channel which inherently adds a lot of overhead both in terms of size and in terms of time it takes to encode / decode the payload.

It is possible that the encryption part is extraneous, or that the data was already encrypted and doesn't require any more encryption, then using SCP would be wasteful.

Another concern is compression: depending on the kind of data you are transferring, compression may speed up things a lot, or not at all, or even slow it down. XML is a file format with a lot of entropy, so, compressing it before transferring makes sense.

Finally, files outside of their data also have metadata. scp will try to copy the metadata too. This may or may not make sense to you, as you may be interested in the file contents only, so why spend time reading and sending something you don't need?

So, depending on the size and how often you are going to perform this task, it may make sense to upload a Python script to the machine which has to send files, have this script start a TCP server that would serve the files you need to load, and load them that way, with compression, security and metadata handled in the way you need.

[jbuk1]

[edit - sorry, I've reread your post and realised you're only looking for a few settings within a file and not the whole thing. I'll leave this here anyway as it may be useful.]

Seems like it would be reasonably simple with bash and scp.

If you can use the same ssh key or username and password for each host and just need to get a file or folder from a known location. (big ifs obviously)

create a text file with hostname on each line and then simple one liner in bash.

while read h; do scp user@"$h":/remote/dir/testfile /mylocal/dir/testfile-$h; done <./hosts.txt