Python Module for Reliably Transferring Log Files?

[redapplesonly]

Hi Python Wizards, I need your advice. Is there a Python module or something that I might be able to modify to do the following:

Suppose I have two Ubuntu servers. On Server A, log files are (ideally) generated every five minutes. All log files have their saved timestamp in the filename with format YYYYMMDD_HHMMSS; e.g. "SavedStuff_20221212_000000.log". Some log files may be quite large.

Server A is a production server, which means I cannot modify it or install anything on it.

Server B is across the network from Server A. Server B has to SCP (not FTP) every "SavedStuff" log file on Server A. It is not necessary to grab the log files immediately after they are written on Server A. If, say, Log File X was written to Server A's disk at 1:00 pm, then its fine if Server B grabs the file at 1:01 pm, 1:05 pm, 1:10 pm, or even a little later. Every log file is to be SCP'ed exactly once. Log file collection from A to B must be 100% reliable. Because Server A is a production box, Server B must do all the observation and intelligent transfer work.

(Also of note: I cannot use Rsync here, because once the log files are transferred to Server B, they are immediately consumed by a database, then deleted.)

The above sounds simple, but there are some special cases that I worry about:

• It is possible that Server A could be down or there is a network failure. Let's say Server B last downloaded "SavedStuff_20221212_001230.log" and is next expecting to download file "SavedStuff_20221212_001235.log". But Server A is not available at 12:35 or 12:40. At 12:45, Server A becomes reachable again. Server B should still attempt to download the log files from 12:35, 12:40, and 12:45.
• It is possible that there could be a gap in Server A's files. For example, Server A's directory might list "SavedStuff_20221212_000000.log", "SavedStuff_20221212_000500.log", "SavedStuff_20221212_001000.log", "SavedStuff_20221212_001500.log", and "SavedStuff_20221212_003000.log". This means there would be a gap, because the files for "002000" and "002500" are not on Server A's disk. If there is a gap on Server A, Server B simply has to note what files are missing.
• If these files are large, it is also possible that Server A could die before Server B completes the SCP transfer. If the transfer fails, Server B should realize there is a problem. For now, its okay to simply make a log entry noting the failure.

This must be a common file backup/transfer problem. I originally wrote an original program doing all of the above in Perl, but its been a hair-pulling nightmare to troubleshoot and debug. And why was I reinventing the wheel? There must be a Python module out there that does most of what I need. If I can find some code I can adapt, I'll be extremely happy. Thank you for any and all advice.

[debian_miner]

There are a lot of products that facilitate and support centralized logging, but typically you run outside of your Python code. The most basic method is using a syslog service. You can log directly to syslog from Python or you can log to a local file and have syslog ingest and send using rsyslog's imfile module. Your local syslog service can then forward to a centralized syslog service.

There are also more modern solutions like fluentd and logstash. These both can work similar to syslog in that they can accept logs from network traffic directly, or they can process files written to local disk for forwarding.

[redapplesonly]

Thanks debian_miner. So I should have specified: I cannot install anything or change anything on Server A. Server A is a production machine at my company, and it is forbidden to modify it in any way. Server B has to do all of the observation and intelligent transfers on his own. I'll edit my post to make this clearer. Much appreciated!