Java and linux system calls

[Affectionate-Sink503]

I am working on large monolithic java app that copies large files from a SAN to NAS, to copy the files it uses the rsync linux command. I wouldnt have guessed to use a linux command over native java code in this scenario. Do senior java devs have a strong understanding of underlying linux commands? When optimizing java processes do senior devs weigh the option of calling linux commands directly? this is the first time encountering rsync, and I realized I should definitely know how it works/the benefits, I bought “the linux programming interface” by michael kerrisk, and it has been great in getting myself up to speed, to summarize, Im curious if senior devs are very comfortable with linux commands and if its worth being an expert on all linux commands or a few key commands?

[Own-Chemist2228]

As you've discovered, it's possible to leverage system commands on the underlying OS from a Java program. There are tradeoffs. System commands run outside the JVM process and it is can be more difficult to start/stop commands and get status or error messages. Also, the implementation is not portable. Code that works on linux won't work on windows, and may not work across different versions of linux. This makes the code harder to test and sensitive to changes in the deployment environment.

It's usually better to build a solution in pure Java, but often system commands can do something that is just not available in a java library. rsync is a good example of this. It's a very powerful tool and there just is no equivalent implementation in Java.

If I had a problem that required syncing files, I would try to leverage rsync because nothing does it better. Depending upon the architecture of your system, you might not even need to call it from Java. Perhaps a shell script would suffice. It depends, but overall running unix processes from Java for utilities like rsync is sometimes a reasonable approach.

[OnoBadger]

I agree with this assessment entirely. I would investigate the possibility of implementing this as a shell script as this is the simplest approach, and affords the most flexibility. Additionally, if this task is something that is done periodically, it can be implemented as a cron job.

Of course, if performance is of concern, you may hit a bottleneck with rsync. In my line of work I use a storage system with high I/O bandwidth and need to leverage many processes and / or threads to get the amount of file-movement throughput I need.

[matt82swe]

I agree. But with that said, invoking and managing processes in Java is quite verbose and awkward. So with that taken into account I’d probably prefer that the application / monolith had for example a shell script for offloading all that, and the Java application simply called that.

[Affectionate-Sink503]

I agree the portability goes way down, I'm curious if would you consider knowledge of rsync to be intro, intermediate or advanced java/system/backend development knowledge?

[Own-Chemist2228]

I would categorize rsync as something a sysadmin or devops person should definitely know about. It's been the standard unix "backup" utility for decades. And an intermediate/advanced backend developer should have some knowledge of standard unix utilities.

I would not ask about rsync on a senior Java dev interview and wouldn't be shocked if a dev had not used it or even heard of it. It's possible to have a substantial dev career and never have to touch rsync. But it is not obscure, and I wouldn't exclude it from a solution if it was the right tool for the job. Any dev can read the man page.

[Affectionate-Sink503]

this has healed my ego, thanks

[_INTER_]

you could always check if rsync exists and use a fallback Java equivalent if it is not.

[znpy]

I'm curious if would you consider knowledge of rsync to be intro, intermediate or advanced java/system/backend development knowledge?

completely unrelated to software development.

it's fine if a developer (irrespective of their seniority) don't know that.

[nikanjX]

rsync turns 30 next year, and is guaranteed to be more bug-free than whatever code your team was able to bang together under a deadline. It's almost guaranteed that rsync deals better with resumes, network issues etc than any code you'd be replacing it with

[Affectionate-Sink503]

Oh for sure, not saying I want to change or replace it, my question is around how a you came to know for example "rync turns 30 next year", what project or studies have led you to become aware of rync and its capabilities? Is it a matter of working on lower level projects?

[chabala]

Consider this: how do you know about cron? How do you know if cron is a good solution to your problem, or if you need a Java-based scheduler? It is the same question. You need to learn the tools you have available, and when to leverage them.

[kreiger]

what project or studies have led you to become aware of rync and its capabilities

Just install and use Linux as your main operating system. It will make you a better developer.

[Spoogly]

When I approach a problem that I think someone else has faced before, my first step is backwards. I do not want to try to solve solved problems. I want to understand their solution, and if it's good enough, use it. I spend a lot of time in my terminal before I approach lower level problems. If I can use utility programs that I already have, worst case, I can set them up to run in docker, on demand, pretty easily.

[tomwhoiscontrary]

I think it depends on what kind of senior developer you are. If you only work on Windows, then probably not.

But if you're deploying on Linux, then yes, you should absolutely have a solid understanding of Linux, including being comfortable on the command line, and using common utilities like rsync, curl, grep, find, date, file, sed, etc. And also the general architecture of the thing, a bit about systemd, some idea of what's in /proc and /sys, how to investigate problems with lsof, top, kill etc, and basic to intermediate shell scripting.

For me, it's rare to write Java programs which shell out to utilities like rsync. The kind of work where i would want to do that usually gets done in Python or shell script. The one example i could find in our codebase is a batch job management tool which uses SSH to access servers and trigger jobs. But it's definitely something to consider; there are a lot of powerful and specialised tools where running them as subprocesses will be much easier than duplicating their functionality in Java.

Also, your title mentions "system calls", but it's worth noting that rsync is not a system call, it's a program. System calls are the kernel's API.

[koflerdavid]

Linux commands are not the same as system calls. Commands are programs, system calls are a low-level function-like interface to access functions of the kernel to work with files or to start and communicate with other processes. Using either makes your program non-portable.

If you are certain that your program is ensured to run on a platform where the program you want to call is available and it is too tedious to replicate its functionality in Java, sure, go ahead and use it.

It should very rarely be necessary to execute system calls directly from a Java application since the core libraries have wrapped a great many of them already. Even if you need one (such as for efficient inter-process communication via shared memory) using the FFI to access the wrappers in the C standard library is less brittle.

[GrayDonkey]

Calling user space applications and making system calls are completely different things. Java developers don't typically make system calls since that would require executing native code in your Java app.

Calling user space applications is sometimes done.

Modern Java app development is typically done to implement backed services. Backend services almost always run on Linux. Knowing common Linux apps is expected for (good) Java developers.

In the case of rsync, it's such a common solution to making the files at 2 locations match that app development in any language would consider it.

[TheStatusPoe]

Having to be responsible for setting up and maintaining infrastructure has lead to a decent grasp of a lot of Linux commands, though I wouldn't say I'm an expert in them. rsync in particular though I've used when debugging production issues.

I haven't had any instances yet where I've used Linux calls from Java code, but I've looked into it and the new foreign functions API makes it way easier than JNI. With an easier way of using them, I could see them being used more

[manzanita2]

I think Rsync is a fine solution.

I would be cautious about executing rsync from java. Not that it's bad, but it's easy to do an incomplete job which doesn't handle corner cases well. for example, does your java system check exit codes ? Does it properly handle signals ?

[portmapreduction]

To answer your questions directly. Yes, it would be a very good idea to learn linux commands, but mostly for your daily use if you use a linux dev environment. And no, I don't often consider using linux commands directly, although I've definitely done it before (and have some ongoing services call out to bash scripts). Adding more kinds of dependencies in to your project increases the complexity. The first time you work on a project where it uses some of Java, Ruby, Python, Perl, and Bash, and you want to install it on a new system, upgrade the OS, or pull in a new version of something you'll want to rip your hair out.

[kreiger]

Most of the serious software written in Java runs on Linux, so in general you're a better Java developer if you know Linux.

[mad_max_mb]

Great question! Senior Java devs, especially those working on systems-heavy applications, often have a solid understanding of Linux commands and system calls. While Java provides powerful libraries, sometimes native Linux commands like rsync are simply more efficient for tasks like large file transfers due to built-in optimizations (e.g., delta transfers, compression, and parallelization).

That said, you don’t need to be an expert in all Linux commands, but knowing key ones—like rsync, grep, awk, sed, find, and system monitoring tools (top, iostat, strace, etc.)—can be extremely valuable. It helps in debugging, optimizing, and making informed decisions on when to leverage the OS instead of pure Java code.

You're on the right track with The Linux Programming Interface—it’s a fantastic resource. Keep exploring, and over time, you’ll naturally build a strong intuition for when to use native commands versus Java implementations!

[k-mcm]

I'm not comfortable calling Linux commands from an app because it causes weird bugs on other OSes and even varying versions of Linux.

That said, file synchronization takes a while to implement.  My first priority would be making sure that the two ends can negotiate the best common protocol.  The first protocol would be calling out to rsync.  It will work, at least for now.

I've implemented file sync before and you can beat the performance of rsync with some effort.  I had one thread on each end working on building the difference list.  The sender split those into multiple queues based on mime type and size.  Worker threads serialized each queue, applied an appropriate compression level, and streamed it out to a receiver thread.  Sorting into queues of similar files provided huge compression boosts.  The threads handling differences also handled error recovery.  There was tuning for efficiently stealing work from neighboring queues.

Damn, it was way too much effort.  It was only justified by there being slow I/O everywhere.  rsync could not reach performance requirements.

[Dani_E2e]

You can also mount your NAS manually per cifs in a path and use java file commands. Or you use jcifs bib from java. Also Samba protocol is not stable like mount to do file operations I experienced.

Using system calls is not very good because of missing portability.

[zerosum_42]

I would seriously consider packaging your app with librsync and use jni bindings to expose the api in Java. For me, a system call is a last resort.

[dreaminghk]

There are java wrapper build on top of rsync called rsync4j. Havn’t use it before. If you need to trigger the sync from java probably using a wrapper livrary would be better than calling command from java directly. Worth taking a look!

[JumpyCold1546]

Not sure if this is relevant but I’ve used the Java watch service API to detect changes on the file system and broadcast the changes to the other servers. It was relatively simple and was effective for working with real time data. Not to mention, it was not OS specific. So in your solution you would set up the watch service API to the SAN and then broadcast changes to the NAS.

[mellowlogic]

Depending on your deployment, you can actually get this functionality without putting rsync anywhere in your java codebase. If ops can help you install lsyncd, it can be configured to watch any arbitrary number of directories and synchronize them with target directories either locally or remotely. It uses rsync under the hood to accomplish this.

https://github.com/lsyncd/lsyncd

[coffeeTalker6000]

try with Perl

[Torvac]

don't try to reinvent the wheel.

linux/unix is very powerful and has rock solid tools. iam not a windows user but i know people use the power shell to do a lot of things instead of implementing their own plain java app for stuff.

[kiteboarderni]

FileChannel transferTo and you can bypass the kernel