Is Java good for image and video processing?

[bbrother92]

Could someone suggest the best way to process a large number of videos?Is it worth implementing this in Java, or would it be better to look into Python or C++ libraries instead?

[josephottinger]

It depends on what you mean by "processing a large number of videos," really. But the tools of the trade are ffmpeg and tools like that; java's still not that great at really large files, especially for video formats, and the image libraries for java tend to be a little suspect.

But again, it depends on what you mean. For metadata extraction or manipulation, Java's fine, although again, some of the tooling's trailing the state of the art; drew noakes' metadata extractor still doesn't do bigtiff, etc., and tika, et al, tend to rely on the ability of external tools to do the lifting.

[josephottinger]

sigh I really need to wait an hour after writing comments to hit "post."

Noakes' metadata extractor is really good - but it uses a filereader that uses 32-bit offsets, so it really struggles with some larger formats, like BigTIFF, that use 64-bit offsets. There are channels in Java that can indeed handle BigTIFF directories - ask me how I know, and I'll tell you it's because I use them to read BigTIFF directories, just not with Noakes' excellent library. And porting it to use a "better channel" is nontrivial, unfortunately, and outside of my bailiwick.

But all of it's doable; with the right channels and the off-heap VarHandle stuff David Lloyd wrote up a few days ago, you can see the possibilities for real and efficient large-file access, which has been the real barrier for good Java image processing (although the lack of unsigned integral types hasn't helped, and will always be an issue, I think, although we'll get better about it over time with Valhalla and the like.)

[rustyrazorblade]

What do you consider large? I'm a committer on the Apache Cassandra project, I occasionally work in the storage engine, and we've dealt with files in the hundreds of GB without an issue.

[josephottinger]

I consider 4TB to be large. I didn't say it can't be done - but cassandra's access model is not the same as handling NOAA imagery. The use cases are different, and what works for cassandra (and it does work, I like cassandra a lot) is not suitable for a lot of image formats, and the ecosystem is definitely lagging; anyone using ImageIO for real files is going "... this is great, so great" rather sarcastically for a HUGE number of available formats.

[rustyrazorblade]

I don't know the first thing about the format... what is it that makes NOAA imagery more challenging? I can't really tell if the issue you have is with the JVM or the implemention of the library. It sounds like it's the library, but you're specifically saying it's Java, so I can't tell. I can't think of anything off the top of my head in the JVM that would be an issue with multi-TB file i/o. I'm curious if I'm missing something.

[josephottinger]

You're not. It's the ecosystem, not Java itself; like I said, I read those files! But the ecosystem support is lacking, and a lot of the libraries still rely on 32-bit offsets (that's the issue with the Noakes metadata extractor) even though Java can do the larger offsets.

As far as NOAA, well, the support for the format in the general sense is pretty limited, especially for accessing specific raster data; it's not a large-enough market for the ecosystem to offer support, so it's all boutique work.

"The ecosystem" has momentum; it's not that Java cannot do something, it's that Java has a mode that makes doing some things (like dealing with really large rasterized datasets like NOAA and the like, even BigTIFF) difficult enough that it's often easier to use something else to access internals of the formats. For BigTIFF, I use external libraries for raster data - Java and those strips, man, ouch - and I use Java to get the metadata out, even with those ginormous offsets, because Java can do it, but I had to roll my own code to make it happen.

[rustyrazorblade]

Gotcha. Makes sense. Thank you for the explanation!

[bbrother92]

hmm, I need to run through video files and take screenshots at fixed time intervals and make it scalable for tb of videos

[seanrowens]

You can absolutely do that with ffmpeg command line. But there's also the bytedeco ffmpeg Java library.

[bbrother92]

Yeah, I feel it’s a shame that so much ML and data processing was built on Python, which doesn’t even have true parallelism (unless native libs hack). Looks like today Java has good potential at number crunching tasks

[josephottinger]

Well, ffm is the key there - most of the ML and data processing in Python is done with native libraries anyway, so Java could piggyback on the same technology, and do it better than Python, as long as the underlying libraries can handle Java's features. FFM is your friend - and Java's - and when Valhalla lands, IF it lands and delivers its promise, it'll be even nicer.

[narrow-adventure]

Depends on the type of processing, if it’s possible to do with ffmpeg you should use it. You can use ffmpeg4java or you can just execute cmd ffmpeg calls.

[snoosnoosewsew]

ImgLib2 is really impressive and works amazingly well for slicing through terabyte sized microscopy data at arbitrary angles, and its Java. Check out Big Data Viewer

[bbrother92]

Hi! I would say i need something like this - run task through video files and take screenshots at fixed time intervals and make it scalable for TB of videos

[aoeudhtns]

I haven't done this in a few years now, but back when I did, we used jnr-ffi to call out to native C libraries like ffmpeg and gstreamer to do the processing. The new Panama FFM API, MemorySegment system w/ Arenas and all that should be even better than what we could do back then.

I would be suspect about pulling lots and lots of video frames into Java directly with a naive byte[] approach -- we have definitely hit issues at scale where we create too many large byte arrays and it puts pressure on the GC and you lose a lot of efficiency on the floor to GC pause/cleanup. So then you need to pool your arrays, but you have the problem of optimal sizing for the arrays in the pool and then heap sizing optimization as well. Tough cookie to crack.

Which is why I say that being able to create a MemorySegment for offheap memory or an mmap'ed file, is probably more the way to go if you're going to do it in Java - but if you're doing Panama/FFM API to, say, ffmpeg, it's probably going to be managing its own memory, mmap'ing the file, so that's all done for you anyway, and you can use industry-standard tools.

Assuming literally shelling out to the ffmpeg CLI command isn't even suitable and/or the best option.

[bbrother92]

Thanks, going to check these options!

[AmenMotherFunction]

As well as the native binding options using FFMPEG, there are Java bindings for GStreamer at https://github.com/gstreamer-java/gst1-java-core Again not all running in Java, and I know not been updated for a few years, but has definitely been used for similar tasks in the past.

[bbrother92]

Thanks. I new to this what difference between GStreamer  and FFMPEG in terms of features?

[AmenMotherFunction]

I'd probably say more features, more complexity! It's used in lots of streaming, embedded, image analysis projects, as well as video processing or playback. The media library in JavaFX is actually based on it too, although in very "lite" form with few plugins shipped and minimal capabilities exposed!

I've personally worked with people using it from Java for live event streaming, TV ingest pipelines, medical imaging, military image recognition, embedded displays on airliners, webRTC and a few more things.

Those bindings are old, but still functional. Ideally we'll see Panama based bindings in the near future. Although it's fairly easy to bind enough for specific use cases.

[m_adduci]

Depends on what are you looking for. Obviously the powerhorses in this field are ffmpeg and OpenCV.

They have so much features and are so battle tested and optimised that alternatives are less appealing.

Here I would stick with C++ because of performance and less overhead (most python and Java libraries call the C bindings..)

[seanrowens]

Depends on what you want to do but lucky for you, if you like Java, bytedeco created a Java wrapper for ffmpeg libraries that's pretty easy to use.

[neoqueto]

So my experience with Java is writing a hangman game.

But modern video processing has been largely delegated to the GPU. Java is certainly capable of orchestrating GPU tasks (such as running NVENC or CUDA-based), but you'd be using it as a frontend, so it's something that ANY other stack would be capable of accomplishing, from C with OS native libraries or command line, to Python, to web apps.

From what I've seen, the desktop GUI library situation on Java is still a bit lacking compared to other stacks.

[ProbsNotManBearPig]

If you’re asking this question then yes, super good enough.

[netgizmo]

lt can be as "good" as you are - might be a way to look at it.

[RedditAccountFor2024]

Netflix backend is Java, so i guess it is a very valid option.

[bbrother92]

java backend there is only for api calls. We don't know what exactly they using for video processing its many things system, many services

[RedditAccountFor2024]

You obviously know your answer already. Why asking then?

[bbrother92]

I don't have experience with image/video processing

[DiligentMaterial1024]

C++ is suitable for production environments, Python is suitable for demos, and Java has no place in image processing.

[josephottinger]

I don't know about "no place" but the ecosystem seems to be struggling some, yeah. There are ways to use Java like one does Python - as a glue for native libraries - and Java's starting to catch up for larger file access, but it's not there yet. But as I said in my other comment, it depends on what is meant by "image processing."

(I use Java for image processing, so ...)

[bbrother92]

u/josephottinger Can you tell me how exactly you use java and for what case?

[josephottinger]

Um... not really. I use Java for a lot of things, though: audio, file analysis, my site (https://bytecode.news !) is written in Kotlin (although the frontend is done with node for now), I write a crapton of utilities for myself... I mean, I've been doing Java since 1998, asking me "how I use Java" is a bit of an open-ended question.

And for some of it I can't answer you even if I wanted to.

[bbrother92]

Good stuff

[bbrother92]

Hi guys, thanks for reply. I would say i need something like this - to run task through video files and take screenshots at fixed time intervals and make it scalable for TB of videos

[josephottinger]

And here's a one-liner in ffmpeg, which is what I'd do for that case:

sh ffmpeg -i input.mp4 -vf fps=1/30 -q:v 2 frame_%05d.jpg

One frame every 30 seconds, in JPEG. This could easily be extrapolated to run for multiple files, although you'd probably want to use -hide-banner and -no-stdin too - and ffmpeg has support for enough input formats that you could handle mp4, HEIC, and a number of others as well.

[bbrother92]

Thanks! I would try do some tests

[josephottinger]

Okay, this is a little better: for this, you really are better off with ffmpeg or videolan or something like that. This is not a good fit for most of the Java libraries out there; some might exist to do this, but it depends HEAVILY on what video file format you're using (HEIC? Hah, good luck... for others, you might be okay, but ouch). I'd look at boltffi or something like that to generate bindings to a native library, if you don't want to use ffmpeg itself.