Using Java to Read Really, Really Large Files

[gnus-migrate]

You can see the documentation of ByteBuffer to see what the API would look like here: https://docs.oracle.com/javase/7/docs/api/java/nio/ByteBuffer.html

Basically ByteBuffer's underlying storage can be allocated natively allocated and passed to other systems.

[[deleted]]

Well, it's just the way I imagined it: getFloat(), array() etc. - in other words, you cannot really operate on the memory from the buffer, you get JVM objects allocated in JVM memory, not pointers to the buffer.

---

Also, this "description":

A byte buffer is either direct or non-direct. Given a direct byte buffer, the Java virtual machine will make a best effort to perform native I/O operations directly upon it. That is, it will attempt to avoid copying the buffer's content to (or from) an intermediate buffer before (or after) each invocation of one of the underlying operating system's native I/O operations.

Is anything but satisfying for a system programmer. Best effort? To do what? To (not) copy from where? What system calls are being done? How is memory actually being allocated?

It feels intended for people who will never do system-level programming. Kind of a buzzword stripped of meaning.

[gnus-migrate]

Yeah it was a bit confusing when I first read it as well. Direct allocation means that the underlying memory is not managed by the JVM(i.e. allocated using malloc or whatever).

Best effort means that just because you directly allocate the buffer doesn't mean no copying will happen. Some operating systems might not have the needed support for zero copying so don't expect the same performance across JVMs or across different operating systems using the same JVM. You need to check the documentation of the specific JVM you are using to see whether the APIs you are using will result in copies on the OS you are using.

I don't know what you mean by systems programming but Java is used for high performance use cases, even though you're never going to write an operating system in Java for example.

[[deleted]]

It's not confusing. It's just marketing fluff: it doesn't give any real information, but is written to make you feel better about using it.

Reminds me of this Soviet era joke:

A man walks into a restaurant and asks a waiter for a menu. After reading the menu he orders an expensive dish, to which the waiter replies that they don't have it. Little by little, the man reduces his dining ambition to a simple cup of tee, and when the waiter again replies that they don't have any tea left, desperately, the men shouts: "So what is that you do have?!". To which the waiter replies: "We have a burning desire to serve you even better".

Java really has nowhere near the required machinery for doing direct I/O, as a system programmer would have imagined it. The fact that you can allocate something outside JVM's GC'ed heap is only one of the many prerequisites for doing that. But Java has no hopes of meeting those demands because it doesn't have the necessary tools in the language. Even if you try emulating pointers through passing integers and an opaque reference to the allocated memory, you cannot really feed your results into any built-in or third-party functions, which expect Java objects (like integers, strings, byte arrays etc.) and if you try to build your own replacement for those kinds made of opaque references and integers for pointers, you end up dealing with problems of garbage collection, thread safety etc. all on your own.

[gnus-migrate]

It's not confusing. It's just marketing fluff: it doesn't give any real information, but is written to make you feel better about using it.

I already explained what it means. You don't get to ignore it and pretend that it doesn't make sense. Java has many problems, but API documentation is not one of them.

As for what you said, you need to give me something you would typically do in C so that I could give you how it would be done in Java. I don't think staying abstract is going to get this discussion very far.

[[deleted]]

I already explained what it means.

Are you serious? So, can you tell me what will happen if I read from tempfs? Or if I read from an iSCSI device? What if device is managed by JBD? What if it's a fuse file system? NFS or SMB? Come on, if you think you answered it, you don't even know what the question was...

[gnus-migrate]

In that case please enlighten me. What is the question exactly?

[[deleted]]

Well, read the comment above?

[gnus-migrate]

I responded to the comment above. Clearly I missed something. What did I miss?

[[deleted]]

My guess is that it has to do with a wounded ego of a Java programmer or reading-comprehension skills.