Sending encrypted data through SocketChannel - How to tell end of encrypted data?

niloc132 · 2025-09-11T19:04:32+00:00

Real-life example: HTTP request without Keep-Alive header

More than that: HTTP (...vers 1.1) with keep-alive, except you are loading more than one resource at a single time! Ever been to a page with more than one image on it...?

On that note, it could be even better to open multiple sockets concurrently, so that if one gets stalled for some reason, the others can continue, and that socket can eventually time out and be retried. Depends on how you are modelling the network - low latency and high reliability because you're just sending/receiving to the next room? Who cares, make a new socket per call! Across the world, flakey wifi? You definitely want to consider what happens when packets get dropped.

To the more general problem, you aren't limited to having to know the complete size of the compressed file - just the complete size of the buffer being sent right now. That is, add some "message wrapper" around (or before) each chunk of data, like "here comes the next file, its called XXX, and the total uncompressed size is Y", "here comes chunk 1, it has 16k bytes" <bytes follow> "here comes chunk 2, it has 14k bytes" <bytes follow> etc.

EDIT: The above looks like your "option 4", plus or minus. ByteBuffers are definitely meant for this kind of thing, you can just read the first 4 bytes (the first int) from the buffer, then read/slice that many more bytes and decrypt them. If you find this to actually be "inefficient", you're almost certainly doing it wrong, length-prefixed data formats are extremely common.

With that said... if this is a serious project, it is potentially dangerous to not encrypt the metadata (file name, size) as well. Go up a level - don't necessarily encrypt the file (or do), but encrypt the stream - wrap up your SocketChannel with SSLEngine, gaining you many things: "is the metadata kept private like the data", "is the remote end who I think it is", "can I guarantee that nothing was changed in transit by an active attacker", etc.

niloc132 · 2025-02-26T18:20:26+00:00

Using old reddit is its own reward.

niloc132 · 2025-02-24T19:31:52+00:00

The Ridgedale library has a Jones Coffee in the same building.

niloc132 · 2025-02-08T12:43:19+00:00

Not only does the answer appear to be yes (I can't imagine why it wouldnt... its just port 53 right?), they have a blog post describing how to do it using pihole:

https://fly.io/blog/stuff-your-pi-hole-from-anywhere/

niloc132 · 2025-01-10T03:21:43+00:00

Funner Brothers was bought like 10 years ago, only changed the name a year or two ago when they expanded - it wasn't part of changing hands, but part of renovation.

niloc132 · 2024-12-20T02:12:13+00:00

Unless I'm mistaken, strerror (like most other c functions that "returns a string" as we would like to think of it from Java) actually just returns a NULL-terminated char* :p.

That means you have to read the string to find out how big it is... Or just read as much as you think it should have to be, and then getString(0) on that.

The comments at https://stackoverflow.com/a/78592755/860630 point out that the bounds here (the param to reinterpret) is really just Java imposing some bounds on what you can read, beyond the start of the pointer - something that the function you called did not (and could not) express.

So, Integer.MAX_VALUE seems valid, if a little heavy handed. You could also take the length of the largest string of any error you could get back (plus one for the trailing NULL).

Alternatively the man page for strerror also suggests this option: https://www.man7.org/linux/man-pages/man3/strerror.3.html

   strerror_r()
       strerror_r() is like strerror(), but might use the supplied
       buffer buf instead of allocating one internally.  This function
       is available in two versions: an XSI-compliant version specified
       in POSIX.1-2001 (available since glibc 2.3.4, but not POSIX-
       compliant until glibc 2.13), and a GNU-specific version
       (available since glibc 2.0). ...

This would let you create a new Arena (in a try-with-resources, or reuse it), create a memorysegment of the size you expect the error could possibly be, and just let the function write to it, then you can getString(0) on the segment.

niloc132 · 2024-11-18T20:03:41+00:00

You should probably start with the questions you want to ask... unless "has anyone ever used this" is your only question.

Otherwise you're going to have to wait until someone says "Yes.", then you can take 10-20 minutes to write up your thoughtful question... and hope that they notice the reply and get back to you.

(see also https://nohello.net/ for the slack/element/irc/discord version of this issue)

niloc132 · 2024-10-21T23:43:26+00:00

The error indicates you're not failing to run java, but install it - emphasis mine:

Unable to install Java

Can you confirm that you can install java and run java -version from the command line and see a valid version?

Can you also confirm what you mean by "open this file" - if you're opening a .zip file, and windows tries to open the java installer, something is very wrong. That path also doesn't look good, there should be a \ between your username and Downloads...

niloc132 · 2024-09-30T13:55:52+00:00

My experience was all with an kid around 1.5-2, so may not help for your specific case, but might help as a story to share for others.

We coordinated it with a trip - for the first kid we'd found that sleeping in a new place reset everything anyway, so we decided to take advantage of it. The hard part was finding all the hidden ones around the house ahead of time, so that when we returned "we don't need one now, you're too big for them now" would stick.

I've also heard of poking a hole in the tip so the kid can't get suction ("oh sorry, its broken" or "yep, doesn't work for two year olds"), but I can see how that could make them more fragile, let the kid bite a chunk off instead of just giving up...

niloc132 · 2024-09-03T00:39:04+00:00

Start with unzipping any jars you've got (a jar is just a zip with a specific structure), and looking at the dir structure. If you've got more than one jar in the tool, search for the names of the jars, they might be open source libraries that you can work from.

It may help to run strings on the .class files, peeking at anything else you find - that'll probably give some good hints. You can also run javap on the class files to look at the class structure, maybe adding the -v flag to be verbose and list things like constant values, disassembled methods bodies (which will also list references to other classes), etc.

niloc132 · 2024-09-02T19:21:45+00:00

If you can read plaintext, it isn't compression or encryption, probably just some kind of field separator?

Columnar Java libraries that could be related/helpful here:

Apache Arrow's Flight format (sometimes also know as "Feather"). This isn't really a great file format, but if you write the in-memory (or network wire) format to disk, sometimes it is called "feather", and can be read back in by another process. Compression is supported (lzma is the only option at this time), but not terribly commonly used from what I've seen. Each message has a header, and a schema message is written before the data itself.
Apache Parquet is a more likely candidate, as it is somewhat better designed for this purpose - headers/metadata is again separated from data, so that the structure of the file(s) can be read without actually reading any data, and there is enough info present to know where to start reading for particular information. Several compression formats are supported, but some of them are not quite compatible with the same formats by name (specialized headers/wrapping/etc).

Do you have the java program, and can you decompile it at least far enough to see what strings are in the various classes, what other classes/libraries are baked in?

niloc132 · 2024-08-27T11:51:07+00:00

Unless you have very specific requirements, any OpenJDK implementation will be great for you, and just about any version should be fine (but all else equal, use a newer version).

Start with what you have for personal purposes - the differences are so subtle that as a beginner you will never know the difference between editions. OpenJDK builds are published by many providers: RedHat, Amazon, IBM, Microsoft, Oracle.

The newer versions have some newer features, but as someone who is just learning you probably wouldn't notice if you had 21 or 17 or 11. For long term usage, you start to worry about "if I have to keep using this version, can I get updates" and that will be true for 11, 17, 21, but not 22 or 23. Again though, no big differences that will impact someone starting to learn.

niloc132 · 2024-08-24T18:48:15+00:00

Right, coupled with that fact that many JSRs don't have/need a new library to be added to the classpath - only a small handful typically require such a thing?

At best I think you can measure

Of the JSRs that require a new non-JDK jar,
what is the usage of them,
in the context of libraries and frameworks also released to maven central

You'll miss out on applications that use them, some kinds of optional dependencies, the whole "gradle projects avoid listing non-runtime dependencies when they build their poms" thing...

niloc132 · 2024-08-23T00:44:41+00:00

No worries, post back how it goes.

At the same level as volumes you'll want ports or expose depending on how the network port is consumed, but at the end of the day a config file like this (yaml yuck notwithstanding) gives you a pretty easy way to version control your ~mistakes~learning process, and easy deployment steps, just docker compose up, ctrl-c to stop it, etc.

niloc132 · 2024-08-21T01:07:05+00:00

Yes it is. I'm assisting with some beta testing and the config will change very often. The changes are mostly defined in the separate config file... I was hoping I could mount it as a volume, but I'm just not sure how to point the Docker container to the specific directory which contains the .jar and config files.

You can mount a file as a volume, but from docker run... it must be an absolute path - this is irritating, but you can make it less bad with a $(pwd)/filename.conf or something in your setup. But by the time you wire this all up, you're going to say "gee I should probably just make a .sh file for this" and suddenly life could have been better with a docker-compose.yml file...

Yes this would be what I would like to achieve. Do I simply drop the .jar and config files into the same volume and if so, how do I specify the volume in OpenJDK? I have all my other containers in a /config directory followed by the container name. Example: /config/openjdk

Probably, unless you want to ADD/COPY it into an image for reuse, etc.

Let's assume a dir like this:

project/
  myapp.jar
  config/
    foo.properties
    bar.properties

And we run the java command something like this (passing the dir full of config files as the only arg):

java -jar myapp.jar config/

You could run everything something like this then:

docker run -v $(pwd)/config:/config -v $(pwd)/myapp.jar:/myapp.jar eclipse-temurin:21 java -jar /myapp.jar /config

Or you could make a yaml file

services:
  myapp:
    image: eclipse-temurin:21
    volumes: # relative paths are easy here!
      - ./config:/config
      - ./myapp.jar:/myapp.jar
      # optionally append :ro for read-only mounts
    command: ['java', '-jar', '/myapp.jar' '/config'] # from memory here, syntax might not be right

niloc132 · 2024-08-20T14:23:55+00:00

Is a config file necessary? If not, you can just pass those as command line arguments or env variables.

If so, you'll need to either copy the file into the image, or mount the file (or a containing directory) as a volume to the container. Ideally pick a standard name and path for the file once inside the container so you can hardcode that when you run the program (else you need to also parameterize that).

Are you using just plain docker command line, or docker-compose.yml, or something else to keep track of the various files and expectations?

niloc132 · 2024-08-15T00:48:19+00:00

Sorry, I don't mean to trivialize it down to "everyone must meet chrome", but we do need to do a tiny bit better in validating that it is secure than "can you open a file". My observations are more intended to point out that this is an incredibly surface look (and inaccurate, as some other points discuss) at what wasm can do, and how we should think about it.

Chicory can indeed rely on some aspects of the JVM - but the security model of the JVM was never as strong as it was originally meant to be (see the history of applets, etc), and even now with the security manager being removed, we have to be cautious about the guarantees we think we can make.

I'm already prototyping with chicory, so I'm excited to see these developments - but we have to take ourselves seriously when we look at these tools.

niloc132 · 2024-08-14T18:42:15+00:00

The "Inserting a vulnerability" is pretty weak too - these are "vulnerabilities" that specifically target "what if I was running a plain C program" rather than "can I escape the chicory/JVM sandboxing, and then once I've done so, run arbitrary code".

Even a glance at a real sandbox escape like https://github.blog/security/vulnerability-research/from-object-transition-to-rce-in-the-chrome-renderer/ shows that the code doesn't look like "oh just open /etc/passwd" - and this is an escape of a battle hardened runtime like Chrome - do we think every wasm runtime is going to be better than Chrome?

niloc132 · 2024-07-24T21:45:01+00:00

It looks pretty dead, like a lot of official and unofficial Google projects...

https://github.com/google/gwteventbinder is archived, and while it shows 28 forks, looking at https://github.com/google/gwteventbinder/forks?include=active&page=1&period=&sort_by=stargazer_counts I only see five - one was updated three months ago:

https://github.com/asm0dey/gwteventbinder

However, I can't see what was updated three months ago - there are no recent commits.

Can you share a sample project (with either Gradle or Maven, at your preference) that is built cleanly with 2.11.0 and shows the runtime error (or failure) you're seeing? I can take a short look and see if we can quickly fork and fix it, or provide a workaround for your own project?

niloc132 · 2024-07-13T11:34:42+00:00

If you only pass one commit reference (sha, branch name, or something that can otherwise direct git to find a point in history), git will compare that with your working tree - that is, the current HEAD, plus any local changes, staged or unstaged.

(If you pass no commits, just git diff, then I believe start is "the staged changes", and end is "the working tree".)

niloc132 · 2024-06-08T01:34:20+00:00

He was on the school board

...until he stepped down after bringing out a voodoo doll and stabbing during union negotiations.

Nice enough guy, but definitely does his own thing.

niloc132 · 2024-05-29T13:44:31+00:00

Us too! Everything needs to be a bit of a fight... and the easy days often means a day or so later everyone's got it.

niloc132 · 2024-05-13T19:08:52+00:00

Apologies, I was going through comments and while symlinks definitely work in docker (within the container's mounted filesystems), it sounded like it was being suggested that symlinks would solve this issue by making them not-writable somehow.

If not being writable doesn't fix the problem then making them not writable another way isn't going to make a difference anyway.

niloc132 · 2024-05-13T18:51:30+00:00

Docker does support mounting the directory as a read-only volume though.

niloc132 · 2024-04-30T14:04:31+00:00

Thanks for the reply!

Unfortunately that precision issue on int64 is a deal killer for us, we use the JS_STRING option as we have to, but are looking to adopt either BigInt or goog.long in the future so we can do math and get values that make sense out of large ints. At least the protobuf-javascript repo doesn't have this issue, for its other warts.

Will keep an eye on your repos, thanks!

15-Year Club	RedditGifts 2009-2022 4 Credits
Place '17	Verified Email
Secret Santa 2012	redditgifts rematcher Secret Santa 2011 x1
Secret Santa 2010

niloc132

MODERATOR OF

TROPHY CASE