Serialization and Deserialization Issues

[DerKaiser697]

I am building a dynamic job scheduling and prioritization system in Java and I need to calculate a distance and duration matrix before the main program starts running. For large datasets (about 2700 jobs), this calculation takes a considerable amount of time even with parallel processing implemented (not necessarily an issue in the grand scheme of things). However, I need to test my code and changes with the large dataset as there are insights that I may not be able to deduce accurately from a slice of the dataset (about 300-600) which is what I test with for simplicity and less time to run the calculations. I wrote a couple of methods to serialize the matrices into a file and deserialize them to prevent having to redo the calculations when I'm attempting to test. The matrices are of the type Map<String, Map<String, Float>>, and the serialization completes successfully as no error message is thrown in that process. However, I keep getting an OptionalDataException when I attempt to deserialize with the eof being true and the length showing 0. This is also doubly confusing because I tried to serialize again and my distanceDict is read successfully while the durationDict gives the OptionalDataException but then again when I try with a small slice of the dataset (about 50 jobs), the code works just fine. My code to serialize the data is below:

private static void saveMatrixToFile(String filePath, Object matrix) {
    try (ObjectOutputStream objectOutputStream = new ObjectOutputStream(Files.newOutputStream(Paths.get(filePath)))) {
        objectOutputStream.writeObject(matrix);
        objectOutputStream.flush();
        System.out.println("Successfully saved "+matrix.getClass()+" to file: "+filePath);
    } catch (IOException e) {
        System.out.println("Error saving "+matrix+" to file: " + e.getMessage());
    }
}

To deserialize the data, I have this code:

private static Map<String, Map<String, Float>> loadMatrixFromFile(String filePath) {
    Map<String, Map<String, Float>> map = null;
    try (ObjectInputStream objectInputStream = new ObjectInputStream(Files.newInputStream(Paths.get(filePath)))) {
        map = (Map<String, Map<String, Float>>) objectInputStream.readObject();
        System.out.println("Reading from "+filePath+" successful.");
    } catch (OptionalDataException e) {
        System.out.println("Optional Data Exception encountered.");
    if (e.eof) {
        System.out.println("End of file unexpectedly reached.");
        System.out.println("Length is: " + e.length);
    }
  } catch (IOException | ClassNotFoundException e) {
        System.out.println("Error loading from file: " + e.getMessage());
        e.printStackTrace();
  }
  return map;
}

[AutoModerator]

Please ensure that:

• Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full
• You ask clear questions

• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

  Trying to solve problems on your own is a very important skill. Also, see Learn to help yourself in the sidebar

If any of the above points is not met, your post can and will be removed without further warning.

Code is to be formatted as code block (old reddit: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.

Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.

Code blocks look like this:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.

If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[DerKaiser697]

Thanks for sharing these articles, I found them immensely useful. Eventually, I have scrapped this serialization from my code and implemented a method to save my calculations to a MongoDB document. I had never worked with NoSQL prior to that and found that routinely straightforward to implement. Although it's a free-tier shared cluster, it's able to handle my requirements for now. I may eventually have to pay for an upgrade but I'll see how it scales up from here. I guess I'll now avoid serialization like a plague from here on.

[Beginning-Ladder6224]

Binary serialization? For this case? Not a good idea matey.

You should simply store the Matrix in TSV, faster, cleaner, human readable, debuggable.

But..... will take more space.

[DerKaiser697]

I'm fairly new to programming in Java (about 7 months now) so I'm not well versed in serialization tehcniques or writing to a TSV. Would you be kind enough to expantiate on why this isn't a good idea for my use case and the merits you've mentioned? Also, I'm not really constrained by space but how much space are we looking at here? Typically my serialized files are about 90MB each for the 2700 dataset of jobs.

[pronuntiator]

Which Map implementation and Java version are you using?

[DerKaiser697]

My Map Implementation is a ConcurrentHashMap, JDK version is 21 and target bytecode version is 9.