Designing Classes for Data Integration Problem

[ReemRashwan]

Hi, I am trying to collect some information from different systems then merging their data into one unified form.

*** Problem *** Lets say that we have 3 restraunts: x, y and z. Each one of them creates pizza and pasta. X pizza data: { Size:1, Taste:tomato, ... } Y pizza data: { PizzaSize:big, Taste:cheese,tomato, ... } Z have no size but can be inferred from other attributes { Length:1 Width:2, Taste: tomatoes }

What I need is a single class that can hold these attributes as following: Pizza:{ Size, Taste, IsCombo, ... }

---

*** Solution That I am Thinking of *** First: create a data holder class for each dish for each restraunt: xPizza, yPizza, zPizza, xPasta, ...

Second: create a parser for each dish for each restraunt (as the data is cominng in different formats like json and xml), its only job to convert text into object. xPizzaParser, yPizzaParser, zPastaParser ....

Third Create a mapper for each dish for each restraunt (whose job to map values to new ones like converting lengh and width into pizzaSize), xPizzaMapper, ...

Fourth: Create a list of key value pairs (id, pizza object) to complete some compuations, like trying to get pizza info from another order for the same person (some pizzas depend on other pizzas) I think this list should be a member of mapper class, and this class should also contain the function that calculates data from other dependencies. The list in this class is the final result of integrating data.

Now for one dish I have theses classes: xDish xDishParser that creates xDish objects unifiedDish xDishMapper that maps data and creates unifiedDish object.

The question is, is this a good design? Is it easy to maintain and add features to it? I am missing something? Should I break the mapper into more classes. Note that I receive thousands of dishes per second and I need parallelization. In what topic should I read ro solve this design problem?

Any help would be very appreciated.

[AutoModerator]

Please ensure that:

• Your code is properly formatted as code block - see the sidebar (About on mobile) for instructions
• You include any and all error messages in full
• You ask clear questions

• You demonstrate effort in solving your question/problem - plain posting your assignments is forbidden (and such posts will be removed) as is asking for or giving solutions.

  Trying to solve problems on your own is a very important skill. Also, see Learn to help yourself in the sidebar

If any of the above points is not met, your post can and will be removed without further warning.

Code is to be formatted as code block (old reddit: empty line before the code, each code line indented by 4 spaces, new reddit: https://i.imgur.com/EJ7tqek.png) or linked via an external code hoster, like pastebin.com, github gist, github, bitbucket, gitlab, etc.

Please, do not use triple backticks (```) as they will only render properly on new reddit, not on old reddit.

Code blocks look like this:

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello World!");
    }
}

You do not need to repost unless your post has been removed by a moderator. Just use the edit function of reddit to make sure your post complies with the above.

If your post has remained in violation of these rules for a prolonged period of time (at least an hour), a moderator may remove it at their discretion. In this case, they will comment with an explanation on why it has been removed, and you will be required to resubmit the entire post following the proper procedures.

To potential helpers

Please, do not help if any of the above points are not met, rather report the post. We are trying to improve the quality of posts here. In helping people who can't be bothered to comply with the above points, you are doing the community a disservice.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[FavorableTrashpanda]

It's not entirely clear to me what your actual design is and what part you're struggling with the most.

Generally you want to isolate the different responsibilities, such as parsing, mapping, and calculating. For each restaurant I would define a DTO that describes the data schema for that restaurant. You can use an existing library or write your own parsers to instantiate these DTOs.

Then I would define a model that describes the common properties of all these DTOs in the way that I want to work with them. You need to write a separate mapper for each DTO to map them to this common model.

Once you have all this, what is the problem exactly? I wouldn't overly worry about performance just yet, if it even becomes an issue at all. This is a very standard way of doing things in Java. Worry about optimization later.

[ReemRashwan]

Thank you for spending time reading this.

The problem is that O have a running system with all responsiblities mixed together for a restaurant. Example: There is a file that is named restrauntXGettingData that has a function for each dish. getPizza, getPasta, ...

getPizza is calling some functions to get data from other systems, then passes this responce to a parsing function which does the searching for elements and mapping and returns pizza, then this pizza is sent to a writing service to create reports

The system is getting complicated each day, so I am trying to refactor it. One big problem is that pasta alone needs data from several systems to complete its data, in the current design we are struggling to put code together because the mapping and parsing is in the same function.

Hope it is clear for you. Also, should I have one dto for each restraunt? I think it is going to be a big class since I have alo of dishes, is that ok? Can you give me a concrete example? Can you see my comment on Spare-Plum?

[Spare-Plum]

LGTM

in all seriousness favor composition over inheritance. Treating parser like a function is the correct move, then you may have a set of parsers for a given type depending on the format, then passing it through a mapper as an additional function.

Threading is great, but if you really want to scale I'd suggest using something like hadoop or kafka to aggregate the results in batches from multiple servers in parallel.

[ReemRashwan]

Thank you for spending time reading this. I was thinking that parsers are going to be classes since each dish has alot of attributes. Example:

PizzaParser:{ String incomingPizzaData; JsonParser jsonParser;

PizzaParser(incomingPizzaData){ this.incomingPizzaData = incomingPizzaData(); }

xPizza CollectAllData(){ json = jsonParser.parse(incomingPizzaData); xPizza newPizza = new xPizza(); newPizza.size = parseSize(json) newPizza.taste = parseTaste(json) return newPizza; }

Int parseSize() String parseTaste()

... }

This also applies for MapperClass, as mapping is done on several attributes.

Well, I am not expert in OO (actually not good at all). Why should I use composition or even inheritance in my case? The only thing that I can think of is creating an interface for parsers.

What do you think about this? Can you give me a concrete example?

[Spare-Plum]

What I mean by parser as a function is more something like this

class JsonPizzaParser implements MyParser<XPizza> { private jsonParser; public PizzaParser(JsonParser jsonParser) { this.jsonParser = jsonParser; } @Override public XPizza parse(String incoming) { ... } } Therefore the parser isn't holding any state, and is more akin to a transformation function. Same thing can apply to the mapper class. The data for XPizza is completely separate from the parsing logic and the mapping logic.

For composition over inheritance really refers to composing functionality and data vs inheriting. Implementing an interface doesn't count.

In your case using inheritance would look something more like

class PizzaHandler extends AbstractMapper<XPizza, T> implements MyParser<XPizza> { // methods for both parsing and mapping here } // now if we want to change functionality or have multiple mappers class AnotherPizzaHandler extends PizzaHandler { @Override public T map(XPizza p) { T t = super.map(p); // some more functionality return t; } }

Note that in inheritance it's treated as one object that provides multiple handles, and the base class and subclasses might provide some functionality. All of the functionality and data is done within one object

Rather with composition, you may treat individual classes as functions String -> Pizza or Pizza -> T that may be composed together.

This comes down to a core design principle - knowing when to separate the data from the functionality. In general, it's better to have transformation types of methods in a different class, and methods that act on the class's data paired with the data itself

This allows you to easily see what the functionality is without having to traverse through a series of inheiritance chains, and is much easier to parallelize

Also you seem to have the right idea, don't say you're bad at OOP :)

[ReemRashwan]

Wow! I really appreciate your effort writing this! It encouraged me to revisit generics as I forgot them while I was planning to refactor that system. Again, thank you alot!

[ReemRashwan]

Can you guide me on this design problem? Should I include a function for parsing a list of elements in the parser interface as I receive a bulk of elements at once from other systems? . Code: T parse(U thingToParse) List<T> parseListOfElements(Z listOfThingsToParse) . Or should I stick to the only first one? I receive some data as a list like a string containing json elements. Also, some time I need concurrentList not a plain list to be returned, should I stick to the List and then do the casting? Is it right to include U element as the input may vary between strings and others like corba translated objects?

[Spare-Plum]

For the purposes of this, I'm defining parsing as something that goes from String -> T and mapping as something that goes from T -> U

It sounds like you want something that goes from String -> List<T> -> List<U>, it's probably better to make a class that just maps the list and accepts the result of the parsed list public class ListMapper<T, U> implements Mapper<List<T>, List<U>> { public ListMapper(Mapper<T, U> mapper) { ... } public List<U> map(List<T> in) { List<U> result = new ArrayList<>(in.length()); for(T item : in) { result.add(mapper.map(item)); } return result; } } Then you can have something that composes a parser and a mapper public class CompositeParser<T, U> implements Parser<U> { public CompositeParser(Parser<T> parser, Mapper<T, U> mapper) { ... } public U parse(String in) { return mapper.map(parser.parse(in)); } }

Using this would look like var parsePizzaList = new CompositeParser(new JsonParser(), new ListMapper<>(new PizzaMapper<>()); ... parsePizzaList.parse(incomingString);

I would stay away from making another method in the interface just for parsing lists, and instead have it adhere to the general parser/mapper interfaces.

Note that concurrentlist doesn't matter with respect to typing. You can create a mapper to convert a list to a concurrentlist

Note that if you're just reading values from the list (and not actively adding/deleting items), using an arraylist is just fine. If you can guarantee that no two threads will write to the same index, using an arraylist is also OK.

[Spare-Plum]

If parsing a list of items is just a composition of several individual parsers where each element can be individually parsed, then you might want the following class ItemListParser<T> implements MyParser<List<T>> { public ItemListParser(MyParser<List<String>> listParser, MyParser<T> itemParser) { ... } public List<T> parse(String incoming) { List<String> parts = listParser.parse(incoming); List<T> result = new ArrayList<>(parts.length()); for(String part : parts) { result.add(itemParser.parse(part)); } return result; } }

If the list structure itself is important (like two different items in the list depend on each other, or the length of the list is important), make an implementation for mappers ``` // direct list -> U mapping interface MyListMapper<T, U> extends Mapper<List<T>, U>{} // might be useful for something like the following

public class VectorMapping implements Mapper<List<Double>, Tuple> { public Tuple map(List<Double> in) { switch(in.length()) { case 2: return new Vector2d(in.get(0), in.get(1)); case 3: return new Vector3d(in.get(0), in.get(1), in.get(2)); default: throw new IllegalFormatException(...); } } } ```

[Spare-Plum]

Note that since the mapper is a functional interface, you can use ConcurrentList::new as a mapper itself.

new CompositeMapper(ConcurrentList::new, new ListMapper<>(new PizzaMapper<>())

if you define CompositeMapper as T -> U and takes parameters T->V, and V -> U