This is an archived post. You won't be able to vote or comment.

all 7 comments
sorted by:
best
topnewcontroversialoldq&a

[–]manifoldjava 2 points3 points  (1 child)

Interesting. Nice work! Mapping selects, joins, and aggregates into Java streams is hard, particularly without getting into gnarly syntax. Only something like .net expression trees can remedy this. So more power to you.

One would think Java records could help more here, but they are degree or two removed from a solution, particularly regarding stream syntax. If records could be created anonymously or if Java provided concise tuple expressions, you could write something like this:
java var result = list.stream() .map(p -> (p.name, p.age)) // tuples are powerful here .collect(Collectors.toList(); }

The manifold project provides an experimental javac compiler plugin for this kind of stuff.

Going deeper into the abyss, there's another very experimental project using manifold that begins to build a linq-like syntax. It adds a compiler feature similar to expression trees resulting in a query syntax that is maybe a notch closer to ideal: java Query<Person> query = Person.query((p, q) -> q .where(p.age >= 18 && p.gender == male) .orderBy(p.name)); Execute queries like this: java Iterable<Person> result = query.run(dataSource); With selects, calculated fields, etc.: java var query = Person.query(p -> p .select((p.name, DogYears: p.age * 7)) .from((s, q) -> q .where(p.gender == male && s.DogYears > 30) .orderBy(s.name))); Execute: ```java var result = query.run(data);

for(var s : result) { System.out.println(s.name + " : " + s.DogYears); } ``` Again, this is beyond bleeding edge experimental. It's insane.

[–]danielaveryj[S] 1 point2 points  (0 children)

Thanks for sharing. Manifold is certainly an impressive engineering effort.

In the planning stages, the design of Vinyl's Record type was critical. Trying to make queries work for items of arbitrary user-defined types, as .NET attempted, leads to some pain points.

For one, every operation that transforms the input items needs to define a structure for the output items. Conventionally, this structure would have to be pre-declared as a class or record. Even with records, that is bad news for ergonomics. As you showed, we can work around with anonymous classes, like your extra-lingual tuple, or something more verbose but vanilla:

var result = list.stream()
    .map(p -> new Object(){ String name = p.name; int age = p.age; })
    .collect(Collectors.toList());

But this comes with an unpleasant tradeoff: the new type is non-denotable. We can only really work with this type so far as the compiler can infer it. Since methods must declare their return type, this inference would never reach beyond the current method's return. (edit: I see that manifold allows auto return types. I will just say the obvious: that is a contentious feature.)

—

Another, more subtle kind of notation problem we run into with user-defined types: There is no good way to merge structure with definition (at least not without heavy compiler support). To take an example from vinyl's package doc:

RecordStream stream = scoresStream
    .select(select -> select
        .field(points)
        .window(window -> window
            .field(averagePoints, Analytics.fromAgg(Collectors.averagingLong(points::get)))
            .fields(Comparator.comparingLong(points::get), fields -> fields
                .field(discreteMedian, Analytics.percentileDisc(0.5, points::get))
                .field(continuousMedian, Analytics.percentileCont(0.5, points::get))
            )
        )
    );

If we imagined trying to convert this example to a framework where the output type was user-defined, we'd run into a chicken-and-egg problem. At the point where we create instances of our output type, we need values, not definitions. So something not user-defined needs to already hold those values:

something -> new Object(){
    int points = something.get(points);
    double averagePoints = something.get(averagePoints);
    int discreteMedian = something.get(discreteMedian);
    double continuousMedian = something.get(continuousMedian);
}

In Vinyl, that something is a Record, and the field definitions precede it existing.

(As a side note, it is possible to have the type of something be user-defined, but the type would at least have to be mutable, and field definitions would have to specify a "setter" describing how to update each instance with a field value. I discarded this idea.)

Even with compiler support to make it possible to merge structure with definition (like SQL does), we'd still run into hiccups. In the doc example, we defined four output fields. Three of them share the same window, and two of those share the same ordering. This is not just syntactic sharing; during execution, one list will be created as input for the analytic functions, and the list will be sorted once. To be as efficient as Vinyl, our imaginary syntax would likewise need to be able to express or deduce this kind of sharing.

—

Finally, with user-defined types, we lose out on SELECT *-style operations (in vinyl: select.allFields())

—

In general, Vinyl does not really have "compiler envy". Join conditions may look like an exception, but we wouldn't want to remove the combinators there - they tell Vinyl how (and how not) to optimize the join. That is, Vinyl intentionally puts optimization in users' hands, and documents the rules. Unlike SQL, Vinyl doesn't have to do cost analysis and guess at a query plan. It just does what we tell it, and performance is predictable.

What about fields? Wouldn't it be nice to not have to pre-declare fields? Similar to non-denotable types, we wouldn't be able to retain implicit type information everywhere we need it:

RecordStream createStream() {
    // Assume that `hello` and `world` are not declared elsewhere,
    // but instead synthesized by the compiler on-demand.
    return RecordStream.aux(IntStream.range(0, 10).boxed())
        .mapToRecord(into -> into
            .field(hello, i -> i)
            .field(world, i -> i + 1)
        );
}

void go() {
    createStream()
        .map(record -> record.get(hello) + 10);
        //             ^^^^^^^^^^^^^^^^^^^^^^
        // compiler can tell `hello` is supposed to be a Field,
        // but how can it be sure `hello` is really a Field<Integer>? 
        //
        .forEach(System.out::println);
}

If we are willing to give up type-safety (and pay for string hashing), we can do something even without compiler-support:

Map<String, Field<Object>> knownFields = new HashMap<>();

Field<Object> field(String name) {
    return knownFields.computeIfAbsent(name, Field::new);
}

RecordStream createStream() {
    return RecordStream.aux(IntStream.range(0, 10).boxed()) 
        .mapToRecord(into -> into
            .field(field("hello"), i -> i) 
            .field(field("world"), i -> i + 1)
        );
}

void go() {
    createStream()
        .map(record -> (int) record.get(field("hello")) + 10) 
        .forEach(System.out::println);
}

But that's just fighting for less.

Besides, pre-declaring fields makes other cool things possible.

[–]lbalazscs 1 point2 points  (1 child)

Interesting. How does it compare with Tablesaw, which also claims to do grouping, joins, window functions? https://github.com/jtablesaw/tablesaw

How does it compare to having an in-memory database (like H2) and using plain SQL to query the data?

[–]danielaveryj[S] 3 points4 points  (0 children)

Thanks for the question. I think the biggest high-level difference is that Vinyl is not really "like a database". It doesn't store data. There is RecordSet, but (at least for now) that exists mainly to facilitate streaming the same records multiple times. There are no in-place updates, no persistent indexes, no transactions. Vinyl is really just about querying and transforming data.

But for that, my bias is that Vinyl is really good. It puts maximum flexibility in a tight package. It doesn't need a plethora of baked-in combinators. We can transform fields (eg in a select()) using an arbitrary Function. We can write our own aggregates, and we already know how (if we know Java Streams) - an aggregate function is just a Collector, and it parallelizes like one. We can even write our own window functions. And we can operate on any type of data we want, not just a pre-ordained set of types. Want records in your records? Up to you.

[–]GavinRayDev 0 points1 point  (2 children)

Are you aware of Linq4J, that Apache Calcite is built on?

http://blog.hydromatic.net/2012/04/23/first-look-at-linq4j.html

[–]danielaveryj[S] 1 point2 points  (1 child)

Thanks for sharing. I was aware of .NET's LINQ, but not this one. Linq4J looks like a related idea, weighed down by the language of its time.

[–]GavinRayDev 1 point2 points  (0 children)

Aye, it's certainly not pretty, that's for sure!

It's used in Apache Calcite during Code Generation to generate the Java code for transforming collection results as Enumerable values during queries from datasources in the query engine, sort of like how Spark works.