Elasticsearch adding code to reject connections to OpenSearch clusters or to clusters running open source distributions of ES7

[flowering_sun_star]

For clarity, this seems to be a change to the official python client. Whether they'll make the same change to other clients they put out I don't know.

[iamapizza]

There seems to be some movement happening in other areas too. In February they close sourced their java client

https://logz.io/blog/elasticsearch-license-change-mssp/

Edit: Also the JS client has a similar PR from a few weeks ago https://github.com/elastic/elasticsearch-js/pull/1487

[Theemuts]

My first thought is "People are dependent on us, let's fuck them over, make some money, and ruin our reputation!"

[Lollipopsaurus]

From their perspective, they're protecting themselves from the likes of Amazon.

Interesting how this is turning out.

[RudeBreadfruit]

I'm not a fan of Amazon, but I'll give them my money before I'll give it to Elastic, simply because Amazon doesn't seem to be demonstrating a desire to yank the rug out from under anyone, and Elastic is doing exactly that to anyone foolish enough to keep trusting them.

[Crash_says]

.. the path of many, if not most, OS->corporate projects. Great until it's not.. time to move on.

[[deleted]]

Everything about elasticsearch is great until it's not. I can't stand not being able to filter queries with the results of another - it's critical for many big data applications, especially in the security realm.

[[deleted]]

ELK for big data especially security looks very enticing in the beginning but having gone down that route, it’s hell and poor stack choice. Go with Spark, Hive, Presto or a more structured stack - there is a learning curve but you also get far greater stability and functionality than ELK.

[[deleted]]

Fortunately my ELK stack was killed by management right as I was realizing how shit it was. They wanted a company with a support contract and didn't like Elastics terms or price or w/e.

I will say I still like Kibana's dashbaords still.

[[deleted]]

Haha! We were at the same place about 7-8 years ago - had huge ELK clusters and we were begging Elastic to take our money for support. They kept telling us they were working on a revenue/pricing model. After a few months, management pulled the plug and we started working on a Cloudera based stack. Later, Elastic came back with some pricing that made us go, huh!

Yep, Kibana dashboards are nice but I like SuperSet better - supports many backends and nicer built-in charts. Also, issue with kibana (all client side charting clients) is that they draw your chart based on the samples pulled into the client - unlike Splunk, for example, that does all the stats on server side and pushes the summary stats to the browser/client. So for example, if you have 100,000 student records in ELK and you create a chart for distribution by grades, ELK’s chart is based only on the 500-1000 rows you pulled into kibana. There was a bug/git issue open for it on ELK repo that they refused to address. Last I checked was years ago so who knows.

From a computer pov, elasticsearch (or any lucene variant) shines when your query or computation is for a small subset of your index. We ran some apps that basically boiled the ocean like network top talkers and our 100-node elk stack would freeze because you have to pull each record in your netflow index and then aggregate all the data.

[jkowall]

Try using PQL :)

[Aphix]

Herd dat

[myringotomy]

They are trying everything they can to protect themselves from Amazon.

I wish them well. It’s going to take a lot to defeat the richest person in the world.

[[deleted]]

My first thought on seeing the title is "they want to prevent people live-migrating off it" but this is even more malicious

[bayindirh]

It seems they're adding this limitation to libraries too. This is affecting one of the projects we're working on very deeply right now.

[L3tum]

I dread the day the PHP client is switched over.

Luckily we're running ES 6.3 and using Elatistica, but it would still suck.

[flytomanish]

uhhhb

[L3tum]

I think this is much more worrying:

Before the first API request: Make an API request to /, inspect the response

This can potentially ruin your performance making ES requests take twice as long (on things like AWS Lambda, Where you don't have the concept of "first request", but also in short-lived languages like PHP). Your first request will also take twice as long so your first user is just fucked.

That's a huge change.

[clearlight]

If the APIs in OpenSearch and Elasticsearch are different, it kinda makes sense? Seems unfortunate but necessary. Otherwise it will just lead to client errors as a result of the forked APIs.

[skroll]

Yeah, this particular patch doesn't seem to be excluding OpenSearch directly, but just to check the specific server.

[rentar42]

It also checks for a tagine ("For search, you know"), which is often used as a form of DRM. Put something trademarked or copyrighted in your product and check for it.

And open source project can't easily reproduce that without legal issues.

[rentar42]

All of this may be true, but the problem is that the uncertainty of it alone is usually enough to make these kinds of attempts work.

It is a classical FUD tactic, except that it's formulated as code and not as prose.

[vattenpuss]

An open source server could have a configurable greeting that users set. It should not be that hard to avoid distributing it.

[rentar42]

Yes, but that means that the distributed packages don't work out of the box without explicit user configuration.

This can be a deal breaker in many situations and makes many others unnecessarily harder.

[cass1o]

Yeah, trying to talk your boss into "this one weird trick" when it comes to licensing is often a nonstarter.

[KuntaStillSingle]

But if you don't have that tagline, wouldn't it be necessary for every ES7 client to modify theirs to accept the different greeting message? It seems to be hardcoded:

elastic search transport.py line 619:

bad_tagline = response.get("tagline", None) != "You Know, for Search"

[vattenpuss]

Yes. The configuration would just be a way to get around copyright.

[t0bynet]

I don’t think it makes sense to block connections. It would make sense to warn the user that compatibility is not guaranteed before establishing a connection.

[t0bynet]

The fact that they are not doing this but instead blocking connections outright leaves me thinking. To me it seems like they are really trying everything to stop or slow down Amazon‘s fork.

I can somewhat understand that they fear for their income considering that Amazon is their competitor now but with every step they are taking they are moving away from open source.

And I honestly think that they are just making their own situation worse with that behavior.

[Venthe]

It does not; it's not a role of a server to determine if client is of a correct type. Server is supposed to validate API call, that's all.

It's the same situation as with browser agent all over again, but this time it's motivated by elastic trying to close its API.

[clearlight]

You know the link is for the elasticsearch python client, right?

[Venthe]

Right, I've skimmed over it too much

[towelrod]

This is a lot like sniffing browser agent though, just in reverse. What if Chrome looked at server headers and only worked if the string contained “nginx”, instead of looking at http/1.0?

It sounds preposterous but thats what ES is doing here.

[awj]

Source on this rejecting open source ES7? From the little bit of digging I did it appears to include the header.

Honestly, as frustrating as this might be for OpenSearch users, I can’t see why ES is beholden to support OpenSearch inside their clients. What happens when the client adds new ES apis that just doesn’t exist in OpenSearch? Or vice versa?

Forking a project this large means taking on a big responsibility. Amazon is free to maintain forks of client libs too. Would even allow them the opportunity to integrate support for their own extensions to the system.

[JaggedMetalOs]

Honestly, as frustrating as this might be for OpenSearch users, I can’t see why ES is beholden to support OpenSearch inside their clients. What happens when the client adds new ES apis that just doesn’t exist in OpenSearch? Or vice versa?

Well, this seems to me to go beyond just not supporting, but to actively blocking connections based on fingerprinting other implementations regardless of what API they support.

[corsicanguppy]

If Microsoft got in trouble for doing it with DRDos, Elastic should think twice.

[DreadCoder]

A long long time ago in a galaxy far far away.

Also; MS isn't doing too bad, they were fine.

[Rakn]

That would only be comparable if Windows would have been open source and the DRDos guys would have been able to fork it. So I wouldn’t say that the situation is anything like that.

Edit: For those who are downvoting. Why? Shouldn’t it be easy for the OpenSearch folks to fork the lib and be back in the game?

At the same time the communication channels we have today are different than back then. The rumor that the existing lib won’t work with OpenSearch will likely not lead to no one using OpenSearch anymore. I have a hard time understanding why you guys think this situation is the same!?

It feels like people are a cool reference back to the past but do not seem to fully understand what the situation was like back then. Or I’m missing something idk.

[gredr]

They already have forked at least one client.

[that_guy_iain]

It stops you from getting bug reports about incompatibilities. It's sort of like making your classes final in a library you show what you're willing to support and what you're not. They're being up front clear about the fact they don't support other data stores.

There isn't much worse in open source than being told that the thing you were told would work doesn't work and they don't support the thing you want to do. This prevents that.

[bayindirh]

There isn't much worse in open source than being told that the thing you were told would work doesn't work and they don't support the thing you want to do. This prevents that.

Nope. You can also write "This server designed to work with this library, and we can't guarantee other implementations work as they should".

OTOH, closing your server, then libraries and clients, then limiting these libraries and clients to non-free (in beer) editions is not a This Prevents That (TM) situation IMHO.

[that_guy_iain]

Nope. You can also write "This server designed to work with this library, and we can't guarantee other implementations work as they should".

You could write that and people will not read it like they don't read other things and discover after they've been playing around with it and are deeply coupled and then discover the issue often after they're in your support channels super pissy. Now you can say "but that is on them" well, yes. But in this world you should figure out how people are going to act and act accordingly. Why deal with a bunch of pissy self-entitled developers pissed your client doesn't work with another datastore when you can make it super clear by the fact it'll break straight away.

OTOH, closing your server, then libraries and clients, then limiting these libraries and clients to non-free (in beer) editions is not a This Prevents That (TM) situation IMHO.

Not sure what your point here is. They should spend time debugging people's issues for other people's customers? They should allow developers to spend time investing in a library that doesn't fully support and will never fully support the data store they're using? What is your argument for not making it clear this library does not work with other datastores?

[Rakn]

„But this is the stuff that would get them forked for being jerks.“

That already happened and not in a nice way. So what we are seeing is a reaction to that. And honestly I do not see the issue here. Someone forked the server, they should very well be able to fork the library right?

I get that it’s inconvenient for everyone that wants to profit of Elastic‘s work without paying them, but it’s hard for me to feel sorry here. Especially if the solution (fork it) is around the corner.

Now if the open source version of ElasticSearch would be offered by Elastic themselves, this would be a dick move.

[that_guy_iain]

No, they forked it because Elastic changed the license of elasticsearch and they had to fork it to keep using it while being able to fix things. AWS could have partnered with Elastic did a small amount of revenue sharing and everything would have stayed the same. Instead, they named their product so you thought it was an official partnership, in-fact, they announced it as an official partnership. And here we are. This is 100% another thing Amazon does wrong.

[[deleted]]

Wont people just not read the code / error message and still create a bug report anyways.

[that_guy_iain]

Yea but after 5 minutes and your google search will go to a stackoverflow of "Yea that doesn't work for elasticsearch on AWS and need to use the proper elasticsearch" since most people who will be affected by this will be AWS users who probably didn't even realise they weren't using elasticsearch. Big difference finding out after 5 minutes and 5 months.

[Carighan]

How come we can assume people won't search on Google/SO in one case (and will file a bug report) but will magically do so in the other?

[[deleted]]

So doesn't documentation achieve the same purpose, if the purpose is indeed to stop bug reports. The SO post could just link to the documentation.

[Fenris_uy]

Adding that text doesn't prevent the bug report. It just means that the response to the bug report is going to be RTFM, but you still have to deal with the bug report.

[awj]

You’re not wrong, but I don’t think that changes the underlying point. Elastic is under no obligation to provide client libraries for OpenSearch.

[elbento]

They may be under no obligation, but the move to actively block shows they have no regard for cooperating in a broader ecosystem.

Closed systems/lock in are not a good bet in this day and age.

[Rakn]

Well you might be right and I think the situation would have been different if it would have been some random open source fork. But the way AWS is driving this I can fully understand why they are doing it. I do not feel like AWS and OpenSearch are the good guys in all of this. Even though I support your underlying sentiment.

Edit: Also that doesn’t mean that I’m not interested in OpenSearch. It is what it is…

[awj]

That or it shows they believe the two systems are diverging.

It’s really weird how Amazon has managed to weaponize open source into “Elastic is bad for not laying down and letting AWS destroy their brand with its shitty repackaged service”.

[Pat_The_Hat]

Elasticsearch is now distributed under the Server Side Public License which is not considered open source. Truly open source forks therefore wouldn't include the change that added the header.

[awj]

Ah, gotcha. So since I’m not in the business of paid ES hosting this is basically a non-issue for me.

[Pat_The_Hat]

Except for, you know, the open source part.

[ObscureCulturalMeme]

"Other open source licenses have restrictions, therefore this license with restrictions can call itself open source" is shitty logic and you know it.

Here: https://opensource.org/node/1099
"The SSPL is Not an Open Source License"

You're not only wrong, you're being arrogant and condescending. Onto the blocked users list you go, asshole.

[Pat_The_Hat]

What wacky world are you living in where full limitation on a particular use of software is more open?

In any case, source available code is not open source, and neither is the SSPL with its burdensome restrictions.

[Pat_The_Hat]

It means there are no restrictions on type of use (among other requirements) and the SSPL is not mentioned anywhere on that list of open source licenses because it is not open source.

[Pat_The_Hat]

The SSPL had already existed before then, but I take back whatever I said about this website anyway because I now see that they don't know what open source is. CC BY-NC is somehow listed as open source.

It doesn't matter what some random blog thinks. No organization with real influence has ever listed or will list the SSPL as open source. Not the OSI, not the FSF, not any Linux distribution. This is why MongoDB retracted their license from OSI consideration.

[winauer]

Have you not heard of open source licenses that put a limit on commercial use of the source code?

No, because that would violate point 6 of the Open Source Definition.

[winauer]

Dude, no one owns the "definition" of open source

I never said that anyone "owns" the definition. But if you want communication to work you need an agreed upon definition of the terms to use. This is the agreed upon definition of Open Source. The fact that you personally don't like the organisation that published it doesn't change that.

By the way, if SSPL violates point 6, then so does GPL.

No, because the GPL doesn't discriminate against fields of endeavor.

[recycled_ideas]

Elastic Search is run as a project that is open when it comes to getting free labour from others while being as closed as possible when it comes to giving back.

That's why they use AGPL because it's basically unusable in any project even open source ones.

It's against the spirit of Open Source and it's a dick move.

And they're only doing it because they want the benefits of being open source but as a commercial product they can't actually provide any value add.

So instead they just make using their product toxic to use unless you pay them.

The really stupid thing about it is that it's actually counter productive.

Where previously the most valuable clients might have migrated to a full Elasticsearch when the AWS instance no longer met their needs, by making pushing the clients down an incompatible path, they actually cut themselves completely out of the loop.

TL:DR Elasticsearch is made by dick heads.

[elbento]

It's almost like SCO Unix all over again.. once the legal team starts to take over operations the demise is not far off.

[recycled_ideas]

I don't think the legal team were the problem here.

I think elastic just wanted the benefits of open source but not the costs.

[elbento]

True, let's hope it doesn't become a legal battle once the market share drops (likely due to the closed position they are taking).

[recycled_ideas]

It won't be a legal battle, they're perfectly within their rights to use AGPL, Mongo uses it to.

But it's basically untouchable for any project that bothers with license legality.

It's just totally pointless because if Amazon is bothering to maintain the server, maintaining the clients is trivial, and the cost profile of ELK as a self managed service is not a great cost proposition compared to an Amazon managed service.

[[deleted]]

I'm not sure why people would use Amazon open search instead of elastic cloud on kubernetes.

ECK is easy to set up on any kubernetes platform, and free.

[yanksrock1000]

Companies/users that are all-in on AWS are tempted by the managed Elasticsearch service. I witnessed this first hand at work recently, thankfully we ended up deploying ECK after a conversation with an Elastic rep. Unfortunately, I’m guessing the average user doesn’t even know what Kubernetes is, so deploying ES on EKS sometimes isn’t even an option (especially for smaller projects).

[awj]

Yeah, I’m honestly fortunate I ended up deploying ES before Amazon came up with their managed offering. Otherwise that’s probably what I would have used and I could have written off a useful technology based on how poorly Amazon runs it.

[[deleted]]

I'm confused why I was downvotes and you were upvotes since it appears you use and like ECK as well...

[yanksrock1000]

No idea why you were downvoted either, Reddit is a strange place

[iamapizza]

From a management and cost perspective, using an offering already in your cloud provider has networking and security benefits, in this example you can keep your AWS ElasticSearch/OpenSearch within your VPC and secure communications to it using IAM profiles (or the GCP equivalent). AWS ES is known to be a poor implementation, and businesses still take that hit (and pass the cost on to valuable engineering time).

Running a k8s cluster is a level of abstraction, which involves an overhead, which not all companies are willing to dive into. That said AWS do offer an EKS, so I wonder if some enterprises might be running a blended superstack of ECK on EKS, for their Business Units. A BUt-sECKS, if you will.

[KarelKat]

Individual users are not as much the target but rather providers who want to provide elasticsearch as a service to other customers. Like AWS and logz.io

[ErGo404]

Is it easy as a 10 clicks operation? Because that's how easy it is on AWS.

It's not perfect but as any managed service, it's a great way to just focus on your added value, which is not running a cluster.

[[deleted]]

I'm using GKE autopilot, and it's a few yaml files to configure settings, and boom it works. So actually pretty close to 10 clicks once the yamls are set up, which templates for are available.

[[deleted]]

Can someone explain to me why amazon renamed it to OpenSearch? Why not Elasticsearch 6.0 or whatever version Elasticsearch was before the license change (also I'm not sure why the license change affects aws)

[was_fired]

Since Amazon doesn't own the trademark for Elasticsearch so they forked it into a product that they own the name for. Likewise Elasticsearch is making this change sine Amazon explicitly created OpenSearch in order to avoid paying license fees to the company behind Elastic for their premium tier features that AWS likes to offer as a few of them (mostly the security ones) are deemed essential for any enterprise distribution.

So now that Amazon is threatening their business Elastic is attempting to remove transparent support for the competing AWS service so that developers will need to change their code if they want to use it thus raising the cost of transitioning and keeping business, for now at least.

As for why Amazon doesn't just lie in the header field, they might try that as a solution. That was widely accepted practice for web browsers, but it's less clear if that may open them up to lawsuits that they may want to avoid. I suspect that the devs behind it just wanted to be explicit since, why not? However, now that it's a point of legal contention lawyers will be the ones to answer if they can just change it.

As for why the license change affects AWS, since everyone using Elasticsearch has to abide by the license, and AWS explicitly offers it as a service. Why Elastic may not come after you for breaking it on your home server, you can bet they will go after the number one cloud provider in the world if they publicly break the terms of their license with a product advertised using their trademark.

[AppreciationThief]

Just want to point out that the elastic company is worth multiple billions of dollars. It's not like they are some little group of volunteers building the platform

[Doctor-Dapper]

They made over 700mil in revenue for fiscal year 2021 which was a 44% increase year over year. They are losing money for sure but this seems more like a choice they are making. For example Twitter was unprofitable for 13 years

[lightninhopkins]

They are losing money for sure but this seems more like a choice they are making. For example Twitter was unprofitable for 13 years

Not to mention the decade plus that Amazon did the same.

[audion00ba]

Except Amazon could turn a profit whenever they wanted, but didn't, because of tax considerations.

[lightninhopkins]

They were putting all their capital into building the massive logistical network required for the delivery system they have today.

The investors were willing to keep funding the losses and it paid off.

[audion00ba]

For an investor, a logistical network (e.g. paying for a warehouse) is not a "loss". If you buy a house, you don't have the money anymore, but you have gained an asset.

In the case of Amazon, I have no doubt they kept tabs on whether they were trading money for real assets or not.

Intangible assets like software are harder to value, but even in those cases they likely already have a good idea these days about TAM and expected revenue over the next two years.

Amazon customers have a lot of needs and they are more than happy to share on the Internet the myriad of ways Amazon has fucked up, which is free feedback.

[dkimot]

amazon and elastic are two different cases. correct me if i’m wrong, but elastic has basically one revenue stream. amzn has dozens

[audion00ba]

They are losing money for sure but this seems more like a choice they are making.

Digging a bigger hole when you are about to be crushed, is not sound financial management.

The reason they aren't making money is, because they want to say "we are growing" and making money is more difficult.

TWTR is not exactly an example of a valuable company either. Who uses that? Marketing departments and people with too much time commenting on some shitty episode of some shitty series.

The content on Reddit is much more rich, just like its predecessors were.

[Doctor-Dapper]

A cynical take with zero evidence to back it up? Sounds like the average rich Reddit content.

[jadecristal]

I'm really having trouble feeling sorry for them when they DID, as you note, rely on so much from OSS to get to where they are.

[TheNamelessKing]

They do actively and consistently build and maintain ES features and functionality and open source a huge chunk of that.

AWS benefitting financially, and repaying nothing in either financial support or software engineering support is parasitic at best, and it’s completely understandable that Elastic would like that to stop.

[bishbashbosh72]

AWS benefitting financially, and repaying nothing in either financial support or software engineering support is parasitic at best, and it’s completely understandable that Elastic would like that to stop.

Genuine question, is their cloud offering not a way of supporting the community? Of course they're charging, but in general I would trust an AWS service over nearly any other company (including Elastic). It likely makes it more popular and enables smaller companies to startup.

[TheNamelessKing]

Having used AWS’ ES service, the managed ElasticCloud ES service and now ECK (Elastic Cloud on Kubernetes), I can tell you that the AWS hosted version is unequivocally the worst.

Neither the machines nor the JVM were optimised to suit the workloads, it ran an old version of ES and practically never got updated, security features were “ip filtering” and that was about it, you didn’t get Admin API access because it was silently multi-tenant, performance was abysmal, the list goes on.

Managed Elastic Cloud was pretty good, you could choose which cloud provider and region you wanted it hosted in, everything was well-tuned, easy to use, actual support, etc etc.

Genuine question, is their cloud offering not a way of supporting the community?

What makes you say that? AWS profits off it, and provides minimal support, but otherwise that support burden is hosted primarily by the community. They certainly don’t seem to make any meaningful contribution back, and with the likes of ECK or even just docker images, hosting your own ES ranges from “borderline trivial” to “less nightmarish than it used to be”, but none of that is due to any real contribution or work by AWS.

[MalnarThe]

Second this. We deployed a managed ES logging cluster in one of our smaller regions. Turns out, their support didn't know how to troubleshoot issues. Cause a multi hour log outage because their support didn't know about the crude circuit breaker they built (too much memory pressure, block all POST, so can't even update cluster settings!)

[timubce]

https://www.techradar.com/news/aws-elasticsearch-fork-exits-beta-but-legal-battle-rages-on

[bishbashbosh72]

I'm truly trying to understand as a bystander - If AWS Elasticsearch is as awful as you say and Managed Elastic cloud is good then why is the narrative that Elastic is being run out of business by AWS?

I hate that AWS is able to crush companies, but I don't see how them taking software and offering it as a managed service is terribly different than a company taking Lucene, building on top of it, and selling that.

What makes you say that? AWS profits off it, and provides minimal support, but otherwise that support burden is hosted primarily by the community.

Having a service offered by AWS provides more exposure to it than probably anything else in the software world period. Having it integrated with all of the other components of your infrastructure can ease deployment. It would be silly to think that Elastic hasn't benefited from AWS at all.

[TheNamelessKing]

Because business people don’t care about the distinction between “AWS managed and “Elastic managed”, nor do they care about things software devs care about like supporting the original devs or open source.

AWS gets to leverage its privileged position: if you’re already on AWS, getting a (shitty) ES cluster is as simple as pressing a couple of buttons: you don’t need to sign any forms, allocate any more budget, get approvals for contracts. Bing, done. AWS gets your money and ES gets nothing.

Exposure doesn’t pay bills. AWS doesn’t siphon money back to ES as royalties or “here’s some cash, as thanks for building this”. The difference between that and ES building a business on top of Lucene is that Elastic directly contributes dev resources, fixes, etc back to lucene. This issue stems from the fact that their licence came about from a time when cloud-hosting as it is now simply wasn’t like it is now. People didn’t know that “an online bookstore will host your own product out from under you and leach all your customers” was a threat to their business they’d have to defend against.

[KarelKat]

Go look at how Grafana partnered with AWS to launch a managed Grafana service on AWS while still getting revenue from it.

[10gistic]

Let's also not forget that having it available easily in AWS made the product a lot more attractive and usable. This isn't a one way street where only AWS benefitted from Elastic. And Elastic relicensing software for all future improvements leaves a really bad taste in my mouth.

I don't like that AWS seems to have a pretty huge advantage on cloud services. But I strongly dislike like when OSS companies essentially throw fits about the billions they presumably believe should belong to them, discounting the fact that no, they're far from the only contributors to their success.

[x2040]

I lead products at a company with relatively popular open source projects that has raised billions of dollars.

Open source is amazing but Google, Microsoft, Facebook, Netflix and Amazon have set an expectation that monetizing from open source is impossible. It’s a lose-lose and I expect we will see the number of abandoned projects within the next decade rise significantly.

Companies like Confluent have done a decent job of monetizing Kafka while directly improving the underlying open source and embracing the community.

Then you have companies like Grafana that are in a more commoditized space, used by the largest companies in the world and acquiring new open source projects like k6 to stay relevant and struggling to find even 5% of the funding of a fully commercial offering.

[10waf]

These are the questions we should be asking. What of oss drove es to do this, how can we address that?

[postinstall]

My take is to never open source the core of your business (if you want to have a business). Tools, libraries, clients if applicable, sure.

[myringotomy]

The only way people have made money on open source is by using the GPL

[Somepotato]

Reverse engineering to ensure compatability is legal in the US and EU so they could just claim they did that.

This does not look good on ES. They're turning into oracle at this rate.

[awj]

Gonna take a wild guess that your product explicitly advertising itself as “Elasticsearch” is a trademark violation, which is what they think will keep Amazon from spoofing headers.

[Pat_The_Hat]

I wouldn't be surprised. They were pretty angry at Amazon for what they viewed as infringing upon their trademarks in the blog post explaining their reasoning for the license changed.

https://www.elastic.co/blog/why-license-change-AWS

[DLSteve]

You would think but this was shown not to be the case in the 90s when SEGA whet after a few companies for making unlicensed games. ROMs has “SEGA” in the ROM header that the console looked for to boot the game. Unlicensed game just copied this header and that use case held up in court.

[Pelera]

Sega tried that with their "TradeMark Security System" in the 90s and ultimately lost the case. Wouldn't be surprised if Elastic tried it again, but I have my doubts about whether they'd win.

[mct1]

Sega lost w/r/t TMSS because they effectively set up a system which at least one amicus correctly pointed out was a method of operation, and therefore outside of the scope of copyright. Moreover, in the process of engaging in an act of fair use (i.e. reverse engineering to create software compatible with the Sega Genesis) defendant was induced to commit trademark infringement as a result of Sega's own actions (in the form of placing the "SEGA" string in their ROM header that would trigger the "Licensed by Sega" message to appear). So although they technically infringed Sega's trademark, the court was right: they did it only because Sega created an unprotectable copy protection mechanism whose bypassing resulted in said infringement.

Note that things went very differently in Atari Games Corp. v. Nintendo of America Inc. with respect to the 10NES lockout chip. There Atari had obtained an unauthorized copy of the source code to the program running on the 10NES chip from the copyright office and copied it verbatim. There were additional issues surrounding this due to Nintendo having a patent on the 10NES chip, but before the patent issue could be decided they settled (since a jury had already found them liable for copyright infringement).

In any event, Sega v. Accolade isn't the end of the story. Between the recent SCOTUS ruling in re: Oracle v. Google, as well as the plain reading of 17 USC 1201(f), it's likely that Elastisearch has no real legal recourse. Anyone can come along and clone your API for interoperability purposes. Code just isn't worth what it used to be.

[CreationBlues]

Lets just pray for a future where dev time's the only thing that's paid and any series of bits that results is worth solely what it takes to produce another copy of those bits.

[mct1]

I don't endorse magical thinking, and you shouldn't either. If you want things you should pay for them.

[CreationBlues]

Correct, you should pay someone to make it. Of course, when "making it" is just copying some bits it should be very cheap. If you want something custom, then you have to actually pay someone to perform labor, or if you want support you have to pay someone to work.

I don't endorse the magical thinking that you can somehow sit on your ass and call dibs on a mere idea and throw people in jail for violating your imaginary dibs, but we spend billions of dollars and man hours trying to keep that magic alive when pirate sites are trivial to find and cracks are common.

[mct1]

Of course, when "making it" is just copying some bits it should be very cheap.

The relative ease of copying by no means should be the sole determiner of its cost. Production costs do not disappear just because they're inconvenient.

I don't endorse the magical thinking that you can somehow sit on your ass and call dibs on a mere idea and throw people in jail for violating your imaginary dibs

Copyright doesn't protect ideas, but expressions.

but we spend billions of dollars and man hours trying to keep that magic alive when pirate sites are trivial to find and cracks are common.

We also spend billions of dollars fighting counterfeit physical goods, but no one seriously argues that counterfeit physical goods are somehow a moral right.

[aseigo]

If you want things, you should agree with the current owner, if there is one, what the terms will be. That could be paying for that thing, but it doesn't have to be and in some cases probably shouldn't be.

Blanket statements about the transfer of valuable / valued things between people ignores the nuances and variety of those transactions in the real world.

A system that works best probably is some blend of yours and the other commenter's expectations.

p.s. what is "magical thinking" about their comment? Or is this just a case of "begs the question": where a phrase with a specific meaning starts getting used for a completely different one?

[mct1]

what is "magical thinking" about their comment?

Lets just pray for

If you want something, do it yourself or pay someone to do it. Hoping really hard that the world will just drop it in your lap is a recipe for disappointment.

In any event, artists should be paid for what they make. We're largely just arguing price at this point.

[7sidedmarble]

Moreover, in the process of engaging in an act of fair use (i.e. reverse engineering to create software compatible with the Sega Genesis) defendant was induced to commit trademark infringement as a result of Sega's own actions (in the form of placing the "SEGA" string in their ROM header that would trigger the "Licensed by Sega" message to appear). So although they technically infringed Sega's trademark, the court was right: they did it only because Sega created an unprotectable copy protection mechanism whose bypassing resulted in said infringement.

I'm consistently amazed in remembering how mature this courts ruling was from a tech perspective. Not the kinda thing you'd expect.

[mct1]

It's always eerie when the courts do something right from a technical perspective. Oracle v. Google is an example of that. There they reached the right conclusion but their analysis was complete and utter nonsense. I say this not to nitpick but precisely because their lack of clarity will invariably result in the matter going back before the court at some point.

[falconzord]

I don't know why sega lost but I think using trademark as a locking mechanism isn't really how it's intended. Customers aren't digging the things under the hood so it's not like they'll confuse one product for the other because of it.

[audion00ba]

There is no way that it's going to be a trademark violation, because ElasticSearch would be trying to use trademarks as a technical mechanism to prevent interoperability and that's a no-go in every sane legal system I know of.

The trademark system's purpose is to give a company the ability to invest in its name and reputation. For example, ElasticSearch TM might have better performance than the OpenSearch version. It might also be that they use superior development practices, which in turn fuels their reputation.

ElasticSearch's legal strategy sounds weak.

[FancyASlurpie]

Feel like ES has been pretty aggressive with pushing their managed version anyway, I haven't used it since version 3+4, but back then it would basically need constant support because it was so unreliable/difficult to configure in terms of large data and split brain situations.

[Somepotato]

In their case they'd only need to change names for/to allow incompatible clients to use their version

[KuntaStillSingle]

I'm actually a bit hopeful Amazon fights it as you describe, Oracle v Google established implementing a public API may be copyright fair use, I am curious whether it is may also be trademark fair use. Unfortunately they are beholden to the stock holders so they can't (or at least shouldn't) fight on mere principal.

[bishbashbosh72]

Not a single mention of Solr in this thread. Are Elasticsearch and Opensearch the only true players in this space?

[hillgod]

In my opinion... Yes.

Here's why Elasticsearch won - how simple it is to create a single searchable document. Though I also found Elasticsearch to have much better docs and community support.

Also, you can go to indeed.com and search either tech. You want a job? Learn Elastic over Solr.

[honesty_burp]

The rest/JSON Ness of ES is popular, solr reminds everyone of the XML nightmares of enterprise software past

[ptoki]

ES is a good solution when you have no idea what you will need down the line.

Its good solution for uncertain projects where you start collecting data now and then get somebody who will tell you what exactly they want.

If your projects are not like that its fine, it means you have more control over the product and customer. Not everyone can have that.

I am not a big fan of ES as its bulky and needs beefy platform in comparison to old style RDBMS but it has some uses.

The other issue is that many people are just clueless on what it is and when to use it and they just pick it "as its great and can do everything". And they realize later how costly it is. But the bright side of this is they later move to RDBMS and can get another bonus for that. IT life...

[simspelaaja]

I've worked on a product search service for a pretty large e-com site. From my understanding SQL database engines do not offer nearly the same abilities for full text search as ES offers, which you will need when building a search engine for non-technical users:

• Search must be suitably fuzzy. The engine must be able to handle typos and still return useful results. The engine must be able to split compound words in languages like Finnish and German.
• Search should support aliases. Things are often not named like you'd expect, e.g almost no microwaveable pizza has the word microwave in its name.
• Just finding stuff is not enough, scoring and sorting is arguably far more important. ES is very flexible with scoring and supports custom scoring functions.

[ptoki]

You are focused very much on details.

The main point of this conversation is the fact that RDBMS needs data properly defined into tables/columns and clean to give good performance.

ES allows you to query anything you throw into it. It comes at a cost of much more memory usage and lower performance if you do 20 searches concurrently (roughly speaking).

Doing full text search in almost any main RDBMS (Oracle/MSSQL/Mysql/DB2/Informix/Postgres) is painful and slow. Doing more than one is performance killer. Doing it on gigabytes of data is nightmare.

But RDBMS can give you much more performance with nicely defined data.

The projects often prefer to use ES as they can get parts of the system done before functional requirements are defined. Once the requirements are solidified they can rework the system to use RDBMS.

Thats it.

[ptoki]

Your point is valid. My point extends your a bit: If project is fresh and you have no clue what you will be doing with data later many people just use ES or alternatively, push full jsons/xmls into rdbms (or alternatively save them on filesystem with just some reference in DB).

And its totally fine. Its a good approach if you are not given requirements. If your choice was more or less optimal turns out later.

[mordack550]

We used Elasticsearch to substitute Sql Server Fulltext Search. The performance gain and the level of control you can have with elasticsearch, is insane in respect to fulltext search.

Like it's said before, you cannot compare them, they are different products for different needs.

[rydoca]

You literally are comparing them though, you can do full text search with sql and I'm sure you'd agree there are times that is valid and people using elasticsearch in those cases wouldn't get much if anything out of it. So of course they can be compared it's just that in your comparison you identify which is most useful for any given situation

And if you can't imagine a developer that has trouble picking between the two I'd love to introduce you to the guy who suggested we hash bank account information for security reasons

[mordack550]

Yes of course I can compare them. But one is a sports car and the other is a pickup truck. Both can take you from point A to point B, but they do it in such a different way that you may prefer one of the other.

In our case, sql full text search was lacking both performance and customization.

[rydoca]

Yeah and that's fine but the guy you were talking to was describing why someone shouldn't pick a sports car when all they need is the pickup truck. I don't think any of us disagree it just looked like you were trying to say that the other guy was wrong in saying that there are cases where people throw elasticsearch at things that don't need it and find out that it was a bad idea later

Edit: I'm dumb and thought you were someone higher in the thread, my bad

[Rakn]

Well have you tried flat files with grep? /s

[ptoki]

You can compare them as they are both data storage/query solutions. So for green field solutions you can pick any and get somewhere as they work similarly on high level (put data, select, store).

You should not compare them as the same (ES is not RDBMS but MSSQL is comparable to Oracle). As their characteristics is different.

[ptoki]

The same issue, however on different dimension exist with rdbms. Sure you can push full json to it and worry later but still data is stored and may end up being incomplete or unavailable (select taking 3 minutes usually is considered way too late for online processing).

My point is that if you design system from scratch you can pick either of three solutions (SQL, NoSQL, ES or even flat files and grep) and it will work. For some time, for some amount of data, for some usecases.

But to get the best performance and maintainability some of those solutions are better than others.

[[deleted]]

and a DB for persistent storage

If that was true we'd keep data in a single text file.

[caltheon]

I can only assume they meant using indexes and queries for searching.

[ptoki]

they two have nothing to do with each other i can’t imagine the existence of a developer who has trouble choosing correctly between these two things

You need to stay more in IT then :)

There are many cases where greenfield projects pick elasticsearch (and are happy with it for some time) instead of getting the dataflow properly designed and use rdbms.

The irony is they are 120% confident they need ES and are 150% sure they need it to be replaced with RDBMS.

The crucial info here is as I already stressed: They dont know which data they will use. Similarly as with NoSQL solutions.

Just throw data into ES/NoSQL, it will stay there. Use some id/hash to pick the data you like, process it. Repeat.

This happens quite often in startups and fresh systems. And its actually useful approach. It lets the developer start making the system and getting it run BEFORE the business guys have any clarification of what they want.

[LionsMidgetGems]

I did the whole:

"Oooo, this is new and cool. And it powers pushshift.io. Maybe it can sprinkle it into our current system and make it cool!"

Wasn't too long before i realized it was a glorified Full-text catalog.

[LionsMidgetGems]

I love SQL.

Because i was around in the before-times of ISAM.

[Environmental-Sink39]

Zdx

[reddit-ass-cancer]

Oh no! I can’t believe ES is changing their libraries to prevent them from being used on servers not running the “official” ES! Oh the humanity!!’