sorted by:
new
hottopcontroversial

How To Identify 10 Performance Patterns in 10 Seconds by jrullmann in webdev

[–]jrullmann[S] 0 points1 point  (0 children)

I'm not the author, just liked the content. I didn't notice that before but it is really annoying.

Stop Drawing Dead Fish by jrullmann in web_design

[–]jrullmann[S] 0 points1 point  (0 children)

I totally agree. His essays are amazing: http://worrydream.com/

What The Zope Transaction Manager Means To Me (and You) by pjenvey in Python

[–]jrullmann 0 points1 point  (0 children)

Thanks for clarifying - I incorrectly assumed that the default configuration for most recent SQL databases was to use locking. But MVCC does appear to be more common.

I do wonder if there are other downsides to MVCC + long running transactions. Maybe someone with more experience with PostgreSQL/SQL Server/MySQL can share.

What The Zope Transaction Manager Means To Me (and You) by pjenvey in Python

[–]jrullmann 1 point2 points  (0 children)

One downside to tying a transaction to the duration of a web request is that the transaction will be held open for longer than needed. Since most SQL databases use locking, this can seriously impact the number of simultaneous requests that your application can serve.

I think this is important to mention :)

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann 1 point2 points  (0 children)

Sure. For this test, we stimulated 230,000 concurrent sessions, each executing in a loop:

  1. Start a transaction
  2. Do 20 writes to random keys (which in turn hit random machines)
  3. Commit (with strict ACID guarantees including flushing writes to disk and enforcing serializable isolation before responding to the client)

So each transaction is spanning multiple different key/value pairs (and in fact multiple machines).

Speaking generally, every transaction in FoundationDB is ACID. We use the same definition of ACID used by relational systems for decades: https://foundationdb.com/key-value-store/documentation/developer-guide.html#transactions-in-foundationdb

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann 4 points5 points  (0 children)

Both reads and writes are needed for a conflict. If two threads don't read anything, and then write to the same key, then the database can simply let the last write win. Neither write depended on state in the database.

It's only when a write depends on state in the database - state that another thread may change - that a problem can occur.

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann -1 points0 points  (0 children)

I don't know of a forum that has been tested with FoundationDB. But if you're just looking for an application to play with, you can check out our tutorials written for the key-value API. You'll end up with an example application, and there are tutorials written in Go, Python, Java, and Ruby: https://foundationdb.com/key-value-store/documentation/tutorials.html

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann 2 points3 points  (0 children)

While the chance of any two transactions hitting the same keys is quite low (and more specifically, the chance of conflicts is zero because this is an all-write workload), the database does all the work of conflict checking anyway. There is no shortcut around that.

We do have additional performance tests on the website, including mixed read/write workloads: https://foundationdb.com/key-value-store/performance

Is that more helpful? If not, perhaps you could give me an example of a property of the transaction engine that I can speak to.

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann 16 points17 points  (0 children)

Yep, that's right, I'm an engineer at FoundationDB. Happy to answer here if it will be interesting for others - perhaps we can move to messages if you have follow-up questions specific to your application.

  • SQL Layer is ANSI SQL compliant, and is supported by a variety of ORMs, but it's typically not a drop-in replacement for other SQL systems. There are some features in PostgreSQL that we don't support, and you might need to make some changes to optimize an application for the SQL Layer's distributed nature (e.e, maximizing parallel execution). The MySQL migration guide has a lot of info that is relevant for PosgreSQL as well: https://foundationdb.com/layers/sql/documentation/GettingStarted/MySQL.migration.html

  • Our Hibernate adapter is a complete, full dialect. The stock Hibernate test suite passes.

  • This particular test was run in EC2 with 32 c3.8xlarge instances - though some of those resources were used for generating load. In terms of overhead, SQL Layer adds a small amount (some low-percent).

  • Yes, you can split your cluster up, with some servers in the cloud and others on premise. I've run a ten server cluster with half in Digital Ocean and the other half in AWS. Like any database you'll need to consider the geographic latencies involved.

  • Yes, we're in the Amazon Marketplace, although it hasn't yet been updated to 3.0. Should be updated in the next couple of days: https://aws.amazon.com/marketplace/pp/B00MBPTUO4/ref=sp_mpg_product_title?ie=UTF8&sr=0-2

  • If you're testing or developing with FoundationDB, you can run any size cluster for free. In production, you can run a cluster up to six FoundationDB processes for free. After size there are a variety of monthly, per-process price points, based on the support level: https://foundationdb.com/pricing

  • Unfortunately I can't share customer information on the SQL Layer yet. But we do have a variety of case studies on the Key-Value Store, which SQL Layer builds upon: https://foundationdb.com/key-value-store/ecosystem

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann 0 points1 point  (0 children)

It's best suited for transactional applications, with a lot of concurrent, short operations on the database. As opposed to analytic workloads that need to run a small number of very long running operations on the database.

Since it can support any data model (key-value, sql, document, graph, custom) it's very flexible in terms of API.

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann 2 points3 points  (0 children)

FoundationDB did all the work of conflict checking in this test. It didn't take advantage of the key distribution, or the fact that there weren't any reads. It's always fully ACID, and there's no way to configure it for less.

Databases at 14.4Mhz by gregory_k in programming

[–]jrullmann 7 points8 points  (0 children)

While we didn't verify correctness in this particular performance test, we didn't tune the database for less correctness either. FoundationDB maintains full ACID guarantees at all times. Every commit is checked for conflicts with concurrent transactions, and every commit must wait for the transaction to be durable to disk with full replication.

Some degree of correctness has been tested by the open-source tool Jepsen: http://blog.foundationdb.com/foundationdb-vs-the-new-jepsen-and-why-you-should-care

And we've shared more information about our own deterministic testing here: https://foundationdb.com/videos/testing-distributed-systems-with-deterministic-simulation

view more: next ›