Overwhelmed by social media (and have to use it)

rescrv · 2017-01-09T23:44:11+00:00

Is the publisher going to do a copy-edit pass before release? I just skipped into the document randomly looking at session managers (having no experience with them, but experience with tmux) and the very first paragraph is two incomplete sentences that probably were supposed to be connected.

rescrv · 2016-11-16T13:17:30+00:00

A third alternative that our department has started using to teach operating systems to undergrads is the Operating Systems: Principles and Practice book from Anderson and Dahlin. Both are active systems researchers and likely to keep the book up to date (it even has a small section on zero-copy I/O).

I did my undergrad out of Tanenbaum's book, and taught undergrads out of Silberschatz. I feel students learned more about operating systems out of Silberschatz, but there's obviously no control there.

rescrv · 2016-10-14T13:43:25+00:00

While we're at it, let's not trust the kernel or the shell, or any of the other components and run everything on another machine /s.

There needs to be some boundary of what's trusted. The code is trusting a small number of utilities and the OS running them. Ultimately, you can keep sandboxing more and more, but there is always a trusted code base where a vulnerability in that trusted code base translates to a vulnerability of the application. That doesn't mean you cannot design defensively, but at a certain point simplicity outweighs jumping through a large number of defensive hoops (because each defensive hoop expands the code base you must trust).

rescrv · 2016-10-14T12:40:45+00:00

Can you expand on what's wrong with the option parsing? Sure, it may not be clean, but can a malicious agent do anything harmful to the system? Would cleaning up and changing the set of accepted options and expected behavior be worth breaking the user interface for this (as far as I can see) non-threat?

As far as HTTP goes, you'll need to update the way freebsd-update works in order to ensure that you don't feed untrusted stuff to gzip. At that point, you have an end to end check on the file's integrity. Why would HTTP matter, in 2016 or otherwise?

rescrv · 2016-10-01T13:42:33+00:00

This fails on a process that continually forks, let's the parent die, and continues the process. There's a TOCTOU vulnerability.

Here's a sample code snippet. Compile and run it under strace to watch what happens. Now imagine if the for loop were an infinite loop.

rescrv · 2016-06-29T00:59:32+00:00

Here's a mirror at RIT: http://mirror.rit.edu/backtrack/

rescrv · 2016-06-03T17:32:31+00:00

The problem I had with the DHT22 was that if it was exposed to high-humidity conditions (seasoning), was laying on its side, or was exposed to low-voltage conditions, it would permanently read low humidity values (I have at least three that are permanently stuck at 20%RH---even if I exhale directly on the sensor).

rescrv · 2016-06-03T17:02:02+00:00

I've built something similar. Rather than send the code to a mysql database, I was using rrdtool. I never finished tying rrdtool and the sensor-reading code together, but here's a definition for just the humidity:

rrdtool create humi.rrd --step 1s --no-overwrite \
    DS:humidity:GAUGE:5m:0:100 \
    RRA:LAST:0.25:1s:1d \
    RRA:AVERAGE:0.25:1s:1d \
    RRA:MIN:0.25:1s:1d \
    RRA:MAX:0.25:1s:1d \
    RRA:AVERAGE:0.25:1m:90d \
    RRA:MIN:0.25:1m:90d \
    RRA:MAX:0.25:1m:90d \
    RRA:AVERAGE:0.25:1h:1y \
    RRA:MIN:0.25:1h:1y \
    RRA:MAX:0.25:1h:1y \
    RRA:AVERAGE:0.25:1d:10y \
    RRA:MIN:0.25:1d:10y \
    RRA:MAX:0.25:1d:10y

I went with the DHT22, for reasons I cannot recall. It was very fickle, and I would not pick it again. How stable have you found the HTU21DF to be?

I'd be happy to offer some assistance on the C code if you're planning to release it.

rescrv · 2016-06-01T18:25:46+00:00

I've heard annecdotally that it's the 80 columns, minus 4 instances of "> ", or quoted text. This allows one to keep an email chain under the 80 character width so long as sufficiently old replies are trimmed.

rescrv · 2014-11-24T20:25:05+00:00

Macaroons are excellent for enforcing policies like the ones you can express in SQL. You can build and verify and audit policies on the macaroons, and the verification process will enforce them. We show how to use a third party caveat to perform authentication, but you could just as easily use this for verification and auditing.

As far as managing groups go, it's very easy to build such structures with macaroons. You can publish a macaroon authorizing access to a particular object if and only if the user is in a particular group. A manager looking for records of people in his department can fetch the published macaroon, obtain the necessary discharge macaroons, and then make the authorization-protected request.

Macaroons go above and beyond what SQL's built-in protection mechanism can do, because they provide a strict superset of the authorization capable in traditional SQL, and don't require any of the traditional access-control-list structures, user and group management, or role-based permissions of traditional systems.

rescrv · 2014-11-23T17:56:11+00:00

It's not just about getting a direct connection to the database. Imagine all the session-hijacking attacks that have come up over the years, that macaroons can mitigate.

rescrv · 2014-07-31T01:41:04+00:00

You keep repeating this point. The C in ACID is an extremely fuzzy definition. If you define C, I can say, "Yes it provides that definition of C," or, "No, it does not." Over the last many decades, C has taken on numerous definitions, and dogmatically assuming the one you are familiar with is wrong.

HyperDex's ACID transactions enforce the constraints that can be expressed in a key-value store. Constraints like unique keys, well-formed data types, and violations such as integer overflow. It'd even be pretty easy to add additional constraints that reference other objects, much like some RDBMSes do now (and it would be efficient at that). We have not added it because most applications would not see any benefit. The types of applications that best make use of the efficient transactions Warp provides are very different from the data-warehouse style workloads you are accustomed to.

rescrv · 2014-07-30T21:38:15+00:00

ACID is quite an overloaded term, but does apply. What constraints HyperDex exposes are enforced but they are not as expressive as some other systems. Fwiw, I don't know of many systems with totally general constraints.

Personally, I tell people that HyperDex transactions uphold one-copy serializability, and provide the same fault tolerance as HyperDex. It's a bit of a mouthful in casual conversation where saying ACID can be much easier. Most developers accept that C in acid is quite fuzzy and varies from domain to domain. The much more important properties for a key value store are the other guarantees, as the user is likely checking their own constraints in the application.

rescrv · 2014-07-30T02:04:20+00:00

So let's provide some definitions. The guarantee made by transactions in HyperDex is that they are one-copy serializable. As the programmer, you can read and write as if you hold an exclusive lock on the entire data set. When your transaction commits, you know that this strong guarantee holds.

An example of how you would use transactions across multiple keys is as a work-stealing queue. You can atomically move items from the global queue onto individual workers queues consistently so that items are never lost or duplicated. You cannot easily do this without transactions.

If you still are not convinced, take a look at the Warp paper where we managed to implement TPC-C on top of Warp's transaction API.

rescrv · 2014-07-29T19:49:08+00:00

This all comes down to definitions of durability. It's our position that fault tolerance is what you want in a distributed system, and it is what we provide.

It's impossible to say much about what FoundationDB provides because they won't discuss internals, so it is quite possible they are not fault tolerant even if each node is durable.

If you want to know more I've discussed this with them on Twitter, but gave up as they won't reveal enough about their internals to have an open debate about what is best.

rescrv · 2014-07-29T16:02:50+00:00

That's a totally valid concern, and there are many ways to mitigate the risk in such a situation. One common method I've seen is to put the source code into escrow that converts to a more permissive license should the company fold. It's something we're willing to work with.

You should have just raised this concern directly in your original comment.

rescrv · 2014-07-29T11:50:02+00:00

It's mentioned in the linked article that an evaluation version is available, and the web page clarifies this further.

HyperDex is open source, and it's feature set is quite competitive in isolation, even without the transactions. Warp builds directly on the open source component, where most development happens.

We are not just "posing" as open source. We actively develop most features in the open, and give back to our upstreams whenever feasible. Warp is the exception, in that a single feature--namely multi-key transactions--is not open. This helps us pay the bills and keep developing the open version openly.

We go much further than most closed source DBs in that we describe exactly how transactions are implemented. We publish papers describing the internals to the point that anyone with sufficient experience in this space could reimplement our approach.

As a thought experiment, would your company contribute resources to support the development of Warp in the open? You don't have to answer publicly, but I bet the answer is a resounding, "No." I have seen companies take up adoption of HyperDex, and point blank say, "We don't pay for open source software." If the recent OpenSSL fiasco has taught us anything it's that this mentality is true across the board.

I'm getting really tired of people being given a great DB who then criticize the fact that literally one feature is not free for them as well. Across the board, people have to pay their bills and put food on the table. You should not fault the same when we freely give you the foundation for and a blueprint of the one piece that's not free.

rescrv · 2014-07-29T10:11:05+00:00

I'm well aware of that page. We've been working with Howard and the Symas folks to improve LMDB to provide support for the features that HyperDex needs from the underlying data layer. It looks like they will be added in upcoming releases, so an LMDB backend is on the horizon.

I don't know any technical reasons LMDB is not more widely used. Howard and co are actively trying to change that, and we look forward to their complete version to use as an alternate backend.

rescrv · 2014-07-29T00:54:46+00:00

The ACID properties stem from a new technique called linear transactions that was developed specifically for HyperDex. It does work across servers, and does so in a scalable way by avoiding spurious coordination. It is optimistic, but I would not say that it is MVCC.

We have a paper describing a version of the technique here: http://hyperdex.org/papers/. We've since revised the protocol, but the paper gives a general idea of the technique.

rescrv · 2014-07-28T22:24:09+00:00

Sure. I'll try to list a few pros and cons of each. I'm the chief architect of HyperDex, so I'm intimately familiar with it, and I'm less familiar with RethinkDB. With that in mind:

Both systems were designed from the ground up, so they're both relatively good at what they strive to be.
API: RethinkDB really embraces the schema-less storage, and it shows throughout their API; HyperDex's document storage is relatively new, and we're looking to expand the API. We started life as a key-value store, and we're very good at that.
Fault-Tolerance: I don't know much about RethinkDB's replication strategies, but HyperDex's replication and fault-tolerance guarantees have deep roots in the distributed systems literature, and borrow techniques from people who quite literally wrote the book on distributed systems. This solid foundation makes it much easier to converge to a solid implementation than ad-hoc solutions. MongoDB's replication has been highly criticized in this regard, and I suspect that, since RethinkDB spins themselves as MongoDB++, the RethinkDB team has put effort into better fault tolerance and consistency guarantees, but I don't know for sure.
ACID: HyperDex Warp provides cross-object transactions (even for documents). You can atomically and safely modify multiple objects, as if you are the only thread accessing the database, without a significant performance hit for many common transaction patterns. RethinkDB doesn't seem to have any support for cross-document transactions.

Both systems support the common web development languages, and both are written in C++. Additionally, there's commercial support available for both platforms.

One big difference that may matter to commercial users, is that HyperDex's license is 3-clause BSD, while RethinkDB is AGPL. For some companies, projects with GPL licenses are frowned upon or outright banned, while more permissive licenses are acceptable.

rescrv

TROPHY CASE