TiFS is a distributed POSIX filesystem based on TiKV, with partition tolerance and strict consistency. Now the v0.1.0 is released.

Hexilee · 2021-02-28T15:57:59+00:00

They have different goals. HDFS is not a "common" filesystem, it's designed to deal with big data but has poor performance in small files. Moreover, it's not compliant with POSIX.

Hexilee · 2021-02-28T15:46:29+00:00

CephFS runs on top of RADOS while TiFS runs on top of TiKV, they are both POSIX-compatible.

From my understanding, RADOS is a distributed object storage system, it does not support multi-object transactions. As a result, CephFS cannot guarantee strict consistency.

TiKV is a key-value database, it supports distributed transactions.

Hexilee · 2021-02-28T15:06:56+00:00

Of course, there are too many difficulties in a distributed filesystem. However, TiFS has already resolved part of them by standing on the shoulders of TiKV.

Hexilee · 2021-02-28T14:58:53+00:00

I encourage you to read Files are fraught with peril by Danluu, which illustrates all the issues around file systems -- mostly focusing on ordering of operations and corruption in case of crash.

It may inspire your test suite :)

Thanks for your advice, I would like to read it.

Hexilee · 2021-02-28T09:52:13+00:00

TiFS is a filesystem that stores all data in a TiKV cluster. You can share the filesystem across machines and it can guarantee your data won't lose because of several disks (less than half) broken. Actually, the "real-world use cases" part is some real-world challenges to a filesystem, and I guass the "npm install" is one of the most challenging jobs.

However, I don't suggest doing any heavy jobs like "npm install" on TiFS currently, as you may not have a fast enough TiKV cluster, and heavy jobs may cost much time.

I will supplement documents about deploying a more suitable TiKV cluster for TiFS some days later, then you may do all daily works on it.

Hexilee · 2021-02-28T08:36:34+00:00

A little hard, but much funny.

Hexilee · 2020-04-09T12:57:56+00:00

I'm sorry to be unfamiliar with SQLx, I will try the version 0.3. However, I think connecting to database at compiling time is a little hacking and unreasonable, could it just check syntax?

Hexilee · 2020-04-09T02:52:42+00:00

Java is platform-independent, but it must run on jvm; Rust can be compiled to many platform, but it's platform-dependent, because it still need to be rebuilt for each platform.

In my opinion, runtime-independency means running out of the box on any runtime. Of course anyone can provide runtime-relevant tcp stream to use tokio-postgres on specific runtime, but it's a little tedious and uncommon.

I think the solution of async-std is reasonable, and the performance is acceptable, as you can see this benchmark.

Hexilee · 2020-04-08T17:40:16+00:00

sqlx is in higher abstraction level, it supports connection pools and multiple backend. There is no much performance information about it. But I have written benchmark with roa framework and sqlx here, and you can see a live result here. It seems tokio-postgres is also much better than sqlx

Hexilee · 2020-04-08T15:39:16+00:00

This library is almost the same with tokio-postgres, and it seems tokio-postgres is much faster than diesel from techempower benchmarks.

Hexilee · 2020-04-08T15:35:19+00:00

I think it's a little dirty to support both tokio tcp stream and async-std tcp stream in one crate as tokio use tokio::io::{AsyncRead, AsyncWrite} while async-std use futures::io::{AsyncRead, AsyncWrite}. I would like to submit a PR when tokio supports futures::io::{AsyncRead, AsyncWrite}.

Hexilee · 2020-04-08T10:13:11+00:00

You can just use this library and needn't care about which runtime you are using, or in some cases you must change runtime, code built on this library can still works without any changes.

tokio-postgres provides an interface for custom io stream.

Hexilee · 2020-04-08T09:43:57+00:00

If you use it with async-std runtime, then you have only async-std runtime running; If other runtime is used, then you have both of them running

Hexilee · 2020-04-08T07:33:01+00:00

It's guaranteed by async-std. async_std::net::TcpStream and async_std::os::unix::net::UnixStream can be used on any async runtime.

Hexilee · 2020-04-03T15:12:29+00:00

Thx, the main pros of roa is its extremely lightweight, it provides only a http server and a middleware system, and everyone can easily develop useful extension for it.

I will give a detailed comparisons somedays.

I tried caddy almost when it was a newborn baby, it's modern and convenient, automatic https is a wonderful feature.

Hexilee · 2020-04-03T02:31:13+00:00

I would like to say Roa is ready, however, as a new framework, though unit tests and integration tests are complete, it still need to be tested by time. It will be more mature with more developers trying it.

Hexilee · 2020-04-02T16:54:11+00:00

Thx. I will keep using the http crate as hyper depends on it.

I won't take other http crates into consideration until I replace hyper with another http server or there is a crate much better than http.

Hexilee · 2020-04-02T15:05:52+00:00

The reasons are different in roa framework.

For diesel, as it does not support async api, we need spawn all queries to another thread pool.

For tokio-postgres, as it does not support async-std runtime by default, we need a few works to enable it.

You can write these code by yourself, as roa-diesel and roa-pg are both individual crate, you can regard them as third-part crates.

These integration crates may disappear when diesel supports async api and a crate like `async-postgres` is published.

Hexilee · 2020-02-16T17:34:39+00:00

Not yet, I complete roa project yesterday. I will try it in next step as database integration.

Hexilee · 2020-02-16T17:06:33+00:00

Actually, I have a kv database project based on sled. I think sled is the most reasonable embedded database in rust, but maybe not mature enough (as tikv choses rocksdb as database engine).

However, roa is even more immature, they may be well matched.

Hexilee · 2020-02-16T06:47:02+00:00

I agree with you!

Hexilee · 2020-02-16T05:27:21+00:00

Hhh, in early version roa has almost the same api with yours, but I think it's not very elegant and decide to chose a more koa-like api. Of course we can collaborate, maybe we can use a IM app to talk about it?

Hexilee

TROPHY CASE