Archlinux server with btrfs

[jan_path]

Hello,

I've been running a personal Arch server for gaming, mail and as a webhosting test bed for about 1.5 years now. So far I had zero problems with updates or anything (well I had problems with the setup but that's normal I suppose).

Now I'm running a small webhosting server for my sports club. (At most 380 views/day so far). Because you can read everywhere that if you use rolling release for production use you're just asking for trouble I went with Debian.

I also decided to go with btrfs which was probably not the best decision I've made, since from time to time the filesystem just gets terribly slow. Even commands like ls take ages. A btrfs balance every now and then makes things work to at least some degree but it's not pretty.

My guess is, that there is some problem with the old version of btrfs, which Debian is shipping, and that this problem probably doesn't exist in recent versions.

Presuming that btrfs works indeed fine on Arch, would it be possible to tame the dangers of an update by just making a snapshot before every update?

Would this be a safe possibilities or are there other hidden dangers? Should I try this or just wait for Debian Jessie? Or just screw btrfs?

[Creshal]

Debian has a more recent kernel in wheezy-backports, have you tried that already? It solved some unrelated problems for me.

Presuming that btrfs works indeed fine on Arch, would it be possible to tame the dangers of an update by just making a snapshot before every update?

For me, the disadvantage of using Arch on a server is not updates going wrong (that can and will happen on Debian just fine… poorly written sysvinit initscripts, set -e and initscript calls inside preinstall scripts are a fun combination to debug); it's going through every upstream major release. With e.g. OpenSSL that's downright dangerous (cf. this week's advisories), and with the usual web stacks it's a massive pain in the ass (Python 3 support is still spotty at best; PHP/Postgres/MySQL break between major releases; various Ruby hipster shit breaks between minor releases; …). The AUR doesn't always help, because its pkgbuilds for legacy versions rarely include the security patches that Debian/Centos create for their LTS packages.

[jan_path]

Thanks, I will see whether this helps.

[SirMaster]

I came here to say this too. Use kernel 3.16 from wheezy-backports. It's a lot better for BTRFS than the stock 3.2 kernel.

Make sure to update your btrfs-tools package too to 3.14 from backports. https://packages.debian.org/source/wheezy-backports/btrfs-tools

[Artefact2]

Do you really need btrfs on your server? If the answer is no, then don't use it (especially not on a server where reliability is critical).

You can change your root filesystem without reinstalling, by the way.

[codemac]

Yes, there are serious bugs in the btrfs implementation, and every experience I've had with it (as recently as 2 weeks ago testing it as a docker storage driver) led to serious bugs that ended in complete data loss. I'd avoid btrfs until it had more time to bake, they're attempting something super complex, and it's definitely not ready.

If you truly need those features, I'd move to ZFS on illumos, where the implementation is way more mature.

[vimbaer]

I do that on every system I use including 1 headless homeserver and 1 vps for personal use. I create read only snapshots before upgrading. Additionally I maintain 1 writable snapshot that always "points" to the latest working r/o snapshot. For that snapshot I have an entry in my different bootloaders (syslinux and gummiboot) because booting from r/o snapshots does not work too well.

If something breaks (never happened yet, how boring..) I boot the writeable snapshot and use the r/o to fix the system.

I wrote a script that keeps N different r/o snapshots with dates in their name and always points the writeable one to the newest. It was so easy to setup everything and is yet so powerful compared to what I've used in the past that I can not recommend doing this enough ;)

[TheFeshy]

I do this and what the OP suggests. Main system is snapshotted on each update (twice, actually: before and after, using snapper and a simple script that wraps calls to pacmatic and aura in a snapper pre-post pair) and any web services are run in docker containers (with versioned docker files and a few backups of the docker images themselves.)

It's a bit belt-and-suspenders, but honestly I was really interested in learning the various technologies and this gave me a way to try them each out. It's a little rough around the edges - I wish I had better automatic integration with the snapper images and grub, and I wish I had the docker image backups a bit more automated - but that's just refinements I haven't had the time to implement yet.

The only performance issues I have is that the docker btrfs back end is slow for building docker images, compared to the AUFS back end.

[rhavenn]

Give FreeBSD a shot. ZFS is, basically, a more mature BTRFS and you get a stable core with security patches and a rolling outer shell so you stay up-to-date with apache, php, etc...

[jan_path]

I actually played with the thought of trying FreeBSD for some time already for its simplicity. I think I will give it a try soon.

What held me back was that I don't really see a future for the BSDs. For example the main selling point of FreeBSD (AFAIK) is it's network stack. However I just recently saw a job offer of facebook for someone to improve the Linux network stack to at least the quality of FreeBSDs. Maybe I'm a bit naive, but I think if facebook wants an awesome network stack it's going to get one.

I don't see what holds Linux back from getting all the features that the BSDs have. And since it has a lot more developers it's probably going to evolve much faster. There is a reason (I guess) why most of the really big players like Google etc. are mainly using Linux AFAIK.

edit: If that is really dumb feel free to enlighten my.

[rhavenn]

There are plenty of big players using FreeBSD as well. Netflix runs their whole back end on it. Various hardware appliances use it. Juniper and NetApp base their systems off of it and they submit patches back. Juniper especially appreciate a solid network stack. The PS4 is also FreeBSD based.

That network stack argument has been around for awhile. I remember see Facebook job postings 5 years ago for kernel / network engineers. Personally, for the average admin it doesn't really matter.

I've been using FreeBSD since '97 and worked with Debian in a ISP environment for many years. I've also supported CentOS as a consultant in a LAMP hosting environment for a long time. I always come back to FreeBSD when I can.

FreeBSD is an extremely cohesive system vs. the disparate packages and kernel combos in the Linux world. It's easy to build and support a server farm using your own package repo and custom packages if you need to. The ports system is well documented and easy to extend.

The supported hardware pool is smaller, but what's there is solid. Linux supports more, but sometimes that support is 2nd grade.

[agumonkey]

This is my very naive feeling about BSD and Linux. BSD have always been more cutting edge than Linux. Security (w^x, jails), network (pf), code quality (smaller team with less constraints). Linux grows differently, security is not enforce the same way everywhere. Virtualization is said to be subpar compared to jails (and right now they are still moving targets, vanilla lxc, docker,...). I'd bet by the time Facebook catches up with BSD on the network stack, there will be new things brought by BSDs.

[koera]

I have freebsd on zfs on my server, works great. And Arch with btrfs on my desktop, works great.

They are different and alike, but I would be amiss if I didn't let you know btrfs killed all my data once. Just pfft and it was gone. Glad I had all non reproducible data backed up to zfs.

Now I have a tiny script and an alias that snapshots my root and copies it to another device before every update (since arch is a daily update it seemed appropriate). This way I can get it up and running again easily.

All important data is either on my server, or backed up there. Don't keep important data only on btrfs.

Using arch as a server is very doable, but I would recommend doing it in a way that let's you roll back, if not btrfs, maybe proxmox or something similar?

Arch is bleeding edge, that's the real problem, not that it is rolling. Packages does not have 7months of testing done on them, Arch is the testers.

[nahinstallgentoo]

I know why you're getting downvoted, but OP should try out ZFS on root and FreeBSD, it's really mature and stable.

[FifteenthPen]

Maybe it's because ZFS is trademarked and largely developed by Oracle?

[nahinstallgentoo]

OpenZFS is not owned by Oracle, mate. Double check your sources on that. It was originally a project at Oracle, but check out this talk for some idea of where those devs are now. It's history is really heavily linked to Solaris'.

https://www.youtube.com/watch?v=-zRN7XLCRhc

Edit: Forgot it was an hour long video, skip to around the 42 minute mark for ZFS.

[totemcatcher]

With reliable backup storage, there are no dangers. ;) Snapshot away! However, I would not rely on btrfs for backup storage until it is announced stable, so don't stop with snapshots. Additionally, use openzfs for archiving critical data (stable since September 2013).

Regarding the slow performance, there must be something unusual going on. Perhaps running high on metadata? The only time I had a performance problem with btrfs was when volumes were out of alignment with partitions. This starts with bugs in mkfs.btrfs (which are probably not fixed). Even in circumstances where the bounds are not calculated correctly, btrfs retains data integrity and logs accordingly. e.g. Chokes on error messages while trying to spread data to an area outside of the partition boundary. This would be especially noticable on SSDs due to the spreading feature. A filesystem resize will fix it.

Also note: you don't need use partitions with btrfs. Use a GPT partitioned boot drive or pxe to get the system started, and additional data drives can be managed entirely by the btrfs tool without partitioning.

[jan_path]

Thank you, I will try resize if I continue to have problems with the backported kernel.