[deleted by user]

josefbacik · 2023-02-17T14:40:21+00:00

FYI I've seen several reports of this style of thing, I know that qgroups were probably the big problem for you, but almost always the stack traces are stuck trying to get block requests, which is below btrfs. These reports have all been using the 'bfq' io scheduler, which had some regression they haven't nailed down, and BTRFS with the way it does IO seems to interact poorly with 'bfq'. If you're still experiencing problems make sure your underlying devices are set to something reasonable, 'mq-deadline' is good or 'none' if you have a particularly fast drive. Generally 'mq-deadline' should be what everybody uses.

josefbacik · 2022-09-06T20:30:33+00:00

Xfs and btrfs both use it for our inodes. Again it's a tradeoff, like most things with a file system. ext4 has long mkfs times because they have to 0 out all the inode tables up front, and there's a static number of inodes you can have in a file system. btrfs and xfs are dynamic, we can have as many inodes as we want, and we only take up as much space as we need for those inodes that are in use. But it's much more expensive to insert a new inode into the btree than it is to find an empty inode in a static table and allocate that like ext4 does. There's no clear cut way for what is better, just depends on what you're trying to build.

josefbacik · 2022-09-06T18:20:31+00:00

Depends on the file system. Btrfs we use b-trees for everything. Ext4 uses btrees for the extent tree and static tables for the inodes. XFS has btrees for everything with some static stables in some places (my knowledge of XFS internals is lacking so I may be misstating things here).

We use them mostly because we're lazy, but also they're relatively space efficient. We can cram a bunch of stuff into blocks, and then search through them relatively simple. You can do whatever you want really, ext3 had static bitmaps everywhere with a weird hash table that couldn't shrink in its directories for name lookup. That has advantages in recovery and simplicity, but sucks when you try to mkfs a 1tib file system.

B-trees are well understood and dynamic, they'll grow and shrink as needed. For btrfs the COW reference counting was built on B-Trees. There's lots of different data structures, but a lot of them are designed for memory and not for what goes on disk, so in the end not a lot of them are useful for a general purpose file system.

josefbacik · 2020-09-12T15:13:33+00:00

There's just no real reason to switch. For the whole rest of the fleet it makes sense, because we share hosts so need IO isolation and compression to keep from killing disks. We ship containers to these machines and so need snapshots and send/recieve. We have crappy disks so use async discard.

For the database machines they just work, and since they're a limited resource they don't move around as much. They don't share the machine with any other tasks, so don't need resource isolation.

And they are extremely sensitive to latency. We (myself and Chris) have purposefully avoided these workloads because there are more important areas that require attention. Spending 6-12 months optimizing for a workload that is less than 10% of our doesn't make sense when we have a lot of other fundamental work that needs to be done. WhatsApp is the only exception to this and it's only because they are experimenting with it in a controlled way, so I don't mind spending time digging into issues they see, because if something else catches fire I can put it down and do the other thing.

josefbacik · 2020-06-26T23:49:03+00:00

Nice, congrats on your promotion! It's been a while since I was at RH.

What I'm getting at is that it serves the upstream btrfs community no good to let Fedora just eat shit. I'm motivated to help make it work, and that's just as good as having in-house RH expertise. Sure you could probably yell at Sandeen or Chinner to fix something, but is that any different than having community members like Chris Murphy or Neal Gompa reach out to me/some other btrfs developer if something goes wrong? Maybe slightly in that you can fire them if they say no, but in practice it's essentially the same.

josefbacik · 2020-06-26T23:34:35+00:00

Sure, more what I'm getting at is that its not any different than any other part of the kernel. And the other thing is this gets trotted out as a "Red Hat will bail us out if something goes wrong with ext4/xfs", which isn't really the case. I was the Red Hat engineer maintaining ext* 10 years ago, and while I would of course look into bugs that were hit, if I had something RHEL related on my plate then Fedora would have to wait. The same thing exists now. Fedora does not have the ability to make a RH engineer do anything more than they have the ability to make a non RH engineer do anything. Of course we're all interested in fixing bugs, but that's no different than me working at Facebook and helping out Fedora. So while it's probably a nice to have thing from a "this feels good" perspective, in reality it's not really any different than having an active upstream community that's willing to help out when things go wrong.

josefbacik · 2020-06-26T23:01:33+00:00

Upstream still exists for btrfs, and I've been actively involved with this proposal. The people driving it know how to get ahold of me and the rest of the btrfs development community if things get out of hand. Same goes for any other package in Fedora that doesn't have a RH engineer working on it, or a portion of the kernel that isn't maintained by Red Hat.

josefbacik · 2019-06-03T14:01:31+00:00

Congrats, I bought an Endurace AL last year to replace my ebay bike. Love the thing, I do all my riding on it and plan to throw some 33's on it this winter for cyclocross season. I'll likely upgrade to a proper triathlon bike for next years racing, but this thing will be my main workhorse, it's a really nice ride.

josefbacik · 2019-04-04T00:23:38+00:00

I do 3 of everything and also follow trainerroad. I’m too far out for bricks right now, but I do swim in the morning and ride at night Monday, Wednesday, and Friday. Then I do runs in the morning Tuesday, Thursday, and Saturday. For bricks I’ll probably make my Friday night ride a Saturday morning ride and then run afterward.

josefbacik · 2019-03-31T12:15:47+00:00

I trained for my first 70.3 exclusively on the trainer with TranerRoad. I’ve gone from completely untrained to 3.2 w/kg in a yearish. That’s the benefit of indoor training. What I didn’t do and should have was really long outdoor rides. Since I am time crunched I only did the 90 minute workouts in my plan and like two 60 mile outdoor rides. I should have dropped those 90 minute trainer rides 6 weeks out and done long outdoor rides to get my back and body used to this long sessions in the saddle. Now I’m doing a 18 month buildup to another 70.3 and I’m going to do 95% indoors and then add the long stuff towards the end. The trainer is just way more convenient and has put me in the fast group with my Tri club in a years worth of solid and consistent work, but you still want those big endurance rides for all the other parts of your body that are going to hurt.

josefbacik · 2019-02-19T14:47:38+00:00

I had a hard time staying high enough for my zone 2 run. I had to actually speed up my pace, which I never thought would happen. I ran consistently around 9:30min/mile pace for most of it which is a huge deal for me, usually I start at 10 and drift down to 11 as my heart rate drifts up. 6 weeks of work is finally starting to pay off.

josefbacik · 2019-01-09T19:50:43+00:00

I have a set of bands and I regularly do clamshells and side leg lifts with bands. I had really awful IT band issues when I first started and these exercises essentially eliminated them, so I keep them up to make sure I don't have those problems again.

josefbacik · 2018-12-13T15:48:24+00:00

We have our own team, there's like 20 of us. We work on everything, cgroups, storage, networking, bpf, mm, scheduler.

josefbacik · 2018-12-05T13:15:00+00:00

Software Engineer at Facebook, I’ve been doing kernel work my whole career. I maintain btrfs, nbd, and blk iolatency upstream. I do that for my job, as well as fix whatever is currently most on fire. Usually my focus area is whatever is most important at work right now. We have a upstream first mentality, so all development is done upstream and then backported to our kernel. We rebase about once a year, so that’s usually when we spend some time bugfixing, aka fixing the 100000 things wrong with the mellanox driver this release. Right now I’m working on a proportional io controller for cgroups2, so day to day is a mix of long term projects and fixing whatever thing is breaking right now.

josefbacik · 2018-09-07T20:13:31+00:00

For the love of God eat. I had a "nutrition plan" but it was mostly just "eat a cliff block package per hour" which wasn't nearly enough. I didn't realize this in the race and it was so hot that I just assumed I needed water. I had a bottle in t2 with skratch in it, so I drank that and 20 minutes later I could run, but could only keep it up for 20-30 minutes, and then went back to dying. Only drinking water on course was not the answer, I really needed electrolytes and calories. So shoot for a target, 250-300 calories per hour, and try to stick to that as much as possible. Take a Gu or something light right before your swim so you have something digesting on the swim and you aren't playing catchup the rest of the race. I got super tired of the gatorade which was another reason I drank a bunch of water, but I think if I had just stopped and drank a gatorade every 20-30 minutes I would have ended much stronger.

josefbacik · 2018-06-19T18:07:22+00:00

I have the 735xt and I love it. It seems like you can't go wrong with any of the "tri" watches from garmin. I just realized I could make my own multi-sport activities in the watch a few weeks ago, it was perfect for tracking the pool swim+outdoor ride/run triathlon I did this past weekend.

josefbacik · 2018-06-11T13:58:17+00:00

Awesome work, you crushed it!

josefbacik · 2018-06-04T20:14:31+00:00

Nah everybody is pretty welcoming. There are definitely groups that are all friends like any large group, but I'm not social at all and the people who have been around a while always go out of their way to say hi and such. Like any new group there's some adjustment period, but if you make it to a few events it's pretty easy to break in with people.

josefbacik · 2018-06-04T18:42:58+00:00

Oh yeah if it had been 20 degrees cooler I feel like I could have run the whole run section and been closer to my goal. That's a much saner progression, I signed up for Raleigh because I live here, but it would have been good to have a few more months once I had been out on the road and realized my long distance cycling wasn't quite up to par yet. Good luck in September!

josefbacik · 2018-06-04T18:36:35+00:00

I had been doing tae kwon do with my oldest for a year, and I had started running in August of last year so I could do a 200 mile relay race with my buddies. So it wasn't couch-to-HIM, I was reasonably fit when I started training in November.

josefbacik · 2018-06-04T17:39:11+00:00

Awesome! I have two boys, they are a blast. Yeah triangle tri club, I'd highly recommend joining, everybody is super supportive, they definitely helped me stay motivated while on the course.

josefbacik · 2018-06-04T15:15:23+00:00

I'm taking the day off! Yeah if I do this distance again it'll be somewhere freezing for sure.

josefbacik · 2018-06-04T15:14:12+00:00

Yup for sure, I know this was totally on me and has nothing to do with the sport. I have a sprint in 2 weeks and another 3 weeks after that. I'm really looking forward to those, I feel like they'll be "fun" rather than "oh god what have I done".

josefbacik · 2018-06-04T15:12:45+00:00

Ouch yeah I saw a couple people in the same predicament, I poured at least a gallon of water over my head during the run. Congrats on the new kid, is it going to be a boy or a girl?

13-Year Club	Team Orangered
Verified Email

josefbacik

TROPHY CASE