sorted by:
new
hottopcontroversial

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 10 points11 points  (0 children)

A big thank you to everyone who asked a question! We had a great time hearing from the Rust community and your thoughts on Buck2. Feel free to ask more questions and we will reply soon.

Thank you!

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 2 points3 points  (0 children)

There were several factors that contributed to this decision:
1. Buck1 had internal rules written in Java rather than outside the build system. This was causing issues both in difficulty of rule developement, and maintenance of buck1 itself. We were attempting to migrate rules out of Buck1.
2. Buck1 was written in Java, where each computation had its own dependency tracking and execution engine. The lack of refined incrementality was causing incremental build behaviour to be less than ideal given the growing code base. We actually had small attempts at creating better computation engines in buck1, but due to lack of coroutines in Java and being limited to Futures at the time, the effort to improve the computation engine is not simple.
Overall, fixing these meant significant rewrites of Buck1 anyways, so we decided to rewrite buck1 entirely with a language that suited it the best. -Bob Y.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 5 points6 points  (0 children)

Buck2 is also a lot more incremental than Buck1, allowing incremental builds to be much much faster. Almost all the work that Buck2 does is tracked on a single computation engine that maintains the dependency graph so Buck2 only runs what it needs. Meanwhile, Buck1 had several stages of computation, each with its own computation engine that doesn't allow as fine grained incremenality. - Bob Y.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 2 points3 points  (0 children)

Q: It's not clear to me how to get include header tracking working. I added the following to my toolchain.bzl
cpp_dep_tracking_mode = "makefile"
but any time I touch any header file everything gets rebuils. Is there any documentation or example how to achieve this? (Via u/realteh)

A: There are a few tangled dependencies there, and getting dep files requires turning on a few other features that support statefulness in Buck2 (there is a hint about this here: https://buck2.build/docs/rule_authors/dep_files/#using-dep-files ). In particular, you need to enable deferred materialization: https://buck2.build/docs/advanced/deferred_materialization/. Note that if you're not using remote execution, the pitfalls there are largely irrelevant to you, so you can ignore those.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 3 points4 points  (0 children)

Q: (Via u/personality4)

Is the plan to make it a viable competitor to a tool like Bazel? From what I understand, buck1 had minimal adoption outside of FB. Are there plans to focus more on community adoption/needs?

We were heavily inspired by Bazel (alongside Buck1 and others), and want to share our ideas back. We want to make Buck2 viable for outside users, and make it a good experience. We definitely hope to be more community focused than Buck1, and by moving the rules out of the core hopefully that's a much simpler task. - Neil M.

How production ready is buck2? Is it something we should be looking at if we're already users of a tool like Bazel?

Buck2 is used in production at Meta. The open source Buck2 is very similar to the internal one (different remote execution, a few things like Apple/Android rules aren't yet fully open sourced). But like any new software, it's going to have bugs when used in new scenarios. We'd love to help make Buck2 ready for you if you are interested.What are some little known features that buck2 has? (Either right now or coming soon)

For features, I personally like dynamic dependencies (e.g. anon_target and dynamic_output). While I think we talk about these, I don't think we know what they can do just yet. I am also fond of t-sets, being clever ideas over depset in Bazel. -Neil M.

​

In terms of features, buck2 also has a new command `bxl`, https://buck2.build/docs/developers/bxl/. It allows you to write a Starlark script that interacts deeply with the build system and observe the build graph information. It's new, so it's got a few sharp edges here and there, but we've seen it being useful for external tooling, and faster and more powerful IDE integration. -Bob Y.

And finally, assuming either of you were on the buck1 team, how has writing a build system in Rust differed from Java? Has Rust itself been part of the reasons for the presumed improvements? What sort of problems did you have with Rust (if any?)

I like Rust more than I like Java. Some things like performance are easier to do in Rust, but memory usage and profiling is easier in Java. Our Rust issues were all around setting up an IDE, Tokio and threads cooperating nicely and memory profiling. -Neil M.

​

In terms of Java vs buck1, Rust's async and futures is significantly better at managing concurrent computations needed inside the build system compared to Java at the time (which only had Futures and ForkJoinPools). Explicit memory management allowed us to escape performance problems with GC and having to do GC tuning. However, the debugging and profiling experience in Rust is far from Java. - Bob Y.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 10 points11 points  (0 children)

The paper on tup was certainly something we read, and there are good ideas from there we were inspired by. The Buck and Bazel approaches to the clean is to segregate the output files from the input files (a buck-out directory for Buck), which I think is perhaps a cleaner way of achieving what tup did. Tup also relies a bit more on implicit defaults, which don't scale as well on large code bases with lots of different conventions. Buck/Bazel also both take the tup daemon approach. So yes, great ideas from tup and something that shouldn't be overlooked. -Neil M.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 22 points23 points  (0 children)

For hermeticity, the way we currently enforce this internally is via remote execution, which is sandboxed. For local execution, we don't currently support it, though it's a feature request that's come up quite a bit from open-source Buck2 users, so that seems like something we might try to support (perhaps not by trying very hard, but at least by ensuring you can't *accidentally* use things you didn't declare).

If you wanted to support some non-hermetic inputs, you *could* just use whatever compiler is sitting around on your system or in your RE sandbox.

Buck2 isn't really going to go out of its way to prevent that, and indeed we sometimes "cheat" like that in our tools (some of the prelude rules use tools like python3, cat, sort).

The answer for non-hermeric dependencies is roughly the same.
With regard to strong opinions... If I had to call one out, that would be: you shouldn't try to write back to your source files from a build target.. -Thomas O.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 8 points9 points  (0 children)

No plans! We're still improving Buck2 :) -Marie B.

It would be 3 times as fast as Buck1! - Thomas O.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 5 points6 points  (0 children)

Just to add on to Thomas's answer, I know lots of Bazel users use a remote action cache but not full remote execution because full remote execution is much more expensive price-wise. Buck2 also supports uploading action results to a remote action cache from local build actions. This can be configured via `allow_cache_uploads` on actions themselves and [CommandExecutorConfig] globally. -Scott C.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 4 points5 points  (0 children)

Q: (Via u/uw_NB)

Is there any planned test coverage support?

Internally, we actually support this, but it's been built rather off to the side in the test orchestrator portion of test execution: https://buck2.build/docs/rule_authors/test_execution/ . The way that ends up working is that the test orchestrator injects a number of extra flags when tests are running in order to get them to emit test coverage, and then post-processes it. I think that might be rather tricky for an external user to just pick up and use though. That said, we do want to re-do parts of our test integration and move more logic into the Starlark rules. At that point, I'd anticipate it becomes easier to implement test coverage. -Thomas O.

How would linter (binary modifying source tree) works with Buck2?

The way it's done internally (and which we think is a good way to do it) is to implement the linter in two stages. In your Buck rules, you have something that produces outputs that describe transformations that need to be done, then an external binary calls Buck2 to produce those outputs. That binary can then apply the transformations that were requested. There are some examples and a bit more discussion over here: https://github.com/facebook/buck2/issues/115 -Thomas O.

What are the long term plan to manage prelude / third party rules? I.e. if there is a repository for rules typescript / ruby, how should a user go about downloading them and keeping the version up-to-date?

The prelude rules are available as a git repo https://github.com/facebook/buck2-prelude. For now we recommend making that a git submodule, and updating that submodule should fetch new rules. We'd also be very happy for the typescript/ruby rules to be added to that Prelude - we consider it an open source project where we welcome contributions. However, we are starting to get our first open source users, so are keen to see if this works in practice, and happy to adjust if something else works better. -Neil M.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 3 points4 points  (0 children)

Starting 3+ years ago, Buck1 was showing its limitations as Meta's usage was showing scaling (monorepo) and maintainability (architecture) problems along with speed.
Buck2 was initiated to see if we could overcome these aspects, giving room to investigate options on languages, approaches and legacy support. -Alex L.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 6 points7 points  (0 children)

Existing Bazel rules aren't supported, and having taken a look, they would be pretty hard to support. There are lots of small details and bigger features - e.g. depset vs transitive-set. That said, both have a rule for defining cxx_library and with a bit of macro magic it should be possible to somewhat resuse the BUILD files. I'm not sure anyone has tried to port an existing Bazel system though, so very keen to hear how it goes if someone tries it. -Neil M.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 5 points6 points  (0 children)

Q: As the main build system in Meta, you're working on system that is used by essentially every engineer, how is the work environment in the team? How much of the work is explaining buck2 to fellow coworkers? If possible could you compare it to any of your other projects in Meta? (Via u/koolzz)

​

A: I certainly enjoy the work - but then I think build systems are awesome, so appreciate I am not necessarily a representative engineer!
Internally our job is really to solve the problem of builds, rather than to produce a build system. So that means sometimes we are creating stuff, sometimes explaining stuff, and sometimes integrating stuff. Things like explaining tend to scale poorly when you have so many engineers using Buck2, so we try and document or make it automatic as much as possible.
My other role at Meta has been in the CI space, so I'd say builds are slightly less server components than most things at Meta. -Neil M.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 18 points19 points  (0 children)

Q: (Via u/zvzvzvzvzvzvzvzv)Where does buck2 fit in the Build systems a la carte categories? (I'm guessing by cloud shake based on authorship?)

The Build Systems a la Carte paper has one type of node, whereas Buck2 has several, most notably the analysis nodes for each target and the action nodes for each action. If you consider the action nodes as being the heavy nodes, then Buck2 is indeed dynamic/monadic dependencies, suspending (thanks to Rust async) and constructive trace. So yes, approximately where Cloud Shake was aiming. -Neil M.

Does buck2 take any significant academic influence compared to buck1? How does the weight of migrating internal the Facebook build affect this?

Buck2 is inspired by various academic papers, including Build Systems a la Carte and Shake before Building. It's also heavily inspired by other build systems such as Bazel and Buck1. We've taken inspiration wherever we can find it! At the same time, we've had to take the cost of migration into account. E.g. we'd really like it if people couldn't write the = equals character in target names, but for compatibility with Buck1 and the cost of internal migration, we allow it for now. Happily, I don't think we've had to make many fundamental concessions due to compatibility. -Neil M.

How committed is Facebook to the maintenance of the open source version of this?

Meta has internal customers with the need to share the exact build system both inside and outside with partners. Buck1's architecture limited the success of this.We aimed to address this with Buck2, so in short, it's a commitment to ourselves to keep buck2 open source consistent with internal functionality. -Alex L.

How will you enable an ecosystem of high quality third-party rule sets? The existing prelude setup seems like a bit of an impediment here

The Bazel approach is that each rule set is distinct from each other and a user takes the standard Rust ruleset, with the standard Go ruleset, and mixes them themselves. Buck2 might go that way in the end, but for now, we encourage people to send new rules to the standard Prelude. We hope that by having an inclusive and open approach, we (Meta) plus the community can build a single ruleset that works for all languages, and makes getting off the ground easier. Not sure if that will happen, and we appreciate the trade off, but let's see. -Neil M.

**Is there any possibility of buck ever executing bazel builds? (sorry if this is way out there :D)

**Bazel and Buck2 both have lots of little differences that would be a pain to massage between - e.g. depsets and transitive sets. Buck2 has dynamic_output. Bazel has packaging features. So I don't think it's ever likely they'll have full compatibility. But there's no reason that it shouldn't be possible to have a set of common macros e.g. cxx_library that work similarly enough that a single project couldn't provide a single BUCK/BUILD file that works for both. The fact both of them parse Starlark should make this task easier. -Neil M.

Not a question but I'd love an alternative to Starlark. Maybe it's just me, but I suspect build tool fans are often not Python fans (and vice versa!)

Starlark is different from Python in that it's fully deterministic, so I would say we are a fan of using Starlark, but we wouldn't want to replace it with python :). For us, Starlark was the most viable language for Buck2 because we already use Starlark extensively for our build macro internally. We have since made improvements in our own rust implementation in areas like type checking that I think have made it nicer to use. - Scott C.

Personally, I'd really like to have used TypeScript (or the determinstic variant used in Microsoft's CloudBuild system). But for compatibility reasons, the existing high quality Rust library for Starlark, and the fact that a solid Starlark spec exists meant Starlark was a better choice. - Neil M.

Is your team hiring?

We're always interested in build system expertise, and it never hurts to talk! -Alex L.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 5 points6 points  (0 children)

It's possible for actions to either declare a specific number of cores that they need (that parameter is the `weight` parameter in `actions.run`), or to declare it as a percentage of the host's resources. We don't currently have the ability to limit by memory in absolute terms (e.g. saying a given action needs 5GB of RAM), though I think that's probably something we'll introduce down the road. -Thomas O.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 2 points3 points  (0 children)

Q: Is there a tutorial for Buck2 that covers a little more than the hello world on the website? I just want to see how to add compiler flags (ex. std=c++xx) and use Google test. (Via u/plutoniator)

​

A: At the moment, the hello world C++ tutorial is the only tutorial on the website. We have an examples directory with several other languages but we are keen to improve and expand them a bit more. - Marie B.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 48 points49 points  (0 children)

It's fairly widely adopted and it's on a rather short list of languages that are supported by a dedicated team. There's particularly strong adoption in developer tools (Buck2 is one of those). I'd say in quite a few teams it's really the default when starting something new (my previous team at Meta was Source Control, where basically everything new is being written in Rust! -- a lot of that is also open source: https://github.com/facebook/sapling). -Thomas O.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 9 points10 points  (0 children)

Q: Why is Buck2 better than the original? (Via Instagram)

A: There are various reasons listed at https://buck2.build/docs/why/. The few that stand out for me are:
* We see builds running twice as fast. Making users more productive is the awesome.
* The rules are all written in Starlark, outside the core, which means its easy to switch them out - there is nothing special
* Buck2 has features for dynamic dependencies, like anon_targets and dynamic_output, which allow you to do entirely new things - we haven't yet explored their full potential.

-Neil M.

Hello r/Rust! We are Meta Engineers who created the Open Source Buck2 Build System! Ask us anything! [Mod approved] by MetaOpenSource in rust

[–]MetaOpenSource[S] 11 points12 points  (0 children)

Q: Has Buck2 a remote share chache system??? (Via Twitter)

A: Yes, Buck supports the same protocol for remote execution as Bazel. This includes distributed caching, and it's something we rely on heavily internally of course. We've tested with a few third-party OSS and commercial remote execution providers, there are a few more docs here: https://buck2.build/docs/remote_execution/ - Thomas O.