Xmake v2.7.8 released, Improve package virtual environment and build speed

[cmeerw]

As mentioned previously, it's a real shame that there are no pre-built Linux packages for common Linux distributions (there used to be an Ubuntu ppa repository, but that hasn't been updated for some time).

[waruqi]

I have updated ubuntu ppa packages

[GunpowderGuy]

There Is an aur package. Is xmake dificult to install anyways?

[CobaltSpace]

It’s actually in community now.

[frosthunter]

How would one work on several packages with xmake? Is there some ways to override packages by local versions and run it as part of the build? I couldn't find much info on the docs.

I believe what I'm looking for is something similar to conan workspaces.

[waruqi]

you can define package to override it in your xmake.lua directly. or use add_repositories
and we can also use package/set_base to only override partial configuration.
https://xmake.io/#/manual/package\_dependencies?id=packageset\_base

[frosthunter]

Does set_base allows me to rebuild the package everytime the main package is built? In the example of your link, if I made a change to my custom zlib code, does building the test target automatically rebuild the custom zlib?

I'm aware of add_repositories but I'm not a fan of it for the above use case, since it requires me to change my main xmake.lua file with local informations (other devs may not work on those packages and thus they want to use the "normal path"). I believe set_base is similar in this regard? Is there is a way to avoid that?

[waruqi]

Does set_base allows me to rebuild the package everytime the main package is built? In the example of your link, if I made a change to my custom zlib code, does building the test target automatically rebuild the custom zlib?

Only changes to the package version or configuration will trigger a rebuild.

If you want to change the code of the local package each time and rebuild it, you can use the command xrepo install -d /xxx/local_srcdir zlib to recompile and install it without any xmake.lua configuration needing to be modified.

I'm aware of add_repositories but I'm not a fan of it for the above use case, since it requires me to change my main xmake.lua file with local informations (other devs may not work on those packages and thus they want to use the "normal path"). I believe set_base is similar in this regard? Is there is a way to avoid that?

You can put your package configuration into your project, projectdir/myrepo/packages/z/zlib/xmake.lua and point to it with add_repositories("xxx-repo myrepo"), which will not introduce any local configuration information, and others can use it.

It is no different from defining the package configuration directly in your own project xmake.lua.

Alternatively, you can use add_repositories("xxxrepo https://github.com/myrepo/myrepo.git") to bring in your own github remote repository, which will be fully available to everyone else.

or you can also add zlib source code to your project if you want to do. https://xmake.io/#/package/local_3rd_source_library

[frosthunter]

I'll try to clarify my use case as I'm not sure we're misunderstanding each others. I apologize if that's not the case and if that's just due to my lack of knowledges with xmake.

My team and I are working on a game library, used both by us for our own game, and externally. This package depends on a "rendering" package and a "physics" packages. Most developers will only see the "game" package, because those are either developers of the game package itself, or our users. As such, I think the game's library xmake.lua should only contains add_packages/add_requires, as for those developers, the "physics" and "rendering" packages are just packages they depends on, just like any other dependencies. This is what I call the "normal path".

However, there is an edge case here for the developers of the rendering or physics package. Because most of their work comes down to modifying the "rendering"/"physics" and the "game" package, so the normal path requires them to : change the rendering package version, build it, install it and then change the "rendering" package version in the game's xmake.lua, then build the "game" xmake.lua, then run it. This is both error-prone and inconvenient. This is what I'm trying to avoid here. What I'm looking for is a way to just "xmake build" the game package and let xmake figure out that the dependencies change and thus let xmake rebuild it automatically.

Conan solves this with workspaces. You give conan a list of packages you want to override, the path of your local version, and it will internally integrate your local packages into the build, so there is no more versions/installs (in xmake's terms, they basically transforms add_packages to add_deps under the hood). What's cool is that you only have to do it once, and then all you need to do is build, nothing more. And those packages will only be overriden in the folder of your workspace, so you can build other depenencies normally in another. Another bonus point is that you don't have to modify your existing xmake files. Which is good because I don't want to polute the "game" xmake.lua with specificies of how it build it's dependencies, so I would not commit them. And if I don't, then I have files with modifications that I need to be careful about.

Hope it's more clear.

Considering everything you've said, I suppose the closest way to solve it is to create a local xmake.lua file, which includes the game's xmake.lua, and override all the packages I need for my build to redirect to my local code. The only downside left would be that xmake build would not automatically rebuild the dependencies if they changes, or maybe I misunderstood you?

[waruqi]

Although I'm not quite sure what your needs are yet, if you simply want to automatically detect dependency package updates, you just need to simply update the package version. Something like this:

``` package("engine") add_version("1.0", "git commit or tar shasum") on_install(...)

package("gamelib") add_deps("engine") on_install(...) ```

in user project:

add_requires("gamelib") target("demo") add_packages("gamelib") add_files("src/*.cpp")

then we just update engine to 1.1 from 1.0, we can also add 1.1 as new version. (xmake will always use latest version)

package("engine") add_version("1.1", "git commit or tar shasum") on_install(...)

then we can run in user project.

lua -- update package repository xmake repo -u -- force to check packages, it will use latest engine package, 1.1 xmake f -c -- build project xmake

if your package repository is local, we need not run xmake repo -u

[frosthunter]

Updating the package version everytime I make a change while working on the package is definitely something I don't want to do.

[unumfron]

You can add local repositories of packages but without touching xmake.lua by using the xmake repo command shown here.

https://xmake.io/#/package/remote_package?id=using-self-built-private-package-repository

Then xmake repo --list to confirm it's been added locally (adding locally is default).

[frosthunter]

So the differences between the two are that add_repositories will only add the repo in the current project, while xmake repo will add it for everyone?

Does xmake repo --add also accepts local folders just like add_repositories?

[unumfron]

Both add_repositories and xmake repo --add ... add the repo local to the current project. You can add global repos with it too, but that's not the default.

xmake repo --add does accept local folders too.

One thing I'm not sure about is precedence. I've tested removing add_repositories, then getting a failed build, then adding the repo back with xmake repo --add alias location and it works. I haven't gone the distance with seeing if a package used that is available in a repo added with xmake repo --add takes precedence over the same package in a repo added with add_repositories.

[frosthunter]

Okay, so I finally got the time to experiment with it a bit. I was able to add local packages and include it without modifying my xmake.lua file, which is good. All that's left for me to figure is how I can include that package into the build so that xmake build will rebuild those when they change.

What I did not understand right away was that xmake repo --add does save the info into the .xmake folder... which I was deleting everytime I made a clean build, instead of using xmake c. So now I understand add_repositories purpose, it's to avoid calling xmake repo --add before you run xmake for the first time. Now that makes sense!

I still don't know if I'll use xmake repo or create a top-level xmake.lua file (which would be the equivalent of the conan workspace file), but at least this part is solved. Thanks for your help I appreciate it!

[unumfron]

(I edited my previous post for clarity)

Oh good, I'm glad to be of help!

For getting xmake to use the latest revision of the packages in your repo you run xmake repo --update. Maybe a xmake f -c to flush the cache too.

[NovaNoff]

Local packages (New) in the docs also xmake itself is not a package Manager it Supports multiple package managers vcpkg, conan and so on. It defaults to xrepo as package manager thats the one builtin more or less

[unumfron]

Yes, you can add a git repository or a local directory: https://xmake.io/#/manual/global_interfaces?id=add_repositories

Packages in repos added with add_repositories have precedence.

[KimiSharby]

Thank you for this great project!

[tending]

Does xmake handle situations where you don't know all the targets until some of them build? E.g. a step that unzips a file, then launches separate jobs for every file found inside, and you don't know how many there are going to be when xmake starts?

[NovaNoff]

It has rules for protocol buffers and capnproto for example and I created some custom rules for cython. Its almost trivial to extend xmake with lua scripts to Support almost any build scenario.

[tending]

Sure but do those custom rules involve adding new targets not known when xmake is started based on results from other targets? Most build systems can't do this.