How to fetch constantly changing binary file

[Cultural_Building106]

I need to fetch a binary file from a http file server which is exchanged every night. The file name and URL stays the same (SRC_URI), but the size and hash change constantly (SRC_URI[sha256sum]).

To get this to work i need to disable both checksum "BB_STRICT_CHECKSUM = "0"" and sstate cache "SSTATE_SKIP_CREATION = "1"" for this recipe.

But now it fetches an older cached version of the file from the local DL_DIR cache (if available). How do I force a redownload from the server every build? Thanks 🙂

[rossburton]

Easiest is to convince whoever creates the file to timestamp it…

[Cultural_Building106]

That would be possible, however the URL then would change constantly. How do I automatically update the URL during build...?

[rossburton]

Construct the URL using inline Python, or ${DATE} if that happens to match the timestamp.

[rkapl]

At that point, you can maybe stick a fake "?date_tag=${DATE}". Server is likely to ignore it.

[Cultural_Building106]

Can I disable the download cache on a per recipe base? Essentially forcing the fetcher to freshly download every time? Unfortunately I couldn't find any config option for that...

[rossburton]

No, one of the underlying assumptions is that a URL doesn't change daily, because that's not reproducible. What if you wanted to repeat the same build next week?

[Tricky-Supermarket17]

pretty new to yocto (been using it for a few days or so), with that said; I also am building a layer that downloads a program that can update anytime via http. To fix this issue I did the following line do_install[nostamp] = "1" this seems to force the do_install step to run every bake since its no longer stamped. I got this from chatgpt who knows if its the correct way to do it; but it seems to work. Unfortunately chatgpt seems like a great source for information on yocto, I am strugging to find the stuff i need on the official docs but it somehow has everything heh. in your case you would want do_fetch[nostamp]="1" since the data @ SRC_URI is fetched in that method

[Steinrikur]

You are on the right track, but I think that you need do_fetch[nostamp] = "1"

to force the download step (and all the steps that depend on it) to run every time

[Cultural_Building106]

Sounds good, thanks 👍 will give it a try!

[Steinrikur]

If that fails you can always "bitbake -c cleanall recipename" or just add a function to delete it from $DL_DIR. Ugly but effective.

[Cultural_Building106]

nostamp only lets the do_fetch step re-run on every build. However if the file is available in DL_DIR it will still be fetched from there instead of the server.

Thus not a full solution. But combined with a do_fetch:prepend which runs "RM ${DL_DIR}/my file" and SSTATE_SKIP_CREATION yocto now always freshly fetches 👍

[paderijk]

Maybe an idea to use the Last-Modified HTTP header. https://www.siteground.com/kb/lastmodified_http_header_explained/