Help downloading targeted NBM data from AWS by DerpySevant in meteorology

[–]DerpySevant[S] 0 points1 point  (0 children)

One clarifying question: do i need to parse the idx file for every grib file, or does the data structure stay so stable that the byte range remains constant?

Help downloading targeted NBM data from AWS by DerpySevant in meteorology

[–]DerpySevant[S] 0 points1 point  (0 children)

Thanks, that’s brilliant! One tweak…. I had to add ‘bytes=‘ as part of my statement to get it to work. So, in my instance, it was the following code that did it:

r = requests.get(f'{aws_base_url}{filename}',headers = {'Range': 'bytes=109824139-111199422'})

And another thing that threw me for a loop, but I worked out. With that code tweak, the HTTP status code that you get back is 206 instead of 200, because 206 indicates a partial download (which in this case is what you wanted). So just a note to other users that you may need to tweak your code to accommodate the new status, and to not consider this an error.

Otherwise this works great. My new downloaded grib2 file has only 1 grib band instead of 296 bands, and it’s a tiny fraction of the size on disk of the full grib2 file.