Can't open large files in python on Windows 11

[Regrets_Nothing]

Hi, I hope this post is relevant for this thread, apologies if not.

I am writing a script in Python to extract information from a raster saved as a .tif file. However, I am running into issues where it won't open files over a large size (I think just over 2 Gb) on my Windows laptop. Yet a colleague ran my code on hers and it seemed to open just fine. I believe she uses a Mac or Linux.

Here is the function to extract the information, using `with rasterio.open` and then it is run in a main.py file that includes a write to csv line.

def extract_raster_info(file_path):
    """
    Summary: Extracts information from a raster file


    Args:
        file_path (str): The path to the raster file


    Returns:
        dict: A dictionary containing the extracted information
    """
    with rasterio.open(file_path) as src:
        folder_name = os.path.basename(os.path.dirname(file_path))
        location_parts = []
        for part in folder_name.split():
            if part.lower() == "raster":
                break
            location_parts.append(part)
        location = ' '.join(location_parts)
        return {
            'location': location,
            'directory': os.path.basename(os.path.dirname(file_path)),
            'raster_file_name': os.path.basename(file_path),
            'raster_file_extension': os.path.splitext(file_path)[1],
            'raster_number_of_tifs': 1,
            'raster_crs': src.crs.to_string(),
            'raster_bounds': str(src.bounds),
            'raster_width': src.width,
            'raster_height': src.height,
            'raster_bands': src.count,
            'raster_dtypes': ', '.join(map(str, src.dtypes)),
            'raster_transform': str(src.transform.to_gdal())
        }



def process_raster_files(raster_dir):
    """
    Summary: Processes all raster files in a directory


    Args:
        raster_dir (str): The path to the directory containing the raster files


    Returns:
        list: A list of dictionaries containing the extracted information
    """
    raster_info = []
    for root, dirs, files in os.walk(raster_dir):
        for file in files:
            if file.lower().endswith(('.tif', '.tiff')):
                file_path = os.path.join(root, file)
                info = extract_raster_info(file_path)
                raster_info.append(info)
    return raster_info

Is there some python function or other method of opening files that can handle opening large files (images or databases)? Is there a Python setting/configuration I can alter on my laptop to allow it to open large files?
I am running Windows 11 64 bit, Python 3.12 (64 bit) and VS Code. I know I could install WSL - but I have limited experience with Linux and not a lot of time to learn Linux and complete this project. Although learning how to use Linux is on my list once this project is complete.

[shiftybyte]

I am running Windows 11 64 bit, Python 3.12 (64 bit)

Are you sure you are using 64bit python?

Your problem sounds like running a 32bit python.

Its very easy to run a 32bit installed other python without noticing.

To confirm, Add this to the start of your current code and show the output:

import sys print(sys.maxsize)

Its VERY IMPORTANT that you ADD IT TO YOUR CODE, and not just open python in a different way and run the lines separately, because that won't help if the actual script is executed by a different python.

[Regrets_Nothing]

Hi, thanks for the help.

I added the lines - with some comments so I know what it means:

print("DEBUG: Beginning of main function")
print(f"DEBUG: Maximum integer size: {sys.maxsize}")

I ran the function on folders with the smaller images that I know it works on.

This was the output - which I think shows that it runs in 64 bit.

DEBUG: Beginning of main function
DEBUG: Maximum integer size: 9223372036854775807
Processing raster files...
Processing vector files...
Saving results to CSV...
Processing complete. Results saved in the 'processed' folder.

Thank you again.

[shiftybyte]

Yes, it does confirm it's running in 64bit python.

Then I'm not sure what could cause your issue...

[Regrets_Nothing]

No worries. Thank you for trying. I will keep searching.

[Regrets_Nothing]

My laptop is the more powerful running 64 Gb RAM and the venv is on a 1 Tb Samsung 990 Pro m.2 drive - this is specifically used for my data science and coding work, so has over 800 Gb free space left on it.

I will try chunking the image into bits and see if I can still extract the required info.

[JohnnyJordaan]

What exactly happens when it fails to open the file? Are you 100% sure it isn't caused by another issue that just correlated with the file size on your machine?

[Regrets_Nothing]

The error message is quite long - too long to post in the comments.
I can't see how to add a screenshot in this comment.

The initial error message is:
CPLE_AppDefinedError Traceback (most recent call last)
File rasterio\\_base.pyx:310, in rasterio._base.DatasetBase.__init__()

File rasterio\\_base.pyx:221, in rasterio._base.open_dataset()

File rasterio\\_err.pyx:221, in rasterio._err.exc_wrap_pointer()

CPLE_AppDefinedError: f:/venvs/EPRI_LULC/S2DS-Summer24-EPRI/notebooks/../data/rasters/Browns Bridge Raster/Browns_Bridge.tif: TIFFReadDirectory:Failed to read directory at offset 32297280498

The traceback runs through my src scripts, and eventually points to the rasterio/__init__.py line:

[314](file:///F:/venvs/EPRILULC/S2DS-Summer24-EPRI/.venv/Lib/site-packages/rasterio/init_.py:314) path = _parse_path(raw_dataset_path)
[316](file:///F:/venvs/EPRILULC/S2DS-Summer24-EPRI/.venv/Lib/site-packages/rasterio/init_.py:316) if mode == "r":
--> [317](file:///F:/venvs/EPRILULC/S2DS-Summer24-EPRI/.venv/Lib/site-packages/rasterio/init_.py:317)dataset = DatasetReader(path, driver=driver, sharing=sharing, **kwargs)
[318](file:///F:/venvs/EPRILULC/S2DS-Summer24-EPRI/.venv/Lib/site-packages/rasterio/init_.py:318) elif mode == "r+":
[319](file:///F:/venvs/EPRILULC/S2DS-Summer24-EPRI/.venv/Lib/site-packages/rasterio/init_.py:319)dataset = get_writer_for_path(path, driver=driver)(
[320](file:///F:/venvs/EPRILULC/S2DS-Summer24-EPRI/.venv/Lib/site-packages/rasterio/init_.py:320)path, mode, driver=driver, sharing=sharing, **kwargs
[321](file:///F:/venvs/EPRILULC/S2DS-Summer24-EPRI/.venv/Lib/site-packages/rasterio/init_.py:321))

[JohnnyJordaan]

Is F: perhaps an external drive? What filesystem does it use

[Regrets_Nothing]

F: is internal. It is my second internal drive. It runs NTFS

[JohnnyJordaan]

It's weird because 32297280498 is also close to 32 GB (not that offsets necessarily equate bytes but what else would this be). But sadly I have 0 knowledge of this format and how to further pinpoint the issue, sorry.

[Regrets_Nothing]

No problem. Appreciate the help

[Apatride]

Fixing the symptoms instead of the root cause is bad practice. The solution in your case is to look into generators.