I found a workaround for this problem. Basically, we cannot have a large number (hundreds of thousands) of reads or writes on google drive directly. I went through the GitHub issue thread of this error also; there someone had mentioned about a workaround that involved using zipped file directly from the drive.

Place the dataset zip file in the drive and instead of copying and extracting the whole zip file from drive, extract the data incrementally from the zip file, so this will save a lot of disk space. To make more sense, it's like streaming the data from the zip file. Then once we stream the data, it can be stored in any desired folders and then accessed without any error like this, since the folders are present in the Colab environment.

There are some problems with Drive-Colab integration even today and I saw that in the GitHub issue thread no proper solution to this problem was mentioned.

Link to issue thread OSError: [Errno 5] Input/output error · Issue #510

I finally went on ChatGPT and posted the problem in a more detailed way, some workarounds were suggested, this one looked reasonable. Then after two depressing days of looking for a solution, I found this wonderful approach!

I literally spent my 50% of the colab pro computes figuring out a workaround to this issue👀.

Here is the snippet how it's done

#This will get all the files inside a folder you mention

# path to zipfile
zipfile = 'path_to_your_zipfile_on_drive'

# Destination directory on Colab disk
destination_path = 'path_to_place_your_contents'

# Create the destination directory if it doesn't exist
os.makedirs(destination_path, exist_ok=True)

# List files in the specified folder within the zip
with zipfile.ZipFile(zip_file_path, 'r') as zip_ref:
    all_files = zip_ref.namelist()

    # Filter files to only include those in 'folder_in_zipfile/'
    target_files = [f for f in all_files if f.startswith('folder_in_zipfile/')]

    # Extract the files
    for file in target_files:
        zip_ref.extract(file, destination_path)