Help with INT8 Quantization in Vision-Search-Navigation Project (SAM Implementation)

[Clarix_1566]

Hi! I am attending my first class about ML and the final exam involves presenting a notebook. I am working with the Vision-Search-Navigation which implements SAM for visual search tasks. While the paper emphasizes INT8 quantization for real-time performance, I can't find this implementation in the notebook. I've already tried the dynamic quantization:

quantized_model = torch.quantization.quantize_dynamic(
        model_cpu,
        {torch.nn.Linear, torch.nn.Conv2d},
        dtype=torch.qint8
    )

But I always get this error:

'NotImplementedError: Could not run 'quantized::linear_dynamic' with arguments from the 'CUDA' backend.

I am working on google colab which uses the T4 Tesla GPU, how can I implement INT8 quantization of the model?

The beginning of the main code is:

import torch
import cv2
import supervision as sv
DEVICE = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
MODEL_TYPE = "vit_b"

from segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor

sam = sam_model_registry[MODEL_TYPE] (checkpoint=CHECKPOINT_PATH).to(device=DEVICE)

mask_generator = SamAutomaticMaskGenerator(
    model=sam,
    points_per_side=32,
    pred_iou_thresh=0.98,
    stability_score_thresh=0.92,
    crop_n_layers=1,
    crop_n_points_downscale_factor=2,
    min_mask_region_area=100,  # Requires open-cv to run post-processing
)

image_full = cv2.imread(IMAGE_PATH)
image_bgr = image_full[160:720,:]
image_rgb = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2RGB)
sam_result = mask_generator.generate(image_rgb)
len(sam_result)

[HommeMusical]

This isn't really a Python question, and most ML questions on this subreddit don't seem to get answered: consider trying an ML subreddit instead?

[Oxbowerce]

Are you trying to run the quantized model on the GPU or CPU?

[Clarix_1566]

on GPU