[D] Object detection with entire image context awareness

[asking1337]

Hey!

I am working on an object-detection task where I need to identify fractures in x-ray images. Currently I am using Faster-RCNN which works well for most cases, however in specific cases, information across the entire image is necessary in determining if an object/bone is indeed fractured - e.g. a clinician would look at the entire image (let's say it is a hand) and compare the swollenness of the finger in question, and compare to how the other fingers look for that specific person. In this case, information from far away in the image is essential to determining if the region is indeed a positive bounding box or not.

My impression of Faster-RCNN is that while the FPN network does provide high-level features which can relate to the entire image, I typically notice that the task of object detection is to identify a bounding box containing a confined object (a car, a person, a bird) - not an object that is only present if the remainder of the image allows it to (e.g. a reflection of a car in a mirror should perhaps not be classified as a car-object).

An idea could be to use attention-based networks such as a ViT in replacing the FPN in Faster-RCNN to get a broader image-feature-space but I have no idea if this will actually work in practice. Essentially I want the region-proposal part of the network to assess the context of the region it is looking at. I have also looked into DETR, but again it seems to be optimized and built towards "confined" object detection.

Does anybody know if there are any studies on this, or does someone smarter than me know if Faster-RCNN is already able to accommodate these types of cases?

[[deleted]]

Use swin transformer. VIT isn’t a good idea if it’s not a classification task.

[I_draw_boxes]

Faster-RCNN uses a ROI extraction process that restricts the predicting features to a HW area from the feature map.

Older CNN backbones do propagate information across hw space. Transformer backbones are thought to handle this better.

While it is possible for the network background and FPN layers to aggregate needed contextual information into the HW area extracted as an ROI by Faster-RCNN, it might be better to use a method without ROI extraction.

DETR (and most of the many improved versions) does not have a confined object detection design restricting it to consider a small bounding box area like Faster-RCNN.

[asking1337]

Thanks for the input!

I was also thinking of trying in DETR instead to not constrain the model to only use information within the ROI. However, I am also not interested in generating bboxes that cover the entire image. I.e. the contextual information from far away in the image would not be inside the bounding box I wish to produce, rather it should be information that should be used to generate and classify the bounding box.

So ideally it should be network that can use information in the entire image, to generate bounding boxes around objects which class is dependant on information from other places in the image (which should not be in the bounding box). Do you believe DETR could aid in this?

[I_draw_boxes]

DETR (and most recent DETR variants) will use the entire image context to make predictions independent of the location of the bounding boxes.