我正在尝试使用detectron2框架提取类别检测超过某个阈值的区域特征。我稍后将在我的流水线中使用这些特征（类似于：VilBert第3.1节训练ViLBERT）。到目前为止，我已经使用这个配置训练了一个Mask R-CNN，并在一些自定义数据上进行了微调。它的表现很好。我希望做的是从我训练的模型中提取生成的边界框的特征。
为什么我只得到一个预测实例，但当我查看预测CLS分数时，有超过1个通过了阈值？
我认为这是生成ROI特征的正确方法：
images = ImageList.from_tensors(lst[:1], size_divisibility=32).to("cuda") # preprocessed input tensor#setup configcfg = get_cfg()cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")cfg.SOLVER.IMS_PER_BATCH = 1cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1 # only has one class (pnumonia)#Just run these lines if you have the trained model im memorycfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set the testing threshold for this model#build modelmodel = build_model(cfg)DetectionCheckpointer(model).load("output/model_final.pth")model.eval()#make sure its in eval mode#run modelwith torch.no_grad(): features = model.backbone(images.tensor.float()) proposals, _ = model.proposal_generator(images, features) instances = model.roi_heads._forward_box(features, proposals)然后
pred_boxes = [x.pred_boxes for x in instances]rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], pred_boxes)这应该是我的ROI特征。
让我非常困惑的是，我可以不使用推理时生成的边界框，而是使用建议框和它们的类别分数来获取这张图片的前n个特征。很好，所以我尝试了以下方法：
proposal_boxes = [x.proposal_boxes for x in proposals]proposal_rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], proposal_boxes)#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.htmlbox_features = model.roi_heads.box_head(proposal_rois)predictions = model.roi_heads.box_predictor(box_features)pred_instances, losses = model.roi_heads.box_predictor.inference(predictions, proposals)我应该在这里获取我的建议框特征及其cls在我的predictions对象中。检查这个predictions对象，我看到了每个框的分数：
预测对象中的CLS分数
(tensor([[ 0.6308, -0.4926], [-1.6662, 1.5430], [-0.2080, 0.4856], ..., [-6.9698, 6.6695], [-5.6361, 5.4046], [-4.4918, 4.3899]], device='cuda:0', grad_fn=<AddmmBackward>),在对这些cls分数进行softmax处理并将其放入数据框后，并设置0.6的阈值，我得到：
pred_df = pd.DataFrame(predictions[0].softmax(-1).tolist())pred_df[pred_df[0] > 0.6] 0 10 0.754618 0.2453826 0.686816 0.31318438 0.722627 0.277373而在我的预测对象中，我得到了相同的最高分，但只有1个实例而不是2个（我设置了cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7）：
预测实例：
[Instances(num_instances=1, image_height=800, image_width=800, fields=[pred_boxes: Boxes(tensor([[548.5992, 341.7193, 756.9728, 438.0507]], device='cuda:0', grad_fn=<IndexBackward>)), scores: tensor([0.7546], device='cuda:0', grad_fn=<IndexBackward>), pred_classes: tensor([0], device='cuda:0')])]预测还包含Tensor: Nx4或Nx(Kx4)边界框回归增量，我并不完全清楚它们是做什么用的，看起来像这样：
预测对象中的边界框回归增量
tensor([[ 0.2502, 0.2461, -0.4559, -0.3304], [-0.1359, -0.1563, -0.2821, 0.0557], [ 0.7802, 0.5719, -1.0790, -1.3001], ..., [-0.8594, 0.0632, 0.2024, -0.6000], [-0.2020, -3.3195, 0.6745, 0.5456], [-0.5542, 1.1727, 1.9679, -2.3912]], device='cuda:0', grad_fn=<AddmmBackward>)另一件奇怪的事情是我的建议框和我的预测框是不同的但相似的：
建议边界框
[Boxes(tensor([[532.9427, 335.8969, 761.2068, 438.8086],#this box vs the instance box [102.7041, 352.5067, 329.4510, 440.7240], [499.2719, 317.9529, 764.1958, 448.1386], ..., [ 25.2890, 379.3329, 28.6030, 429.9694], [127.1215, 392.6055, 328.6081, 489.0793], [164.5633, 275.6021, 295.0134, 462.7395]], device='cuda:0'))]回答：
你几乎已经接近目标了。查看roi_heads.box_predictor.inference()，你会发现它不仅仅是按分数对框候选进行排序。首先，它应用框增量来重新调整建议框。然后，它计算非最大抑制来移除不重叠的框（同时也应用其他超参数设置，如分数阈值）。最后，它根据分数对前k个框进行排序。这可能解释了为什么你的方法产生相同的框分数但输出框的数量和坐标不同。
回到你的原始问题，这里是如何在一个推理过程中提取建议框的特征的方法：
image = cv2.imread('my_image.jpg')height, width = image.shape[:2]image = torch.as_tensor(image.astype("float32").transpose(2, 0, 1))inputs = [{"image": image, "height": height, "width": width}]with torch.no_grad(): images = model.preprocess_image(inputs) # don't forget to preprocess features = model.backbone(images.tensor) # set of cnn features proposals, _ = model.proposal_generator(images, features, None) # RPN features_ = [features[f] for f in model.roi_heads.box_in_features] box_features = model.roi_heads.box_pooler(features_, [x.proposal_boxes for x in proposals]) box_features = model.roi_heads.box_head(box_features) # features of all 1k candidates predictions = model.roi_heads.box_predictor(box_features) pred_instances, pred_inds = model.roi_heads.box_predictor.inference(predictions, proposals) pred_instances = model.roi_heads.forward_with_given_boxes(features, pred_instances) # output boxes, masks, scores, etc pred_instances = model._postprocess(pred_instances, inputs, images.image_sizes) # scale box to orig size # features of the proposed boxes feats = box_features[pred_inds]