How to apply threshold within multiple rectangular bounding boxes in an image?

Color thresholding using cv2.inRange() should work here. I'm assuming you want to isolate the green area

Here's the main idea

• Convert image to HSV format since it is easier to represent color than RBG
• Perform color segmentation with a lower/upper threshold

You could also perform morphological operations to smooth or remove noise after obtaining the mask

---

import numpy as np
import cv2

image = cv2.imread('1.jpg')
result = image.copy()
image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
lower = np.array([18, 0, 0])
upper = np.array([179, 255, 255])
mask = cv2.inRange(image, lower, upper)
result = cv2.bitwise_and(result,result, mask=mask)
cv2.imshow('result', result)
cv2.imwrite('result.png', result)
cv2.waitKey()

You can use a HSV color thresholder script to isolate the desired color range

import cv2
import sys
import numpy as np

def nothing(x):
    pass

# Create a window
cv2.namedWindow('image')

# create trackbars for color change
cv2.createTrackbar('HMin','image',0,179,nothing) # Hue is from 0-179 for Opencv
cv2.createTrackbar('SMin','image',0,255,nothing)
cv2.createTrackbar('VMin','image',0,255,nothing)
cv2.createTrackbar('HMax','image',0,179,nothing)
cv2.createTrackbar('SMax','image',0,255,nothing)
cv2.createTrackbar('VMax','image',0,255,nothing)

# Set default value for MAX HSV trackbars.
cv2.setTrackbarPos('HMax', 'image', 179)
cv2.setTrackbarPos('SMax', 'image', 255)
cv2.setTrackbarPos('VMax', 'image', 255)

# Initialize to check if HSV min/max value changes
hMin = sMin = vMin = hMax = sMax = vMax = 0
phMin = psMin = pvMin = phMax = psMax = pvMax = 0

img = cv2.imread('1.jpg')
output = img
waitTime = 33

while(1):

    # get current positions of all trackbars
    hMin = cv2.getTrackbarPos('HMin','image')
    sMin = cv2.getTrackbarPos('SMin','image')
    vMin = cv2.getTrackbarPos('VMin','image')

    hMax = cv2.getTrackbarPos('HMax','image')
    sMax = cv2.getTrackbarPos('SMax','image')
    vMax = cv2.getTrackbarPos('VMax','image')

    # Set minimum and max HSV values to display
    lower = np.array([hMin, sMin, vMin])
    upper = np.array([hMax, sMax, vMax])

    # Create HSV Image and threshold into a range.
    hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
    mask = cv2.inRange(hsv, lower, upper)
    output = cv2.bitwise_and(img,img, mask= mask)

    # Print if there is a change in HSV value
    if( (phMin != hMin) | (psMin != sMin) | (pvMin != vMin) | (phMax != hMax) | (psMax != sMax) | (pvMax != vMax) ):
        print("(hMin = %d , sMin = %d, vMin = %d), (hMax = %d , sMax = %d, vMax = %d)" % (hMin , sMin , vMin, hMax, sMax , vMax))
        phMin = hMin
        psMin = sMin
        pvMin = vMin
        phMax = hMax
        psMax = sMax
        pvMax = vMax

    # Display output image
    cv2.imshow('image',output)

    # Wait longer to prevent freeze for videos.
    if cv2.waitKey(waitTime) & 0xFF == ord('q'):
        break

cv2.destroyAllWindows()

Here's the result on the original image

In your TensorFlow detection, the output dictionary you get after you run the prediction has a field, "detection_scores".

output_dict = sess.run(tensor_dict,feed_dict={image_tensor: image})

Set a threshold on that,

 indexes=np.where(output_dict['detection_scores']>0.5)

Use the boxes, i.e. output_dict['detection_boxes'] only on those specific indexes which you filtered in the previous step.

[EDIT] Adding more code after the discussion in comments

#convert the image to hsv
hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)
#tune the numbers below accordingly
lower_green = np.array([60, 100, 50])
upper_green = np.array([60 , 255, 255])

mask = cv2.inRange(hsv, lower_green, upper_green)
res = cv2.bitwise_and(frame,frame, mask= mask)
#res has the output masked image

[EDIT] editing with the actual image given in the question

img=cv2.imread("idJyc.jpg")
lower_green = np.array([0, 10, 0])
upper_green = np.array([255 , 100, 255])
mask = cv2.inRange(img, lower_green, upper_green)
mask = np.abs(255-mask)
res = cv2.bitwise_and(img,img, mask=mask)
cv2.imshow("a",res)
cv2.waitKey(0)

Adding the output image for your reference.

If you re familiar with applying neural networks and you re having enough data. This task is perfectly suited for segmentation.

I recommend U-Net, since it works with a small amount of labelled data for training. It is also fast, with few operations for this task with comparably low complexity. And has shown good performance on various tasks.

I also found a full code pipeline, in this case for heart segmentation in zebrafish, but in my opinion they did a good job explaining how to prepare the data (proposing labeling tools, etc.) and train the model.

Also taking a step back, you could also think of interpreting your task as a segmentation task from the beginning. Especially for the U-Net it shouldn't be a problem to segment multiple instances in the same image.