Is there a way to get the color of a recognized object inside a picture?

You are using a code which predicts the label of the given image, i.e. classifies the image from some trained classes So you don't know the exact pixels of your object.

So, I suggest you do any of the following,

1. Use an object detector to detect the location of the object and get the bounding box. Then get the color of the most pixels.
2. Use a pixel-wise classification (segmentation) like this to get the exact pixels of your object.

Note, you may need to manually train the network (or model) for your object

Edit:

For Java object detection examples, have a look at this project which is coded for android, but it should be straightforward to use them in desktop applications. More specifically look into this part.

You don't need both object detection and segmentation at the same time but if you want, I think first try to train a model for segmentation using python (the link is provided above) then use the model in java similarly as the object detection models.

Edit 2:

I have added a simple object detection client in java which uses Tensorflow Object detection API models just to show you that you can use any frozen model in java.

Also, check this beautiful repository which uses pixel wise segmentation.