Converting YUV->RGB(Image processing)->YUV during onPreviewFrame in android?

Although the documentation suggests that you can set which format the image data should arrive from the camera in, in practice you often have a choice of one: NV21, a YUV format. For lots of information on this format see http://www.fourcc.org/yuv.php#NV21 and for information on the theory behind converting it to RGB see http://www.fourcc.org/fccyvrgb.php. There is a picture based explanation at Extract black and white image from android camera's NV21 format. There is an android specific section on a wikipedia page about the subject (thanks @AlexCohn): YUV#Y'UV420sp (NV21) to RGB conversion (Android).

However, once you've set up your onPreviewFrame routine, the mechanics of going from the byte array it sends you to useful data is somewhat, ummmm, unclear. From API 8 onwards, the following solution is available, to get to a ByteStream holiding a JPEG of the image (compressToJpeg is the only conversion option offered by YuvImage):

// pWidth and pHeight define the size of the preview Frame
ByteArrayOutputStream out = new ByteArrayOutputStream();

// Alter the second parameter of this to the actual format you are receiving
YuvImage yuv = new YuvImage(data, ImageFormat.NV21, pWidth, pHeight, null);

// bWidth and bHeight define the size of the bitmap you wish the fill with the preview image
yuv.compressToJpeg(new Rect(0, 0, bWidth, bHeight), 50, out);

This JPEG may then need to be converted into the format you want. If you want a Bitmap:

byte[] bytes = out.toByteArray();
Bitmap bitmap= BitmapFactory.decodeByteArray(bytes, 0, bytes.length);

If, for whatever reason, you are unable to do this, you can do the conversion manually. Some problems to be overcome in doing this:

1. The data arrives in a byte array. By definition, bytes are signed numbers, meaning that they go from -128 to 127. However, the data is actually unsigned bytes (0 to 255). If this isn't dealt with, the outcome is doomed to have some odd clipping effects.

2. The data is in a very specific order (as per the previously mentioned web pages) and each pixel needs to be extracted carefully.

3. Each pixel needs to be put into the right place on a bitmap, say. This also requires a rather messy (in my view) approach of building a buffer of the data and then filling a bitmap from it.

4. In principle, the values should be stored [16..240], but it appears that they are stored [0..255] in the data sent to onPreviewFrame

5. Just about every web page on the matter proposes different coefficients, even allowing for [16..240] vs [0..255] options.

6. If you've actually got NV12 (another variant on YUV420), then you will need to swap the reads for U and V.

I present a solution (which seems to work), with requests for corrections, improvements and ways of making the whole thing less costly to run. I have set it out to hopefully make clear what is happening, rather than to optimise it for speed. It creates a bitmap the size of the preview image:

The data variable is coming from the call to onPreviewFrame

// Define whether expecting [16..240] or [0..255]
boolean dataIs16To240 = false;

// the bitmap we want to fill with the image
Bitmap bitmap = Bitmap.createBitmap(imageWidth, imageHeight, Bitmap.Config.ARGB_8888);
int numPixels = imageWidth*imageHeight;

// the buffer we fill up which we then fill the bitmap with
IntBuffer intBuffer = IntBuffer.allocate(imageWidth*imageHeight);
// If you're reusing a buffer, next line imperative to refill from the start,
// if not good practice
intBuffer.position(0);

// Set the alpha for the image: 0 is transparent, 255 fully opaque
final byte alpha = (byte) 255;

// Holding variables for the loop calculation
int R = 0;
int G = 0;
int B = 0;

// Get each pixel, one at a time
for (int y = 0; y < imageHeight; y++) {
    for (int x = 0; x < imageWidth; x++) {
        // Get the Y value, stored in the first block of data
        // The logical "AND 0xff" is needed to deal with the signed issue
        float Y = (float) (data[y*imageWidth + x] & 0xff);

        // Get U and V values, stored after Y values, one per 2x2 block
        // of pixels, interleaved. Prepare them as floats with correct range
        // ready for calculation later.
        int xby2 = x/2;
        int yby2 = y/2;

        // make this V for NV12/420SP
        float U = (float)(data[numPixels + 2*xby2 + yby2*imageWidth] & 0xff) - 128.0f;

        // make this U for NV12/420SP
        float V = (float)(data[numPixels + 2*xby2 + 1 + yby2*imageWidth] & 0xff) - 128.0f;

        if (dataIs16To240) {
            // Correct Y to allow for the fact that it is [16..235] and not [0..255]
            Y = 1.164*(Y - 16.0);

            // Do the YUV -> RGB conversion
            // These seem to work, but other variations are quoted
            // out there.
            R = (int)(Yf + 1.596f*V);
            G = (int)(Yf - 0.813f*V - 0.391f*U);
            B = (int)(Yf            + 2.018f*U);
        }
        else {
            // No need to correct Y
            // These are the coefficients proposed by @AlexCohn
            // for [0..255], as per the wikipedia page referenced
            // above
            R = (int)(Yf + 1.370705f*V);
            G = (int)(Yf - 0.698001f*V - 0.337633f*U);
            B = (int)(Yf               + 1.732446f*U);
        }
              
        // Clip rgb values to 0-255
        R = R < 0 ? 0 : R > 255 ? 255 : R;
        G = G < 0 ? 0 : G > 255 ? 255 : G;
        B = B < 0 ? 0 : B > 255 ? 255 : B;

        // Put that pixel in the buffer
        intBuffer.put(alpha*16777216 + R*65536 + G*256 + B);
    }
}

// Get buffer ready to be read
intBuffer.flip();

// Push the pixel information from the buffer onto the bitmap.
bitmap.copyPixelsFromBuffer(intBuffer);

As @Timmmm points out below, you could do the conversion in int by multiplying the scaling factors by 1000 (ie. 1.164 becomes 1164) and then dividng the end results by 1000.

Why not specify that camera preview should provide RGB images?

i.e. Camera.Parameters.setPreviewFormat(ImageFormat.RGB_565);

You can use RenderScript -> ScriptIntrinsicYuvToRGB

Kotlin Sample

val rs = RenderScript.create(CONTEXT_HERE)
val yuvToRgbIntrinsic = ScriptIntrinsicYuvToRGB.create(rs, Element.U8_4(rs))

val yuvType = Type.Builder(rs, Element.U8(rs)).setX(byteArray.size)
val inData = Allocation.createTyped(rs, yuvType.create(), Allocation.USAGE_SCRIPT)

val rgbaType = Type.Builder(rs, Element.RGBA_8888(rs)).setX(width).setY(height)
val outData = Allocation.createTyped(rs, rgbaType.create(), Allocation.USAGE_SCRIPT)

inData.copyFrom(byteArray)

yuvToRgbIntrinsic.setInput(inData)
yuvToRgbIntrinsic.forEach(outData)

val bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888)
outData.copyTo(bitmap)