How to unpack COMP-3 digits using Java?

COMP-3 (or "packed decimal") data looks like this: 0x12345s, where "s" is C for positive, D for negative, or F for unsigned. Thus 0x12345c -> "12345", x012345d -> "-12345", and 0x12345f -> "12345".

You've got one obvious error: You're ignoring the nybble in the byte that contains the sign nybble (e.g., "5" above) if the sign is negative. In addition, you're working too hard at manipulating the nybbles, it's a simple bitwise-and or a 4-bit shift to isolate a nybble.

Try something like this (untested):

public String unpackData(String packedData, int decimalPointLocation) {
    String unpackedData = "";
    char[] characters = packedData.toCharArray();
    final int negativeSign = 13;
    for (int currentCharIndex = 0; currentCharIndex < characters.length; currentCharIndex++) {
        byte firstDigit = ((byte) characters[currentCharIndex]) >>> 4);
        byte secondDigit = ((byte) characters[currentCharIndex]) & 0x0F;
        unpackedData += String.valueOf(firstDigit);
        if (currentCharIndex == (characters.length - 1)) {
            if (secondDigit == negativeSign) {
                unpackedData = "-" + unpackedData;
            }
        } else {
            unpackedData += String.valueOf(secondDigit);
        }
    }
    if (decimalPointLocation > 0) {
        unpackedData = unpackedData.substring(0, (decimalPointLocation - 1)) + 
                        "." + 
                        unpackedData.substring(decimalPointLocation);
    }
    return unpackedData;
}

public static final int UNSIGNED_BYTE = 0xff;
public static final int BITS_RIGHT = 0xf;

public long parseComp3(byte[] data) {
    long val = 0L;
    boolean negative = false;
    for (int i = 0; i < data.length; i++) {
        int raw = data[i] & UNSIGNED_BYTE;
        int digitA = raw >> 4;
        int digitB = raw & BITS_RIGHT;

        if (digitA < 10) {
            val *= 10L;
            val += (long) digitA;

        } else if (digitA == 11 || digitA == 13) { // Some non-IBM systems store the sign on left or use 11 for negative.
            negative = true;
        }

        if (digitB < 10) {
            val *= 10L;
            val += (long) digitB;

        } else if (digitB == 11 || digitB == 13) {
            negative = true;
        }
    }
    if (negative)
        val = -val;
    return val;
}

The Ross Paterson solution has a bug when it moves the first 4 bits to the right. The mask 0x0F must be applied.

Here is the corrected method:

private static String unpackData(byte[] packedData, int decimalPointLocation) {
    String unpackedData = "";

    final int negativeSign = 13;
    for (int currentCharIndex = 0; currentCharIndex < packedData.length; currentCharIndex++) {
        byte firstDigit = (byte) ((packedData[currentCharIndex] >>> 4) & 0x0F);
        byte secondDigit = (byte) (packedData[currentCharIndex] & 0x0F);
        unpackedData += String.valueOf(firstDigit);
        if (currentCharIndex == (packedData.length - 1)) {
            if (secondDigit == negativeSign) {
                unpackedData = "-" + unpackedData;
            }
        } else {
            unpackedData += String.valueOf(secondDigit);
        }
    }

    if (decimalPointLocation > 0) {
        int position = unpackedData.length() - decimalPointLocation;
        unpackedData = unpackedData.substring(0, position) + "." + unpackedData.substring(position);
    }
    return unpackedData;
}