create an UTF-8 string with BOM

Try embedding the BOM directly in the format string as escaped character literals:

NSString *sourceString = [[NSString alloc] initWithFormat:@"\357\273\277%@", str];

You might have to add the BOM to the NSData object, not the NSString. Something like this:

char BOM[] = {0xEF, 0xBB, 0xBF};
NSMutableData* data = [NSMutableData data];
[data appendBytes:BOM length:3];
[data appendData:[strMd5 dataUsingEncoding:NSUTF8StringEncoding]];