get back a string representation from computeDigest(algorithm, value) byte[]

If we put Logger.log(signature); right after the call to computeDigest(), we get:

[8, 30, -43, 124, -101, 114, -37, 10, 78, -13, -102, 51, 65, -24, -83, 81]

As represented in javascript, the digest includes both positive and negative integers, so we can't simply treat them as ascii characters. The MD5 algorithm, however, should provide us with 8-bit values, in the range 0x00 to 0xFF (255). Those negative values, then, are just a misinterpretation of the high-order bit; taking it to be a sign bit. To correct, we need to add 256 to any negative value.

How to convert decimal to hex in JavaScript? gives us this for retrieving hex characters:

hexString = yourNumber.toString(16);

Putting that together, here's your sign() function, which is also available as a gist:

function sign(message){     
  message = message || "thisisteststring";
  var signature = Utilities.computeDigest(
                       Utilities.DigestAlgorithm.MD5,
                       message,
                       Utilities.Charset.US_ASCII);
  Logger.log(signature);
  var signatureStr = '';
    for (i = 0; i < signature.length; i++) {
      var byte = signature[i];
      if (byte < 0)
        byte += 256;
      var byteStr = byte.toString(16);
      // Ensure we have 2 chars in our byte, pad with 0
      if (byteStr.length == 1) byteStr = '0'+byteStr;
      signatureStr += byteStr;
    }   
  Logger.log(signatureStr);
  return signatureStr;
}

And here's what the logs contain:

[13-04-25 21:46:55:787 EDT] [8, 30, -43, 124, -101, 114, -37, 10, 78, -13, -102, 51, 65, -24, -83, 81]
[13-04-25 21:46:55:788 EDT] 081ed57c9b72db0a4ef39a3341e8ad51

Let's see what we get from this on-line MD5 Hash Generator:

I tried it with a few other strings, and they consistently matched the result from the on-line generator.

Just in case this is helpful to anyone else, I've put together a more succinct version of Mogsdad's solution:

function md5(str) {
  return Utilities.computeDigest(Utilities.DigestAlgorithm.MD5, str).reduce(function(str,chr){
    chr = (chr < 0 ? chr + 256 : chr).toString(16);
    return str + (chr.length==1?'0':'') + chr;
  },'');
}

One-liner:

Utilities.computeDigest(Utilities.DigestAlgorithm.MD5, "teststring")
  .map(function(b) {return ("0" + (b < 0 && b + 256 || b).toString(16)).substr(-2)})
  .join("")