justjulian justjulian - 1 year ago 93
reST (reStructuredText) Question

Generating Content-MD5 for AWS S3 REST in Haxe

I'm trying to add a Content-MD5 header to my REST calls to AWS S3 in Haxe (compiling to PHP). It's generated by

var contentMD5 = haxe.crypto.Base64.encode(haxe.io.Bytes.ofString(haxe.crypto.Md5.encode(_data)));


with
_data
in my example being

<Delete><Object><Key>nathan/storage/72ENgrtnpA5VAoy7zEpzPRNEChN0TRGc</Key></Object><Object><Key>nathan/storage/7rlZZSJFvZ7AxUhQZsh4ufn9M2x8m1ae</Key></Object><Object><Key>nathan/storage/HN8NFlUnJiiGo7qlddvRrlGE6hPmWMnZ</Key></Object><Object><Key>nathan/storage/SFsZ8z63DswEVFJQJqmUwbenaWyfZ8zb</Key></Object><Object><Key>nathan/storage/YSYXXgYbSZixOKo27PL65ii6nCeiFesl</Key></Object></Delete>


My full request sent to AWS (for a multiple delete call as described here http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html), using AWS signature version 4 (bucket, signature, and credential shortened):

POST /?delete= HTTP/1.1
Host: mybucket.s3-eu-central-1.amazonaws.com
Content-Length: 392
x-amz-content-sha256: 53da469cb6fc9d0701a1c6ff98d48edd361cd8a90d8a290a2dd224b2681bf7fb
x-amz-date: 20150923T195117Z
Authorization: AWS4-HMAC-SHA256 Credential=zzzzz/20150923/eu-central-1/s3/aws4_request, SignedHeaders=content-md5;host;x-amz-date, Signature=xxxxx
Content-MD5: MDAzNDZmZjJiMGJkMDFkNzVjYzFiOGE4MzI5NTc0NGY=

<Delete><Object><Key>nathan/storage/72ENgrtnpA5VAoy7zEpzPRNEChN0TRGc</Key></Object><Object><Key>nathan/storage/7rlZZSJFvZ7AxUhQZsh4ufn9M2x8m1ae</Key></Object><Object><Key>nathan/storage/HN8NFlUnJiiGo7qlddvRrlGE6hPmWMnZ</Key></Object><Object><Key>nathan/storage/SFsZ8z63DswEVFJQJqmUwbenaWyfZ8zb</Key></Object><Object><Key>nathan/storage/YSYXXgYbSZixOKo27PL65ii6nCeiFesl</Key></Object></Delete>


The response is as follows (RequestId and HostId shortened)

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>InvalidDigest</Code><Message>The Content-MD5 you specified was invalid.</Message><Content-MD5>MDAzNDZmZjJiMGJkMDFkNzVjYzFiOGE4MzI5NTc0NGY=</Content-MD5><RequestId>rrrrrr</RequestId><HostId>ccccccc</HostId></Error>


In my opinion, the generated MD5 is correct. I verified the value with other tools. Also, note that
x-amz-content-sha256
is based on the same
_data
and AWS accepted that header in my previous (non delete) calls.

What am I missing here? Why does my MD5 value differ from the one AWS generates?

Answer Source

You're very close.

Here's the problem:

An md5 hash is 16 bytes in binary representation, 32 characters in hexadecimal representation, and 24 characters (Including padding) in base64.

Yours is approximately twice as long. You appear to be taking the 32 character hex md5, and base64-encoding that, resulting in a base64 string of about 44 characters, instead of just encoding the binary form.

Note that the length of the output, 44 vs 24 is not a factor of two in spite of my assertion that you are encoding 32 initial bytes instead of 16. That's expected, because base64 output_bytes = ceil(input_bytes/3) * 4.

My expertise is with the S3 API -- not haxe, which I've never used -- so the above is almost certainly correct, but the following is wild speculation.

var contentMD5 = haxe.crypto.Base64.encode(haxe.crypto.Md5.make(_data));
Recommended from our users: Dynamic Network Monitoring from WhatsUp Gold from IPSwitch. Free Download