daoudc daoudc - 23 days ago 13
Python Question

Getting unencoded data from Google cloud Pub/Sub instead of base64

I'm attempting to use the Python library for Pub/Sub, but I keep getting this error:

TypeError: Incorrect padding
. Some quick googling revealed this issue: https://github.com/GoogleCloudPlatform/google-cloud-python/pull/2527

However, this doesn't resolve the issue - in fact, printing out the data revealed that the data was not even base64 encoded at all, and setting
data = raw_data
in the library resolved the issue.

We're sending the message from Java, here is the code we're using:

PCollection<String> userActionsJson = userActionsRaw.apply(ParDo.of(new BigQueryRowToJson()));
String topicNameFull = "projects/" + options.getProject() + "/topics/" +
options.getUsersActionsTopicName() + "-" + options.getProduct();
userActionsJson.apply(PubsubIO.Write.named("PublishToPubSub")
.topic(topicNameFull));


however, the same thing happens attempting to send a message through the console.

Has something changed recently to mean that data is no longer base64 encoded?

Answer

This was an issue that has now been resolved in master: https://github.com/GoogleCloudPlatform/google-cloud-python/issues/2577

You can install it using pip like so:

pip install -e git+git://github.com/GoogleCloudPlatform/google-cloud-python.git#egg=pubsub
Comments