Image is uploading with errors through google cloud cpp client, but can upload successfully with gsutil

[elamacchia]

I haven't found much about specifically uploading images - I'm trying to include this functionality within my c++ code, but not sure if I'm missing something. My upload code looks like this:

auto client = gcs::Client();
StatusOr<gcs::ObjectMetadata> metadata = client.UploadFile(
            file_name, bucket_name, target_path,  gcs::ContentType("image/jpeg"));.

where file_name is my local file, bucket_name is the bucket name and target_path is the name to save the file as. The upload seems to work correctly (metadata is correct) and generates a file in the bucket with the target name, but clicking on or downloading the actual image says it contains errors. In contrast, when I run gsutil cp file_name gs://bucket_name, it uploads correctly without errors.

[coryan]

but clicking on or downloading the actual image says it contains errors.

What are you using the "click on" or download the file? And what is the error message? And could you share the OS you used? I just wrote a repro for this and:

• I was able to see an uploaded (JPEG) image using the GCS console.
• I was also able to download it using:

curl -v -H "Authorization: Bearer $(gcloud auth print-access-token)" https://www.googleapis.com/storage/v1/b/${BUCKET_NAME}/o/${OBJECT_NAME}?alt=media -o /dev/null

And the content-type and other headers look correct to me.

[elamacchia]

This is on ubuntu 18.04. I uploaded a screenshot below of the error.

[elamacchia]

[elamacchia]

This is what appears when clicking on the actual image in the bucket, whereas clicking on an image uploaded with gsutil will display correctly. Additionally, downloading an errored image will just download an empty image.

[coryan]

When you upload the object with gsutil what is the contentType metadata attribute set to? One possibility is that gsutil made a better guess at the actual format of the file.

The other thing is that the data is really corrupted, comparing the checksum of the uploaded object vs. the local object should answer that.

[elamacchia]

gsutil also uploads it as an image/jpeg. I did also try adding checksum verification and got the same result, where it seemingly uploaded but didn't actually get the image - gsutil is able to upload the same images, so I don't think the files are corrupted. I did try your repro script and got the same results where the image file just shows an error.

[coryan]

I think you found a bug. The checksums and size for the file uploaded via the C++ client library do not match.

[elamacchia]

in that case, how was your script able to upload an image?

[coryan]

Most likely I just misread the results.

[elamacchia]

But you were able to view the image in the cloud storage?

…

________________________________ From: coryan ***@***.***> Sent: Tuesday, October 26, 2021 1:39 PM To: googleapis/google-cloud-cpp ***@***.***> Cc: Elizabeth LaMacchia ***@***.***>; Author ***@***.***> Subject: Re: [googleapis/google-cloud-cpp] Image is uploading with errors through google cloud cpp client, but can upload successfully with gsutil (Discussion #7519) Most likely I just misread the results. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#7519 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVQYO3MUYP6BAHB4UDDONALUI4GXXANCNFSM5GYRLVSQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[coryan]

I saw an image. I picked some random jpeg file, did not care (or checked) what it looked like, just that it would not "error". And what I got was some blank square and I just assumed that were the contents. Why you got an error and I got some random blank square? No idea.

[elamacchia]

I see, thank you for clarifying, just wanted to make sure we were experiencing the same results

…

________________________________ From: coryan ***@***.***> Sent: Tuesday, October 26, 2021 1:51 PM To: googleapis/google-cloud-cpp ***@***.***> Cc: Elizabeth LaMacchia ***@***.***>; Author ***@***.***> Subject: Re: [googleapis/google-cloud-cpp] Image is uploading with errors through google cloud cpp client, but can upload successfully with gsutil (Discussion #7519) I saw an image. I picked some random jpeg file, did not care (or checked) what it looked like, just that it would not "error". And what I got was some blank square and I just assumed that were the contents. Why you got an error and I got some random blank square? No idea. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#7519 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVQYO3JSQ4S4CCC5HWWNMULUI4IGNANCNFSM5GYRLVSQ>. Triage notifications on the go with GitHub Mobile for iOS<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[coryan]

As a workaround use gcs::WithObjectMetadata

diff --git a/quickstart.cc b/quickstart.cc
index 6df793a..8ab34b9 100644
--- a/quickstart.cc
+++ b/quickstart.cc
@@ -1,7 +1,7 @@
 #include <google/cloud/storage/client.h>
 #include <iostream>
 
-int main(int argc, char* argv[]) try {
+int main(int argc, char *argv[]) try {
   if (argc != 3) {
     std::cerr << "Missing bucket name or image filename.\n";
     std::cerr << "Usage: quickstart <image-file> <bucket-name>\n";
@@ -16,12 +16,17 @@ int main(int argc, char* argv[]) try {
   // Create a client to communicate with Google Cloud Storage. This client
   // uses the default configuration for authentication and project id.
   auto client = gcs::Client();
-  auto metadata = client.UploadFile(image_filename, bucket_name, image_filename, gcs::ContentType("image/jpeg")).value();
+  auto metadata =
+      client
+          .UploadFile(image_filename, bucket_name, image_filename,
+                      gcs::WithObjectMetadata(
+                          gcs::ObjectMetadata{}.set_content_type("image/jpeg")))
+          .value();
 
   std::cout << "Image uploaded: " << metadata << "\n";
 
   return 0;
-} catch (std::exception const& ex) {
+} catch (std::exception const &ex) {
   std::cerr << "Standard C++ exception thrown: " << ex.what() << std::endl;
   return 1;
 }

[coryan]

I merged the fixes for this last night, there will be a new release next week if you need to wait for that.

[coryan]

In case anybody finds this later, this is a bug, continued discussion in #7520