Resumable uploads

Setup

This page discusses resumable uploads in Cloud Storage. Resumable uploads are the recommended method for uploading large files, because you don't have to restart them from the beginning if there is a network failure while the upload is underway.

Introduction

A resumable upload lets you resume data transfer operations to Cloud Storage after a communication failure has interrupted the flow of data. Resumable uploads work by sending multiple requests, each of which contains a portion of the object you're uploading. This is different from a single-request upload, which contains all of the object's data in a single request and must restart from the beginning if it fails part way through.

A resumable upload operates through a resumable upload session. When you initiate an upload, Cloud Storage creates a session and returns a unique session URI. You or a delegated client (such as a web browser) then send the object data to this session URI in one or more requests. Each session remains active for up to a week, allowing uploads to be resumed if they get interrupted by network dropouts or client restarts.

  • Use a resumable upload if you are uploading large files or uploading over a slow connection. For example file size cutoffs for using resumable uploads, see upload size considerations.

  • A resumable upload must be completed within a week of being initiated, but can be cancelled at any time.

  • Only a completed resumable upload appears in your bucket and, if applicable, replaces an existing object with the same name.

    • The creation time for the object is based on when the upload completes.

    • Object metadata set by the user is specified in the initial request. This metadata is applied to the object once the upload completes.

    • The JSON API also supports setting custom metadata in the final request if you include headers prefixed with X-Goog-Meta- in that request.

How tools and APIs use resumable uploads

Depending on how you interact with Cloud Storage, resumable uploads might be managed automatically on your behalf. This section describes the resumable upload behavior for different tools and provides guidance on configuring the appropriate buffer size for your application.

Console

The Google Cloud console manages resumable uploads automatically on your behalf. However, if you refresh or navigate away from the Google Cloud console while an upload is underway, the upload is cancelled.

Command line

The gcloud CLI uses resumable uploads in the gcloud storage cp and gcloud storage rsync commands when uploading data to Cloud Storage. If your upload is interrupted, you can resume it by running the same command that you used to start the upload. When resuming such an upload that includes multiple files, use the --no-clobber flag to prevent re-uploading files that already completed successfully.

Client libraries

When performing resumable uploads, client libraries function as wrappers around the Cloud Storage JSON API.

C++

Functions in storage::Client perform with different behavior:

By default, UploadFile() performs a resumable upload when the object is larger than 20 MiB. Otherwise, it performs a simple upload or multipart upload. You can configure this threshold by setting MaximumSimpleUploadsSizeOption when creating a storage::Client.

8 MiB is the default buffer size, which you can modify with the UploadBufferSizeOption option.

The C++ client library uses a buffer size that's equal to the chunk size. The buffer size must be a multiple of 256 KiB (256 x 1024 bytes). When using WriteObject() and UploadFile(), you might want to consider the tradeoffs between upload speed and memory usage. Using small buffers to upload large objects can make the upload slow. For more information on the relationship between upload speed and buffer size for C++, see the detailed analysis in GitHub.

C#

When uploading, the C# client library always performs resumable uploads. You can initiate a resumable upload with CreateObjectUploader.

The C# client library uses a buffer size that's equal to the chunk size. The default buffer size is 10 MB and you can change this value by setting ChunkSize on UploadObjectOptions. The buffer size must be a multiple of 256 KiB (256 x 1024 bytes). Larger buffer sizes typically make uploads faster, but note that there's a tradeoff between speed and memory usage.

Go

By default, resumable uploads occur automatically when the file is larger than 16 MiB. You change the cutoff for performing resumable uploads with Writer.ChunkSize. Resumable uploads are always chunked when using the Go client library.

Multipart uploads occur when the object is smaller than Writer.ChunkSize or when Writer.ChunkSize is set to 0, where chunking becomes disabled. The Writer is unable to retry requests if ChunkSize is set to 0.

The Go client library uses a buffer size that's equal to the chunk size. The buffer size must be a multiple of 256 KiB (256 x 1024 bytes). Larger buffer sizes typically make uploads faster, but note that there's a tradeoff between speed and memory usage. If you're running several resumable uploads concurrently, you should set Writer.ChunkSize to a value that's smaller than 16 MiB to avoid memory bloat.

Note that the object is not finalized in Cloud Storage until you call Writer.Close() and receive a success response. Writer.Close returns an error if the request isn't successful.

Java

The Java client library has separate methods for multipart and resumable uploads. The following methods always perform a resumable upload:

The default buffer size is 15 MiB. You can set the buffer size either by using the WriteChannel#setChunkSize(int) method, or by passing in a bufferSize parameter to the Storage#createFrom method. The buffer size has a hard minimum of 256KiB. When calling WriteChannel#setChunkSize(int) internally, the buffer size is shifted to a multiple of 256 KiB.

Buffering for resumable uploads functions as a minimum flush threshold, where writes smaller than the buffer size are buffered until a write pushes the number of buffered bytes above the buffer size.

If uploading smaller amounts of data, consider using Storage#create(BlobInfo, byte[], Storage.BlobTargetOption...) or Storage#create(BlobInfo, byte[], int, int, Storage.BlobTargetOption...).

Node.js

Resumable uploads occur automatically. You can turn off resumable uploads by setting resumable on UploadOptions to false. Resumable uploads are automatically managed when using the createWriteStream method.

There is no default buffer size and chunked uploads must be manually invoked by setting the