com.google.cloud.hadoop.gcsio

Interface GoogleCloudStorage

All Known Implementing Classes:

ForwardingGoogleCloudStorage, GoogleCloudStorageClientImpl, GoogleCloudStorageImpl, InMemoryGoogleCloudStorage, PerformanceCachingGoogleCloudStorage

public interface GoogleCloudStorage

Interface for exposing the Google Cloud Storage API behavior in a way more amenable to writing filesystem semantics on top of it, without having to deal with API-specific considerations such as HttpTransports, credentials, network errors, batching, etc.

Please read the following document to get familiarity with basic GCS concepts: https://developers.google.com/storage/docs/concepts-techniques

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	GoogleCloudStorage.ListPage<T> Paged list request response

Field Summary

Fields

Modifier and Type	Field and Description
static int	MAX_COMPOSE_OBJECTS The maximum number of objects that can be composed in one operation.
static long	MAX_RESULTS_UNLIMITED Value indicating all objects should be returned from GCS, no limit.
static String	PATH_DELIMITER Pseudo path delimiter.

Method Summary

Modifier and Type	Method and Description
void	close() Releases resources used by this instance.
void	compose(String bucketName, List<String> sources, String destination, String contentType) Composes inputs into a single GCS object.
GoogleCloudStorageItemInfo	composeObjects(List<StorageResourceId> sources, StorageResourceId destination, CreateObjectOptions options) Composes inputs into a single GCS object.
default void	copy(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap) Copies metadata of the given objects.
void	copy(String srcBucketName, List<String> srcObjectNames, String dstBucketName, List<String> dstObjectNames) Copies metadata of the given objects.
default WritableByteChannel	create(StorageResourceId resourceId) Creates and opens an object for writing.
WritableByteChannel	create(StorageResourceId resourceId, CreateObjectOptions options) Creates and opens an object for writing.
default void	createBucket(String bucketName) Creates a bucket.
void	createBucket(String bucketName, CreateBucketOptions options) Creates a bucket.
void	createEmptyObject(StorageResourceId resourceId) Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObject(StorageResourceId resourceId, CreateObjectOptions options) Creates an empty object, useful for placeholders representing, for example, directories.
void	createEmptyObjects(List<StorageResourceId> resourceIds) Creates a list of empty objects; see createEmptyObject(StorageResourceId) for the single-item version of this method.
void	createEmptyObjects(List<StorageResourceId> resourceIds, CreateObjectOptions options) Creates a list of empty objects; see createEmptyObject(StorageResourceId) for the single-item version of this method.
void	deleteBuckets(List<String> bucketNames) Deletes a list of buckets.
void	deleteFolders(List<FolderInfo> folders) Deletes the given folder resources.
void	deleteObjects(List<StorageResourceId> fullObjectNames) Deletes the given objects.
GoogleCloudStorageItemInfo	getItemInfo(StorageResourceId resourceId) Gets information about an object or a bucket.
List<GoogleCloudStorageItemInfo>	getItemInfos(List<StorageResourceId> resourceIds) Gets information about multiple objects and/or buckets.
GoogleCloudStorageOptions	getOptions() Retrieve the options that were used to create this GoogleCloudStorage.
boolean	isHnBucket(URI path) Checks if resourceId belongs to a Hierarchical namespace enabled bucket.
List<GoogleCloudStorageItemInfo>	listBucketInfo() Gets a list of GoogleCloudStorageItemInfo for all buckets of this project.
List<String>	listBucketNames() Gets a list of names of buckets in this project.
GoogleCloudStorage.ListPage<FolderInfo>	listFolderInfoForPrefixPage(String bucketName, String folderNamePrefix, ListFolderOptions listFolderOptions, String pageToken) Returns the list of folder resources.
default List<GoogleCloudStorageItemInfo>	listObjectInfo(String bucketName, String objectNamePrefix) Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.
List<GoogleCloudStorageItemInfo>	listObjectInfo(String bucketName, String objectNamePrefix, ListObjectOptions listOptions) Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.
GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo>	listObjectInfoPage(String bucketName, String objectNamePrefix, ListObjectOptions listOptions, String pageToken) The same semantics as listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).
default GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo>	listObjectInfoPage(String bucketName, String objectNamePrefix, String pageToken) The same semantics as listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).
default SeekableByteChannel	open(StorageResourceId resourceId) Opens an object for reading.
SeekableByteChannel	open(StorageResourceId resourceId, GoogleCloudStorageReadOptions readOptions) Opens an object for reading.
void	renameHnFolder(URI src, URI dst) Renames src to dst using the rename LRO API.
List<GoogleCloudStorageItemInfo>	updateItems(List<UpdatableItemInfo> itemInfoList) Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Field Detail

PATH_DELIMITER

static final String PATH_DELIMITER

Pseudo path delimiter.

GCS does not implement full concept of file system paths but it does expose some notion of a delimiter that can be used with ${code Storage.Objects.List} to control which items are listed.

See Also:

Constant Field Values

MAX_RESULTS_UNLIMITED

static final long MAX_RESULTS_UNLIMITED

Value indicating all objects should be returned from GCS, no limit.

See Also:

Constant Field Values

MAX_COMPOSE_OBJECTS

static final int MAX_COMPOSE_OBJECTS

The maximum number of objects that can be composed in one operation.

See Also:

Constant Field Values

Method Detail

getOptions

GoogleCloudStorageOptions getOptions()

Retrieve the options that were used to create this GoogleCloudStorage.

create

default WritableByteChannel create(StorageResourceId resourceId)
                            throws IOException

Creates and opens an object for writing. The bucket must already exist. If the object already exists and resourceId doesn't have a explicit generationId set, it is deleted. If a generationId is provided in resourceId, that is used in place of normal overwrite semantics (overwrite will only succeed if the provided generationId matches the existing object, where a generationId of 0 indicates no existing object expected).

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

create

WritableByteChannel create(StorageResourceId resourceId,
                           CreateObjectOptions options)
                    throws IOException

Creates and opens an object for writing. The bucket must already exist. If resourceId contains a known generationId or a generationId of 0, it is used instead of any "overwrite" settings in the provided CreateObjectOptions (overwrite will only succeed if the provided generationId matches the existing object, where a generationId of 0 indicates no existing object expected).

Parameters:

resourceId - identifies a StorageObject

options - Options to use when creating the object

Returns:

a channel for writing to the given object

Throws:

IOException - on IO error

createBucket

default void createBucket(String bucketName)
                   throws IOException

Creates a bucket.

Parameters:

bucketName - name of the bucket to create

Throws:

IOException - on IO error

createBucket

void createBucket(String bucketName,
                  CreateBucketOptions options)
           throws IOException

Creates a bucket.

Parameters:

bucketName - name of the bucket to create

options - options to use when creating bucket

Throws:

IOException - on IO error

createEmptyObject

void createEmptyObject(StorageResourceId resourceId)
                throws IOException

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten. See create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Parameters:

resourceId - identifies a StorageObject

Throws:

IOException - on IO error

createEmptyObject

void createEmptyObject(StorageResourceId resourceId,
                       CreateObjectOptions options)
                throws IOException

Creates an empty object, useful for placeholders representing, for example, directories. The bucket must already exist. If the object already exists, it is overwritten. See create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Parameters:

resourceId - identifies a StorageObject

options - options to use when creating the object

Throws:

IOException - on IO error

createEmptyObjects

void createEmptyObjects(List<StorageResourceId> resourceIds)
                 throws IOException

Creates a list of empty objects; see createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency. See create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Throws:

IOException

createEmptyObjects

void createEmptyObjects(List<StorageResourceId> resourceIds,
                        CreateObjectOptions options)
                 throws IOException

Creates a list of empty objects; see createEmptyObject(StorageResourceId) for the single-item version of this method. Implementations may use different flow than the single-item version for greater efficiency. See create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set.

Throws:

IOException

open

default SeekableByteChannel open(StorageResourceId resourceId)
                          throws IOException

Opens an object for reading.

Parameters:

resourceId - identifies a StorageObject

Returns:

a channel for reading from the given object

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be opened

open

SeekableByteChannel open(StorageResourceId resourceId,
                         GoogleCloudStorageReadOptions readOptions)
                  throws IOException

Opens an object for reading.

Parameters:

resourceId - identifies a StorageObject

readOptions - Fine-grained options for behaviors of retries, buffering, etc.

Returns:

a channel for reading from the given object

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be opened

deleteBuckets

void deleteBuckets(List<String> bucketNames)
            throws IOException

Deletes a list of buckets. Does not throw any exception for "bucket not found" errors.

Parameters:

bucketNames - name of the buckets to delete

Throws:

FileNotFoundException - if the given bucket does not exist

IOException - on IO error

deleteObjects

void deleteObjects(List<StorageResourceId> fullObjectNames)
            throws IOException

Deletes the given objects. Does not throw any exception for "object not found" errors.

Parameters:

fullObjectNames - names of objects to delete with their respective bucketNames.

Throws:

FileNotFoundException - if the given object does not exist

IOException - if object exists but cannot be deleted

deleteFolders

void deleteFolders(List<FolderInfo> folders)
            throws IOException

Deletes the given folder resources. Does not throw any exception for "folders not found" errors.

Parameters:

folders - names of folder resources to delete

Throws:

IOException - if folder exists but cannot be deleted

copy

void copy(String srcBucketName,
          List<String> srcObjectNames,
          String dstBucketName,
          List<String> dstObjectNames)
   throws IOException

Copies metadata of the given objects. After the copy is successfully complete, each object blob is reachable by two different names. Copying between two different locations or between two different storage classes is not allowed.

Parameters:

srcBucketName - name of the bucket containing the objects to copy

srcObjectNames - names of the objects to copy

dstBucketName - name of the bucket to copy to

dstObjectNames - names of the objects after copy

Throws:

FileNotFoundException - if the source object or the destination bucket does not exist

IOException - in all other error cases

copy

default void copy(Map<StorageResourceId,StorageResourceId> sourceToDestinationObjectsMap)
           throws IOException

Copies metadata of the given objects. After the copy is successfully complete, each object blob is reachable by two different names. Copying between two different locations or between two different storage classes is not allowed.

Parameters:

sourceToDestinationObjectsMap - map of destination objects to be copied, keyed by source

Throws:

FileNotFoundException - if the source object or the destination bucket does not exist

IOException - in all other error cases

isHnBucket

boolean isHnBucket(URI path)
            throws IOException

Checks if resourceId belongs to a Hierarchical namespace enabled bucket. This takes a path and not the bucket name since the caller may not have permission to query the bucket.

Parameters:

path - Path for which the check need to be performed

Returns:

Throws:

IOException

listBucketNames

List<String> listBucketNames()
                      throws IOException

Gets a list of names of buckets in this project.

Throws:

IOException

renameHnFolder

void renameHnFolder(URI src,
                    URI dst)
             throws IOException

Renames src to dst using the rename LRO API. This should be called only on an Hierarchical namespace enabled bucket.

Parameters:

src - source path

dst - destination path

Throws:

IOException

listBucketInfo

List<GoogleCloudStorageItemInfo> listBucketInfo()
                                         throws IOException

Gets a list of GoogleCloudStorageItemInfo for all buckets of this project. This is no more expensive than calling listBucketNames(), since the list API for buckets already retrieves all the relevant bucket metadata.

Throws:

IOException

listObjectInfo

default List<GoogleCloudStorageItemInfo> listObjectInfo(String bucketName,
                                                        String objectNamePrefix)
                                                 throws IOException

Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.

Note: Although GCS does not implement a file system, it treats objects that contain a delimiter (ListObjectOptions.getDelimiter()) as different from other objects when listing objects. This will be clearer with an example.

Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

• With prefix == null and delimiter == /, we get: d1/, o1
• With prefix == null and delimiter == null, we get: o1, d1/, d1/o1, d1/o2

Thus when delimiter is null, the entire key name is considered an opaque string, otherwise only the part up to the first delimiter is considered.

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

Returns:

list of objects

Throws:

IOException - on IO error

listObjectInfo

List<GoogleCloudStorageItemInfo> listObjectInfo(String bucketName,
                                                String objectNamePrefix,
                                                ListObjectOptions listOptions)
                                         throws IOException

Lists GoogleCloudStorageItemInfo of objects contained in the given bucket and whose names begin with the given prefix.

Note: Although GCS does not implement a file system, it treats objects that contain a delimiter (ListObjectOptions.getDelimiter()) as different from other objects when listing objects. This will be clearer with an example.

Consider a bucket with objects: o1, d1/, d1/o1, d1/o2

• With prefix == null and delimiter == /, we get: d1/, o1
• With prefix == null and delimiter == null, we get: o1, d1/, d1/o1, d1/o2

Thus when delimiter is null, the entire key name is considered an opaque string, otherwise only the part up to the first delimiter is considered.

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

listOptions - options to use when listing objects

Returns:

list of objects

Throws:

IOException - on IO error

listObjectInfoPage

default GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo> listObjectInfoPage(String bucketName,
                                                                                   String objectNamePrefix,
                                                                                   String pageToken)
                                                                            throws IOException

The same semantics as listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

pageToken - the page token

Returns:

GoogleCloudStorage.ListPage object with listed GoogleCloudStorageItemInfos and next page token if any

Throws:

IOException - on IO error

listObjectInfoPage

GoogleCloudStorage.ListPage<GoogleCloudStorageItemInfo> listObjectInfoPage(String bucketName,
                                                                           String objectNamePrefix,
                                                                           ListObjectOptions listOptions,
                                                                           String pageToken)
                                                                    throws IOException

The same semantics as listObjectInfo(java.lang.String, java.lang.String), but returns only result of single list request (1 page).

Parameters:

bucketName - bucket name

objectNamePrefix - object name prefix or null if all objects in the bucket are desired

listOptions - options to use when listing objects

pageToken - the page token

Returns:

GoogleCloudStorage.ListPage object with listed GoogleCloudStorageItemInfos and next page token if any

Throws:

IOException - on IO error

listFolderInfoForPrefixPage

GoogleCloudStorage.ListPage<FolderInfo> listFolderInfoForPrefixPage(String bucketName,
                                                                    String folderNamePrefix,
                                                                    ListFolderOptions listFolderOptions,
                                                                    String pageToken)
                                                             throws IOException

Returns the list of folder resources. Applicable only for HN enabled bucket

Parameters:

bucketName - bucket name

folderNamePrefix - folder resource name prefix

listFolderOptions - options to use when listing folder resources

pageToken - the page token

Returns:

GoogleCloudStorage.ListPage folder resources with listed Folders

Throws:

IOException - on IO error

getItemInfo

GoogleCloudStorageItemInfo getItemInfo(StorageResourceId resourceId)
                                throws IOException

Gets information about an object or a bucket.

Parameters:

resourceId - identifies either root, a Bucket, or a StorageObject

Returns:

information about the given item

Throws:

IOException - on IO error

getItemInfos

List<GoogleCloudStorageItemInfo> getItemInfos(List<StorageResourceId> resourceIds)
                                       throws IOException

Gets information about multiple objects and/or buckets. Items that are "not found" will still have an entry in the returned list; exists() will return false for these entries. The return list will have the same ordering as the input arguments.

Parameters:

resourceIds - names of the GCS StorageObjects or Buckets for which to retrieve info.

Returns:

information about the given resourceIds.

Throws:

IOException - on IO error

updateItems

List<GoogleCloudStorageItemInfo> updateItems(List<UpdatableItemInfo> itemInfoList)
                                      throws IOException

Attempt to update metadata of the objects referenced within the passed itemInfo objects.

Returns:

Updated GoogleCloudStorageItemInfo objects for the referenced objects.

Throws:

IOException - on IO error

compose

void compose(String bucketName,
             List<String> sources,
             String destination,
             String contentType)
      throws IOException

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object, if already present, will be overwritten. Sources and destination are assumed to be in the same bucket.

Parameters:

bucketName - name of the bucket containing the sources and the destination

sources - names of the objects to be composed

destination - name of the resulting object

contentType - content-type of composed object

Throws:

IOException - if the Compose operation was unsuccessful

composeObjects

GoogleCloudStorageItemInfo composeObjects(List<StorageResourceId> sources,
                                          StorageResourceId destination,
                                          CreateObjectOptions options)
                                   throws IOException

Composes inputs into a single GCS object. This performs a GCS Compose. Objects will be composed according to the order they appear in the input. The destination object will have metadata set according to options. Overwrite semantics for the destination object will follow the same semantics as create(StorageResourceId, CreateObjectOptions). See create(StorageResourceId) for the behavior if StorageResourceId.getGenerationId() is explicitly set. The bucket must be the same for all sources and the destination.

Throws:

IOException

close

void close()

Releases resources used by this instance.