Package org.apache.hadoop.ozone.om

Interface KeyManager

All Superinterfaces:
IOzoneAcl, OzoneManagerFS
All Known Implementing Classes:
KeyManagerImpl
public interface KeyManager extends OzoneManagerFS, IOzoneAcl
Handles key level commands.

  • Method Summary

    Modifier and Type
    Method
    Description
    CompactionService
    getCompactionService()
    Returns the instance of CompactionService.
    default org.apache.hadoop.hdds.utils.db.TableIterator<String,? extends org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>>
    getDeletedDirEntries()
    Returns an iterator for pending deleted directories all buckets.
    org.apache.hadoop.hdds.utils.db.TableIterator<String,? extends org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>>
    getDeletedDirEntries(String volume, String bucket)
    Returns an iterator for pending deleted directories for volume and bucket.
    default List<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>>
    getDeletedDirEntries(String volume, String bucket, int size)
     
    List<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,List<org.apache.hadoop.ozone.om.helpers.OmKeyInfo>>>
    getDeletedKeyEntries(String volume, String bucket, String startKey, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo>,Boolean,IOException> filter, int count)
    Returns a list deleted entries from the deletedTable.
    KeyDeletingService
    getDeletingService()
    Returns the instance of Deleting Service.
    DirectoryDeletingService
    getDirDeletingService()
    Returns the instance of Directory Deleting Service.
    List<org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ExpiredMultipartUploadsBucket>
    getExpiredMultipartUploads(Duration expireThreshold, int maxParts)
    Returns the MPU infos of up to count whose age is greater than or equal to expireThreshold.
    org.apache.hadoop.ozone.om.ExpiredOpenKeys
    getExpiredOpenKeys(Duration expireThreshold, int count, org.apache.hadoop.ozone.om.helpers.BucketLayout bucketLayout, Duration leaseThreshold)
    Returns the names of up to count open keys whose age is greater than or equal to expireThreshold.
    org.apache.hadoop.ozone.om.helpers.OmKeyInfo
    getKeyInfo(org.apache.hadoop.ozone.om.helpers.OmKeyArgs args, ResolvedBucket buctket, String clientAddress)
    Return info of an existing key to client side to access to data on datanodes.
    org.apache.hadoop.ozone.om.OMMetadataManager
    getMetadataManager()
    Returns the metadataManager.
    org.apache.hadoop.hdds.utils.BackgroundService
    getMultipartUploadCleanupService()
    Returns the instance of Multipart Upload Cleanup Service.
    Map<String,String>
    getObjectTagging(org.apache.hadoop.ozone.om.helpers.OmKeyArgs args, ResolvedBucket bucket)
    Look up an existing key from the OM table and retrieve the tags from the key info.
    org.apache.hadoop.hdds.utils.BackgroundService
    getOpenKeyCleanupService()
    Returns the instance of Open Key Cleanup Service.
    PendingKeysDeletion
    getPendingDeletionKeys(String volume, String bucket, String startKey, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int count)
    Retrieves the keys that are pending deletion in a specified bucket and volume.
    PendingKeysDeletion
    getPendingDeletionKeys(org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int count)
    Retrieves pending deletion keys that match a given filter function.
    DeleteKeysResult
    getPendingDeletionSubDirs(long volumeId, long bucketId, org.apache.hadoop.ozone.om.helpers.OmKeyInfo parentInfo, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int remainingNum)
    Returns all sub directories under the given parent directory.
    DeleteKeysResult
    getPendingDeletionSubFiles(long volumeId, long bucketId, org.apache.hadoop.ozone.om.helpers.OmKeyInfo parentInfo, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int remainingNum)
    Returns all sub files under the given parent directory.
    org.apache.ratis.util.function.CheckedFunction<KeyManager,org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo,IOException>
    getPreviousSnapshotOzoneDirInfo(long volumeId, org.apache.hadoop.ozone.om.helpers.OmBucketInfo bucketInfo, org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo directoryInfo)
    Returns the previous snapshot's ozone directorInfo corresponding for the object.
    org.apache.ratis.util.function.CheckedFunction<KeyManager,org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo,IOException>
    getPreviousSnapshotOzoneDirInfo(long volumeId, org.apache.hadoop.ozone.om.helpers.OmBucketInfo bucketInfo, org.apache.hadoop.ozone.om.helpers.OmKeyInfo directoryInfo)
    Returns the previous snapshot's ozone directoryInfo corresponding for the object.
    org.apache.ratis.util.function.CheckedFunction<KeyManager,org.apache.hadoop.ozone.om.helpers.OmKeyInfo,IOException>
    getPreviousSnapshotOzoneKeyInfo(long volumeId, org.apache.hadoop.ozone.om.helpers.OmBucketInfo bucketInfo, org.apache.hadoop.ozone.om.helpers.OmKeyInfo keyInfo)
    Returns the previous snapshot's ozone keyInfo corresponding for the object.
    List<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,String>>
    getRenamesKeyEntries(String volume, String bucket, String startKey, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,String>,Boolean,IOException> filter, int count)
    Returns a list rename entries from the snapshotRenamedTable.
    SnapshotDefragService
    getSnapshotDefragService()
    Returns the instance of Snapshot Defrag service.
    SnapshotDeletingService
    getSnapshotDeletingService()
    Returns the instance of Snapshot Deleting service.
    SstFilteringService
    getSnapshotSstFilteringService()
    Returns the instance of Snapshot SST Filtering service.
    org.apache.hadoop.ozone.om.helpers.ListKeysResult
    listKeys(String volumeName, String bucketName, String startKey, String keyPrefix, int maxKeys)
    Returns a list of keys represented by OmKeyInfo in the given bucket.
    org.apache.hadoop.ozone.om.helpers.OmMultipartUploadList
    listMultipartUploads(String volumeName, String bucketName, String prefix, String keyMarker, String uploadIdMarker, int maxUploads, boolean withPagination)
     
    org.apache.hadoop.ozone.om.helpers.OmMultipartUploadListParts
    listParts(String volumeName, String bucketName, String keyName, String uploadID, int partNumberMarker, int maxParts)
    Returns list of parts of a multipart upload key.
    org.apache.hadoop.ozone.om.helpers.OmKeyInfo
    lookupKey(org.apache.hadoop.ozone.om.helpers.OmKeyArgs args, ResolvedBucket bucketLayout, String clientAddress)
    Look up an existing key.
    void
    refresh(org.apache.hadoop.ozone.om.helpers.OmKeyInfo key)
    Refresh the key block location information by get latest info from SCM.
    void
    start(org.apache.hadoop.hdds.conf.OzoneConfiguration configuration)
    Start key manager.
    void
    stop()
    Stop key manager.

    Methods inherited from interface org.apache.hadoop.ozone.om.IOzoneAcl

    checkAccess, getAcl

    Methods inherited from interface org.apache.hadoop.ozone.om.fs.OzoneManagerFS

    getFileStatus, getFileStatus, listStatus, listStatus, listStatus, lookupFile

  • Method Details

    • start

      void start(org.apache.hadoop.hdds.conf.OzoneConfiguration configuration)
      Start key manager.
      Parameters:
      configuration -

    • stop

      void stop() throws IOException
      Stop key manager.
      Throws:
      IOException

    • lookupKey

      org.apache.hadoop.ozone.om.helpers.OmKeyInfo lookupKey(org.apache.hadoop.ozone.om.helpers.OmKeyArgs args, ResolvedBucket bucketLayout, String clientAddress) throws IOException
      Look up an existing key. Return the info of the key to client side, which DistributedStorageHandler will use to access the data on datanode.
      Parameters:
      args - the args of the key provided by client.
      clientAddress - a hint to key manager, order the datanode in returned pipeline by distance between client and datanode.
      Returns:
      a OmKeyInfo instance client uses to talk to container.
      Throws:
      IOException

    • getKeyInfo

      org.apache.hadoop.ozone.om.helpers.OmKeyInfo getKeyInfo(org.apache.hadoop.ozone.om.helpers.OmKeyArgs args, ResolvedBucket buctket, String clientAddress) throws IOException
      Return info of an existing key to client side to access to data on datanodes.
      Parameters:
      args - the args of the key provided by client.
      clientAddress - a hint to key manager, order the datanode in returned pipeline by distance between client and datanode.
      Returns:
      a OmKeyInfo instance client uses to talk to container.
      Throws:
      IOException

    • listKeys

      org.apache.hadoop.ozone.om.helpers.ListKeysResult listKeys(String volumeName, String bucketName, String startKey, String keyPrefix, int maxKeys) throws IOException
      Returns a list of keys represented by OmKeyInfo in the given bucket.
      Parameters:
      volumeName - the name of the volume.
      bucketName - the name of the bucket.
      startKey - the start key name, only the keys whose name is after this value will be included in the result. This key is excluded from the result.
      keyPrefix - key name prefix, only the keys whose name has this prefix will be included in the result.
      maxKeys - the maximum number of keys to return. It ensures the size of the result will not exceed this limit.
      Returns:
      a list of keys.
      Throws:
      IOException

    • getPendingDeletionKeys

      PendingKeysDeletion getPendingDeletionKeys(org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int count) throws IOException
      Retrieves pending deletion keys that match a given filter function.
      Parameters:
      filter - a functional interface specifying the filter condition to apply to the keys. It takes a KeyValue pair containing a string key and an OmKeyInfo object, and returns a boolean value indicating whether the key meets the filter criteria.
      count - the maximum number of keys to retrieve.
      Returns:
      a PendingKeysDeletion object containing the keys that satisfy the filter criteria, up to the specified count.
      Throws:
      IOException - if an I/O error occurs while fetching the keys.

    • getPendingDeletionKeys

      PendingKeysDeletion getPendingDeletionKeys(String volume, String bucket, String startKey, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int count) throws IOException
      Retrieves the keys that are pending deletion in a specified bucket and volume.
      Parameters:
      volume - the name of the volume that contains the bucket.
      bucket - the name of the bucket within the volume where keys are located.
      startKey - the key from which to start retrieving pending deletions.
      filter - a filter function to determine which keys should be included in the pending deletion list.
      count - the maximum number of keys to retrieve that are pending deletion.
      Returns:
      a PendingKeysDeletion object containing the list of keys pending deletion based on the specified parameters.
      Throws:
      IOException - if an I/O error occurs during the operation.

    • getRenamesKeyEntries

      List<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,String>> getRenamesKeyEntries(String volume, String bucket, String startKey, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,String>,Boolean,IOException> filter, int count) throws IOException
      Returns a list rename entries from the snapshotRenamedTable.
      Parameters:
      filter - filter to apply on the entries.
      count - max number of keys to return.
      Returns:
      a Pair of list of Table.KeyValue representing the keys in the underlying metadataManager.
      Throws:
      IOException

    • getPreviousSnapshotOzoneDirInfo

      org.apache.ratis.util.function.CheckedFunction<KeyManager,org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo,IOException> getPreviousSnapshotOzoneDirInfo(long volumeId, org.apache.hadoop.ozone.om.helpers.OmBucketInfo bucketInfo, org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo directoryInfo) throws IOException
      Returns the previous snapshot's ozone directorInfo corresponding for the object.
      Throws:
      IOException

    • getPreviousSnapshotOzoneDirInfo

      org.apache.ratis.util.function.CheckedFunction<KeyManager,org.apache.hadoop.ozone.om.helpers.OmDirectoryInfo,IOException> getPreviousSnapshotOzoneDirInfo(long volumeId, org.apache.hadoop.ozone.om.helpers.OmBucketInfo bucketInfo, org.apache.hadoop.ozone.om.helpers.OmKeyInfo directoryInfo) throws IOException
      Returns the previous snapshot's ozone directoryInfo corresponding for the object.
      Throws:
      IOException

    • getPreviousSnapshotOzoneKeyInfo

      org.apache.ratis.util.function.CheckedFunction<KeyManager,org.apache.hadoop.ozone.om.helpers.OmKeyInfo,IOException> getPreviousSnapshotOzoneKeyInfo(long volumeId, org.apache.hadoop.ozone.om.helpers.OmBucketInfo bucketInfo, org.apache.hadoop.ozone.om.helpers.OmKeyInfo keyInfo) throws IOException
      Returns the previous snapshot's ozone keyInfo corresponding for the object.
      Throws:
      IOException

    • getDeletedKeyEntries

      List<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,List<org.apache.hadoop.ozone.om.helpers.OmKeyInfo>>> getDeletedKeyEntries(String volume, String bucket, String startKey, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.RepeatedOmKeyInfo>,Boolean,IOException> filter, int count) throws IOException
      Returns a list deleted entries from the deletedTable.
      Parameters:
      filter - filter to apply on the entries.
      count - max number of keys to return.
      Returns:
      a Pair of list of Table.KeyValue representing the keys in the underlying metadataManager.
      Throws:
      IOException

    • getExpiredOpenKeys

      org.apache.hadoop.ozone.om.ExpiredOpenKeys getExpiredOpenKeys(Duration expireThreshold, int count, org.apache.hadoop.ozone.om.helpers.BucketLayout bucketLayout, Duration leaseThreshold) throws IOException
      Returns the names of up to count open keys whose age is greater than or equal to expireThreshold.
      Parameters:
      expireThreshold - The threshold of open key expiration age.
      count - The maximum number of expired open keys to return.
      bucketLayout - The type of open keys to get (e.g. DEFAULT or FSO).
      leaseThreshold - The threshold of hsync key.
      Returns:
      the expired open keys.
      Throws:
      IOException

    • getExpiredMultipartUploads

      List<org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos.ExpiredMultipartUploadsBucket> getExpiredMultipartUploads(Duration expireThreshold, int maxParts) throws IOException
      Returns the MPU infos of up to count whose age is greater than or equal to expireThreshold.
      Parameters:
      expireThreshold - The threshold of expired multipart uploads to return.
      maxParts - The threshold of number of MPU parts to return. This is a soft limit, which means if the last MPU has parts that exceed this limit, it will be included in the returned expired MPUs. This is to handle situation where MPU that has more parts than maxParts never get deleted.
      Returns:
      a List of the expired OzoneManagerProtocolProtos.ExpiredMultipartUploadsBucket, the expired multipart uploads, grouped by volume and bucket.
      Throws:
      IOException

    • getObjectTagging

      Map<String,String> getObjectTagging(org.apache.hadoop.ozone.om.helpers.OmKeyArgs args, ResolvedBucket bucket) throws IOException
      Look up an existing key from the OM table and retrieve the tags from the key info.
      Parameters:
      args - the args of the key provided by client.
      bucket - the resolved parent bucket of the key.
      Returns:
      Map of the tag set associated with the key.
      Throws:
      IOException

    • getMetadataManager

      org.apache.hadoop.ozone.om.OMMetadataManager getMetadataManager()
      Returns the metadataManager.
      Returns:
      OMMetadataManager.

    • getDeletingService

      KeyDeletingService getDeletingService()
      Returns the instance of Deleting Service.
      Returns:
      Background service.

    • listMultipartUploads

      org.apache.hadoop.ozone.om.helpers.OmMultipartUploadList listMultipartUploads(String volumeName, String bucketName, String prefix, String keyMarker, String uploadIdMarker, int maxUploads, boolean withPagination) throws org.apache.hadoop.ozone.om.exceptions.OMException
      Throws:
      org.apache.hadoop.ozone.om.exceptions.OMException

    • listParts

      org.apache.hadoop.ozone.om.helpers.OmMultipartUploadListParts listParts(String volumeName, String bucketName, String keyName, String uploadID, int partNumberMarker, int maxParts) throws IOException
      Returns list of parts of a multipart upload key.
      Parameters:
      volumeName -
      bucketName -
      keyName -
      uploadID -
      partNumberMarker -
      maxParts -
      Returns:
      OmMultipartUploadListParts
      Throws:
      IOException

    • refresh

      void refresh(org.apache.hadoop.ozone.om.helpers.OmKeyInfo key) throws IOException
      Refresh the key block location information by get latest info from SCM.
      Parameters:
      key -
      Throws:
      IOException

    • getDeletedDirEntries

      default org.apache.hadoop.hdds.utils.db.TableIterator<String,? extends org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>> getDeletedDirEntries() throws IOException
      Returns an iterator for pending deleted directories all buckets.
      Throws:
      IOException

    • getDeletedDirEntries

      org.apache.hadoop.hdds.utils.db.TableIterator<String,? extends org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>> getDeletedDirEntries(String volume, String bucket) throws IOException
      Returns an iterator for pending deleted directories for volume and bucket.
      Throws:
      IOException

    • getDeletedDirEntries

      default List<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>> getDeletedDirEntries(String volume, String bucket, int size) throws IOException
      Throws:
      IOException

    • getPendingDeletionSubDirs

      DeleteKeysResult getPendingDeletionSubDirs(long volumeId, long bucketId, org.apache.hadoop.ozone.om.helpers.OmKeyInfo parentInfo, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int remainingNum) throws IOException
      Returns all sub directories under the given parent directory.
      Parameters:
      parentInfo -
      Returns:
      list of dirs
      Throws:
      IOException

    • getPendingDeletionSubFiles

      DeleteKeysResult getPendingDeletionSubFiles(long volumeId, long bucketId, org.apache.hadoop.ozone.om.helpers.OmKeyInfo parentInfo, org.apache.ratis.util.function.CheckedFunction<org.apache.hadoop.hdds.utils.db.Table.KeyValue<String,org.apache.hadoop.ozone.om.helpers.OmKeyInfo>,Boolean,IOException> filter, int remainingNum) throws IOException
      Returns all sub files under the given parent directory.
      Parameters:
      parentInfo -
      Returns:
      list of files
      Throws:
      IOException

    • getDirDeletingService

      DirectoryDeletingService getDirDeletingService()
      Returns the instance of Directory Deleting Service.
      Returns:
      Background service.

    • getOpenKeyCleanupService

      org.apache.hadoop.hdds.utils.BackgroundService getOpenKeyCleanupService()
      Returns the instance of Open Key Cleanup Service.
      Returns:
      Background service.

    • getMultipartUploadCleanupService

      org.apache.hadoop.hdds.utils.BackgroundService getMultipartUploadCleanupService()
      Returns the instance of Multipart Upload Cleanup Service.
      Returns:
      Background service.

    • getSnapshotSstFilteringService

      SstFilteringService getSnapshotSstFilteringService()
      Returns the instance of Snapshot SST Filtering service.
      Returns:
      Background service.

    • getSnapshotDefragService

      SnapshotDefragService getSnapshotDefragService()
      Returns the instance of Snapshot Defrag service.
      Returns:
      Background service.

    • getSnapshotDeletingService

      SnapshotDeletingService getSnapshotDeletingService()
      Returns the instance of Snapshot Deleting service.
      Returns:
      Background service.

    • getCompactionService

      CompactionService getCompactionService()
      Returns the instance of CompactionService.
      Returns:
      BackgroundService