Package org.apache.hadoop.ozone.container.checksum

Class ContainerChecksumTreeManager

java.lang.Object
org.apache.hadoop.ozone.container.checksum.ContainerChecksumTreeManager
public class ContainerChecksumTreeManager extends Object
This class coordinates reading and writing Container checksum information for all containers.

  • Constructor Summary

    Constructors
    Constructor
    Description
    ContainerChecksumTreeManager(org.apache.hadoop.hdds.conf.ConfigurationSource conf)
    Creates one instance that should be used to coordinate all container checksum info within a datanode.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    addDeletedBlocks(ContainerData data, Collection<org.apache.hadoop.ozone.container.common.helpers.BlockData> blocks)
    Called by block deletion to update the merkle tree persisted to disk with more deleted blocks.
    ContainerDiffReport
    diff(org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo thisChecksumInfo, org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo peerChecksumInfo)
    Compares the checksum info of the container with the peer's checksum info and returns a report of the differences.
    static File
    getContainerChecksumFile(ContainerData data)
    Returns the container checksum tree file for the specified container without deserializing it.
    org.apache.ratis.thirdparty.com.google.protobuf.ByteString
    getContainerChecksumInfo(KeyValueContainerData data)
    Reads the container checksum info file from the disk as bytes.
    static long
    getDataChecksum(org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo checksumInfo)
     
    ContainerMerkleTreeMetrics
    getMetrics()
     
    static File
    getTmpContainerChecksumFile(ContainerData data)
     
    static boolean
    hasDataChecksum(org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo checksumInfo)
    Returns true if the ContainerProtos.ContainerChecksumInfo provided is not null, and its merkle tree has a data checksum field present.
    org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo
    read(ContainerData data)
    Reads the checksum info of the specified container.
    static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo
    readChecksumInfo(ContainerData data)
    Reads the container checksum info file (containerID.tree) from the disk.
    void
    stop()
     
    org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo
    updateTree(ContainerData data, ContainerMerkleTreeWriter treeWriter)
    Called by the container scanner and reconciliation to update the merkle tree persisted to disk.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • ContainerChecksumTreeManager

      public ContainerChecksumTreeManager(org.apache.hadoop.hdds.conf.ConfigurationSource conf)
      Creates one instance that should be used to coordinate all container checksum info within a datanode.

  • Method Details

    • stop

      public void stop()

    • diff

      public ContainerDiffReport diff(org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo thisChecksumInfo, org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo peerChecksumInfo) throws org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException
      Compares the checksum info of the container with the peer's checksum info and returns a report of the differences.
      Parameters:
      thisChecksumInfo - The checksum info of the container on this datanode.
      peerChecksumInfo - The checksum info of the container on the peer datanode.
      Throws:
      org.apache.hadoop.hdds.scm.container.common.helpers.StorageContainerException

    • getDataChecksum

      public static long getDataChecksum(org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo checksumInfo)

    • getContainerChecksumFile

      public static File getContainerChecksumFile(ContainerData data)
      Returns the container checksum tree file for the specified container without deserializing it.

    • hasDataChecksum

      public static boolean hasDataChecksum(org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo checksumInfo)
      Returns true if the ContainerProtos.ContainerChecksumInfo provided is not null, and its merkle tree has a data checksum field present. Returns false otherwise, indicating a scan of the data in this container has not yet been done.

    • getTmpContainerChecksumFile

      public static File getTmpContainerChecksumFile(ContainerData data)

    • read

      public org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo read(ContainerData data) throws IOException
      Reads the checksum info of the specified container. If the tree file with the information does not exist, an empty instance is returned. Callers are not required to hold a lock while calling this since writes are done to a tmp file and atomically swapped into place.
      Throws:
      IOException

    • updateTree

      public org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo updateTree(ContainerData data, ContainerMerkleTreeWriter treeWriter) throws IOException
      Called by the container scanner and reconciliation to update the merkle tree persisted to disk. For live (non-deleted) blocks, only those in the incoming treeWriter parameter are used. For deleted blocks, those in the incoming treeWriter are merged with those on disk.
      Throws:
      IOException

    • addDeletedBlocks

      public void addDeletedBlocks(ContainerData data, Collection<org.apache.hadoop.ozone.container.common.helpers.BlockData> blocks) throws IOException
      Called by block deletion to update the merkle tree persisted to disk with more deleted blocks. If a block with the same ID already exists in the tree, it is overwritten as deleted with the checksum computed from the chunk checksums in the BlockData. The top level container data checksum is only updated if the existing tree on disk already has this value present. This lets the block deleting service add blocks to the tree before the scanner has reached the container, and that list of deleted blocks will not be mistaken for the list of all blocks seen in the container. See hasDataChecksum(ContainerProtos.ContainerChecksumInfo).
      Throws:
      IOException

    • getContainerChecksumInfo

      public org.apache.ratis.thirdparty.com.google.protobuf.ByteString getContainerChecksumInfo(KeyValueContainerData data) throws IOException
      Reads the container checksum info file from the disk as bytes. Callers are not required to hold a lock while calling this since writes are done to a tmp file and atomically swapped into place.
      Throws:
      FileNotFoundException - When the file does not exist. It may not have been generated yet for this container.
      IOException - On error reading the file.

    • readChecksumInfo

      public static org.apache.hadoop.hdds.protocol.datanode.proto.ContainerProtos.ContainerChecksumInfo readChecksumInfo(ContainerData data) throws IOException
      Reads the container checksum info file (containerID.tree) from the disk. Callers are not required to hold a lock while calling this since writes are done to a tmp file and atomically swapped into place.
      Throws:
      IOException

    • getMetrics

      public ContainerMerkleTreeMetrics getMetrics()