Package org.h2.mvstore

Class FileStore<C extends Chunk<C>>

  • Direct Known Subclasses:
    AppendOnlyMultiFileStore, RandomAccessStore
    public abstract class FileStore<C extends Chunk<C>>
extends java.lang.Object
    Class FileStore is a base class to allow for different store implementations. FileStore concept revolves around notion of a "chunk", which is a piece of data written into the store at once.

    • Field Detail

      • META_ID_KEY

        public static final java.lang.String META_ID_KEY
        The key for the entry within "layout" map, which contains id of "meta" map. Entry value (hex encoded) is usually equal to 1, unless it's a legacy (upgraded) database and id 1 has been taken already by another map.
        See Also:
        Constant Field Values

      • readCount

        protected final java.util.concurrent.atomic.AtomicLong readCount
        The number of read operations.

      • readBytes

        protected final java.util.concurrent.atomic.AtomicLong readBytes
        The number of read bytes.

      • writeCount

        protected final java.util.concurrent.atomic.AtomicLong writeCount
        The number of write operations.

      • writeBytes

        protected final java.util.concurrent.atomic.AtomicLong writeBytes
        The number of written bytes.

      • lastChunk

        protected volatile C extends Chunk<C> lastChunk
        The newest chunk. If nothing was stored yet, this field is not set.

      • saveChunkLock

        protected final java.util.concurrent.locks.ReentrantLock saveChunkLock

      • storeHeader

        protected final java.util.HashMap<java.lang.String,​java.lang.Object> storeHeader

      • recoveryMode

        protected final boolean recoveryMode

    • Constructor Detail

      • FileStore

        protected FileStore​(java.util.Map<java.lang.String,​java.lang.Object> config)

    • Method Detail

      • open

        public abstract void open​(java.lang.String fileName,
                          boolean readOnly,
                          char[] encryptionKey)

      • open

        public abstract FileStore<C> open​(java.lang.String fileName,
                                  boolean readOnly)

      • init

        protected final void init​(java.lang.String fileName,
                          boolean readOnly)

      • bind

        public final void bind​(MVStore mvStore)

      • stop

        public final void stop​(long allowedCompactionTime)

      • close

        public void close()

      • getMetaMapId

        public final int getMetaMapId​(java.util.function.IntSupplier nextIdSupplier)

      • getLayoutMap

        public final java.util.Map<java.lang.String,​java.lang.String> getLayoutMap()
        Get this store's layout map. This data is for informational purposes only. The data is subject to change in future versions.

        The data in this map should not be modified (changing system data may corrupt the store).

        The layout map contains the following entries: 

         chunk.{chunkId} = {chunk metadata}
 root.{mapId} = {root position}
        Returns:
        the metadata map

      • isRegularMap

        public final boolean isRegularMap​(MVMap<?,​?> map)

      • getRootPos

        public final long getRootPos​(int mapId)
        Get "position" of the root page for the specified map
        Parameters:
        mapId - to get root position for
        Returns:
        opaque "position" value, that should be used to read the page

      • deregisterMapRoot

        public final boolean deregisterMapRoot​(int mapId)
        Performs final stage of map removal - delete root location info from the layout map. Specified map is supposedly closed, is anonymous and has no outstanding usage by now.
        Parameters:
        mapId - to deregister
        Returns:
        true if root was removed, false if it is not there

      • hasChangesSince

        public final boolean hasChangesSince​(long lastStoredVersion)
        Check whether there are any unsaved changes since specified version.
        Parameters:
        lastStoredVersion - version to take as a base for changes
        Returns:
        if there are any changes

      • lastChunkVersion

        public final long lastChunkVersion()

      • getMaxPageSize

        public final long getMaxPageSize()

      • getRetentionTime

        public final int getRetentionTime()

      • setRetentionTime

        public final void setRetentionTime​(int ms)
        How long to retain old, persisted chunks, in milliseconds. Chunks that are older may be overwritten once they contain no live data.

        The default value is 45000 (45 seconds) when using the default file store. It is assumed that a file system and hard disk will flush all write buffers within this time. Using a lower value might be dangerous, unless the file system and hard disk flush the buffers earlier. To manually flush the buffers, use MVStore.getFile().force(true), however please note that according to various tests this does not always work as expected depending on the operating system and hardware.

        The retention time needs to be long enough to allow reading old chunks while traversing over the entries of a map.

        This setting is not persisted.

        Parameters:
        ms - how many milliseconds to retain old chunks (0 to overwrite them as early as possible)

      • shouldSaveNow

        public abstract boolean shouldSaveNow​(int unsavedMemory,
                                      int autoCommitMemory)
        Decision about autocommit is delegated to store
        Parameters:
        unsavedMemory - amount of unsaved memory, so far
        autoCommitMemory - configured limit on amount of unsaved memory
        Returns:
        true if commit should happen now

      • getAutoCommitDelay

        public final int getAutoCommitDelay()
        Get the auto-commit delay.
        Returns:
        the delay in milliseconds, or 0 if auto-commit is disabled.

      • setAutoCommitDelay

        public final void setAutoCommitDelay​(int millis)
        Set the maximum delay in milliseconds to auto-commit changes.

        To disable auto-commit, set the value to 0. In this case, changes are only committed when explicitly calling commit.

        The default is 1000, meaning all changes are committed after at most one second.

        Parameters:
        millis - the maximum delay

      • isKnownVersion

        public final boolean isKnownVersion​(long version)
        Check whether all data can be read from this version. This requires that all chunks referenced by this version are still available (not overwritten).
        Parameters:
        version - the version
        Returns:
        true if all data can be read

      • rollbackTo

        public final void rollbackTo​(long version)

      • initializeCommonHeaderAttributes

        protected final void initializeCommonHeaderAttributes​(long time)

      • processCommonHeaderAttributes

        protected final void processCommonHeaderAttributes()

      • hasPersistentData

        protected final boolean hasPersistentData()

      • isIdle

        protected final boolean isIdle()

      • setLastChunk

        protected final void setLastChunk​(C last)

      • registerDeadChunk

        protected final void registerDeadChunk​(C chunk)

      • dropUnusedChunks

        public final void dropUnusedChunks()

      • writeFully

        protected abstract void writeFully​(C chunk,
                                   long pos,
                                   java.nio.ByteBuffer src)
        Write to the file.
        Parameters:
        chunk - to write
        pos - the write position
        src - the source buffer

      • readFully

        public abstract java.nio.ByteBuffer readFully​(C chunk,
                                              long pos,
                                              int len)
        Read data from the store.
        Parameters:
        chunk - that owns data to be read
        pos - the read "position"
        len - the number of bytes to read
        Returns:
        the byte buffer with data requested

      • readFully

        protected final java.nio.ByteBuffer readFully​(java.nio.channels.FileChannel file,
                                              long pos,
                                              int len)

      • allocateChunkSpace

        protected abstract void allocateChunkSpace​(C chunk,
                                           WriteBuffer buff)
        Allocate logical space and assign position of the buffer within the store.
        Parameters:
        chunk - to allocate space for
        buff - to allocate space for

      • writeChunk

        protected abstract void writeChunk​(C chunk,
                                   WriteBuffer buffer)
        Write buffer associated with chunk into store at chunk's allocated position
        Parameters:
        chunk - chunk to write
        buffer - to write

      • writeCleanShutdownMark

        protected abstract void writeCleanShutdownMark()
        Performs final preparation before store is closed normally

      • adjustStoreToLastChunk

        protected abstract void adjustStoreToLastChunk()
        Make persistent changes after lastChunk was reset

      • getStoreHeader

        public java.util.Map<java.lang.String,​java.lang.Object> getStoreHeader()
        Get the store header. This data is for informational purposes only. The data is subject to change in future versions. The data should not be modified (doing so may corrupt the store).
        Returns:
        the store header

      • createChunk

        protected abstract C createChunk​(int id)

      • createChunk

        public abstract C createChunk​(java.lang.String s)
        Build a Chunk from the given string.
        Parameters:
        s - the string
        Returns:
        the Chunk created

      • createChunk

        protected abstract C createChunk​(java.util.Map<java.lang.String,​java.lang.String> map)

      • writeCleanShutdown

        protected void writeCleanShutdown()

      • saveChunkMetadataChanges

        public void saveChunkMetadataChanges​(C chunk)
        Store chunk's serialized metadata as an entry in a layout map. Key for this entry would be "chunk.<id>"
        Parameters:
        chunk - to save

      • freeChunkSpace

        protected abstract void freeChunkSpace​(java.lang.Iterable<C> chunks)
        Mark the space occupied by specified chunks as free.
        Parameters:
        chunks - chunks to be processed

      • validateFileLength

        protected abstract boolean validateFileLength​(java.lang.String msg)

      • compact

        public boolean compact​(int targetFillRate,
                       int write)
        Try to increase the fill rate by re-writing partially full chunks. Chunks with a low number of live items are re-written.

        If the current fill rate is higher than the target fill rate, nothing is done.

        Please note this method will not necessarily reduce the file size, as empty chunks are not overwritten.

        Only data of open maps can be moved. For maps that are not open, the old chunk is still referenced. Therefore, it is recommended to open all maps before calling this method.

        Parameters:
        targetFillRate - the minimum percentage of live entries
        write - the minimum number of bytes to write
        Returns:
        if any chunk was re-written

      • compactStore

        public void compactStore​(long maxCompactTime)

      • compactStore

        protected abstract void compactStore​(int thresholdFillRate,
                                     long maxCompactTime,
                                     int maxWriteSize,
                                     MVStore mvStore)
        Compact store file, that is, compact blocks that have a low fill rate, and move chunks next to each other. This will typically shrink the file. Changes are flushed to the file, and old chunks are overwritten.
        Parameters:
        thresholdFillRate - do not compact if store fill rate above this value (0-100)
        maxCompactTime - the maximum time in milliseconds to compact
        maxWriteSize - the maximum amount of data to be written as part of this call
        mvStore - that owns this FileStore

      • doHousekeeping

        protected abstract void doHousekeeping​(MVStore mvStore)
                                throws java.lang.InterruptedException
        Throws:
        java.lang.InterruptedException

      • start

        public MVMap<java.lang.String,​java.lang.String> start()

      • initializeStoreHeader

        protected abstract void initializeStoreHeader​(long time)

      • readStoreHeader

        protected abstract void readStoreHeader​(boolean recoveryMode)

      • discoverChunk

        protected final C discoverChunk​(long block)
        Discover a valid chunk, searching file backwards from the given block
        Parameters:
        block - to start search from (found chunk footer should be no further than block-1)
        Returns:
        valid chunk or null if none found

      • findLastChunkWithCompleteValidChunkSet

        protected final boolean findLastChunkWithCompleteValidChunkSet​(java.util.Comparator<C> chunkComparator,
                                                               java.util.Map<java.lang.Long,​C> validChunksByLocation,
                                                               boolean afterFullScan)

      • getChunksFromLayoutMap

        protected java.lang.Iterable<C> getChunksFromLayoutMap()

      • readChunkHeaderAndFooter

        protected final C readChunkHeaderAndFooter​(long block,
                                           int expectedId)
        Read a chunk header and footer, and verify the stored data is consistent.
        Parameters:
        block - the block
        expectedId - of the chunk
        Returns:
        the chunk, or null if the header or footer don't match or are not consistent

      • readChunkHeaderOptionally

        protected final C readChunkHeaderOptionally​(long block,
                                            int expectedId)

      • readChunkHeaderOptionally

        protected final C readChunkHeaderOptionally​(long block)

      • readChunkFooter

        protected final C readChunkFooter​(long block)
        Try to read a chunk footer.
        Parameters:
        block - the index of the next block after the chunk
        Returns:
        the chunk, or null if not successful

      • getWriteBuffer

        public WriteBuffer getWriteBuffer()
        Get a buffer for writing. This caller must synchronize on the store before calling the method and until after using the buffer.
        Returns:
        the buffer

      • releaseWriteBuffer

        public void releaseWriteBuffer​(WriteBuffer buff)
        Release a buffer for writing. This caller must synchronize on the store before calling the method and until after using the buffer.
        Parameters:
        buff - the buffer than can be re-used

      • getCreationTime

        public long getCreationTime()
        The time the store was created, in milliseconds since 1970.
        Returns:
        creation time

      • getAutoCompactFillRate

        protected final int getAutoCompactFillRate()

      • sync

        public void sync()

      • getFillRate

        public abstract int getFillRate()

      • shrinkStoreIfPossible

        protected abstract void shrinkStoreIfPossible​(int minPercent)
        Shrink store if possible, and if at least a given percentage can be saved.
        Parameters:
        minPercent - the minimum percentage to save

      • size

        public long size()
        Get the file size.
        Returns:
        the file size

      • setSize

        protected final void setSize​(long size)

      • getWriteCount

        public long getWriteCount()
        Get the number of write operations since this store was opened. For file based stores, this is the number of file write operations.
        Returns:
        the number of write operations

      • getReadCount

        public long getReadCount()
        Get the number of read operations since this store was opened. For file based stores, this is the number of file read operations.
        Returns:
        the number of read operations

      • getReadBytes

        public long getReadBytes()
        Get the number of read bytes since this store was opened.
        Returns:
        the number of write operations

      • isReadOnly

        public boolean isReadOnly()

      • getDefaultRetentionTime

        public int getDefaultRetentionTime()
        Get the default retention time for this store in milliseconds.
        Returns:
        the retention time

      • clear

        public void clear()

      • getFileName

        public java.lang.String getFileName()
        Get the file name.
        Returns:
        the file name

      • getMvStore

        protected final MVStore getMvStore()

      • markUsed

        protected abstract void markUsed​(long pos,
                                 int length)
        Mark the space as in use.
        Parameters:
        pos - the position in bytes
        length - the number of bytes

      • backup

        public abstract void backup​(java.util.zip.ZipOutputStream out)
                     throws java.io.IOException
        Throws:
        java.io.IOException

      • getChunks

        protected final java.util.concurrent.ConcurrentMap<java.lang.Integer,​C> getChunks()

      • getRewriteCandidates

        protected java.util.Collection<C> getRewriteCandidates()

      • isSpaceReused

        public boolean isSpaceReused()

      • setReuseSpace

        public void setReuseSpace​(boolean reuseSpace)

      • store

        protected final void store()

      • submitOrRun

        public static void submitOrRun​(java.util.concurrent.ThreadPoolExecutor executor,
                               java.lang.Runnable action,
                               boolean syncRun,
                               int threshold)
                        throws java.util.concurrent.ExecutionException
        Throws:
        java.util.concurrent.ExecutionException

      • getChunksFillRate

        public int getChunksFillRate()
        Get the current fill rate (percentage of used space in the file). Unlike the fill rate of the store, here we only account for chunk data; the fill rate here is how much of the chunk data is live (still referenced). Young chunks are considered live.
        Returns:
        the fill rate, in percent (100 is completely full)

      • getCacheSize

        public int getCacheSize()
        Get the maximum cache size, in MB. Note that this does not include the page chunk references cache, which is 25% of the size of the page cache.
        Returns:
        the cache size

      • getCacheSizeUsed

        public int getCacheSizeUsed()
        Get the amount of memory used for caching, in MB. Note that this does not include the page chunk references cache, which is 25% of the size of the page cache.
        Returns:
        the amount of memory used for caching

      • setCacheSize

        public void setCacheSize​(int mb)
        Set the read cache size in MB.
        Parameters:
        mb - the cache size in MB.

      • populateInfo

        public void populateInfo​(java.util.function.BiConsumer<java.lang.String,​java.lang.String> consumer)

      • getCacheHitRatio

        public int getCacheHitRatio()

      • getTocCacheHitRatio

        public int getTocCacheHitRatio()

      • rewriteChunks

        protected boolean rewriteChunks​(int writeLimit,
                                int targetFillRate)

      • executeFileStoreOperation

        public void executeFileStoreOperation​(java.lang.Runnable operation)

      • accountForRemovedPage

        public void accountForRemovedPage​(long pos,
                                  long version,
                                  boolean pinned,
                                  int pageNo)
        Remove a page.
        Parameters:
        pos - the position of the page
        version - at which page was removed
        pinned - whether page is considered pinned
        pageNo - sequential page number within chunk