org.apache.hadoop.hbase.wal

Class WALSplitter

java.lang.Object

org.apache.hadoop.hbase.wal.WALSplitter

@InterfaceAudience.Private
public class WALSplitter
extends Object

This class is responsible for splitting up a bunch of regionserver commit log files that are no longer being written to, into new files, one per region, for recovering data on startup. Delete the old log files when finished.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	WALSplitter.EntryBuffers Class which accumulates edits and separates them into a buffer per region while simultaneously accounting RAM usage.
static class	WALSplitter.MutationReplay A struct used by getMutationsFromWALEntry
static class	WALSplitter.OutputSink The following class is an abstraction class to provide a common interface to support different ways of consuming recovered edits.
static class	WALSplitter.PipelineController Contains some methods to control WAL-entries producer / consumer interactions
static class	WALSplitter.RegionEntryBuffer A buffer of some number of edits for a given region.
static class	WALSplitter.SinkWriter Class wraps the actual writer which writes data out and related statistics
static class	WALSplitter.WriterThread

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.conf.Configuration	conf
protected org.apache.hadoop.fs.FileSystem	fs
protected Map<String,Long>	lastFlushedSequenceIds
protected Map<String,Map<byte[],Long>>	regionMaxSeqIdInStores
protected org.apache.hadoop.fs.Path	rootDir
protected LastSequenceId	sequenceIdChecker
static boolean	SPLIT_SKIP_ERRORS_DEFAULT By default we retry errors in splitting, rather than skipping.
static String	SPLIT_WRITER_CREATION_BOUNDED

Method Summary

Modifier and Type	Method and Description
protected WALProvider.Writer	createWriter(org.apache.hadoop.fs.Path logfile) Create a new WALProvider.Writer for writing log splits.
static void	finishSplitLogFile(String logfile, org.apache.hadoop.conf.Configuration conf) Completes the work done by splitLogFile by archiving logs
static long	getMaxRegionSequenceId(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path regionDir) Get the max sequence id which is stored in the region directory.
static List<WALSplitter.MutationReplay>	getMutationsFromWALEntry(org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.WALEntry entry, CellScanner cells, Pair<WALKey,WALEdit> logEntry, Durability durability) This function is used to construct mutations from a WALEntry.
protected WAL.Reader	getReader(org.apache.hadoop.fs.FileStatus file, boolean skipErrors, CancelableProgressable reporter) Create a new WAL.Reader for reading logs to split.
protected WAL.Reader	getReader(org.apache.hadoop.fs.Path curLogFile, CancelableProgressable reporter) Create a new WAL.Reader for reading logs to split.
static org.apache.hadoop.fs.Path	getRegionDirRecoveredEditsDir(org.apache.hadoop.fs.Path regiondir)
static NavigableSet<org.apache.hadoop.fs.Path>	getSplitEditFilesSorted(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path regiondir) Returns sorted set of edit files made by splitter, excluding files with '.temp' suffix.
static boolean	isSequenceIdFile(org.apache.hadoop.fs.Path file) Is the given file a region open sequence id file.
static org.apache.hadoop.fs.Path	moveAsideBadEditsFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path edits) Move aside a bad edits file.
static List<org.apache.hadoop.fs.Path>	split(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.Path logDir, org.apache.hadoop.fs.Path oldLogDir, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, WALFactory factory)
static boolean	splitLogFile(org.apache.hadoop.fs.Path rootDir, org.apache.hadoop.fs.FileStatus logfile, org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.conf.Configuration conf, CancelableProgressable reporter, LastSequenceId idChecker, SplitLogWorkerCoordination splitLogWorkerCoordination, WALFactory factory) Splits a WAL file into region's recovered-edits directory.
static void	writeRegionSequenceIdFile(org.apache.hadoop.fs.FileSystem fs, org.apache.hadoop.fs.Path regionDir, long newMaxSeqId) Create a file with name as region's max sequence id

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPLIT_SKIP_ERRORS_DEFAULT

public static final boolean SPLIT_SKIP_ERRORS_DEFAULT

By default we retry errors in splitting, rather than skipping.

See Also:

Constant Field Values

rootDir

protected final org.apache.hadoop.fs.Path rootDir

fs

protected final org.apache.hadoop.fs.FileSystem fs

conf

protected final org.apache.hadoop.conf.Configuration conf

sequenceIdChecker

protected final LastSequenceId sequenceIdChecker

lastFlushedSequenceIds

protected Map<String,Long> lastFlushedSequenceIds

regionMaxSeqIdInStores

protected Map<String,Map<byte[],Long>> regionMaxSeqIdInStores

SPLIT_WRITER_CREATION_BOUNDED

public static final String SPLIT_WRITER_CREATION_BOUNDED

See Also:

Constant Field Values

Method Detail

splitLogFile

public static boolean splitLogFile(org.apache.hadoop.fs.Path rootDir,
                                   org.apache.hadoop.fs.FileStatus logfile,
                                   org.apache.hadoop.fs.FileSystem fs,
                                   org.apache.hadoop.conf.Configuration conf,
                                   CancelableProgressable reporter,
                                   LastSequenceId idChecker,
                                   SplitLogWorkerCoordination splitLogWorkerCoordination,
                                   WALFactory factory)
                            throws IOException

Splits a WAL file into region's recovered-edits directory. This is the main entry point for distributed log splitting from SplitLogWorker.

If the log file has N regions then N recovered.edits files will be produced.

Returns:

false if it is interrupted by the progress-able.

Throws:

IOException

split

public static List<org.apache.hadoop.fs.Path> split(org.apache.hadoop.fs.Path rootDir,
                                                    org.apache.hadoop.fs.Path logDir,
                                                    org.apache.hadoop.fs.Path oldLogDir,
                                                    org.apache.hadoop.fs.FileSystem fs,
                                                    org.apache.hadoop.conf.Configuration conf,
                                                    WALFactory factory)
                                             throws IOException

Throws:

IOException

finishSplitLogFile

public static void finishSplitLogFile(String logfile,
                                      org.apache.hadoop.conf.Configuration conf)
                               throws IOException

Completes the work done by splitLogFile by archiving logs

It is invoked by SplitLogManager once it knows that one of the SplitLogWorkers have completed the splitLogFile() part. If the master crashes then this function might get called multiple times.

Parameters:

logfile -

conf -

Throws:

IOException

getRegionDirRecoveredEditsDir

public static org.apache.hadoop.fs.Path getRegionDirRecoveredEditsDir(org.apache.hadoop.fs.Path regiondir)

Parameters:

regiondir - This regions directory in the filesystem.

Returns:

The directory that holds recovered edits files for the region regiondir

getSplitEditFilesSorted

public static NavigableSet<org.apache.hadoop.fs.Path> getSplitEditFilesSorted(org.apache.hadoop.fs.FileSystem fs,
                                                                              org.apache.hadoop.fs.Path regiondir)
                                                                       throws IOException

Returns sorted set of edit files made by splitter, excluding files with '.temp' suffix.

Parameters:

fs -

regiondir -

Returns:

Files in passed regiondir as a sorted set.

Throws:

IOException

moveAsideBadEditsFile

public static org.apache.hadoop.fs.Path moveAsideBadEditsFile(org.apache.hadoop.fs.FileSystem fs,
                                                              org.apache.hadoop.fs.Path edits)
                                                       throws IOException

Move aside a bad edits file.

Parameters:

fs -

edits - Edits file to move aside.

Returns:

The name of the moved aside file.

Throws:

IOException

isSequenceIdFile

public static boolean isSequenceIdFile(org.apache.hadoop.fs.Path file)

Is the given file a region open sequence id file.

getMaxRegionSequenceId

public static long getMaxRegionSequenceId(org.apache.hadoop.fs.FileSystem fs,
                                          org.apache.hadoop.fs.Path regionDir)
                                   throws IOException

Get the max sequence id which is stored in the region directory. -1 if none.

Throws:

IOException

writeRegionSequenceIdFile

public static void writeRegionSequenceIdFile(org.apache.hadoop.fs.FileSystem fs,
                                             org.apache.hadoop.fs.Path regionDir,
                                             long newMaxSeqId)
                                      throws IOException

Create a file with name as region's max sequence id

Throws:

IOException

getReader

protected WAL.Reader getReader(org.apache.hadoop.fs.FileStatus file,
                               boolean skipErrors,
                               CancelableProgressable reporter)
                        throws IOException,
                               org.apache.hadoop.hbase.wal.WALSplitter.CorruptedLogFileException

Create a new WAL.Reader for reading logs to split.

Parameters:

file -

Returns:

A new Reader instance, caller should close

Throws:

IOException

CorruptedLogFileException

org.apache.hadoop.hbase.wal.WALSplitter.CorruptedLogFileException

createWriter

protected WALProvider.Writer createWriter(org.apache.hadoop.fs.Path logfile)
                                   throws IOException

Create a new WALProvider.Writer for writing log splits.

Returns:

a new Writer instance, caller should close

Throws:

IOException

getReader

protected WAL.Reader getReader(org.apache.hadoop.fs.Path curLogFile,
                               CancelableProgressable reporter)
                        throws IOException

Create a new WAL.Reader for reading logs to split.

Returns:

new Reader instance, caller should close

Throws:

IOException

getMutationsFromWALEntry

public static List<WALSplitter.MutationReplay> getMutationsFromWALEntry(org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos.WALEntry entry,
                                                                        CellScanner cells,
                                                                        Pair<WALKey,WALEdit> logEntry,
                                                                        Durability durability)
                                                                 throws IOException

This function is used to construct mutations from a WALEntry. It also reconstructs WALKey & WALEdit from the passed in WALEntry

Parameters:

entry -

cells -

logEntry - pair of WALKey and WALEdit instance stores WALKey and WALEdit instances extracted from the passed in WALEntry.

Returns:

list of Pair<MutationType, Mutation> to be replayed

Throws:

IOException