org.apache.derby.impl.store.raw.log
Class LogToFile

java.lang.Object
  extended byorg.apache.derby.impl.store.raw.log.LogToFile
All Implemented Interfaces:
Corruptable, LogFactory, ModuleControl, ModuleSupportable, java.security.PrivilegedExceptionAction, Serviceable

public class LogToFile
extends java.lang.Object
implements LogFactory, ModuleControl, ModuleSupportable, Serviceable, java.security.PrivilegedExceptionAction

This is an implementation of the log using a non-circular file system file. No support for incremental log backup or media recovery. Only crash recovery is supported.

The 'log' is a stream of log records. The 'log' is implemented as a series of numbered log files. These numbered log files are logically continuous so a transaction can have log records that span multiple log files. A single log record cannot span more then one log file. The log file number is monotonically increasing.

The log belongs to a log factory of a RawStore. In the current implementation, each RawStore only has one log factory, so each RawStore only has one log (which composed of multiple log files). At any given time, a log factory only writes new log records to one log file, this log file is called the 'current log file'.

A log file is named loglogNumber.dat

Everytime a checkpoint is taken, a new log file is created and all subsequent log records will go to the new log file. After a checkpoint is taken, old and useless log files will be deleted.

RawStore exposes a checkpoint method which clients can call, or a checkpoint is taken automatically by the RawStore when

  1. the log file grows beyond a certain size (configurable, default 100K bytes)
  2. RawStore is shutdown and a checkpoint hasn't been done "for a while"
  3. RawStore is recovered and a checkpoint hasn't been done "for a while"

This LogFactory is responsible for the formats of 2 kinds of file: the log file and the log control file. And it is responsible for the format of the log record wrapper.


        Format of log control file

FILE_STREAM_LOG_FILEnone. The format is implied by the FILE_STREAM_LOG_FILEThe log file contains log record which record all the changes to the database. The complete transaction log is composed of a series of log files.The log record wrapper provides information for the log scan.int format id - the format Id of this log file int obsolete log file version - not used long log file number - this number orders the log files in a series to form the complete transaction log long prevLogRecord - log instant of the previous log record, in the previous log file. [log record wrapper]* one or more log records with wrapper int endMarker - value of zero. The beginning of a log record wrapper is the length of the log record, therefore it is never zero [int fuzzy end]* zero or more int's of value 0, in case this log file has been recovered and any incomplete log record set to zero.length(int) length of the log record (for forward scan) instant(long) LogInstant of the log record logRecord(byte[length]) byte array that is written by the FileLogger length(int) length of the log record (for backward scan)
Format IDFILE_STREAM_LOG_FILE
PurposeThe log control file contains information about which log files are present and where the last checkpoint log record is located.
Upgrade
Disk Layout(pre-v15) int format id int log file version long the log instant (LogCounter) of the last completed checkpoint (v15 onward) int format id int obsolete log file version long the log instant (LogCounter) of the last completed checkpoint int JBMS version int checkpoint interval long spare (value set to 0) long spare (value set to 0) long spare (value set to 0)

Field Summary
private  int action
           
private  StorageFile activeFile
           
private  java.lang.String activePerms
           
private static int CHECKPOINT_INTERVAL_MAX
           
private static int CHECKPOINT_INTERVAL_MIN
           
private  DaemonService checkpointDaemon
           
private  boolean checkpointDaemonCalled
           
(package private)  long checkpointInstant
           
private  int checkpointInterval
           
private  java.util.zip.CRC32 checksum
           
protected  StandardException corrupt
          If not null then something is corrupt in the raw store and this represents the original error.
private  CheckpointOperation currentCheckpoint
           
private  boolean databaseEncrypted
           
(package private)  java.lang.String dataDirectory
           
protected  DataFactory dataFactory
           
static java.lang.String DBG_FLAG
           
private static int DEFAULT_CHECKPOINT_INTERVAL
           
private static java.lang.String DEFAULT_LOG_ARCHIVE_DIRECTORY
           
private static int DEFAULT_LOG_BUFFER_SIZE
           
private static int DEFAULT_LOG_SWITCH_INTERVAL
           
static java.lang.String DUMP_LOG_FROM_LOG_FILE
           
static java.lang.String DUMP_LOG_ONLY
           
protected  long endPosition
           
private static int fid
           
(package private)  long firstLogFileNumber
           
private  boolean inCheckpoint
           
private  boolean inLogSwitch
           
private  boolean inRedo
           
private static byte IS_BETA_FLAG
           
private  boolean isFrozen
          If frozen, don't allow anything on disk to change.
private  boolean isWriteSynced
          Note: Why logging system support file sync and write sync ?
(package private)  ProductVersionHolder jbmsVersion
          Product Version information.
private  boolean keepAllLogs
           
(package private)  long lastFlush
           
private static int LOG_BUFFER_SIZE_MAX
           
private static int LOG_BUFFER_SIZE_MIN
           
protected static int LOG_FILE_HEADER_PREVIOUS_LOG_INSTANT_OFFSET
           
static int LOG_FILE_HEADER_SIZE
           
static int LOG_RECORD_OVERHEAD
           
private static int LOG_SWITCH_INTERVAL_MAX
           
private static int LOG_SWITCH_INTERVAL_MIN
           
protected static java.lang.String LOG_SYNC_STATISTICS
           
private  boolean logArchived
           
private  boolean logBeingFlushed
           
private  int logBufferSize
           
(package private)  java.lang.String logDevice
           
(package private)  long logFileNumber
           
private  boolean logNotSynced
           
protected  LogAccessFile logOut
           
private  WritableStorageFactory logStorageFactory
           
private  int logSwitchInterval
           
private  boolean logSwitchRequired
           
private  long logWrittenFromLastCheckPoint
           
private  int mon_flushCalls
           
private  boolean mon_LogSyncStatistics
           
private  int mon_numBytesToLog
           
private  int mon_numLogFlushWaits
           
private  int mon_syncCalls
           
private  int myClientNumber
           
private static int OBSOLETE_LOG_VERSION_NUMBER
           
private  boolean onDiskBeta
           
private  int onDiskMajorVersion
          On disk database version information.
private  int onDiskMinorVersion
           
private  RawStoreFactory rawStoreFactory
           
protected  boolean ReadOnlyDB
           
private  boolean recoveryNeeded
           
private  boolean stopped
           
static java.lang.String TEST_LOG_FULL
          Set to true if we want to simulate a log full condition
static java.lang.String TEST_LOG_INCOMPLETE_LOG_WRITE
          Set to true if we want the up comming log record to be only partially written.
static java.lang.String TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES
          Set to the number of bytes we want the next log record to actually write out, only used when TEST_LOG_INCOMPLETE_LOG_WRITE is on.
static java.lang.String TEST_LOG_SWITCH_LOG
          Set to true if we want the checkpoint to only switch the log but not actually do the checkpoint
(package private)  int test_logWritten
          DEBUG test only
(package private)  int test_numRecordToFillLog
           
static java.lang.String TEST_RECORD_TO_FILL_LOG
          Set to the number of log record we want to write before the log is simulated to be full.
static java.lang.String TEST_SWITCH_LOG_FAIL1
          Set to true if we want to simulate a log full condition while switching log
static java.lang.String TEST_SWITCH_LOG_FAIL2
           
private  java.io.File toFile
           
 
Fields inherited from interface org.apache.derby.iapi.store.raw.log.LogFactory
LOG_DIRECTORY_NAME, MODULE, RT_READONLY, RUNTIME_ATTRIBUTES
 
Fields inherited from interface org.apache.derby.iapi.services.daemon.Serviceable
DONE, REQUEUE
 
Constructor Summary
LogToFile()
          MT- not needed for constructor
 
Method Summary
protected  long appendLogRecord(byte[] data, int offset, int length, byte[] optionalData, int optionalDataOffset, int optionalDataLength)
          Append length bytes of data to the log prepended by a long log instant and followed by 4 bytes of length information.
 void boot(boolean create, java.util.Properties startParams)
          Boot up the log factory.
 boolean canSupport(java.util.Properties startParams)
          See if this implementation can support any attributes that are listed in properties.
private  void checkCorrupt()
           
 boolean checkpoint(RawStoreFactory rsf, DataFactory df, TransactionFactory tf, boolean wait)
          Checkpoint the rawStore.
 void checkpointInRFR(LogInstant cinstant, long redoLWM, DataFactory df)
          redo a checkpoint during rollforward recovery
protected  boolean checkpointWithTran(RawTransaction cptran, RawStoreFactory rsf, DataFactory df, TransactionFactory tf)
          checkpoint with pre-start transaction
 boolean copyActiveLogFiles(java.io.File toDir)
           
protected  long currentInstant()
          Get the current log instant - this is the log instant of the Next log record to be written out MT - This method is synchronized to ensure that it always points to the end of a log record, not the middle of one.
 boolean databaseEncrypted()
           
 int decrypt(byte[] ciphertext, int offset, int length, byte[] cleartext, int outputOffset)
           
private  void deleteObsoleteLogfiles()
           
 void deleteOnlineArchivedLogFiles()
           
 void disableLogArchiveMode()
           
 void enableLogArchiveMode()
           
 int encrypt(byte[] cleartext, int offset, int length, byte[] ciphertext, int outputOffset)
           
protected  long endPosition()
           
private  CheckpointOperation findCheckpoint(long checkpointInstant, FileLogger logger)
           
private  long firstLogInstant()
          Get the first valid log instant - this is the beginning of the first log file MT- synchronized on this
 void flush(LogInstant where)
          Flush all unwritten log record up to the log instance indicated to disk and sync.
protected  void flush(long fileNumber, long wherePosition)
          Flush the log such that the log record written with the instant wherePosition is guaranteed to be on disk.
 void flushAll()
          Flush all unwritten log record to disk and sync.
private  void flushBuffer(long fileNumber, long wherePosition)
          Flush all unwritten log record up to the log instance indicated to disk without syncing.
 void freezePersistentStore()
          Backup restore - stop sending log record to the log stream
 java.lang.String getCanonicalLogPath()
          Return the canonical directory of the PARENT of the log directory.
private  StorageFile getControlFileName()
          Return the control file name MT- read only
 int getEncryptionBlockSize()
          return the encryption block size used during encrypted db creation
private  long getFirstLogNeeded(CheckpointOperation checkpoint)
           
 LogInstant getFirstUnflushedInstant()
          Get the instant of the first record which was not flushed.
 StorageFile getLogDirectory()
          Return the location of the log directory.
private  java.lang.String getLogDirPath(StorageFile logDir)
           
 void getLogFactoryProperties(PersistentSet set)
          Get JBMS properties relavent to the log factory
protected  StorageRandomAccessFile getLogFileAtBeginning(long filenumber)
          Open a log file and position the file at the beginning.
protected  StorageRandomAccessFile getLogFileAtPosition(long logInstant)
          Get a read-only handle to the log file positioned at the stated position MT- read only
private  StorageFile getLogFileName(long filenumber)
          Given a log file number, return its file name MT- read only
private  long getLogFileNumber()
          Return the current log file number.
 Logger getLogger()
          MT- not needed
private  void getLogStorageFactory()
           
private  StorageFile getMirrorControlFileName()
          Return the mirror control file name MT- read only
 int getTypeFormatId()
          Return my format identifier.
private  boolean initLogFile(StorageRandomAccessFile newlog, long number, long prevLogRecordEndInstant)
          Initialize the log to the correct format with the given version and log file number.
 boolean inRFR()
           
 boolean logArchived()
          Backup restore - is the log being archived to some directory?
protected  void logErrMsg(java.lang.String msg)
          Print error message to user about the log MT - not needed, informational only
protected  void logErrMsg(java.lang.Throwable t)
          Print error message to user about the log MT - not needed, informational only
private  long logtest_appendPartialLogRecord(byte[] data, int offset, int length, byte[] optionalData, int optionalDataOffset, int optionalDataLength)
          Writes out a partial log record - takes the appendLogRecord.
 StandardException markCorrupt(StandardException originalError)
          Once the log factory is makred as corrupt then the raw sto
protected  LogScan openBackwardsScan(LogInstant stopAt)
          Scan backward from end of log.
protected  LogScan openBackwardsScan(long startAt, LogInstant stopAt)
          Scan backward from start position.
 ScanHandle openFlushedScan(DatabaseInstant start, int groupsIWant)
          Get a ScanHandle to scan flushed records from the log.
 LogScan openForwardsFlushedScan(LogInstant startAt)
          Open a forward scan of the transaction log.
 LogScan openForwardsScan(LogInstant startAt, LogInstant stopAt)
          Get a forwards scan
protected  LogScan openForwardsScan(long startAt, LogInstant stopAt)
          Scan Forward from start position.
 StorageRandomAccessFile openLogFileInWriteMode(StorageFile logFile)
           
 int performWork(ContextManager context)
          Do whatever it is that you want the daemon to do for you.
private  void preAllocateNewLogFile(StorageRandomAccessFile log)
           
private  void printErrorStack(java.lang.Throwable t)
           
protected  boolean privCanWrite(StorageFile file)
           
protected  boolean privCopyFile(java.io.File from, StorageFile to)
           
protected  boolean privCopyFile(StorageFile from, java.io.File to)
           
protected  boolean privDelete(StorageFile file)
           
protected  boolean privExists(StorageFile file)
           
protected  java.lang.String[] privList(java.io.File file)
           
protected  java.lang.String[] privList(StorageFile file)
           
protected  boolean privMkdirs(StorageFile file)
           
protected  StorageRandomAccessFile privRandomAccessFile(StorageFile file, java.lang.String perms)
           
protected  boolean privRemoveDirectory(StorageFile file)
           
private  long readControlFile(StorageFile logControlFileName, java.util.Properties startParams)
           
 void recover(RawStoreFactory rsf, DataFactory df, TransactionFactory tf)
          Recover the rawStore to a consistent state using the log.
private  void restoreLogs(java.util.Properties properties)
          This function restores logs based on the following attributes are specified on connection URL: Attribute.CREATE_FROM (Create database from backup if it does not exist) Attribute.RESTORE_FROM (Delete the whole database if it exists and then restore it from backup) Attribute.ROLL_FORWARD_RECOVERY_FROM:(Perform Rollforward Recovery; except for the log directory everthing else is replced by the copy from backup. log files in the backup are copied to the existing online log directory.
 java.lang.Object run()
           
private  boolean runBooleanAction(int action, StorageFile file)
           
 boolean serviceASAP()
          If this work should be done as soon as possible, then return true.
 boolean serviceImmediately()
          If this work should be done immediately on the user thread then return true.
 void stop()
          Stop the log factory MT- caller provide synchronization (RESOLVE: this should be called AFTER dataFactory and transFactory are stopped)
private  void switchLogFile()
          Switch to the next log file if possible.
private  void syncFile(StorageRandomAccessFile raf)
          Utility routine to call sync() on the input file descriptor.
protected  void testLogFull()
          Simulate a log full condition if TEST_LOG_FULL is set to true, then the property TEST_RECORD_TO_FILL_LOG indicates the number of times this function is call before an IOException simulating a log full condition is raised.
private  void truncateLog(CheckpointOperation checkpoint)
          Get rid of old and unnecessary log files MT- only one truncate log is allowed to be taking place at any given time.
 void unfreezePersistentStore()
          Backup restore - start sending log record to the log stream
private  boolean verifyLogFormat(StorageFile logFileName, long number)
          Verify that we the log file is of the right format and of the right version and log file number.
private  boolean verifyLogFormat(StorageRandomAccessFile log, long number)
          Verify that we the log file is of the right format and of the right version and log file number.
(package private)  boolean writeControlFile(StorageFile logControlFileName, long value)
          Carefully write out this value to the control file.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

fid

private static int fid

LOG_FILE_HEADER_SIZE

public static final int LOG_FILE_HEADER_SIZE
See Also:
Constant Field Values

LOG_FILE_HEADER_PREVIOUS_LOG_INSTANT_OFFSET

protected static final int LOG_FILE_HEADER_PREVIOUS_LOG_INSTANT_OFFSET
See Also:
Constant Field Values

LOG_RECORD_OVERHEAD

public static final int LOG_RECORD_OVERHEAD
See Also:
Constant Field Values

DBG_FLAG

public static final java.lang.String DBG_FLAG

DUMP_LOG_ONLY

public static final java.lang.String DUMP_LOG_ONLY

DUMP_LOG_FROM_LOG_FILE

public static final java.lang.String DUMP_LOG_FROM_LOG_FILE

LOG_SYNC_STATISTICS

protected static final java.lang.String LOG_SYNC_STATISTICS
See Also:
Constant Field Values

OBSOLETE_LOG_VERSION_NUMBER

private static final int OBSOLETE_LOG_VERSION_NUMBER
See Also:
Constant Field Values

DEFAULT_LOG_SWITCH_INTERVAL

private static final int DEFAULT_LOG_SWITCH_INTERVAL
See Also:
Constant Field Values

LOG_SWITCH_INTERVAL_MIN

private static final int LOG_SWITCH_INTERVAL_MIN
See Also:
Constant Field Values

LOG_SWITCH_INTERVAL_MAX

private static final int LOG_SWITCH_INTERVAL_MAX
See Also:
Constant Field Values

CHECKPOINT_INTERVAL_MIN

private static final int CHECKPOINT_INTERVAL_MIN
See Also:
Constant Field Values

CHECKPOINT_INTERVAL_MAX

private static final int CHECKPOINT_INTERVAL_MAX
See Also:
Constant Field Values

DEFAULT_CHECKPOINT_INTERVAL

private static final int DEFAULT_CHECKPOINT_INTERVAL
See Also:
Constant Field Values

DEFAULT_LOG_BUFFER_SIZE

private static final int DEFAULT_LOG_BUFFER_SIZE
See Also:
Constant Field Values

LOG_BUFFER_SIZE_MIN

private static final int LOG_BUFFER_SIZE_MIN
See Also:
Constant Field Values

LOG_BUFFER_SIZE_MAX

private static final int LOG_BUFFER_SIZE_MAX
See Also:
Constant Field Values

logBufferSize

private int logBufferSize

IS_BETA_FLAG

private static final byte IS_BETA_FLAG
See Also:
Constant Field Values

DEFAULT_LOG_ARCHIVE_DIRECTORY

private static final java.lang.String DEFAULT_LOG_ARCHIVE_DIRECTORY
See Also:
Constant Field Values

logSwitchInterval

private int logSwitchInterval

checkpointInterval

private int checkpointInterval

dataDirectory

java.lang.String dataDirectory

logStorageFactory

private WritableStorageFactory logStorageFactory

logBeingFlushed

private boolean logBeingFlushed

logOut

protected LogAccessFile logOut

endPosition

protected long endPosition

lastFlush

long lastFlush

logFileNumber

long logFileNumber

firstLogFileNumber

long firstLogFileNumber

currentCheckpoint

private CheckpointOperation currentCheckpoint

checkpointInstant

long checkpointInstant

checkpointDaemon

private DaemonService checkpointDaemon

myClientNumber

private int myClientNumber

checkpointDaemonCalled

private volatile boolean checkpointDaemonCalled

logWrittenFromLastCheckPoint

private long logWrittenFromLastCheckPoint

rawStoreFactory

private RawStoreFactory rawStoreFactory

dataFactory

protected DataFactory dataFactory

ReadOnlyDB

protected boolean ReadOnlyDB

keepAllLogs

private boolean keepAllLogs

databaseEncrypted

private boolean databaseEncrypted

recoveryNeeded

private boolean recoveryNeeded

inCheckpoint

private boolean inCheckpoint

inRedo

private boolean inRedo

inLogSwitch

private boolean inLogSwitch

stopped

private boolean stopped

logDevice

java.lang.String logDevice

logNotSynced

private boolean logNotSynced

logArchived

private boolean logArchived

logSwitchRequired

private boolean logSwitchRequired

test_logWritten

int test_logWritten
DEBUG test only


test_numRecordToFillLog

int test_numRecordToFillLog

mon_flushCalls

private int mon_flushCalls

mon_syncCalls

private int mon_syncCalls

mon_numLogFlushWaits

private int mon_numLogFlushWaits

mon_LogSyncStatistics

private boolean mon_LogSyncStatistics

mon_numBytesToLog

private int mon_numBytesToLog

corrupt

protected volatile StandardException corrupt
If not null then something is corrupt in the raw store and this represents the original error.


isFrozen

private boolean isFrozen
If frozen, don't allow anything on disk to change.


jbmsVersion

ProductVersionHolder jbmsVersion
Product Version information. Invarient after boot.


onDiskMajorVersion

private int onDiskMajorVersion
On disk database version information. When running in soft upgrade this version may be different to jbmsVersion.


onDiskMinorVersion

private int onDiskMinorVersion

onDiskBeta

private boolean onDiskBeta

checksum

private java.util.zip.CRC32 checksum

isWriteSynced

private boolean isWriteSynced
Note: Why logging system support file sync and write sync ? Note : The reason to support file and write sync of logs is there was no support to do write sync until jdk1.4 and then there was write sync jvm bug in jdk1.4.1, only in jdk1.4.2 write sync(rws mode) mechanism can be used corretly. Default in JVMS >= jdk1.4.2 is write sync(see the boot method for jvm checks). Write sync mechanism support is added for performance reasons. On commits, logging system has to make sure the log for committed transaction is on disk. With out write sync , log is written to the disk and then fsync() is used on commits to make log is written to the disk for sure. On most of the OS , fsync() calls are expensive. On heavey commit oriented systems , file sync make the system run slow. This problem is solved by using write sync on preallocated log file. write sync is much faster than doing write and file sync to a file. File should be preallocated for write syncs to perform better than the file sync method. Whenever a new log file is created, logSwitchInterval size is preallocated by writing zeros after file after the header.


TEST_LOG_SWITCH_LOG

public static final java.lang.String TEST_LOG_SWITCH_LOG
Set to true if we want the checkpoint to only switch the log but not actually do the checkpoint


TEST_LOG_INCOMPLETE_LOG_WRITE

public static final java.lang.String TEST_LOG_INCOMPLETE_LOG_WRITE
Set to true if we want the up comming log record to be only partially written. The database is corrupted if not immediately shutdown. Set TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES to the number of bytes to write out, default is 1 byte.


TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES

public static final java.lang.String TEST_LOG_PARTIAL_LOG_WRITE_NUM_BYTES
Set to the number of bytes we want the next log record to actually write out, only used when TEST_LOG_INCOMPLETE_LOG_WRITE is on. Default is 1 byte.


TEST_LOG_FULL

public static final java.lang.String TEST_LOG_FULL
Set to true if we want to simulate a log full condition


TEST_SWITCH_LOG_FAIL1

public static final java.lang.String TEST_SWITCH_LOG_FAIL1
Set to true if we want to simulate a log full condition while switching log


TEST_SWITCH_LOG_FAIL2

public static final java.lang.String TEST_SWITCH_LOG_FAIL2

TEST_RECORD_TO_FILL_LOG

public static final java.lang.String TEST_RECORD_TO_FILL_LOG
Set to the number of log record we want to write before the log is simulated to be full.


action

private int action

activeFile

private StorageFile activeFile

toFile

private java.io.File toFile

activePerms

private java.lang.String activePerms
Constructor Detail

LogToFile

public LogToFile()
MT- not needed for constructor

Method Detail

getTypeFormatId

public int getTypeFormatId()
Return my format identifier.


markCorrupt

public StandardException markCorrupt(StandardException originalError)
Once the log factory is makred as corrupt then the raw sto

Specified by:
markCorrupt in interface Corruptable
Returns:
Must always return its parameter.

checkCorrupt

private void checkCorrupt()
                   throws StandardException
Throws:
StandardException

getLogger

public Logger getLogger()
MT- not needed

Specified by:
getLogger in interface LogFactory

recover

public void recover(RawStoreFactory rsf,
                    DataFactory df,
                    TransactionFactory tf)
             throws StandardException
Recover the rawStore to a consistent state using the log.

In this implementation, the log is a stream of log records stored in one or more flat files. Recovery is done in 2 passes: redo and undo.
Redo pass
In the redo pass, reconstruct the state of the rawstore by repeating exactly what happened before as recorded in the log.
Undo pass
In the undo pass, all incomplete transactions are rolled back in the order from the most recently started to the oldest.

MT - synchronization provided by caller - RawStore boot. This method is guaranteed to be the only method being called and can assume single thread access on all fields.

Specified by:
recover in interface LogFactory
Parameters:
rsf - - the raw store
df - - the data factory
tf - - the transaction factory
Throws:
StandardException - Standard Cloudscape error policy
See Also:
Loggable.needsRedo(org.apache.derby.iapi.store.raw.Transaction), FileLogger.redo(org.apache.derby.iapi.store.raw.xact.RawTransaction, org.apache.derby.iapi.store.raw.xact.TransactionFactory, org.apache.derby.impl.store.raw.log.StreamLogScan, long, long)

checkpoint

public boolean checkpoint(RawStoreFactory rsf,
                          DataFactory df,
                          TransactionFactory tf,
                          boolean wait)
                   throws StandardException
Checkpoint the rawStore.

MT- Only one checkpoint is to be taking place at any given time.

The steps of a checkpoint are

  1. switch to a new log file if possible
                            freeze the log (for the transition to a new log file)
                                    flush current log file
                                    create and flush the new log file (with file number 1 higher
                    than the previous log file). The new log file becomes the
                    current log file.
                            unfreeze the log
                    
  2. start checkpoint transaction
  3. gather interesting information about the rawStore: the current log instant (redoLWM) the earliest active transaction begin tran log record instant (undoLWM), all the truncation LWM set by clients of raw store (replication)
  4. clean the buffer cache
  5. log the next checkpoint log record, which contains (repPoint, undoLWM, redoLWM) and commit checkpoint transaction.
  6. synchronously write the control file containing the next checkpoint log record log instant
  7. the new checkpoint becomes the current checkpoint. Somewhere near the beginning of each log file should be a checkpoint log record (not guarenteed to be there)
  8. see if the log can be truncated

    The earliest useful log record is determined by the repPoint and the undoLWM, whichever is earlier.

    Every log file whose log file number is smaller than the earliest useful log record's log file number can be deleted.

                            Transactions can be at the following states w/r to a checkpoint -
                            consider the log as a continous stream and not as series of log 
                files for the sake of clarity.  
                            |(BT)-------(ET)| marks the begin and end of a transaction.
                            .                          checkpoint started
                            .       |__undoLWM          |
                            .       V                   |___redoLWM
                            .                           |___TruncationLWM
                            .                           |
                            .                           V
                            1 |-----------------|
                            2       |--------------------------------|
                            3           |-------|
                            4               |--------------------------------------(end of log)
                            5                                       |-^-|
                            .                                   Checkpoint Log Record
                            ---A--->|<-------B--------->|<-------------C-----------
                    

    There are only 3 periods of interest :
    A) before undoLWM, B) between undo and redo LWM, C) after redoLWM.

    Transaction 1 started in A and terminates in B.
    During redo, we should only see log records and endXact from this transaction in the first phase (between undoLWM and redoLWM). No beginXact log record for this transaction will be seen.

    Transaction 2 started in B (right on the undoLWM) and terminated in C.
    Any transaction that terminates in C must have a beginXact at or after undoLWM. In other words, no transaction can span A, B and C. During redo, we will see beginXact, other log records and endXact for this transaction.

    Transaction 3 started in B and ended in B.
    During redo, we will see beginXact, other log records and endXact for this transaction.

    Transaction 4 begins in B and never ends.
    During redo, we will see beginXact, other log records. In undo, this loser transaction will be rolled back.

    Transaction 5 is the transaction taking the checkpoint.
    The checkpoint action started way back in time but the checkpoint log record is only written after the buffer cache has been flushed.

    Note that if any time elapse between taking the undoLWM and the redoLWM, then it will create a 4th period of interest.

    Specified by:
    checkpoint in interface LogFactory
    Parameters:
    rsf - - the raw store
    df - - the data factory
    tf - - the transaction factory
    wait - - if true; waits for the checkpoint to completed even if it is being done my an another thread.
    Returns:
    true if checkpoint is successful
    Throws:
    StandardException - - encounter exception while doing checkpoint.

checkpointWithTran

protected boolean checkpointWithTran(RawTransaction cptran,
                                     RawStoreFactory rsf,
                                     DataFactory df,
                                     TransactionFactory tf)
                              throws StandardException
checkpoint with pre-start transaction

Throws:
StandardException - Cloudscape Standard Error Policy

flush

public void flush(LogInstant where)
           throws StandardException
Flush all unwritten log record up to the log instance indicated to disk and sync. Also check to see if database is frozen or corrupt.

MT - not needed, wrapper method

Specified by:
flush in interface LogFactory
Parameters:
where - flush log up to here
Throws:
StandardException - Standard Cloudscape error policy

flushAll

public void flushAll()
              throws StandardException
Flush all unwritten log record to disk and sync. Also check to see if database is frozen or corrupt.

MT - not needed, wrapper method

Throws:
StandardException - Standard Cloudscape error policy

verifyLogFormat

private boolean verifyLogFormat(StorageFile logFileName,
                                long number)
                         throws StandardException
Verify that we the log file is of the right format and of the right version and log file number.

MT - not needed, no global variables used

Parameters:
logFileName - the name of the log file
number - the log file number
Returns:
true if the log file is of the current version and of the correct format
Throws:
StandardException - Standard Cloudscape error policy

verifyLogFormat

private boolean verifyLogFormat(StorageRandomAccessFile log,
                                long number)
                         throws StandardException
Verify that we the log file is of the right format and of the right version and log file number. The log file position is set to the beginning.

MT - MT-unsafe, caller must synchronize

Parameters:
log - the log file
number - the log file number
Returns:
true if the log file is of the current version and of the correct format
Throws:
StandardException - Standard Cloudscape error policy

initLogFile

private boolean initLogFile(StorageRandomAccessFile newlog,
                            long number,
                            long prevLogRecordEndInstant)
                     throws java.io.IOException,
                            StandardException
Initialize the log to the correct format with the given version and log file number. The new log file must be empty. After initializing, the file is synchronously written to disk.

MT - synchornization provided by caller

Parameters:
number - the log file number
prevLogRecordEndInstant - the end position of the previous log record
Returns:
true if the log file is empty, else false.
Throws:
java.io.IOException - if new log file cannot be accessed or initialized
StandardException

switchLogFile

private void switchLogFile()
                    throws StandardException
Switch to the next log file if possible.

MT - log factory is single threaded thru a log file switch, the log is frozen for the duration of the switch

Throws:
StandardException

flushBuffer

private void flushBuffer(long fileNumber,
                         long wherePosition)
                  throws java.io.IOException
Flush all unwritten log record up to the log instance indicated to disk without syncing.

MT - not needed, wrapper method

Throws:
java.io.IOException - Failed to flush to the log

truncateLog

private void truncateLog(CheckpointOperation checkpoint)
Get rid of old and unnecessary log files

MT- only one truncate log is allowed to be taking place at any given time. Synchronized on this.


getFirstLogNeeded

private long getFirstLogNeeded(CheckpointOperation checkpoint)

writeControlFile

boolean writeControlFile(StorageFile logControlFileName,
                         long value)
                   throws java.io.IOException,
                          StandardException
Carefully write out this value to the control file. We do safe write of this data by writing the data into two files every time we write the control data. we write checksum at the end of the file, so if by chance system crashes while writing into the file, using the checksum we find that the control file is hosed then we use the mirror file, which will have the condrol data written at last check point. see comment at beginning of file for log control file format.

MT- synchronized by caller

Throws:
java.io.IOException
StandardException

readControlFile

private long readControlFile(StorageFile logControlFileName,
                             java.util.Properties startParams)
                      throws java.io.IOException,
                             StandardException
Throws:
java.io.IOException
StandardException

getLogDirectory

public StorageFile getLogDirectory()
                            throws StandardException
Description copied from interface: LogFactory
Return the location of the log directory.

Specified by:
getLogDirectory in interface LogFactory
Throws:
StandardException - Standard Cloudscape Error Policy

getCanonicalLogPath

public java.lang.String getCanonicalLogPath()
Description copied from interface: LogFactory
Return the canonical directory of the PARENT of the log directory. The log directory live in the "log" subdirectory of this path. If the log is at the default location (underneath the database directory), this returns null. Should only be called after the log factory is booted.

Specified by:
getCanonicalLogPath in interface LogFactory

getControlFileName

private StorageFile getControlFileName()
                                throws StandardException
Return the control file name

MT- read only

Throws:
StandardException

getMirrorControlFileName

private StorageFile getMirrorControlFileName()
                                      throws StandardException
Return the mirror control file name

MT- read only

Throws:
StandardException

getLogFileName

private StorageFile getLogFileName(long filenumber)
                            throws StandardException
Given a log file number, return its file name

MT- read only

Throws:
StandardException

findCheckpoint

private CheckpointOperation findCheckpoint(long checkpointInstant,
                                           FileLogger logger)
                                    throws java.io.IOException,
                                           StandardException,
                                           java.lang.ClassNotFoundException
Throws:
java.io.IOException
StandardException
java.lang.ClassNotFoundException

openBackwardsScan

protected LogScan openBackwardsScan(long startAt,
                                    LogInstant stopAt)
                             throws java.io.IOException,
                                    StandardException
Scan backward from start position.

MT- read only

Throws:
java.io.IOException - cannot access the log
StandardException - Standard Cloudscape error policy

openBackwardsScan

protected LogScan openBackwardsScan(LogInstant stopAt)
                             throws java.io.IOException,
                                    StandardException
Scan backward from end of log.

MT- read only

Throws:
java.io.IOException - cannot access the log
StandardException - Standard Cloudscape error policy

openFlushedScan

public ScanHandle openFlushedScan(DatabaseInstant start,
                                  int groupsIWant)
                           throws StandardException
Description copied from interface: LogFactory
Get a ScanHandle to scan flushed records from the log.

MT- read only

Specified by:
openFlushedScan in interface LogFactory
Parameters:
groupsIWant - - log record groups the scanner wants.
Returns:
the LogScan.
Throws:
StandardException - Ooops.
See Also:
LogFactory.openFlushedScan(org.apache.derby.iapi.store.access.DatabaseInstant, int)

openForwardsScan

protected LogScan openForwardsScan(long startAt,
                                   LogInstant stopAt)
                            throws java.io.IOException,
                                   StandardException
Scan Forward from start position.

MT- read only

Parameters:
startAt - - if startAt == INVALID_LOG_INSTANT, start from the beginning of the log. Otherwise, start scan from startAt.
stopAt - - if not null, stop at this log instant (inclusive). Otherwise, stop at the end of the log
Throws:
java.io.IOException - cannot access the log
StandardException - Standard Cloudscape error policy

getLogFileAtBeginning

protected StorageRandomAccessFile getLogFileAtBeginning(long filenumber)
                                                 throws java.io.IOException,
                                                        StandardException
Open a log file and position the file at the beginning. Used by scan to switch to the next log file

MT- read only

Throws:
StandardException - Standard Cloudscape error policy
java.io.IOException - cannot access the log at the new position.

getLogFileAtPosition

protected StorageRandomAccessFile getLogFileAtPosition(long logInstant)
                                                throws java.io.IOException,
                                                       StandardException
Get a read-only handle to the log file positioned at the stated position

MT- read only

Returns:
null if file does not exist or of the wrong format
Throws:
java.io.IOException - cannot access the log at the new position.
StandardException - Standard Cloudscape error policy

canSupport

public boolean canSupport(java.util.Properties startParams)
Description copied from interface: ModuleSupportable
See if this implementation can support any attributes that are listed in properties. This call may be made on a newly created instance before the boot() method has been called, or after the boot method has been called for a running module.

The module can check for attributes in the properties to see if it can fulfill the required behaviour. E.g. the raw store may define an attribute called RawStore.Recoverable. If a temporary raw store is required the property RawStore.recoverable=false would be added to the properties before calling bootServiceModule. If a raw store cannot support this attribute its canSupport method would return null. Also see the Monitor class's prologue to see how the identifier is used in looking up properties.
Actually a better way maybe to have properties of the form RawStore.Attributes.mandatory=recoverable,smallfootprint and RawStore.Attributes.requested=oltp,fast

Specified by:
canSupport in interface ModuleSupportable
Returns:
true if this instance can be used, false otherwise.

boot

public void boot(boolean create,
                 java.util.Properties startParams)
          throws StandardException
Boot up the log factory.

MT- caller provide synchronization

Specified by:
boot in interface ModuleControl
Throws:
StandardException - log factory cannot start up
See Also:
Monitor, ModuleFactory

getLogStorageFactory

private void getLogStorageFactory()
                           throws StandardException
Throws:
StandardException

stop

public void stop()
Stop the log factory

MT- caller provide synchronization (RESOLVE: this should be called AFTER dataFactory and transFactory are stopped)

Specified by:
stop in interface ModuleControl
See Also:
Monitor, ModuleFactory

deleteObsoleteLogfiles

private void deleteObsoleteLogfiles()

serviceASAP

public boolean serviceASAP()
Description copied from interface: Serviceable
If this work should be done as soon as possible, then return true. If it doesn't make any difference if it is done sooner rather than later, then return false. The difference is whether or not the daemon service will be notified to work on this when this work is enqueued or subscribed, in case the serviceable work is put together but not sent to the daemon service directly, like in post commit processing

MT - MT safe

Specified by:
serviceASAP in interface Serviceable

serviceImmediately

public boolean serviceImmediately()
Description copied from interface: Serviceable
If this work should be done immediately on the user thread then return true. If it doesn't make any difference if this work is done on a the user thread immediately or if it is performed by another thread asynchronously later, then return false.

Specified by:
serviceImmediately in interface Serviceable

getLogFactoryProperties

public void getLogFactoryProperties(PersistentSet set)
                             throws StandardException
Description copied from interface: LogFactory
Get JBMS properties relavent to the log factory

Specified by:
getLogFactoryProperties in interface LogFactory
Throws:
StandardException - Standard Cloudscape Error Policy

performWork

public int performWork(ContextManager context)
Description copied from interface: Serviceable
Do whatever it is that you want the daemon to do for you. There may be multiple daemon objects on different thread calling performWork at the same time. The DaemonService will always call performWork with a context manager set up. the DaemonService will clean up the context if an exception is thrown. However, it is up to performWork to manage its own transaction. If you start a transaction in performWork, you must commit or abort it at the end. You may leave the transaction open so that other serviceable may use the transaction and context without starting a new one. On the same token, there may already be an opened transaction on the context. Serviceable performWork should always check the state of the context before use. A Serviceable object should be well behaved while it is performing the daemon work, i.e., it should not take too many resources or hog the CPU for too long or deadlock with anyone else.

Specified by:
performWork in interface Serviceable
Parameters:
context - the contextManager set up by the DaemonService. There may or may not be the necessary context on it, depending on which other Serviceable object it has done work for.
Returns:
the return status is only significant if the Serviceable client was enqueued instead of subscribed. For subscribed client, the return status is ignored. For enqueue client, it returns DONE or REQUEUE. If a REQUEUEd is returned, it would be desirable if this should not be serviceASAP, although no harm is done if this still maintains that this should be serviced ASAP ...

appendLogRecord

protected long appendLogRecord(byte[] data,
                               int offset,
                               int length,
                               byte[] optionalData,
                               int optionalDataOffset,
                               int optionalDataLength)
                        throws StandardException
Append length bytes of data to the log prepended by a long log instant and followed by 4 bytes of length information.

This method is synchronized to ensure log records are added sequentially to the end of the log.

MT- single threaded through this log factory. Log records are appended one at a time.

Throws:
StandardException - Log Full.

currentInstant

protected long currentInstant()
Get the current log instant - this is the log instant of the Next log record to be written out

MT - This method is synchronized to ensure that it always points to the end of a log record, not the middle of one.


endPosition

protected long endPosition()

getLogFileNumber

private long getLogFileNumber()
Return the current log file number.

MT - this method is synchronized so that it is not in the middle of being changed by swithLogFile


firstLogInstant

private long firstLogInstant()
Get the first valid log instant - this is the beginning of the first log file

MT- synchronized on this


flush

protected void flush(long fileNumber,
                     long wherePosition)
              throws StandardException
Flush the log such that the log record written with the instant wherePosition is guaranteed to be on disk.

MT - only one flush is allowed to be taking place at any given time (RESOLVE: right now it single thread thru the log factory while the log is frozen)

Throws:
StandardException - cannot sync log file

syncFile

private void syncFile(StorageRandomAccessFile raf)
               throws StandardException
Utility routine to call sync() on the input file descriptor.

Throws:
StandardException

openForwardsFlushedScan

public LogScan openForwardsFlushedScan(LogInstant startAt)
                                throws StandardException
Open a forward scan of the transaction log.

MT- read only

Specified by:
openForwardsFlushedScan in interface LogFactory
Parameters:
startAt - - the LogInstant where we start our scan. null means start at the beginning of the log. This function raises an error if startAt is a LogInstant which is not in the log.
Returns:
the LogScan.
Throws:
StandardException - Standard cloudscape exception policy

openForwardsScan

public LogScan openForwardsScan(LogInstant startAt,
                                LogInstant stopAt)
                         throws StandardException
Get a forwards scan

Specified by:
openForwardsScan in interface LogFactory
Parameters:
startAt - - the LogInstant where we start our scan. null means start at the beginning of the log. This function raises an error if startAt is a LogInstant which is not in the log.
stopAt - - the LogInstant where we stop our scan. null means stop at the end of the log. This function raises an error if stopAt is a LogInstant which is not in the log.
Returns:
the LogScan.
Throws:
StandardException - Standard Cloudscape error policy

databaseEncrypted

public final boolean databaseEncrypted()

encrypt

public int encrypt(byte[] cleartext,
                   int offset,
                   int length,
                   byte[] ciphertext,
                   int outputOffset)
            throws StandardException
Throws:
StandardException - Standard Cloudscape Error Policy
See Also:
RawStoreFactory.encrypt(byte[], int, int, byte[], int)

decrypt

public int decrypt(byte[] ciphertext,
                   int offset,
                   int length,
                   byte[] cleartext,
                   int outputOffset)
            throws StandardException
Throws:
StandardException - Standard Cloudscape Error Policy
See Also:
RawStoreFactory.decrypt(byte[], int, int, byte[], int)

getEncryptionBlockSize

public int getEncryptionBlockSize()
return the encryption block size used during encrypted db creation


getFirstUnflushedInstant

public LogInstant getFirstUnflushedInstant()
Get the instant of the first record which was not flushed.

This only works after running recovery the first time.

MT - RESOLVE:

Specified by:
getFirstUnflushedInstant in interface LogFactory

freezePersistentStore

public void freezePersistentStore()
                           throws StandardException
Backup restore - stop sending log record to the log stream

Specified by:
freezePersistentStore in interface LogFactory
Throws:
StandardException - Standard Cloudscape error policy

unfreezePersistentStore

public void unfreezePersistentStore()
                             throws StandardException
Backup restore - start sending log record to the log stream

Specified by:
unfreezePersistentStore in interface LogFactory
Throws:
StandardException - Standard Cloudscape error policy

logArchived

public boolean logArchived()
Backup restore - is the log being archived to some directory? if log archive mode is enabled return true else false

Specified by:
logArchived in interface LogFactory
Returns:
true if the log is being archived.

logErrMsg

protected void logErrMsg(java.lang.String msg)
Print error message to user about the log MT - not needed, informational only


logErrMsg

protected void logErrMsg(java.lang.Throwable t)
Print error message to user about the log MT - not needed, informational only


printErrorStack

private void printErrorStack(java.lang.Throwable t)

logtest_appendPartialLogRecord

private long logtest_appendPartialLogRecord(byte[] data,
                                            int offset,
                                            int length,
                                            byte[] optionalData,
                                            int optionalDataOffset,
                                            int optionalDataLength)
                                     throws StandardException
Writes out a partial log record - takes the appendLogRecord. Need to shutdown the database before another log record gets written, or the database is not recoverable.

Throws:
StandardException

testLogFull

protected void testLogFull()
                    throws java.io.IOException
Simulate a log full condition if TEST_LOG_FULL is set to true, then the property TEST_RECORD_TO_FILL_LOG indicates the number of times this function is call before an IOException simulating a log full condition is raised. If TEST_RECORD_TO_FILL_LOG is not set, it defaults to 100 log record

Throws:
java.io.IOException

enableLogArchiveMode

public void enableLogArchiveMode()
                          throws StandardException
Specified by:
enableLogArchiveMode in interface LogFactory
Throws:
StandardException

disableLogArchiveMode

public void disableLogArchiveMode()
                           throws StandardException
Specified by:
disableLogArchiveMode in interface LogFactory
Throws:
StandardException

deleteOnlineArchivedLogFiles

public void deleteOnlineArchivedLogFiles()
Specified by:
deleteOnlineArchivedLogFiles in interface LogFactory

copyActiveLogFiles

public boolean copyActiveLogFiles(java.io.File toDir)
                           throws StandardException
Specified by:
copyActiveLogFiles in interface LogFactory
Throws:
StandardException

inRFR

public boolean inRFR()
Specified by:
inRFR in interface LogFactory

checkpointInRFR

public void checkpointInRFR(LogInstant cinstant,
                            long redoLWM,
                            DataFactory df)
                     throws StandardException
redo a checkpoint during rollforward recovery

Specified by:
checkpointInRFR in interface LogFactory
Parameters:
cinstant - The LogInstant of the checkpoint
redoLWM - Redo Low Water Mark in the check point record
Throws:
StandardException - - encounter exception during checkpoint

restoreLogs

private void restoreLogs(java.util.Properties properties)
                  throws StandardException
This function restores logs based on the following attributes are specified on connection URL: Attribute.CREATE_FROM (Create database from backup if it does not exist) Attribute.RESTORE_FROM (Delete the whole database if it exists and then restore it from backup) Attribute.ROLL_FORWARD_RECOVERY_FROM:(Perform Rollforward Recovery; except for the log directory everthing else is replced by the copy from backup. log files in the backup are copied to the existing online log directory. In cases of RESTORE_FROM whole databases directoy is is removed in Directory.java while restoring service.properties so even the log directory is removed. In case of CREATE_FROM , log directoy will not exist if we came so far bacause it should fail if a database already exists. In case ROLL_FORWARD_RECOVERY_FROM log directotry should not be removed. So only thing that needs to be done here is create a a log directory if it does not exists and copy the log files(including control files) that exists in the backup from which we are are trying to restore the database to the onlie log directory.

Throws:
StandardException

preAllocateNewLogFile

private void preAllocateNewLogFile(StorageRandomAccessFile log)
                            throws java.io.IOException,
                                   StandardException
Throws:
java.io.IOException
StandardException

openLogFileInWriteMode

public StorageRandomAccessFile openLogFileInWriteMode(StorageFile logFile)
                                               throws java.io.IOException
Throws:
java.io.IOException

getLogDirPath

private java.lang.String getLogDirPath(StorageFile logDir)

privExists

protected boolean privExists(StorageFile file)

privDelete

protected boolean privDelete(StorageFile file)

privRandomAccessFile

protected StorageRandomAccessFile privRandomAccessFile(StorageFile file,
                                                       java.lang.String perms)
                                                throws java.io.IOException
Throws:
java.io.IOException

privCanWrite

protected boolean privCanWrite(StorageFile file)

privMkdirs

protected boolean privMkdirs(StorageFile file)

privList

protected java.lang.String[] privList(java.io.File file)

privList

protected java.lang.String[] privList(StorageFile file)

privCopyFile

protected boolean privCopyFile(StorageFile from,
                               java.io.File to)

privCopyFile

protected boolean privCopyFile(java.io.File from,
                               StorageFile to)

privRemoveDirectory

protected boolean privRemoveDirectory(StorageFile file)

runBooleanAction

private boolean runBooleanAction(int action,
                                 StorageFile file)

run

public java.lang.Object run()
                     throws java.io.IOException
Specified by:
run in interface java.security.PrivilegedExceptionAction
Throws:
java.io.IOException


Apache Derby V10.0 Engine Documentation - Copyright © 1997,2004 The Apache Software Foundation or its licensors, as applicable.