public class KCVSLog extends Object implements Log, BackendOperation.TransactionalProvider
Log wrapped around a KeyColumnValueStore. Each message is written as a column-value pair (Entry)
into a timeslice slot. A timeslice slot is uniquely identified by:
GraphDatabaseConfiguration.LOG_NUM_BUCKETS. Messages are written to the buckets
in round-robin fashion and each bucket is identified by a bucket id.
Having multiple buckets per timeslice allows for load balancing across multiple keys in the storage backend.TIMESLICE_INTERVAL microseconds long. And all messages that are added between
start-time and start-time+TIMESLICE_INTERVAL end up in the same timeslice. For high throughput logs that might be more messages
than the underlying storage backend can handle per key. In that case, ensure that (2^(partition-bit-width) x (num-bucekts) is large enough
to distribute the load.KCVSLogManager), and the
message id (which is auto-incrementing). These three data points comprise the column of a log message. The actual content of the message
is written into the value.
When MessageReader are registered, one reader thread per partition id and bucket is created which periodically (as configured) checks for
new messages in the storage backend and invokes the reader.
Read-markers are maintained (for each partition-id & bucket id combination) under a dedicated key in the same KeyColumnValueStoreManager as the
log messages. The read markers are updated to the current position before each new iteration of reading messages from the log. If the system fails
while reading a batch of messages, a subsequently restarted log reader may therefore read messages twice. Hence, MessageReader implementations
should exhibit correct behavior for the (rare) circumstance that messages are read twice.
Note: All time values in this class are in mircoseconds. Hence, there are many cases where milliseconds are converted to microseconds.| Modifier and Type | Field and Description |
|---|---|
static ConfigOption<Boolean> |
LOG_KEY_CONSISTENT |
static ConfigOption<java.time.Duration> |
LOG_MAX_READ_TIME |
static ConfigOption<java.time.Duration> |
LOG_MAX_WRITE_TIME |
static ConfigOption<java.time.Duration> |
LOG_READ_LAG_TIME |
static long |
TIMESLICE_INTERVAL
The time period that is stored under one key in the underlying KCVS.
|
| Constructor and Description |
|---|
KCVSLog(String name,
KCVSLogManager manager,
KeyColumnValueStore store,
Configuration config) |
| Modifier and Type | Method and Description |
|---|---|
Future<Message> |
add(StaticBuffer content)
###################################
Message Sending
###################################
|
Future<Message> |
add(StaticBuffer content,
StaticBuffer key)
Attempts to add the given content to the log and returns a
Future for this action. |
Future<Message> |
add(StaticBuffer content,
StaticBuffer key,
ExternalPersistor persistor) |
void |
close()
Closes the log by terminating all threads and waiting for their termination.
|
String |
getName()
Returns the name of this log
|
StoreTransaction |
openTx() |
void |
registerReader(ReadMarker readMarker,
MessageReader... reader)
###################################
Message Reading
###################################
|
void |
registerReaders(ReadMarker readMarker,
Iterable<MessageReader> readers)
Registers the given readers with this log.
|
boolean |
unregisterReader(MessageReader reader)
Removes the given reader from the list of registered readers and returns whether this reader was registered in the
first place.
|
public static final ConfigOption<java.time.Duration> LOG_MAX_WRITE_TIME
public static final ConfigOption<java.time.Duration> LOG_MAX_READ_TIME
public static final ConfigOption<java.time.Duration> LOG_READ_LAG_TIME
public static final ConfigOption<Boolean> LOG_KEY_CONSISTENT
public static final long TIMESLICE_INTERVAL
public KCVSLog(String name, KCVSLogManager manager, KeyColumnValueStore store, Configuration config)
public void close()
throws BackendException
close in interface Logclose in interface BackendOperation.TransactionalProviderBackendExceptionpublic StoreTransaction openTx() throws BackendException
openTx in interface BackendOperation.TransactionalProviderBackendExceptionpublic Future<Message> add(StaticBuffer content)
public Future<Message> add(StaticBuffer content, StaticBuffer key)
LogFuture for this action.
In addition, a key is provided to signal the recipient of the log message in partitioned logging systems.
If the log is configured for immediate sending, then any exception encountered during this process is thrown
by this method. Otherwise, encountered exceptions are attached to the returned future.public Future<Message> add(StaticBuffer content, StaticBuffer key, ExternalPersistor persistor)
public void registerReader(ReadMarker readMarker, MessageReader... reader)
registerReader in interface LogreadMarker - Indicates where to start reading from the log once message readers are registeredreader - The readers to register (all at once)Log.registerReaders(ReadMarker, Iterable)public void registerReaders(ReadMarker readMarker, Iterable<MessageReader> readers)
LogReadMarker.
If no previous readers were registered, invoking this method triggers reader threads to be instantiated.
If readers have been previously registered, then the provided ReadMarker must be compatible with the
previous ReadMarker or an exception will be thrown.registerReaders in interface LogreadMarker - Indicates where to start reading from the log once message readers are registeredreaders - The readers to register (all at once)public boolean unregisterReader(MessageReader reader)
LogLog.close() instead.unregisterReader in interface LogCopyright © 2012–2015. All rights reserved.