Margaret

Margaret is go-ssb's append-only log* provider, and greatly inspired by flumedb. Compatible with Go 1.13+.

the project name is inspired by Twin Peaks's character Margaret aka the log lady

Margaret has the following facilities:

• an append-only log interface .Append(interface{}), .Get(int64)
• queries .Query(...QuerySpec) for retrieving ranges based on sequence numbers e.g. .Gt(int64), or limiting the amount of data returned .Limit(int64)
• a variety of index mechanisms, both for categorizing log entries into buckets and for creating virtual logs (aka sublogs)

Margaret is one of a few key components that make the go implementation of ssb tick, for example:

• ssb/sbot uses margaret for storing each peer's data

Log storage

Margaret outputs data according to the offset2 format, which is inspired by (but significantly differs from) flumelog-offset.

In brief: margaret stores the data of all logs in the three following files:

• data stores the actual data (with a length-prefix before each entry)
• ofst indexes the starting locations for each data entry in data
• jrnl an integrity checking mechanism for all three files; a checksum of sorts, more details

More details

There are a few concepts that might be tough to digest for newcomers on first approach:

• multilogs, a kind of tree-based index, where each leaf is a margaret.Log
  • in other words: it creates virtual sublogs that map to entries in an offset log (see log storage above)
• margaret/indexes similar to leveldb indexes (arbitrary key-value stores)
• sublogs (and rxLog/receiveLog/offsetLog and its equivalence to offset.log)
• queries
• zeroing out, or replacing, written data

For more on these concepts, visit the dev.scutttlebutt.nz portal for in-depth explanations.

* margaret is technically an append-based log, as there is support for both zeroing out and replacing items in the log after they have been written. Given the relative ubiquity of append-only logs & their uses, it's easier to just say append-only log.