Skip to main content

Implementation Notes

The following notes may be useful for users of the MCAP format, including implementers of readers and writers.

MCAP is intended to support efficient, indexed reading of messages and generation of summary data in both local and remote contexts. "Seeking" should be imagined to incur either a disk seek or an HTTP range request to an object store—the latter being significantly more costly. In both random access and summarization, features may be unsupported due to choices taken by the writer of the file. For instance, statistics may not include channel message counts, or there may be no message index present. If the index data section is empty, the index_offset in the file footer will be set to zero.

Scanning for records on specific topics within an interval​

The index is designed to support fast local and remote seek/filter operations with minimal seeking or range request overhead. The operation of the index for message reading is as follows:

  1. Client queries for all messages on topics /a, /b, /c between t0 and t1
  2. Reader reads the fixed-length footer off the end of the file
  3. Reader parses the index_offset from the footer, and starts reading from that offset to the end of the file. During this read it will encounter the following in order:
    • A run of Channel records, one per channel in the file
    • A run of Message Group Index records, one per chunk in the file
    • The attachment index records
    • The statistics record

The reader in this case will stop after the chunk index records.

  1. Using the channel records at the start of the read, the reader converts topic names to channel IDs.
  2. Using the chunk index records, the reader locates the chunks that must be read, based on the requested start times, channel IDs, and end times. These chunks will be a contiguous run.
  3. Readers may access the message data in at least two ways,
    • “full scan”: Seek from the chunk index to the start of the chunk using chunk_offset. Read/decompress the entire chunk, discarding messages not on the requested channels. Skip through the index data and into the next chunk if it is targeted too.
    • “index scan”: Consult the message_index_offsets field in the chunk index record, and use it to locate specific message indexes after the chunk for the channels of interest. These message indexes can be used to obtain a list of offsets, which the reader can seek to and extract messages from.

Which of these options is preferable will tend to depend on the proportion of topics in use, as well as potentially whether the storage system is local or remote.

Listing and accessing attachments​

The format provides the ability to list attachments contained within the file, and quickly extract them from the file contents. To list/select attachments in the file:

  1. Read the fixed-length footer and seek to the start of the index data section.
  2. Scan forward until encountering the attachment index, then read attachment index records until encountering a record that is not an attachment index.
  3. The records covered in the previous read will include attachment names, types, sizes, and timestamps. These can be used to fill out a list of attachments for selection.
  4. To select an attachment from the file, seek to the associated offset in the file and unpack the file content from the attachment record.

Accessing summary statistics​

The format provides for fast local or remote access to summary information in the same style as "rosbag info", with the intent of functional parity with rosbag info. For reference, here is an example of the rosbag info output:

path:         demo.bag
version:      2.0
duration:     7.8s
start:        Mar 21 2017 19:26:20.10 (1490149580.10)
end:          Mar 21 2017 19:26:27.88 (1490149587.88)
size:         67.1 MB
messages:     1606
compression:        lz4 [79/79 chunks; 56.23%]
uncompressed:       119.1 MB @ 15.3 MB/s
compressed:    67.0 MB @  8.6 MB/s (56.23%)
types:        diagnostic_msgs/DiagnosticArray [60810da900de1dd6ddd437c3503511da]
              radar_driver/RadarTracks        [6a2de2f790cb8bb0e149d45d297462f8]
              sensor_msgs/CompressedImage     [8f7a12909da2c9d3332d540a0977563f]
              sensor_msgs/PointCloud2         [1158d486dd51d683ce2f1be655c3c181]
              sensor_msgs/Range               [c005c34273dc426c67a020a87bc24148]
              tf2_msgs/TFMessage              [94810edda583a504dfda3829e70d7eec]
topics:        /diagnostics               52 msgs    : diagnostic_msgs/DiagnosticArray
              /image_color/compressed       234 msgs    : sensor_msgs/CompressedImage
              /radar/points             156 msgs    : sensor_msgs/PointCloud2
              /radar/range              156 msgs    : sensor_msgs/Range
              /radar/tracks             156 msgs    : radar_driver/RadarTracks
              /tf                       774 msgs    : tf2_msgs/TFMessage
              /velodyne_points           78 msgs    : sensor_msgs/PointCloud2

The reader will recover this data from the index as follows:

  1. Read the fixed length footer and seek to the index_offset.
  2. Read the run of channel records that follow to get topic names, types, and MD5 data (which in case of ROS1 will be in the user data section), as well as channel IDs to interpret the chunk index records.
  3. After the channel are the chunk index records, if the file is chunked. From each chunk index record extract the compression algorithm and compressed/uncompressed size. From these the reader can compute the compression statistics shown in the rosbag info summary. For unchunked files this field is omitted.
  4. The MCAP version of “rosbag info” will display information about included attachments as well. After reading the chunk index records, the attachment index records will be scanned and incorporated into the summary.
  5. Finally, the statistics record is used to compute the start, end, total, and per-channel message counts. The per-channel message counts must be grouped/summed over topics for display.

The only difference between the chunked and unchunked versions of this output will be the chunk compression statistics (“compressed”, “uncompressed”, “compression”), which will be omitted in the case of unchunked files. The summary should be very fast to generate in either local or remote contexts, requiring no seeking around the file to visit chunks.

The above is not meant to prescribe a summary formatting, but to demonstrate that parity with the rosbag summary is supported by MCAP. There are other details we may consider including, like references to per-channel encryption or compression if these features get uptake. We could also enable more interaction with the channel records, such as quickly obtaining schemas from the file for particular topics.

Auxiliary timestamps​

Every Message record carries exactly two timestamps: log_time and publish_time. Auxiliary timestamps allow a message to carry an arbitrary number of additional named timestamps without modifying the (frozen) Message record. The feature is designed to be fully backward compatible: it is built entirely from records with new opcodes, which existing readers skip, so existing readers and writers are unaffected and only readers that wish to access the additional timestamps need updating.

The feature is composed of four records:

  • Timestamp Name registers a uint16 ID for each named timestamp, file-globally. It is written like a Schema or Channel record (before first use, and optionally duplicated in the summary).
  • Message Auxiliary Timestamps carries the (timestamp_id, value) pairs for one message and is written immediately after that message.
  • Auxiliary Message Index and Auxiliary Chunk Index mirror the standard Message Index and Chunk Index, but key on an auxiliary timestamp instead of log_time, so a reader can seek and prune by it.

Reading auxiliary timestamps during a linear scan​

While iterating records (in the data section or within a decompressed chunk), a reader pairs each Message with the Message Auxiliary Timestamps record that immediately follows it:

  1. Read a Message record.
  2. Peek at the next record. If it is a Message Auxiliary Timestamps record whose channel_id matches, attach its timestamps to the message just read; otherwise the message has no auxiliary timestamps.
  3. Resolve each timestamp_id to a name using the Timestamp Name records seen so far.

Readers that do not understand opcode 0x11 skip it and observe only log_time and publish_time.

Seeking by an auxiliary timestamp​

To seek to messages by an auxiliary timestamp T (rather than log_time):

  1. Read the summary section and collect the Timestamp Name records to map the desired name to its ID, plus the Auxiliary Chunk Index records for that ID.
  2. Use the min_time/max_time of each Auxiliary Chunk Index record to select candidate chunks. Note that, unlike log_time, an auxiliary timestamp is not guaranteed to be monotonic, so candidate chunk ranges may overlap and a query interval may match more chunks than the equivalent log_time query would.
  3. For each candidate chunk, either full-scan the chunk, or consult its Auxiliary Message Index records (located via message_index_offsets) to obtain message offsets directly, seek to each Message, and read the trailing Message Auxiliary Timestamps record.

Writing considerations​

  • Because association is positional, a Message Auxiliary Timestamps record MUST be written immediately after its Message, with no records in between, in the same record stream. Tools that copy records through verbatim preserve this pairing; tools that re-emit messages via a message-level API must be updated to carry the auxiliary records along.
  • Declaring timestamp IDs per file (and listing them where convenient, e.g. in channel metadata) keeps merge operations cheap: messages from channels without a given auxiliary timestamp simply omit it rather than padding with zero values.