KSecureMessage client storage and local key lifecycle

This document describes how a KSecureMessage client keeps its local protocol state: the identity key, prekeys and sessions, and since milestone 8 its message reliability state. It also defines what a ClientStorage implementation must guarantee.

Contracts: storage/core/.../storage/ClientStorage.kt. Lifecycle: client/core/.../client/SecureMessageClient.kt and PreKeyManager.kt.

Sensitive data at rest

Client storage holds secret key material (the identity private key, signed and one-time prekey private keys), session state (SecureSession.state, which contains the ratchet chain keys) and, since milestone 8, application message content: the plaintext of every sent message stays in PendingOutboundStore until the recipient acknowledges it (message-reliability.md), and since milestone 20 the plaintext of every received message stays in PendingInboundStore until the application commits it (application-delivery.md) or, since milestone 21, discards it (message-discard.md); a discard deletes the plaintext. ProcessedInboundStore holds the terminal tombstones: sender addresses, logical message IDs, finalization times and, since milestone 21, the outcome (committed or discarded) and discard reason (metadata), and, since milestone 20, a digest of each finalized body (a content fingerprint, so it is sealed too).

Since milestone 9 SqlDelightClientStorage encrypts these records with AES-256-GCM under a storage key from an application-supplied StorageKeyProvider, bound to record type and row key (storage-encryption.md). This is record-level encryption, not database encryption: IDs, addresses, timestamps, pins, retired initiations and processed message IDs stay plaintext metadata, and row deletion or rollback of the whole database is not detected. The protection is only as good as the key storage. InMemoryClientStorage does not encrypt (it persists nothing).

Key bytes, session state and message plaintext never appear in logs, exception messages or toString().

Client lifecycle

val client = SecureMessageClient(
    localAddress = address,
    storage = storage,              // InMemoryClientStorage, SqlDelightClientStorage, ...
    protocol = KodiumProtocolEngine(),
    transport = transport,
)
client.initialize()                 // on every start
val bundle = client.currentPreKeyBundle()
val oneTimePreKeys = client.publicOneTimePreKeys()

initialize() runs one storage transaction:

1. Load the local identity.
2. If it exists, keep it.
3. If it is missing and no prekey state exists, create it (ProtocolEngine.createIdentity) and store it.
4. If it is missing but prekeys exist, fail with SecureMessageClientException.InconsistentStorage. A new identity would not match the stored prekeys, so the client never creates one silently.
5. Signed prekey maintenance (milestone 7, see signed-prekey-lifecycle.md): stamp keys stored before milestone 7 with the current time, create a current signed prekey if there is none or rotate it once it reached PreKeyConfiguration.signedPreKeyRotationAge (default 7 days), delete replaced keys whose signedPreKeyGracePeriod (default 30 days) is over, and prune retired session initiations that only those keys could accept. New signed prekeys are signed by the identity.
6. If fewer than PreKeyConfiguration.oneTimePreKeyTarget one-time prekeys (default 100) are stored, create only the missing number.

initialize() is idempotent. A second call, or a call after an app restart on the same storage, changes nothing unless one-time prekeys were consumed or time moved past a rotation or expiry deadline. Sessions are never touched. Time comes from the client's clock parameter (Clock.System by default).

Initialization is explicit. Other operations do not create state. Without a stored identity they throw SecureMessageClientException.NotInitialized. The client caches nothing, and every operation reads storage.

Key material is always created by the ProtocolEngine. Storage only persists it.

Prekeys

Signed prekeys

The current signed prekey is the one to publish. rotateSignedPreKey(), or initialize() once the current key reached the rotation age, creates a new current signed prekey. The replaced one stays stored and available by ID for the grace period. A PreKeyMessage created from an older bundle names the old ID, and its private key is needed to accept it. After the grace period, new initiations naming it fail with ExpiredSignedPreKey, and initialize() deletes it. Established sessions are not affected. Each key stores createdAt and, once replaced, replacedAt. See signed-prekey-lifecycle.md.

One-time prekeys

A one-time prekey is deleted when a session is accepted with it. That happens in the same transaction that stores the new session. The next initialize() replaces it with a new key under a new ID.

ID allocation

SignedPreKeyId and OneTimePreKeyId are in 0..Int.MAX_VALUE. Each store keeps a persisted high-water mark, the highest ID ever stored (highestSignedPreKeyId, highestOneTimePreKeyId). Deleting keys does not lower it. New IDs are allocated upward from highest + 1, starting at 0.

• IDs of consumed one-time prekeys and deleted signed prekeys are never reused.
• Stores reject a key whose ID is not above the high-water mark.
• There is no wraparound. When the next IDs would pass Int.MAX_VALUE, initialize() or rotateSignedPreKey() fails with SecureMessageClientException.PreKeyIdsExhausted and writes nothing.

Publication boundary

publishPreKeys() uploads this public material (see prekey-publication.md). It reads it in one transaction and does the network call afterwards:

• currentPreKeyBundle(): the identity public key and the current signed prekey (ID, public key, signature). oneTimePreKey is always null.
• publicOneTimePreKeys(): the public halves of all unused local one-time prekeys.

The client does not select, reserve or mark one-time prekeys as handed out. Which one a fetcher receives is a server concern. Locally a key is only removed when a session is accepted with it.

ClientStorage contract

interface ClientStorage {
    val identity: IdentityStore
    val remoteIdentities: RemoteIdentityStore
    val sessions: SessionStore
    val sessionInitiations: SessionInitiationStore
    val preKeys: PreKeyStore
    val pendingOutbound: PendingOutboundStore
    val pendingInbound: PendingInboundStore
    val processedInbound: ProcessedInboundStore
    suspend fun <T> transaction(block: suspend ClientStorage.() -> T): T
}

Every implementation must provide:

• Atomic transactions. If transaction's block throws, none of its writes become visible. Otherwise all of them do. This is what keeps the ratchet state consistent with sent and received messages. It also makes first contact safe: storing the session and removing the one-time prekey commit together or not at all. A failure never leaves a stored session with its one-time prekey still available, or a removed one-time prekey with no session. The first remote identity pin commits with the same transaction (see identity-trust.md). A replaced session, its retired initiation and the consumed one-time prekey commit together too (see session-lifecycle.md). An accepted identity change replaces the pin, removes the session and retires its initiation in one transaction (see identity-verification.md).
• Nesting. A transaction call inside a running transaction of the same storage joins it, through the receiver or the storage object.
• Single-call writes. Store calls outside transaction behave as single-call transactions.
• No identity replacement. IdentityStore.store fails if an identity exists.
• No silent pin replacement. RemoteIdentityStore.store is a no-op for the key already pinned for an address and fails with IllegalStateException for a different key. Pins are per DeviceAddress and are never removed. New pins are UNVERIFIED. setVerification and replace are compare-and-set on the pinned key; replace (only for an explicitly accepted identity change) makes the new pin UNVERIFIED. A verification never moves to another key.
• Retired initiations are kept until pruned. SessionInitiationStore.retire is idempotent (the first entry wins), and entries are per DeviceAddress. Each entry may name a local signed prekey. Entries are removed only by removeRetiredFor(signedPreKeyId), which the client calls after that key was deleted.
• Signed prekey lifecycle. storeCurrentSignedPreKey(key, createdAt) sets the previous current key's replacedAt to createdAt in the same call. signedPreKeyInfos() returns metadata without key material. removeSignedPreKey refuses the current key and keeps the high-water mark. stampLegacySignedPreKeys only fills missing timestamps.
• Pending outbound messages (milestone 8). store(recipient, id, frame) fails with IllegalArgumentException if (recipient, id) is pending and returns a sequence number that is higher than every earlier one, also after removals and restarts. page(afterSequence, limit, recipient?) returns at most limit entries with a greater sequence in ascending order, to one recipient if given, and opens only those records (milestone 22; it replaced the unbounded list(recipient)); count(recipient?) opens none. remove deletes exactly (recipient, id) without opening it and returns whether it removed something. The client stores a message in the transaction that first encrypts it and removes it in the transaction that decrypts its acknowledgement or in the abandon transaction (outbound-message-lifecycle.md).
• Pending inbound messages (milestone 20). store assigns a sequence number that increases with every call and is never reused, also after removals and restarts; a second store of the same (sender, ID) throws IllegalArgumentException. list() and list(sender) are ordered by sequence; contains never opens a record. The client stores a message in the transaction that decrypts it and removes it only in the commit transaction. Nothing expires them.
• Processed inbound messages (milestones 8 and 20). Scoped by sender. markProcessed(sender, id, digest, committedAt) throws if the entry exists (never replaced). isProcessed never opens a record; get returns the digest and commit time (null for entries from before milestone 20). stampLegacyCommitTimes(at) sets only missing commit times. pruneCommittedAtOrBefore(cutoff) removes entries with a commit time at or before cutoff, never entries without one and never pending messages. The client writes the entry in the commit transaction, together with removing the pending inbound message.
• Monotonic prekey IDs, as described above.
• No aliasing. Mutating an array after storing it, or an array returned by a load, must not change stored state.

Rules for transaction blocks:

• No network I/O or other long suspension. A transaction holds a lock, and SecureMessageClient fetches prekey bundles before its transaction starts.
• Do not switch threads or dispatchers. The SQLite drivers bind a transaction to the thread that started it.

storage:testing contains ClientStorageContractTest. Every adapter's tests extend it.

Adapters

Client storage adapters live under storage:client:*, server repository adapters under storage:server:*; the contracts they implement stay in storage:core. storage:encryption, storage:keyprovider:* and storage:rotation:core are storage security infrastructure; today they serve only the persistent client adapter.

storage:client:inmemory

InMemoryClientStorage is not persistent. It is meant for tests and examples. Its committed state is one immutable snapshot. A transaction works on a copy and replaces the snapshot only when the block returns normally. A failed block drops the copy. A Mutex serializes transactions. Byte arrays are copied on store and on load.

storage:client:sqldelight

SqlDelightClientStorage persists to SQLite through SQLDelight 2.4.0 with async code generation. It supports all client targets: JVM, Android, iOS, macOS, linuxX64, mingwX64, JS and Wasm. The application creates the platform driver and closes it:

Platform	Driver artifact	Example
JVM	app.cash.sqldelight:sqlite-driver	JdbcSqliteDriver("jdbc:sqlite:client.db", Properties(), SqlDelightClientStorage.Schema.synchronous())
Android	app.cash.sqldelight:android-driver	AndroidSqliteDriver(SqlDelightClientStorage.Schema.synchronous(), context, "client.db")
Apple, Linux, Windows	app.cash.sqldelight:native-driver	NativeSqliteDriver(SqlDelightClientStorage.Schema.synchronous(), "client.db")
JS, Wasm	app.cash.sqldelight:web-worker-driver	web worker driver, async schema

Use one SqlDelightClientStorage instance per database.

Each transaction is a SQLite transaction (SuspendingTransacter.transactionWithResult). If the block throws, SQLite rolls back. A Mutex serializes transactions of the instance.

Tables:

• storage_encryption: one row with the storage format and the bound storage key ID
• local_identity: one row, the sealed key pair
• device_authentication_key: one row, the sealed device authentication key pair (schema version 8, milestone 12, server-authentication.md)
• device_authentication_recovery_key: at most one row, the sealed pending replacement key of a device recovery (record type 8, schema version 9, milestone 14, device-recovery.md)
• device_authentication_rotation_key: at most one row, the sealed pending replacement key of a routine rotation (record type 9, schema version 11, milestone 16, device-authentication-rotation.md); never filled together with the recovery key
• device_authentication_last_device_recovery_key: at most one row, the sealed pending replacement key of a last-device recovery (record type 10, schema version 12, milestone 18, last-device-recovery.md); never filled together with the recovery or the rotation key. The offline recovery key is never stored
• device_authentication_state: one row, awaits_upgrade_key (plaintext flag: set by the version 8 migration when an identity existed without a key)
• signed_pre_key: sealed key pair, with created_at and replaced_at (epoch milliseconds, nullable)
• one_time_pre_key: sealed key pair
• pre_key_state: one row holding the current signed prekey ID and both high-water marks
• session: sealed state, keyed by remote user and device ID
• remote_identity: pinned remote identity public keys, keyed by remote user and device ID, with verification (0 unverified, 1 verified; schema version 10)
• retired_session_initiation: retired 32-byte session initiation IDs, keyed by remote user, device ID and initiation ID, with the nullable local signed_pre_key_id used for pruning
• pending_outbound_message: sequence INTEGER PRIMARY KEY AUTOINCREMENT (never reused), recipient user and device ID, 16-byte message_id, and the sealed frame (encrypted; the frame holds the application plaintext); unique per recipient and message ID; since schema version 15 indexed by (recipient, sequence) for recipient-filtered pages
• processed_inbound_message: the terminal tombstone per sender user and device ID and 16-byte message_id; since schema version 13 committed_at (the finalization time, commit or discard; epoch milliseconds, nullable for legacy rows, indexed) and sealed_digest (record type 12 for both outcomes, NULL for legacy rows); since schema version 14 finalization (0 committed, 1 discarded; CHECK, default 0) and discard_reason (1..5, set exactly for discarded rows, CHECK; plaintext metadata)
• pending_inbound_message (schema version 13, milestone 20): sequence INTEGER PRIMARY KEY AUTOINCREMENT (never reused), sender user and device ID, 16-byte message_id, received_at (epoch milliseconds) and the sealed frame (record type 11; holds the application plaintext); unique per sender and message ID; since schema version 14 indexed by (sender, sequence) for sender-filtered pages

Since schema version 6 the key pairs, session state and pending frames are in sealed_* columns holding encrypted storage records, and storage_encryption (one row) records the storage format and the bound storage key ID (storage-encryption.md). The schema does not depend on Kodium internals. All IDs have a CHECK (id BETWEEN 0 AND 2147483647) constraint.

Open the storage with SqlDelightClientStorage.open(driver, keyProvider). There is no unencrypted mode.

The schema version is 15. Version 1 (milestones 3 and 4) had no remote_identity table; 1.sqm adds it and changes nothing else. Version 2 (milestone 5) had no retired_session_initiation table; 2.sqm adds it and changes nothing else. Version 3 (milestone 6) had no lifecycle columns; 3.sqm adds signed_pre_key.created_at, signed_pre_key.replaced_at and retired_session_initiation.signed_pre_key_id, all nullable, and changes nothing else. Upgraded signed prekeys get their timestamps on the next initialize(): the current key counts as created then, older keys start a full grace period then, and nothing is deleted on that first start (signed-prekey-lifecycle.md). Version 4 (milestone 7) had no reliability tables; 4.sqm creates pending_outbound_message and processed_inbound_message, empty, and changes nothing else. Messages from before the upgrade are neither pending nor processed. Version 5 (milestone 8) stored key pairs, session state and pending frames in plaintext; 5.sqm only adds storage_encryption with format 0, and the next SqlDelightClientStorage.open encrypts every sensitive record in one transaction, rebuilding the five sensitive tables (storage-encryption.md). Plaintext that the old version wrote may survive in free pages, the journal and backups. Version 6 (milestones 9 and 10) had no storage key rotation state; 6.sqm adds it to storage_encryption (storage-key-rotation.md). Version 7 (milestone 11) had no device authentication key; 7.sqm creates device_authentication_key (empty) and device_authentication_state, whose flag is 1 if the database held an identity. The migration creates no key material; the next initialize() creates the key once, and a key that goes missing later fails closed (server-authentication.md). Version 8 (milestones 12 and 13) had no pending recovery key; 8.sqm creates device_authentication_recovery_key, empty (device-recovery.md). Version 9 (milestone 14) had no verification state; 9.sqm adds remote_identity.verification with default 0, so every existing pin becomes unverified and nothing else changes (identity-verification.md). Version 10 (milestone 15) had no pending rotation key; 10.sqm creates device_authentication_rotation_key, empty, and changes no existing row (device-authentication-rotation.md). Version 11 (milestones 16 and 17) had no pending last-device recovery key; 11.sqm creates device_authentication_last_device_recovery_key, empty, and changes no existing row (last-device-recovery.md). Version 12 (milestones 18 and 19) had no application commit boundary; 12.sqm creates pending_inbound_message, empty, and adds the nullable processed_inbound_message.committed_at and sealed_digest plus an index on committed_at. Existing processed IDs are kept; the next initialize() stamps their commit time once with the client clock, their digest stays unknown (application-delivery.md). Version 13 (milestone 20) had no discard; 13.sqm adds processed_inbound_message.finalization (default 0, so every existing row is committed) and discard_reason (NULL) and the pending sender index. No existing value changes, digests and times included (message-discard.md). Version 14 (milestone 21) had no recipient index on pending outbound messages; 14.sqm adds pending_outbound_message_recipient and changes no row (outbound-message-lifecycle.md). Frozen copies of versions 1–14 (Version1Schema … Version14Schema) back the migration tests.

The offline recovery key lifecycle (milestone 19) and the delayed recovery key reset (milestone 23, recovery-key-reset.md) change no client schema: the client never stores the offline recovery key, a replacement key or a pending reset; that state lives on the server only. A driver created with SqlDelightClientStorage.Schema, as in the table above, reads SQLite's user_version on open and runs the migrations itself. An application that manages schema versions on its own calls SqlDelightClientStorage.Schema.migrate(driver, oldVersion, SqlDelightClientStorage.Schema.version). Existing identities, prekeys, sessions, pins and retired initiations are kept. Sessions from version 1 have no pin, see identity-trust.md. Session BLOBs written before milestone 6 use the older local state format and stay readable (see session-lifecycle.md).

The tests run on JVM (file database) and Apple/native targets. They cover the contract, closing and reopening the database, on-disk rollback, a client restart that continues an existing session, remote trust across restarts, replay protection and simultaneous-initiation resolution across restarts, the signed prekey lifecycle (grace, expiry, pruning, clock set back) across restarts, pending messages, duplicate suppression and collision recovery across restarts, and migrating version 1, 2, 3 and 4 databases. Since milestone 9 they also check the persisted bytes: sensitive columns hold only encrypted records, corrupted and swapped records fail closed, wrong and missing keys fail at open, encryption failures roll back, and a milestone 8 database migrates completely or not at all. JS/Wasm have no SQLDelight tests: the web worker driver needs a browser worker. The linuxX64 and mingwX64 test binaries link against the target's libsqlite3, so they are only linked and run on a Linux or Windows host.

Not covered yet

• Messaging identity recovery, reset or backup (a changed remote identity can be accepted, see identity-verification.md).
• Tracking locally what the server handed out (the server tombstones consumed one-time prekey IDs instead, see prekey-publication.md).
• Pruning of retired session initiations that name no local signed prekey (initiations this device started, entries from before milestone 7).
• A secure or monotonic clock for the signed prekey lifecycle.
• Automatic pruning of processed message IDs (only explicit pruneProcessedMessages, see application-delivery.md), and any expiry of uncommitted pending inbound or unacknowledged pending outbound messages (only explicit discard and abandon).
• Platform storage key providers, storage key rotation, rollback protection and integrity of plaintext metadata (storage-encryption.md).