--- title: CSF Message Delivery Policy tags: [Standalone, Message Delivery, Retry, Operational] --- # CSF Message Delivery Policy > Connected Services Framework (CSF) — Standalone Companion Document — Version 1.0 > > Maintained by the Telecom Technical Architecture Group (TAG). This document consolidates the CSF's message-delivery rules into a single self-contained reference. The canonical normative source is [Part 2 §4.7 Message Delivery Policy](../part2-implementation/04-message-api.md#47-message-delivery-policy); this standalone is the readable summary that does not require the reader to navigate the full Master Document. --- ## Document Control | | | |---|---| | **Audience** | MAP implementers; CPs preparing to become a MAP-of-1; SI partners building or integrating CSF endpoints; the GPLB Steering Group; TOTSCo (for the HMAP-boundary context). | | **Status** | Published — ratified by the TAG and consolidated from the framework on 2026-06-24. | | **Companion to** | The full normative text in [Part 2 §4.7](../part2-implementation/04-message-api.md#47-message-delivery-policy); the test suite in [Onboarding & Testing §8.7](./onboarding-testing-process.md#87-retry--delivery-policy); the architectural analysis in `wiki/syntheses/csf-compliance-with-gplb-message-delivery-principles.md`. | | **Purpose** | Set out, in one place, how a CSF MAP delivers JAM messages to another CSF MAP, what happens when delivery fails, how persistent failures escalate, and how the rules differ at the HMAP boundary with TOTSCo. | --- ## 1. Purpose and scope This document is the **CSF Message Delivery Policy** — the contract every MAP follows when sending a JAM-envelope message to a peer MAP's letterbox. It covers: - The lifecycle of a single delivery attempt (connect → send → response). - The timeouts that govern that attempt. - What happens when a delivery attempt fails (retry / circuit-breaker / escalation). - The operational-engagement path when failure persists beyond the retry window. - The asymmetric behaviour at the **HMAP boundary** where a CSF MAP exchanges with TOTSCo. - The architectural defence for why CSF retains **exponential backoff with jitter** rather than the linear / fixed-cadence schedules used by centralised hubs. It does **not** cover: - The JAM envelope structure — see the OTA2 JAM specification. - DKIM signing of outbound messages — see [Part 2 §3 PKI & DKIM Signing](../part2-implementation/03-pki-dkim-signing.md). - OAuth 2.0 authentication — see [Part 2 §2 OAuth 2.0 Implementation](../part2-implementation/02-oauth2-implementation.md). - TLS transport security — mandated as TLS 1.3 throughout the framework (see [Part 1 §6 Security Overview](../part1-framework/06-security-overview.md)). - Directory and registry exchange between MAPs — see [Part 2 §5 Directory API](../part2-implementation/05-directory-api.md). - The error-code taxonomy itself (`9xxx` industry / `8xxx` CSF PKI) — see [Part 2 §4.6](../part2-implementation/04-message-api.md#46-asynchronous-error-handling--message-delivery-failures). --- ## 2. Architectural framing The CSF is a **peer-to-peer synchronous** transport. Two MAPs exchange JAM messages directly over TLS 1.3 / OAuth 2.0 / DKIM-signed HTTP. The sending MAP knows the result of every delivery attempt **in the same HTTP exchange**: | Outcome | What it means | |---|---| | `HTTP 202 Accepted` | The receiving MAP has accepted the message for downstream processing. Delivery is confirmed at the transport layer. | | `HTTP 4xx` + `9xxx` industry error code | The receiving MAP has rejected the message for a documented envelope / process reason. Sender does **not** retry. | | `HTTP 4xx` + `8xxx` CSF error code | The receiving MAP has rejected the message for a signature / verification reason. Retry rules vary by sub-code (see [Part 2 §3](../part2-implementation/03-pki-dkim-signing.md)). | | Network / connection / response timeout | The receiving MAP is unreachable. Sender enters the retry loop. | This is materially different from a **centralised hub** model — where a hub sits between sender and destination, absorbs the asynchrony, queues messages per-endpoint, retries on a per-protocol schedule, ages stale queue items, and emits asynchronous failure notifications back to the sender on a separate channel. **CSF MAPs do none of that at the framework layer.** The synchronous response IS the delivery indication. The asymmetry has one important boundary: the **HMAP interface** between a CSF MAP and TOTSCo. TOTSCo is a hub by its own design, so traffic crossing that boundary follows the OTA2 GPLB Message Delivery Principles on TOTSCo's side. See §7 below. ### Two-tier scope at a glance | Boundary | Delivery model | Rules in this document | |---|---|---| | **CSF MAP ↔ CSF MAP** | Synchronous P2P | §§3–6 (the bulk of this document) | | **CSF MAP ↔ TOTSCo (HMAP)** | Asymmetric — TOTSCo applies hub schedules; the CSF MAP applies CSF rules on its own side | §7 | --- ## 3. The delivery lifecycle A single CSF MAP-to-MAP delivery attempt proceeds as follows. The sender's responsibility is to attempt delivery; the receiver's responsibility is to acknowledge synchronously. ### 3.1 Connection The sender opens a TCP / TLS 1.3 connection to the destination MAP's letterbox URL (published in the destination's CP Registry — see [Part 2 §5.4](../part2-implementation/05-directory-api.md#54-map-section--field-reference)). - **Connection timeout: 1 second.** If the handshake does not complete within 1 second, the sender treats the attempt as a failed delivery and feeds it into the retry loop (§4). ### 3.2 Request Once connected, the sender POSTs the signed JAM envelope to the letterbox URL with: - An OAuth 2.0 bearer token in the `Authorization` header. - The DKIM signature in the `X-CSF-SIGNATURE` header (per [Part 2 §3](../part2-implementation/03-pki-dkim-signing.md)). - The optional `X-CSF-ROUTE` chain (for HMAP-bridged traffic — see [Part 2 §4.4](../part2-implementation/04-message-api.md)). ### 3.3 Response The sender waits for the receiving MAP's response. - **Response timeout: 3 seconds.** If no response is received within 3 seconds, the attempt is treated as a failed delivery and feeds into the retry loop. The response is one of: - `HTTP 202 Accepted` — delivery confirmed. No body needed. The sender removes the message from its outbound queue. - `HTTP 4xx` + structured JSON `{ "code": "9xxx", "description": "..." }` — synchronous industry-level rejection. The sender does NOT retry; it surfaces the failure to the originating CP via its internal mechanism. - `HTTP 4xx` + structured JSON `{ "code": "8xxx", "description": "..." }` — synchronous CSF signature-level rejection. The sender does NOT retry permanent codes (`8101`); it retries with backoff on transient codes (`8102`). - `HTTP 429 Too Many Requests` with `Retry-After` — the destination is rate-limited. The sender backs off respecting the header. - `HTTP 503 Service Unavailable` with `Retry-After` — the destination is temporarily unhealthy. The sender backs off respecting the header. - Any `HTTP 5xx` without `Retry-After` — the sender retries with exponential backoff (§4). --- ## 4. Retry policy (CSF MAP-to-MAP) When a delivery attempt fails for a retryable reason, the sender retries. The rules in this section are the framework's **MUST** statements for every CSF MAP at the P2P boundary. ### 4.1 Exponential backoff with jitter Retries grow **exponentially** in delay (each interval geometrically larger than the previous, typically doubling) and carry **jitter** (a random component of ±25 % added to each delay so independent senders do not align on identical retry timestamps). ``` attempt delay (typical) 1 1 s ±25 % 2 2 s ±25 % 3 4 s ±25 % 4 8 s ±25 % 5 16 s ±25 % 6 32 s ±25 % ← capped here in most implementations 7 32 s ±25 % … ``` The specific base, multiplier, and cap are MAP implementation choices. The **shape** of the curve — geometric growth with randomisation — is required. MAPs **MUST NOT** use linear or fixed-cadence retry schedules at the CSF P2P boundary. The architectural rationale is in §6. ### 4.2 Respect HTTP semantics MAPs **MUST** honour: - `HTTP 429 Too Many Requests` with `Retry-After` — back off for at least the period specified by the header. - `HTTP 503 Service Unavailable` with `Retry-After` — same. A `Retry-After` value overrides the sender's normal backoff schedule for that attempt. ### 4.3 Circuit-breaker After **N consecutive failures** to the same destination (recommended `N = 5`, tunable per MAP), the sender: 1. Stops active retry to that destination. 2. Fails subsequent messages destined for it FAST (within ~100 ms) without attempting delivery. 3. Raises an operational alert. 4. Holds the outbound queue for that destination — does not drop pending messages. After a recovery period the circuit transitions to **half-open**: the sender issues a single probe attempt. If the probe succeeds the circuit closes and normal flow resumes; if it fails the circuit returns cleanly to **open** for the next recovery period. ### 4.4 Bounded retry window MAPs **MUST NOT** "wait forever". The framework recommends a maximum retry window in the **order of hours, not days** — typically 1 to 4 hours for production traffic. Beyond the window the sender: 1. Stops retrying. 2. Raises a `9008` Delivery Failure for the originating message. 3. Increments the persistent-failure counter for that destination. 4. Triggers the operational-engagement path (§5). ### 4.5 Logging Every delivery attempt **MUST** be logged for audit. The log entry includes, at minimum: - Destination URL. - Timestamp of the attempt. - Attempt sequence number. - Outcome (`202`, `4xx` + code, network error). - Any `Retry-After` value observed. - The originating message's correlation IDs. Audit retention follows [Part 2 §9 Audit Log requirements](../part2-implementation/09-operations.md). --- ## 5. Operational-engagement escalation When delivery persistently fails beyond the sender's retry window, the framework's **primary recovery mechanism is human operational engagement between MAP teams** — not silent retry. The procedure: 1. **The failing MAP publishes a `map/outage[]` entry** in its CP Registry per [Part 2 §5.4](../part2-implementation/05-directory-api.md#54-map-section--field-reference). The entry carries an ITIL-style priority (P1, P2, P3), start/end timestamps, affected services, a short description, and escalation contacts. 2. **Every other MAP picks the entry up on the next directory poll** (typical poll interval: 5 minutes; max recommended: hourly) and downgrades its retry behaviour to the destination — see [Part 2 §9.3 Outage Notification](../part2-implementation/09-operations.md#93-outage-notification). 3. **The sender raises an internal alert** when its retry window is exhausted, naming the destination RCPID, duration of consecutive failure, and the last synchronous error code observed. 4. **Operations teams from both sides engage** via the published `map/contact[]` channels (phone, email, ticketing URL — see [Part 2 §5.6 CP Contact Object](../part2-implementation/05-directory-api.md#56-cp-contact-object)). Contact entries include availability hours, days, and purpose tags so the sender knows which channel is appropriate at the time of the incident. 5. **The framework's standard incident response process applies** — see [Part 1 §10.6.5 Incident Response Coordination](../part1-framework/10-governance.md#1065-incident-response-coordination). This sequence is the deliberate replacement for "wait 12 days" hub-side queue ageing. **CSF MAPs do not silently retry for days; they escalate.** A persistent destination failure beyond a few hours is treated as a P1 / P2 incident, not a routine retry condition. --- ## 6. Why exponential backoff and not linear / fixed-cadence retry The CSF retains exponential backoff with jitter as its P2P default. Linear retry schedules (e.g. retries every 5 seconds for 30 seconds, then fail) and fixed-cadence tails (e.g. every 60 seconds for 12 days) are architecturally appropriate for a single centralised hub but **inappropriate** for a distributed P2P network for two distinct reasons: ### 6.1 Linear schedules synchronise retry storms When a destination is recovering from an outage, the **identical retry intervals across many independent senders** create a "thundering herd" — a synchronised load spike every 5 s (or every 60 s) until the destination falls back over. This is precisely the failure mode exponential-with-jitter is designed to prevent. Under exponential-with-jitter pressure, the load is **spread across the recovery window**: each sender backs off on its own random schedule, the destination sees a manageable trickle of probes rather than a coordinated burst, and recovery is far more likely to complete cleanly. ### 6.2 Exponential-with-jitter is the industry-standard distributed-systems pattern Every modern cloud SDK (AWS, Google, Microsoft, Cloudflare, …) uses this pattern as its default retry policy. The pattern is documented in the SRE literature (Google SRE book, "Implementing Service Level Objectives", Amazon Builders' Library) and is the recommended approach for any service operating across multiple independent clients. The CSF aligns with that practice. The alignment is deliberate, not incidental. ### 6.3 What the framework permits and what it does not | Pattern | Allowed at the CSF P2P boundary? | |---|---| | Exponential backoff with jitter, bounded by a maximum retry window | **MUST** | | Linear retry schedule (e.g. every 5 s, every 30 s) | **MUST NOT** | | Fixed-cadence retry tail (e.g. every 60 s indefinitely) | **MUST NOT** | | Honouring `Retry-After` from a destination | **MUST** | | Per-implementation tuning of base / cap / jitter range | Permitted | | Circuit-breaker after N consecutive failures | **MUST** | | Operational escalation beyond the retry window | **MUST** | --- ## 7. The HMAP boundary When a CSF MAP exchanges traffic with TOTSCo via the dedicated HMAP interface, the boundary inherits **hub behaviour** by definition. TOTSCo is a centralised hub, and the OTA2 *Switching for Business Message Delivery Principles V0.2* bind TOTSCo's hub-side delivery to the destination RCP. ### 7.1 What applies on TOTSCo's side TOTSCo applies its own hub-side delivery schedule per the GPLB principles. Summarised: | GPLB principle | Match Requests | All other SforB messages | | ------------------------------------- | ---------------------------------------------------------- | ---------------------------------------------------------- | | Connection timeout | 1 s | 1 s | | Response timeout | 3 s | 3 s | | Retries at (from hub's 202 to sender) | 5, 10, 15, 20, 25 s | 10, 20, 30, 60 s, then every 60 s | | Total hold window | **30 s** | **12 days** | | Failure notification | Async, to sender's pre-configured failure endpoint | Async, to sender's pre-configured failure endpoint | | Queueing | Single-flight per endpoint; queue items age at same window | Single-flight per endpoint; queue items age at same window | These are TOTSCo's rules for the hub it operates. No change is required of TOTSCo when integrating CSF — the GPLB principles already govern TOTSCo's existing operation. ### 7.2 What the CSF MAP does on its side The CSF MAP behaves as a **standard TOTSCo hub client**: - Posts JAM envelopes to TOTSCo's letterbox with the same OAuth + TLS + JAM-envelope conventions used by every TOTSCo MAP today. - Receives `HTTP 202` synchronously on hub-side acceptance. - Receives **asynchronous failure notifications** from TOTSCo at a sender-configured failure endpoint when TOTSCo's hub-side delivery to the destination RCP fails (per GPLB principles). - Applies CSF's own retry policy (§§4–5) on its own retries to TOTSCo's letterbox if a delivery to TOTSCo itself fails for connection / response timeout reasons. ### 7.3 The locked-down interface The HMAP interface is **locked down by design** — see [Part 2 §6.3.1 Dedicated, locked-down HMAP interfaces — both directions](../part2-implementation/06-totsco-integration.md#631-dedicated-locked-down-hmap-interfaces--both-directions) and the [TOTSCo HMAP Integration Guide §1](./totsco-hmap-integration-guide.md). This means: - CSF version increments do **NOT** propagate through the dedicated TOTSCo endpoints. - The GPLB hub schedules at the boundary stay stable for TOTSCo. - The CSF P2P rules in §§3–6 stay stable for the wider framework. - The asymmetry is bilateral, predictable, and durable. In other words: the GPLB hub schedules **do not bleed** into the CSF P2P transport. They bind only TOTSCo's side of the HMAP boundary. --- ## 8. Summary of bindings This table is the practical reference. Each row states which rule applies on each side of the boundary. | Rule | CSF MAP ↔ CSF MAP (P2P) | HMAP boundary (CSF MAP ↔ TOTSCo) | |---|---|---| | 1 s connection timeout | **MUST** | **MUST** (aligned with GPLB on TOTSCo's side) | | 3 s response timeout | **MUST** | **MUST** (aligned with GPLB on TOTSCo's side) | | Exponential backoff with jitter | **MUST** on retry | CSF MAP applies on its retries to TOTSCo. TOTSCo applies the GPLB linear / fixed-cadence schedule on its hub side. | | Respect `Retry-After` on `429` / `503` | **MUST** | **MUST** | | Circuit-breaker after N consecutive failures | **MUST** (CSF MAP side; recommended N = 5) | **MUST** (CSF MAP side) — n/a on TOTSCo's hub side | | Maximum retry period bounded | **MUST** — typically hours | TOTSCo applies the GPLB 30 s Match / 12-day other-message holds | | Failure notification to sender | Synchronous — `9xxx` / `8xxx` codes in the original HTTP response | Asynchronous from TOTSCo when hub-side delivery fails; CSF MAP consumes as standard hub client | | Operational escalation via `map/contact[]` / `map/outage[]` | Primary recovery mechanism beyond the retry window | Applies on the CSF MAP side; TOTSCo follows its own incident process | | Single-flight per endpoint + queue ageing | Not required; MAP implementation choice | TOTSCo applies hub-side single-flight + queue ageing | | Logging of every delivery attempt | **MUST** | **MUST** | --- ## 9. Testing The TAG Baseline Test Suite formally exercises this policy. The test set is at [Onboarding & Testing Process §8.7](./onboarding-testing-process.md#87-retry--delivery-policy) — tests **R-01..R-18**, summarised here for context: | ID | Type | What it validates | |---|---|---| | R-01..R-03 | P | `Retry-After` honouring on `503` / `429`; transient network failure retry with exponential backoff and jitter | | R-04..R-06 | N | No retry on `400` (`9006`) / `404` (`9007`); max retry window exhausted raises `9008` | | R-07..R-09 | E | Circuit-breaker open / close behaviour; persistent `8102` (DNS) failure | | **R-10** | P | **1 s connection timeout** (TCP / TLS SYN drop) | | **R-11** | P | **3 s response timeout** (connection established, no `202`) | | **R-12** | P | **Exponential backoff shape** — geometric growth, each step ≥ 1.5× the previous | | **R-13** | P | **Jitter present** — ±15 % variance across independent senders | | **R-14** | P | **Circuit-breaker activation** after N consecutive failures | | **R-15** | P | **Circuit-breaker recovery** (half-open → closed) on probe success | | **R-16** | E | **Half-open probe failure** returns cleanly to open | | **R-17** | P | **Bounded retry window** — raises `9008` and triggers operational escalation | | **R-18** | E | **Operational engagement** — sender downgrades retry on `map/outage[]` | A MAP cannot complete onboarding (Phase 3 → Phase 4) without these tests passing. --- ## 10. Cross-references - **Canonical normative source:** [Part 2 §4.7 Message Delivery Policy](../part2-implementation/04-message-api.md#47-message-delivery-policy) — the ratified clause this standalone summarises. - **Retry-related operational rules:** [Part 1 §9 Operational Excellence](../part1-framework/09-operational-excellence.md), specifically the *Retry Policy* paragraph. - **Test suite:** [Onboarding & Testing Process §8.7](./onboarding-testing-process.md#87-retry--delivery-policy) — tests R-01..R-18. - **HMAP boundary operating model:** [TOTSCo HMAP Integration Guide](./totsco-hmap-integration-guide.md), specifically §1 (locked-down interface) and Phase 3 (worked bidirectional example). - **Locked-down interface ratification:** [Part 2 §6.3.1 Dedicated, locked-down HMAP interfaces](../part2-implementation/06-totsco-integration.md#631-dedicated-locked-down-hmap-interfaces--both-directions). - **Outage publication:** [Part 2 §5.4 MAP Section — Field Reference](../part2-implementation/05-directory-api.md#54-map-section--field-reference) for the `map/outage[]` schema; [Part 2 §9.3](../part2-implementation/09-operations.md#93-outage-notification) for the operational rules. - **Operational engagement contacts:** [Part 2 §5.6 CP Contact Object](../part2-implementation/05-directory-api.md#56-cp-contact-object). - **Incident response coordination:** [Part 1 §10.6.5](../part1-framework/10-governance.md#1065-incident-response-coordination). - **Architectural analysis (CSF vs OTA2 GPLB MDP V0.2):** `wiki/syntheses/csf-compliance-with-gplb-message-delivery-principles.md` — the synthesis that grounds the CSF P2P / HMAP-boundary split documented here. --- ## 11. Document Control | Version | Date | Description | |---|---|---| | 1.0 | 2026-06-24 | First-draft CSF Message Delivery Policy as a standalone consolidation. Faithful summary of the ratified [Part 2 §4.7](../part2-implementation/04-message-api.md#47-message-delivery-policy), the [Part 1 §9 Retry Policy paragraph](../part1-framework/09-operational-excellence.md), the [§8.7 test suite](./onboarding-testing-process.md#87-retry--delivery-policy) (R-01..R-18), the [HMAP-boundary callout](./totsco-hmap-integration-guide.md) in the TOTSCo Integration Guide, and the compliance synthesis at `wiki/syntheses/csf-compliance-with-gplb-message-delivery-principles.md`. No new policy claims — consolidation only. For TAG review. |