Documentation

Core concepts

Pub/Sub

Pub/Sub is short for "Publish/Subscribe", is a messaging style in software design. Imagine it like this: one or more senders (called publishers) share messages on a central topic, and lots of receivers (called subscribers) get and respond to those messages. It's like a way for different parts of a system to talk to each other without knowing who they're talking to. This makes it easier to make big systems that can grow, stay dependable, and be easy to take care of.

The WebSocket protocol

WebSocket is a computer communications protocol, providing full-duplex communication channels over a single TCP connection. The WebSocket protocol was standardized by the IETF as RFC 6455 in 2011. The current API specification allowing web applications to use this protocol is known as WebSockets. It is a living standard maintained by the WHATWG and a successor to The WebSocket API from the W3C.

See: https://en.wikipedia.org/wiki/WebSocket

Common uses for WebSockets are:

• Real-time Communication: WebSockets provide full-duplex communication between client and server, enabling real-time communication in web applications such as online gaming, chat applications, and collaboration tools.
• Dynamic Data Updates: WebSockets can be used to update dynamic data in real-time without the need for continuous polling. This is useful in applications such as stock tickers, weather updates, and sports scores.
• Multimedia Streaming: WebSockets can be used to stream multimedia data in real-time, such as audio or video, enabling low-latency communication in applications such as video conferencing and online broadcast platforms.
• IoT (Internet of Things) Applications: WebSockets can be used to communicate with IoT devices in real-time, allowing for real-time monitoring and control of these devices, as well as sending and receiving data from them.
• Interactive Dashboards: WebSockets can be used to create dynamic, real-time dashboards that can update in response to user actions or new data. This is useful in fields such as finance, marketing, and business intelligence, where real-time data analysis is critical.
• Live Multiplayer Gaming Experiences: WebSockets enable live, interactive gameplay for mobile gaming apps by facilitating instant communication between the game server and multiple players.

The WebSocket API

The WebSocket API is an advanced technology that makes it possible to open a two-way interactive communication session between the user's browser and a server. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

Most browsers and JavaScript runtimes fully support the WebSockets API: https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API#browser_compatibility

See: https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API

JSON Web Tokens

JSON Web Token (JWT) is a compact, URL-safe means of representing claims to be transferred between two parties. The claims in a JWT are encoded as a JSON object that is used as the payload of a JSON Web Signature (JWS) structure or as the plaintext of a JSON Web Encryption (JWE) structure, enabling the claims to be digitally signed or integrity protected with a Message Authentication Code (MAC) and/or encrypted.

See: https://www.rfc-editor.org/rfc/rfc7519 and https://jwt.io/introduction

JWT implementations exist for many languages and frameworks, including but not limited to: https://jwt.io/libraries

Server

In Realtime Pub/Sub, "server" is the software component that is responsible for managing WebSocket connections, application topics and publications. Each server is highly optimized for low-latency and can handle several thousands of concurrent connections and message flows with minimal impact on hardware resources. Multiple server instances work together as a cluster to increase sockets capacity, throughput and availability.

Application

An "application" is the virtual representation of an isolated and secure namespace for "subscribers", "topics", "publishers" and "messages". Among other properties, an application defines authentication configuration using JWT, both for WebSocket clients and administrative operations via REST.

Application configuration example:

{
  "enabled": true,
  "appId": "f36b57561fc006e958e521008764a570",
  "name": "My Example App",
  "description": "This is an example application that uses ES512 JWT-based authentication",
  "tier": "free",
  "createdAt": "2023-09-17T15:26:35.666Z",
  "updatedAt": "2023-10-05T13:05:38.238Z",
  "adminKey": {
    "source": "raw",
    "key": "-----BEGIN PUBLIC KEY-----\nMIGbMBAGBy...RzY1gqGo=\n-----END PUBLIC KEY-----",
    "algorithms": [
      "ES512"
    ],
    "issuer": []
  },
  "adminSourceAddress": [
    "0.0.0.0/0"
  ],
  "clientSourceAddress": [
    "0.0.0.0/0"
  ],
  "authKey": {
    "source": "raw",
    "key": "-----BEGIN PUBLIC KEY-----\nMIGbMBAGBy...RzY1gqGo=\n-----END PUBLIC KEY-----",
    "algorithms": [
      "ES512"
    ],
    "issuer": []
  },
  "enableWebSocketInboundTopic": true,
  "amazonSnsForwarder": {},
  "amazonSqsForwarder": {}
  ...
}

Supported configuration attributes:

• enabled: A boolean value indicating whether the application is enabled or not.
• name: The unique name of the application.
• description: Description of the application.
• authKey: An object that provides the public key and algorithms used for client authentication.
• adminKey: An object that provides the public key and algorithms used for administrative endpoints authentication.
• adminSourceAddress: An array containing source IP addresses and CIDR masks that either permit or deny (with a preceding "!") access for administrative connections.
• clientSourceAddress: An array of source IP addresses and CIDR masks that either allow or deny (preceded by "!") access for client (subscriber) connections.
• enableWebSocketInboundTopic: A boolean flag used to control the enabling or disabling of topic forwarding for inbound WebSocket messages. When set to TRUE, inbound messages are broadcasted on the secure/inbound topic.
• clientInboundAckEnabled: A boolean flag used to control the enabling or disabling of WebSocket inbound message acknowledgements.
• amazonSnsForwarder: An object that provides the configuration for forwarding inbound WebSocket messages to an Amazon SNS Topic.
• amazonSqsForwarder: An object that provides the configuration for forwarding inbound WebSocket messages to an Amazon SQS Queue.

Topics

A topic is another virtual entity that exists within the context of an application, representing isolated namespaces for messages, subscriptions and publications. Each application can hold hundreds of topics, each of them with up to thousands of subscribers.

The following are possible topics categories:

• Private (prefixed with priv/)
• Secure (prefixed with secure/)
• Public

Note that each WebSocket connection (aka: Subscriber) is ALWAYS subscribed to the following private topics:

• priv/WEBSOCKET_CONNECTION_ID, for example: priv/f774e704c1eb
• priv/WEBSOCKET_CONNECTION_ACCESS_TOKEN_SUBJECT, for example: priv/user0@example.com

  NOTE: Subscribing to a private topic is controlled by the backend and is prohibited in the client subscription API.

Clients can subscribe or unsubscribe from public topics as needed. By default, clients are automatically subscribed to the main topic.

See Subscribing to secure topics below

Subscribers

WebSocket connections that subscribe to one or multiple topics are referred to as "subscribers". A successful subscription to a topic is referred to as a "subscription".

See Pricing/Plan features for subscriptions related quotas

const ws = new WebSocket(`wss://genesis.r7.21no.de/apps/${APP_ID}?access_token=${ACCESS_TOKEN}`)
ws.onmessage = (event) => {
  // Parse incoming WebSocket messages
  const { topic, messageType, data } = JSON.parse(event.data)

  if (topic === 'main' && messageType === 'welcome') {
    // SESSION IS NOW READY, YOU CAN SUBSCRIBE TO APP TOPICS...
    ws.send(JSON.stringify({
      type: 'subscribe',
      data: {
        topic: 'topic1'
      }
    }))
    ws.send(JSON.stringify({
      type: 'subscribe',
      data: {
        topic: 'topic2'
      }
    }))
    ws.send(JSON.stringify({
      type: 'subscribe',
      data: {
        topic: 'topicN'
      }
    }))
  }
}

Connections can also unsubscribe topics on-demand:

ws.send(JSON.stringify({
  type: 'unsubscribe',
  data: {
    topic: 'topic1'
  }
}))
ws.send(JSON.stringify({
  type: 'unsubscribe',
  data: {
    topic: 'topic2'
  }
}))

When clients subscribe to a topic, they'll get a welcome message, and afterwards they'll receive all messages broadcasted to that topic. Let's dive into a detailed example to illustrate this:

const ws = new WebSocket(`wss://genesis.r7.21no.de/apps/${APP_ID}?access_token=${ACCESS_TOKEN}`)

ws.on('message', (event) => {
  // Parse incoming WebSocket messages
  const { topic, messageType, data } = JSON.parse(event.data)
  logger.info(`> received bytes from '${topic}' topic: ${event.data.length}`)

  if (topic === 'main' && messageType === 'welcome') {
    ws.id = data.connection.id
    logger.info(`> connection '${data.connection.id}' established ...`)

    // SUBSCRIBE TO TOPICS HERE
  } else if (messageType === 'welcome') {
    logger.info(`> topic '${topic}' subscribed...`)

    // ON SUBSCRIPTION LOGIC HERE
  } else if (messageType === 'broadcast') {
    logger.info(`> processing message from topic '${topic}'...`)
    logger.info(data)

    // MESSAGE PROCESSING LOGIC HERE
  }
})

Publishers

WebSocket clients or backend services (via the HTTP publish endpoint) that publish messages into application topics are referred to as "publishers".

When using the HTTP publish endpoint, distributed services would trigger HTTP requests to the publication endpoint using a valid authorization token. For a token to be valid, it is required to be verifiable using the application adminKey configuration.

Upon successful operation, broadcasted topic messages can be received by subscribers who have subscribed to the same topic.

Example event payload received at a subscriber side (WebSocket API):

ws.on('message', (event) => {
  const { topic, messageType, data } = JSON.parse(event.data)
  /*
    "messageType"   = "broadcast"
    "topic"         = "a-topic-name"
    "data"          = "Hello Pub/Sub!"
  */
}

Continue reading at: https://realtime.21no.de/getting-started/#the-subscription-and-publication-workflow

Granting publisher permissions on topics

Granting publishers access to topics is essential for controlling who can publish on them. Publishers, require the realtime:publisher:write:topic permission, with explicit topic names or patterns. For example:

• realtime:publisher:write:topic:main: Grants the publisher permission to publish messages on the main topic.
• realtime:publisher:write:topic:secure/*: Grants the publisher permission to publish messages on all topics that match the secure/* pattern.

Let's review an example authorization token (JWT) below, describing a publisher who can publish messages into the main and secure topics:

{
  "permissions": [
    "realtime:publisher:write:topic:secure/*",
    "realtime:publisher:write:topic:main"
  ],
  "iat": 1673028921,
  "exp": 1673028926,
  "iss": "https://signer.example.com",
  "sub": "service-name",
  ...
}

Topic patterns are matched using the matcher module. Use * to allow the publisher to broadcast messages on all topics.

WebSocket Inbound Messaging

You can enable WebSocket clients to emit messages as input to backend applications. These messages are subsequently routed to enabled forwarding targets for real-time processing.
The following example demonstrates how to send messages from a WebSocket client for processing by the backend services:

ws.send(JSON.stringify({
  type: 'message',
  data: {
    // your message payload, expected type: object or string
    payload: 'hello',
    // message id, auto-generated if not provided
    id: '52',
    // compression, default value: false
    compress: true
  }
}))

Supported messages forwarding destinations:

• Inbound Topic: Application owners can forward inbound WebSocket messages into the secure/inbound topic.
• Amazon SNS Topic or SQS Queue: Application owners can forward inbound WebSocket messages to an Amazon SNS Topic or SQS Queue.

Please note that the maximum payload size for a single WebSocket message is limited to 5KB.

Inbound TopicAmazon SQS QueueAmazon SNS Topic

Applications can forward WebSocket clients inbound messaging to the secure/inbound topic by setting enableWebSocketInboundTopic to TRUE in application configuration. Afterwards, any WebSocket client with the realtime:subscriber:read:topic:secure/inbound permission in its access token can receive and process those messages:

// NOTE: A valid client access_token is still required here
const ws = new WebSocket(`wss://genesis.r7.21no.de/apps/${APP_ID}?access_token=${ACCESS_TOKEN}`)

ws.on('message', (event) => {
  // Parse incoming WebSocket messages
  const { topic, messageType, data } = JSON.parse(event.data)

  if (topic === 'main' && messageType === 'welcome') {
    ws.send(JSON.stringify({
      type: 'subscribe',
      data: {
        topic: 'secure/inbound'
      }
    }))
  } else if (topic === 'secure/inbound' && messageType === 'broadcast') {
    // PROCESS WEBSOCKET MESSAGES HERE...
  }
})

Also see: Subscribing to secure topics below

Applications can forward WebSocket clients inbound messaging to a target Amazon SQS Queue, leveraging SQS scalability, consumer parallelism and reliability.
To enable Amazon SQS Queue forwarding, go to Modify Application / WebSocket Inbound Messaging / Amazon SQS Forwarding Target Configuration and provide the requested configuration.

Learn more through a full example backend service that demonstrates how to respond to client requests via SQS integration: SQS Backend Example

Example SQS consumer message format:

{
  "client": {
    "connectionId": "a5c07d1d1a310eb55d9d038e71e9c986",
    "subject": "user-01",
    "permissions": ["..."]
  },
  "payload": "Saying hi from WS client... ",
  "id": "3bb24d51855461cb55511b789cc1418e"
}

Applications can forward WebSocket clients inbound messaging to a target Amazon SNS Topic, leveraging SNS's extensive range of supported event targets.
To enable Amazon SNS Topic forwarding, go to Modify Application / WebSocket Inbound Messaging / Amazon SNS Forwarding Target Configuration and provide the requested configuration.

---

WebSocket Inbound ACK

When enabling WebSocket Inbound ACK in application configuration, the server will acknowledge each successful processed message. The payload will contain the submitted (or generated if absent) message ID and is deliverd to the client via the priv/acks topic.

This functionality not only confirms the successful receipt and forwarding of messages but also enhances tracking and integrity of data flow within your application and clients.

ws.on('message', (event) => {
  // Parse incoming WebSocket messages
  const { topic, messageType, data } = JSON.parse(event.data)

  if (topic === 'main' && messageType === 'welcome') {
    ws.send(JSON.stringify({
    type: 'message',
      data: {
        payload: 'hello',
        id: '528c3s4-3d8y'
      }
    }))
  } else if (topic === 'priv/acks' && messageType === 'ack') {
    // retrieving acknowledged message ID
    const messageId = data

    // PROCESS ACKNOWLEDGED MESSAGES HERE...
  }
})

Presence Protocol

When WebSocket Inbound Messaging is enabled, the platform will emit presence events to the secure/inboud topic using presence as messageType attribute value. These messages are used to notify clients or servers about the connection status of other clients within the application. The payload of the presence message contains attributes such as connection ID, subject, and status of the client.

Client Connected EventClient Disconnected Event

    ...
    data: {
      client: {
        connectionId: '4389b2ed1a7e',
        subject: 'user0@example.com',
        permissions: [Array]
      },
      subProtocol: 1,
      payload: { code: 0, status: 'connected' }
    }

    ...
    data: {
      client: {
        connectionId: '4389b2ed1a7e',
        subject: 'user0@example.com',
        permissions: [Array]
      },
      subProtocol: 1,
      payload: { code: -1, status: 'disconnected' }
    }

The following example demonstrates how to process presence events in a WebSocket client:

ws.on('message', (event) => {
  // Parse incoming WebSocket messages
  const { topic, messageType, data } = JSON.parse(event.data)

  if (topic === 'main' && messageType === 'welcome') {
    ws.send(JSON.stringify({
      type: 'subscribe',
      data: {
        topic: 'secure/inbound'
      }
    }))
  } else if (topic === 'secure/inbound') {
    if ( && messageType === 'broadcast') {
    // PROCESS WEBSOCKET MESSAGES HERE...
    } else if (messageType === 'presence') {
      // PROCESS PRESENCE MESSAGES HERE...

      const clientId = data.client.connectionId
      const statusCode = data.payload.code

      if (statusCode === 0) {
        logger.info(`> client '${clientId}' connected...`)
      } else if (statusCode === -1) {
        logger.info(`> client '${clientId}' disconnected...`)
      }
    }
  }
})

The presence protocol events are also delivered via configured forwarding targets such as Amazon SNS or SQS.

Management Services

Management services refer to administration and telemetry APIs supporting the platform beyond the servers, APIs which are mainly used for administrative and reporting purposes. Example of management services are:

• Apps Manager API
• Telemetry processors
• Telemetry and Reporting API
• Settings API

Checkout our OpenAPI documentation: https://api-docs.r7.21no.de

OpenAPI UI works with the Chrome browser only!

API

The following APIs are provided by the real-time servers and are designed for use by application subscribers, admins, publishers or operators.

See Management Services below for other administrative APIs

WebSockets

• Open session:

  const ws = new WebSocket(`wss://genesis.r7.21no.de/apps/${APP_ID}?access_token=${ACCESS_TOKEN}`)
  ws.onmessage = (event) => {
    // Parse incoming WebSocket messages
    const { topic, messageType, data } = JSON.parse(event.data)
  
    if (topic === 'main' && messageType === 'welcome') {
      // SESSION IS NOW READY, YOU CAN SUBSCRIBE TO APP TOPICS...
    }
  } 
  

• Subscribe topics:

  // subscribing public topics
  ws.send(JSON.stringify({
    type: 'subscribe',
    data: {
      topic: TOPIC_NAME
    }
  }))
  

  // subscribing secure topics
  ws.send(JSON.stringify({
    type: 'subscribe',
    data: {
      topic: `secure/${TOPIC_NAME}`
    }
  }))
  

• Unsubscribe topics:

  // unsubscribing public topics
  ws.send(JSON.stringify({
    type: 'unsubscribe',
    data: {
      topic: TOPIC_NAME
    }
  }))
  

  // unsubscribing secure topics
  ws.send(JSON.stringify({
    type: 'unsubscribe',
    data: {
      topic: `secure/${TOPIC_NAME}`
    }
  }))
  

• Process incoming messages:

  // processing incoming messages
  ws.onmessage = (event) => {
    // Parse incoming WebSocket messages
    const { topic, messageType, data } = JSON.parse(event.data)
  
    if (topic === 'main' && messageType === 'welcome') {
      // SESSION IS NOW READY, YOU CAN SUBSCRIBE TO APP TOPICS...
    } else if (messageType === 'broadcast') {
      logger.info(`> processing message from topic '${topic}'...`)
      logger.info(data)
  
      // MESSAGE PROCESSING LOGIC HERE
    }
  } 
  

• Publish on topics:

  // subscribing public topics
  ws.send(JSON.stringify({
    type: 'publish',
    data: {
      // target topic name
      topic: 'main',
      // message type (optional, default: 'broadcast')
      messageType: 'greeting',
      // your message payload, expected type: object or string
      payload: 'hello',
      // message id, auto-generated if not provided
      id: '52',
      // compression, default value: false
      compress: true 
    }
  }))
  

• Send messages:

  // subscribing public topics
  ws.send(JSON.stringify({
    type: 'message',
    data: {
      // message type (optional, default: 'broadcast')
      messageType: 'greeting',
      // your message payload, expected type: object or string
      payload: 'hello',
      // message id, auto-generated if not provided
      id: '52',
      // compression, default value: false
      compress: true 
    }
  }))
  

• Close session:

  ws.close()
  

IMPORTANT: When any WebSocket client input message (using send method) fails the validation process, the client is immediately disconnected with the reason: Invalid message format!

WebSocket clients and payload compression

WebSocket clients can vary in their support for message payload compression, often reflected in the different API abstractions available to developers. The following examples illustrate these differences across various libraries:

• Web Browsers (Google Chrome, Mozilla Firefox and Safari):

    ws.onmessage = async (event) => {
      // Parse incoming WebSocket messages
      const { topic, messageType, data } = event.data instanceof Blob
        ? JSON.parse(await event.data.text()) // compression is enabled
        : JSON.parse(event.data)
  
      // Check if it's a welcome message from the 'main' topic
      if (topic === 'main' && messageType === 'welcome') {
        console.log('> Connected!')
      }
  
      // Log incoming WebSocket messages
      console.log('> Incoming message:', { topic, messageType, data, compression: event.data instanceof Blob })
    }
  

  Full demo: https://github.com/BackendStack21/realtime-forum/blob/main/demos/browser/subscriber.html

• Node.js with ws and reconnecting-websocket libraries:

    ws.onmessage = (event) => {
      // Parse incoming WebSocket messages
      // Compression is handled transparently, no need for decoding on handler
      const { topic, messageType, data } = JSON.parse(event.data)
  
      // Check if it's a welcome message from the 'main' topic
      if (topic === 'main' && messageType === 'welcome') {
        console.log('> Connected!')
      }
  
      // Log incoming WebSocket messages
      console.log('> Incoming message:', { topic, messageType, data })
    }
  

  Full demo: https://github.com/BackendStack21/realtime-forum/blob/main/demos/node/auto-reconnection.js

• Node.js with websocket library:

    ws.onmessage = (event) => {
      // Parse incoming WebSocket messages
      const { topic, messageType, data } = event.data instanceof ArrayBuffer
        ? JSON.parse(new TextDecoder().decode(event.data)) // compression is enabled
        : JSON.parse(event.data)
  
      // Check if it's a welcome message from the 'main' topic
      if (topic === 'main' && messageType === 'welcome') {
        console.log('> Connected!')
  
        // Subscribe to a custom topic
        ws.send(JSON.stringify({
          type: 'subscribe',
          data: {
            topic: 'my-custom-topic'
          }
        }))
      }
  
      // Log incoming WebSocket messages
      console.log('> Incoming message:', { topic, messageType, data, compression: event.data instanceof ArrayBuffer })
    }
  

  Full demo: https://github.com/BackendStack21/realtime-forum/blob/main/demos/node/subscriber.js

HTTP

The following considerations are effective when invoking protected endpoints below:

• Max message payload size is 5KB
• Each endpoint invocation reduces your API usage quota by a certain amount of bytes
• Authentication tokens are required to be verifiable using the JWT verification settings defined in application adminKey configuration.

Sample JWT token format with all supported roles:

{
  "permissions": [
    "realtime:publisher:write:topic:*",
    "realtime:admin:manage:subscribers",
    "realtime:admin:manage:connections"
  ],
  "iat": 1673028921,
  "exp": 1673028926,
  "iss": "https://signer.example.com",
  "sub": "service-name",
  ...
}

Endpoints

• Server health status: GET /health/status, unprotected

  curl -X GET \
  -H "Accept:application/json" \
  "https://genesis.r7.21no.de/api/health/status"
  

• Publish a message on a topic: POST /topics/:appId/publish, required permission: realtime:publisher:write:topic:*, cost in bytes: data.length
  

  curl -X POST \
  -H "Accept:application/json" \
  -H "Content-Type:application/json" \
  -H "Authorization:Bearer XXX" \
  --data '{"topic":"main","message":{"hello":"world"},"compress":false}' \
  "https://genesis.r7.21no.de/api/topics/$APPPLICATION_ID/publish"
  

• Unsubscribe clients from one or many topics: POST /topics/:appId/unsubscribe, required permission: realtime:admin:manage:subscribers, cost in bytes: 256
  

  curl -X POST \
  -H "Accept:application/json" \
  -H "Content-Type:application/json" \
  -H "Authorization:Bearer XXX" \
  --data '{"topicPatterns":["secure/logs/*"],"subject":"user0@example.com"}' \
  "https://genesis.r7.21no.de/api/topics/$APPPLICATION_ID/unsubscribe"
  

  The request above would unsubscribe clients where subject = user0@example.com from all topics matching the pattern secure/logs/*. Read more about supported matching patterns at https://github.com/sindresorhus/matcher#patterns:

  Use * to match zero or more characters.
  A leading ! negates the pattern.
  

• Terminate a connection (close WebSocket client) by it's connection identifier: DELETE /connections/:appId/:connectionId, required permission: realtime:admin:manage:connections, cost in bytes: 256
  (please note that a re-connection logic might exist on the client)

  curl -X DELETE \
  -H "Accept:application/json" \
  -H "Authorization:Bearer XXX" \
  "https://genesis.r7.21no.de/api/connections/$APPPLICATION_ID/$CONNECTION_ID"
  

• Terminate all connections to an application: DELETE /connections/:appId, required permission: realtime:admin:manage:connections, cost in bytes: 256
  (please note that a re-connection logic might exist on the client)

  curl -X DELETE \
  -H "Accept:application/json" \
  -H "Authorization:Bearer XXX" \
  "https://genesis.r7.21no.de/api/connections/$APPPLICATION_ID"
  

Accessing application telemetry and reports via REST

Application owners can also access their application telemetry reports via REST in order to enable or extend existing monitoring capabilities.

The following example illustrates how to query the telemetry API endpoints using administrative tokens:

const jwt = require('jsonwebtoken')
const axios = require('axios').default
const APPLICATION_ADMIN_PERMISSION = 'realtime:admin:manage:application'

function getAppAggregatedTelemetry (cfg, days) {
  console.info(`Getting aggregated telemetry from ${cfg.appId} application...`)

  const AUTH_TOKEN = getAppAdminToken(cfg.adminSigningKey, cfg.tokenExpiresInSeconds)
  const options = {
    method: 'GET',
    url: `https://${cfg.apiGatewayUrl}/telemetry/aggregates/${cfg.appId}/${days}`,
    headers: {
      'Content-Type': 'application/json',
      Authorization: AUTH_TOKEN,
      'X-App-Id': cfg.appId
    }
  }

  return axios.request(options)
}

function getAppAdminToken (secret, expiresIn) {
  return 'Bearer ' + jwt.sign({ permissions: [APPLICATION_ADMIN_PERMISSION] }, secret, {
    algorithm: 'ES512', 
    expiresIn,
    issuer: 'https://signer.example.com',
    subject: 'service-name'
  })
}

getAppAggregatedTelemetry({
  appId: 'YOUR APPLICATION ID',
  adminSigningKey: 'APPLICATION ADMIN SECRET/PRIVATE KEY',
  tokenExpiresInSeconds: 5,
  apiGatewayUrl: 'api-gateway.r7.21no.de' // OR PROVIDED CLUSTER API GATEWAY URL
}, 1)
  .then(({ data }) => console.log(data))
  .catch(err => console.error(err))

Depending on the environment, administrators and operators may issue tokens with longer expiration periods. In the previous example, we used short-lived tokens (5 seconds).
Also note that the responsibility of issuing tokens commonly lies with a separate service and not with the telemetry consumer itself, included here for demonstration purposes.

Security Considerations

Embrace public key authentication with JWT

We would like application owners to keep in control of their authentication strategies, including their JWT signing keys. Always use asymmetric keys for production applications!

Applications related authentication takes place in two stages:

• WebSocket clients authentication: Uses JWT verification details defined in the authKey application configuration attribute to decide if a subscriber can connect to a target application

  ...
  "authKey": {
    "source": "raw",
    "key": "-----BEGIN PUBLIC KEY-----\nMIGbMBAGBy...RzY1gqGo=\n-----END PUBLIC KEY-----",
    "algorithms": [
      "ES512"
    ],
    "issuer": []
  }
  ...
  

• Invoking administrative HTTP endpoints: Uses JWT verification details defined in the adminKey application configuration attribute to authorize operations supported by the HTTP endpoints in real-time servers (see above)

  ...
  "adminKey": {
    "source": "raw",
    "key": "-----BEGIN PUBLIC KEY-----\nMIGbMBAGBy...RzY1gqGo=\n-----END PUBLIC KEY-----",
    "algorithms": [
      "ES512"
    ],
    "issuer": []
  }
  ...
  

When necessary, application owners can re-use the same configuration for both attributes (authKey and adminKey), it works as intended.

In both cases, application owners are encouraged to use asymmetric encryption and digital signature algorithms. Opposed to HS* algorithms, RSA and ECDSA based algorithms require application owners to submit only their public key, retaining cryptographic control and authority over the issuing of short-living authentication tokens.

🔐 The use of asymmetric algorithms ensure stronger security features to application owners, being PS384, PS512, ES384, ES512 preferred candidates. Optionally, you can generate JWT keys at: https://jwt-keys.21no.de

Subscribing to secure topics

As previously mentioned, "secure" topics are those with a secure/ prefix and are designed to be the channel for restricted messages which are only available to subscribers with required permissions.

Subscribing to a secure topic requires that a client’s access token (JWT) contains the necessary permissions to read from the topic. The token should include the realtime:subscriber:read:topic:secure/* permission or a specific topic pattern, such as realtime:subscriber:read:topic:secure/logs/*.

Let's review an example below:

{
  "permissions": [
    "realtime:subscriber:read:topic:secure/logs/*"
  ],
  "iat": 1675335604,
  "exp": 1675335609,
  "iss": "https://jwt-signer.r7.21no.de",
  "sub": "user0@example.com"
}

A valid token with the structure above would allow clients to subscribe to secure topics which name matches the secure/logs/* pattern. Patterns are evaluate using the matcher module.

End to end encryption

At this time, the platform does not support built-in end-to-end encryption. Nevertheless, developers with specific encryption needs, implementing a custom encryption layer is straightforward and feasible.

It's important to clarify that the platform does not retain messages nor does it extract metadata from the content. Instead, we gather data primarily for reporting and billing purposes. Here's a JSON sample showcasing the kind of data we collect, which includes the message length and the count of subscribers for a given topic:

// portion of the response payload returned by the publication endpoint (Telemetry API)

{
  "bytes": 1175641,
  "subscribers": 17101,
  "records": [
    {
      "_id": "2023-02-01T17:00:00.000Z",
      "ops": 24,
      "bytes": 11476,
      "subscribers": 164
    },
    {
      "_id": "2023-02-01T17:30:00.000Z",
      "ops": 45,
      "bytes": 23317,
      "subscribers": 337
    },
    {
      "_id": "2023-02-01T18:00:00.000Z",
      "ops": 45,
      "bytes": 21975,
      "subscribers": 315
    },
    ...

For more information, please refer to our Privacy Policy to know about the data that we collect.

Integrating with IDPs and SSO providers using JSON Web Key Sets (JWKS)

Our platform supports JSON Web Key Sets (JWKS) for authentication and authorization, providing a reliable solution for enterprise integrations. This feature enables application owners to seamlessly integrate with Identity Providers (IDPs) and Single Sign-On (SSO) providers that support JWKS, ensuring secure and efficient user identity and access management.

To use JWKS, application owners need to provide the URI of the JWKS endpoint in the authKey and adminKey configuration attributes. The platform will then fetch the public keys from the JWKS endpoint and use them to verify the JWT tokens.

Here is an example of how to configure the authKey and adminKey attributes with a JWKS URI:

{
  ...
  "authKey": {
    "source": "jwks",
    "jwksUri": "https://example.com/.well-known/jwks.json",
    "audience": "https://example.com",
    "issuer": []
  },
  "adminKey": {
    "source": "jwks",
    "jwksUri": "https://example.com/.well-known/jwks.json",
    "audience": "https://example.com",
    "issuer": []
  }
  ...
}

In this example, the jwksUri attribute specifies the URI of the JWKS endpoint, and the audience attribute specifies the expected audience of the JWT tokens. The issuer attribute can be used to specify the expected issuer of the JWT tokens.

Application configuration the Admin Console:

For more information on JWKS, see the JSON Web Key Set (JWKS) RFC.

How are permissions extracted from JWT tokens?

The platform extracts permissions from JWT tokens by decoding the token and reading the permissions, scp or scope claim (in that order). The permissions or scp claims are an array of strings that represent the permissions granted to the token holder.

If only the scope claim is passed, the platform will convert it to an array of strings and use it as the permissions claim.

Other low-level WebSocket server settings

The following are low-level WebSocket server settings that are not configurable by application owners:

• Max "Back-Pressure" buffer size per connection is 64KB, on buffer overflow, the connection is terminated.
• Max message payload size is 5KB.
• Server will send pings automatically to uphold a stable connection.
• Connection idle timeout is 30 seconds, afterwards the connection is closed.
• Each user is restricted to only one WebSocket connection for each server instance.

  A minimum of five server instances are consistently operational in our PROD environment.