Channels¶

After Icinga Notifications decides to send a notification of any kind, it is passed to a channel plugin. Such a plugin submits the notification event to a domain-specific channel, such as email or a chat client.

Icinga Notifications comes packed with channels, but also enables you to develop your own channels.

To make these channels available to Icinga Notifications, they must be placed in the channels directory, which is done automatically during package installations. At startup, Icinga Notifications scans this directory, starts each channel once to query its configuration options and stores those options in the database. Using this information, Icinga Notifications Web allows channels to be configured, which are then started, configured, and finally used to send notification events from Icinga Notifications.

Available Channels¶

Icinga Notifications comes with multiple channels out of the box:

email: Email submission via SMTP
rocketchat: Rocket.Chat
telegram: Telegram
webhook: Configurable HTTP/HTTPS queries for your backend

Most of them are configured entirely in Icinga Notifications Web. The sections below cover those channels that need some preparation on the other side first.

Telegram¶

Telegram messages are sent by a bot, which is created and owned by a regular Telegram user account. Each contact receives their notifications in their personal chat from this bot.

Create a bot by talking to @BotFather and using its /newbot command, as described in the Telegram bot documentation. BotFather answers with an authentication token, which is the credential for the bot’s whole API access. Treat it like a password.
Create a new channel of the type Telegram in Icinga Notifications Web and set the Bot token to that token.
Let every contact who should be notified open a chat with the bot and send it a message, for example /start. A bot is not allowed to write messages to someone who has never written to it, so this step can not be skipped.
Look up the chat ID of each contact and configure it as their telegram address in Icinga Notifications Web.

The messages from the previous step are queued as updates for the bot and can be read with the getUpdates method:

curl -s 'https://api.telegram.org/bot<token>/getUpdates'

In each update, message.chat.id is the chat ID to configure.

A group can be notified the same way: add the bot to the group, post a message there, and use the group’s chat ID.

Technical Channel Description¶

Channel plugins are independent processes that run continuously, started and supervised by Icinga Notifications. They receive JSON-RPC 2.0 requests on stdin and reply with JSON-RPC 2.0 responses on stdout. For each request that includes an id, the channel must return a response, even if the request was invalid or could not be processed. Requests without an id are JSON-RPC Notifications and must not be responded to. Channel plugins can be written in any programming language, as long as they adhere to the communication protocol and implement the required methods, which are described below.

Architecture¶

The channel plugin architecture is designed to be simple and flexible, allowing for easy integration of new channels into Icinga Notifications. The channel plugin acts as a server, receiving requests from Icinga Notifications and sending back responses. The channel plugin can also send requests to Icinga Notifications, allowing for bidirectional communication. Please refer to the Available Icinga Notifications Methods section for more information on the methods available to channel plugins.

This documentation uses beautified JSON for ease of reading.

Request¶

Icinga Notifications sends requests to the channel plugin as JSON objects with the following fields, which are defined in the JSON-RPC 2.0 specification:

Field	Type	Description
jsonrpc	String	Required. Must be `"2.0"`.
method	String	Required. Request method to call.
params	JSON object	Optional. Params for the request method.
id	Unsigned integer	Optional. Unique identifier.

The jsonrpc field denotes the version of the JSON-RPC protocol used, which must be "2.0" for all requests and responses. The method field specifies the name of the request method to call, which is one of the methods described in the RPC Methods section for channel plugins. The params field contains the parameters for the request method, which is a JSON object, and may be omitted entirely if the request method does not require any parameters. Lastly, the id field is a unique identifier for the request, which is used to match the response to the request, and will always be set by Icinga Notifications when sending requests to the channel plugin.

Examples:

Simple request without any params:

{
  "jsonrpc": "2.0",
  "method": "ping",
  "id": 1000
}

Request with params of different types:

{
  "jsonrpc": "2.0",
  "method": "sleep",
  "params": {
    "duration": 30000000000,
    "message": "Sleeping for 30 seconds"
  },
  "id": 2000
}

Response¶

Each request must be answered by the channel plugin with a JSON object containing the following fields, which are defined in the JSON-RPC 2.0 specification:

Field	Type	Description
jsonrpc	String	Required. Must be `"2.0"`.
result	JSON value (any)	Optional. Output of a successful method call.
error	JSON object	Optional. Error object if the method call failed.
id	Unsigned integer	Required. Unique identifier of the request being responded to.

The result and the error fields are mutually exclusive, meaning that either one of them must be present, but not both. For a successful method call, the result field must always be present, and the error field must be omitted. For a failed method call, on the other hand, the error field must always be present, and the result field must be omitted. The error field is a JSON object according to the JSON-RPC 2.0 specification, which contains a code and a message describing the error. The id field must always be present and must match the id of the request being responded to.

Examples:

Successful response message:

{
  "jsonrpc": "2.0",
  "result": "pong",
  "id": 1000
}

Response with an error:

{
  "jsonrpc": "2.0",
  "error": {
    "code": -32601,
    "message": "method not found"
  },
  "id": 2000
}

Available Icinga Notifications Methods¶

Currently, Icinga Notifications provides the following method for channel plugins to call:

Method	Description
channel::Log	Logs a message to Icinga Notifications’ logging system.

Channel plugins can make use of this method by sending their own JSON-RPC requests to Icinga Notifications. It is described in detail below, including the expected request format.

Log¶

The channel::Log method allows channel plugins to log messages to Icinga Notifications’ logging system. The request’s params field must be a JSON object with the following fields:

Field	Type	Description
level	String	Required. Log level, one of `"debug"`, `"info"`, `"warn"`, or `"error"`.
message	String	Required. Log message to be logged.
fields	Array	Optional. Additional fields to be logged as key-value pairs.

If fields are defined, they must be an array of an even number of elements, with each pair representing a key-value pair. Also, the request’s id field must be omitted, as the request is expected to be a JSON-RPC Notification, meaning that no response is expected from Icinga Notifications.

Example Log Request:

{
  "jsonrpc": "2.0",
  "method": "channel::Log",
  "params": {
    "level": "debug",
    "message": "Sending notification to Slack channel #general",
    "fields": [
      "notification_id",
      12345
    ]
  }
}

This request will result in a log message in Icinga Notifications’ logging system with the following format:

... DEBUG   channel Sending notification to Slack channel #general  {"channel": {"id": 2, "name": "slack", "type": "slack"}, "pid": 2382, "notification_id": 12345}

Icinga Notifications will also log the channel’s id and name, as well as the process ID of the channel plugin process, which can be useful for debugging purposes.

RPC Methods¶

The following methods must be implemented by a channel plugin, and are called by Icinga Notifications to interact with the channel:

Method	Description
GetInfo	Returns meta information about the channel, such as its name, version etc.
SetConfig	Configures the channel with the provided configuration options.
SendNotification	Requests the channel to send a notification.

Each of these methods is described in detail below, including the expected request and response formats.

GetInfo¶

The parameterless GetInfo method returns information about the channel.

Its result is expected to be a JSON object with the json fields defined in the Info type.

Field	Type	Description
name	String	Required. The display name of the channel.
version	String	Required. The version of the channel.
author	String	Required. The author of the channel.
supported_request_versions	Array	Required. The supported request versions of the channel.
config_attrs	Array	Required. The configuration options of the channel.

Everything except supported_request_versions will be persisted in the database and used to configure the channel in Icinga Notifications Web. The supported_request_versions field tells Icinga Notifications which request versions the channel plugin supports, and is used to ensure compatibility between the channel plugin and Icinga Notifications. Icinga Notifications is expected to support all past request versions the channel plugin advertises, but will refuse versions it does not support. For example, if a channel plugin advertises support for request version 2 (doesn’t exist yet) and Icinga Notifications knows only about request version 1, it will refuse to start the channel plugin and log an error message.

The config_attrs field must be an array of JSON objects according to the ConfigOption type. These attributes define configuration options for the channel to be set by the SetConfig method. They are also used for channel configuration in Icinga Notifications Web.

Example GetInfo Request¶

{
  "jsonrpc": "2.0",
  "method": "GetInfo",
  "id": 1
}

Example GetInfo Response¶

{
  "jsonrpc": "2.0",
  "result": {
    "name": "Minified Webhook",
    "version": "0.0.0-gf369a11-dirty",
    "author": "Icinga GmbH",
    "supported_request_versions": [1],
    "config_attrs": [
      {
        "name": "url_template",
        "type": "string",
        "label": {
          "de_DE": "URL-Template",
          "en_US": "URL Template"
        },
        "help": {
          "de_DE": "URL, optional als Go-Template über das zu verarbeitende plugin.NotificationRequest.",
          "en_US": "URL, optionally as a Go template over the current plugin.NotificationRequest."
        },
        "required": true,
        "min": null,
        "max": null
      },
      {
        "name": "response_status_codes",
        "type": "string",
        "label": {
          "de_DE": "Antwort-Status-Codes",
          "en_US": "Response Status Codes"
        },
        "help": {
          "de_DE": "Kommaseparierte Liste erwarteter Status-Code der HTTP-Antwort, z.B.: 200,201,202,208,418",
          "en_US": "Comma separated list of expected HTTP response status code, e.g., 200,201,202,208,418"
        },
        "default": "200",
        "min": null,
        "max": null
      }
    ]
  },
  "id": 1
}

SetConfig¶

The SetConfig method configures the channel.

Icinga Notifications will call this method at least once on each channel before sending the first notification to initialize the channel plugin. However, it may also be called multiple times during the lifetime of a channel plugin, for example, if the user changes the configuration in Icinga Notifications Web. Therefore, the channel plugin must be able to handle multiple calls to SetConfig and update its configuration accordingly at any given time.

The passed JSON object in the request’s params field reflects the objects from GetInfo‘s config_attrs. Each object in the config_attrs array must be configurable, using its name attribute as a key together along with the desired configuration value, which must be of the type specified in the type field.

To illustrate, the URL template from the above output is configurable with the following JSON object passed in params:

{
  "url_template": "http://localhost:8000/update/{{.Object.Name}}"
}

After the channel plugin has successfully applied the configuration, it must return a valid JSON-RPC response with the result field set to a simple success message or just to JSON null. If the channel plugin is unable to apply the configuration, it must return a valid JSON-RPC response with the error field set to the appropriate error code and message.

Example SetConfig Request¶

{
  "jsonrpc": "2.0",
  "method": "SetConfig",
  "params": {
    "url_template": "http://localhost:8000/update/{{.Object.Name}}",
    "response_status_codes": "200"
  },
  "id": 2
}

Example SetConfig Response¶

{
  "jsonrpc": "2.0",
  "result": null,
  "id": 2
}

SendNotification¶

The SendNotification method requests the channel to dispatch notifications.

Within the request’s params, a JSON object representing a NotificationRequest is passed.

The incident field is only set when the notification relates to an incident. That is the case if:

the event opened or escalated an incident, or
the contact is notified as a recipient of the object’s open incident (via an escalation, as a subscriber or as the manager).

Otherwise, for example for a contact that is notified only because of a notification rule, incident is null. Channel plugins (and templates referencing .Incident) must handle this null case explicitly.

The request’s contact.addresses only contains the contact’s addresses whose type matches the channel plugin type. For example, the email channel plugin only receives addresses of type email. If the contact has no such address, the array is empty, and it’s up to the channel plugin to decide how to handle such a case.

If the request refers to an incident and the contact has at least one matching address, the request also contains a state object. See Channel State for how to use it.

If the channel is unable to send a notification, an error response must be replied with the appropriate error code and message. This may be due to channel-specific reasons, such as an email channel where the SMTP server is unavailable, or if the channel is missing required configuration values. The error message is stored in the notification history, so it should be meaningful to the user.

On success, the result may either be JSON null or a DeliveryResult object with the following optional fields:

Field	Type	Description
state	Object	Optional. The new or updated channel state to persist for this incident and contact.
details	Array	Optional. Delivery details to show in the notification history.

Example SendNotification Request¶

{
  "jsonrpc": "2.0",
  "method": "SendNotification",
  "params": {
    "version": 1,
    "payload": {
      "contact": {
        "full_name": "icingaadmin",
        "addresses": [
          {
            "type": "email",
            "address": "icingaaadmin@example.com"
          }
        ]
      },
      "object": {
        "name": "example-809!disk /",
        "url": "http://localhost/icingaweb2/icingadb/service?name=disk%20%2F&host.name=example-809",
        "sources": [
          "icinga2-master"
        ],
        "tags": {
          "host": "example-809",
          "service": "disk /"
        }
      },
      "incident": {
        "severity": "critical",
        "muted": false,
        "recovered": false,
        "message": "[CRITICAL] - Filesystem checks failed\n        \\_[CRITICAL] - / (ffs)\n                \\_[CRITICAL] - Free space absolute: 42 MiB (of 985 MiB)\n                \\_[CRITICAL] - Free space percentage: 4.264%\n                \\_[OK] - Inodes free: 97.4923% (151618 of 155518)|'/'=943718400B;826918502.400000;930283315.200000;0;1033648128",
        "summary": "[CRITICAL] - Filesystem checks failed",
        "started_at": "2024-07-12T10:40:00Z",
        "managed": false,
        "opened_by_source": "icinga2-master"
      },
      "event": {
        "time": "2024-07-12T10:47:30.445439055Z",
        "message": "[CRITICAL] - Filesystem checks failed\n        \\_[CRITICAL] - / (ffs)\n                \\_[CRITICAL] - Free space absolute: 42 MiB (of 985 MiB)\n                \\_[CRITICAL] - Free space percentage: 4.264%\n                \\_[OK] - Inodes free: 97.4923% (151618 of 155518)|'/'=943718400B;826918502.400000;930283315.200000;0;1033648128",
        "summary": "[CRITICAL] - Filesystem checks failed"
      },
      "summary": "[example-809!disk /] Incident opened: [CRITICAL] - Filesystem checks failed",
      "body": "A new incident has been opened by icinga2-master for example-809!disk /.\n\nSeverity: Critical\nOpened:   2024-07-12 10:40:00 +0000\nDetails:  http://localhost/icingaweb2/icingadb/service?name=disk%20%2F&host.name=example-809\n\n[CRITICAL] - Filesystem checks failed\n        \\_[CRITICAL] - / (ffs)\n                \\_[CRITICAL] - Free space absolute: 42 MiB (of 985 MiB)\n                \\_[CRITICAL] - Free space percentage: 4.264%\n                \\_[OK] - Inodes free: 97.4923% (151618 of 155518)|'/'=943718400B;826918502.400000;930283315.200000;0;1033648128",
      "state": {
        "state_key": "0d6a5bd1-3c1e-5f0e-9d43-6f1f6c2b7a10",
        "value": "{\"last_message_id\": \"<bb7364a5-8f24-4b6a-a2d8-4658f42cd5cd-sender@example.com>\"}"
      }
    }
  },
  "id": 3
}

Example SendNotification Response¶

{
  "jsonrpc": "2.0",
  "result": {
    "state": {
      "state_key": "0d6a5bd1-3c1e-5f0e-9d43-6f1f6c2b7a10",
      "value": "{\"last_message_id\": \"<2e9dd0b2-9555-4627-871e-c8696e742486-sender@example.com>\"}"
    },
    "details": [
      {
        "type": "text",
        "label": "Message-ID",
        "value": "<2e9dd0b2-9555-4627-871e-c8696e742486-sender@example.com>"
      }
    ]
  },
  "id": 3
}

Info

A channel that neither needs state nor has details to report can simply reply with "result": null instead of a DeliveryResult object.

Delivery Details¶

Besides its state, a channel plugin can report details about a successful delivery, such as the ID of the sent message or a link to the ticket created in an external system. Unlike the state, these details are never passed back to the plugin. They are meant for users, who can inspect them in the notification history of Icinga Notifications Web.

The plugin returns them in the details field of its SendNotification result, as an array of JSON objects according to the DeliveryDetail type:

Field	Type	Description
type	String	Required. The type of the detail, e.g., `"text"` or `"url"`.
label	String	Required. Describes the detail, e.g., `"Ticket"`.
value	String	Required. The actual value of the detail, e.g., `"#123"`.
anchor_text	String	Optional. The text to display for a clickable link, if `type` is `"url"`.

In order Icinga Notifications Web to interpret the details correctly, the detail type must be provided. The supported types are:

number: The detail is a number and will be displayed as such.
boolean: The detail is a boolean value and will be displayed as such.
severity: It is a severity level and will be displayed with a corresponding color.
text: The detail is plain text and will be displayed as such.
url: The detail is a clickable link and will be displayed as such. The value field must be an absolute http or https URL. If the optional anchor_text field is set, it will be used as the link text instead of the URL.
datetime: The detail is a date and time in ISO 8601 format and will be displayed as such.

The serialized details must not exceed 1 MiB. Larger details are discarded with a logged warning, but the notification itself is still considered successful. For example, a channel creating tickets in an external system might return the following details, where the ticket is linked and the queue it was filed in is shown as plain text:

{
  "details": [
    {
      "type": "url",
      "label": "Ticket",
      "value": "https://rt.example.com/Ticket/Display.html?id=123",
      "anchor_text": "#123"
    },
    {
      "type": "text",
      "label": "Queue",
      "text": "Operations"
    }
  ]
}

Channel Configuration¶

A channel offers its configuration options through its response to the GetInfo method call. Each configuration option is described by a ConfigOption entry in the config_attrs array. This information will be queried once by Icinga Notifications upon startup and stored in the database.

Depending on the ConfigOption‘s type, Icinga Notifications Web will render different form element. For example, a text will result in an input box, while options will result in a multi-select list.

When a user configures a Channel in Icinga Notifications Web, the configuration will be stored in the database as JSON. More specifically, a JSON object mapping config_attrs.name to the configured value is stored, as expected as the params for the SetConfig method.

Since ConfigOptions may have defaults defined, Icinga Notifications Web will not add unchanged defaults to the configuration JSON object. Therefore, the channel plugin is expected to use their offered default value if the key-value pair is absent.

A SetConfig implementation may follow this logic by first setting its defaults and then overwriting its state based on the configuration received via the params.

Finally, Icinga Notifications will start a new process for this channel and pass the stored JSON object to the channel by calling the SetConfig method. The process is kept alive and receives occasional SendNotification method calls until Icinga Notifications is stopped or the channel plugin is restarted due to a configuration change. Note that Icinga Notifications will not restart the process on every configuration change, unless it has no other choice. For instance, if the channel plugin config changes at runtime, Icinga Notifications will call SetConfig again with the new config except when the channel plugin type has changed to a different one, in which case the old channel plugin process will be stopped and a new one will be started.

Channel State¶

Some channel plugins need to remember information about previously sent notifications. For example, the email channel remembers the Message-Id of the last email sent for an incident so that it can set the In-Reply-To and References headers on subsequent emails, and mail clients group them into one thread. Likewise, the telegram channel replies to the last message sent for an incident. Icinga Notifications manages this state on behalf of the channel plugins and stores it in its database, so it survives plugin restarts and is shared in HA setups.

A state is always bound to a single channel, incident and contact. It is represented by the State type:

Field	Type	Description
state_key	String	A UUID generated by Icinga Notifications that identifies the state.
value	String	The plugin-defined state value, empty if no state has been stored yet.

Icinga Notifications derives the state key from the channel, the incident and the contact’s addresses passed to the plugin, so channel plugins must treat it as opaque and never generate or modify it themselves. With every SendNotification request for an incident, Icinga Notifications passes the contact’s state object to the plugin. Its state_key is always set, while its value holds the previously stored value, or is empty if nothing has been stored yet. Note that changing a contact’s addresses of the channel plugin’s type while an incident is ongoing results in a new state key, so the next notification arrives without a state and starts a new thread.

To store or update the state, the channel plugin returns it in the state field of its SendNotification result, using the exact state_key it received. The value can be any string, such as JSON, between 1 and 4096 characters long. If the returned state has a different key or an invalid value, Icinga Notifications discards it and logs a warning, but the notification itself is still considered successful.

Once an incident has recovered, Icinga Notifications deletes its state, since no further notifications will reference it. The final notification still receives the last stored value, so the plugin can refer to it one last time, but any state the plugin returns for a recovered incident is ignored.

There is no state if the request doesn’t refer to an incident or the contact has no address matching the channel plugin type. In that case, the plugin must not return a state either.

Any remaining state is also removed when the channel is deleted, when its plugin type changes, or when the incident is removed by the retention policy. Channel plugins don’t need to clean up after themselves.

Writing Channel Plugins¶

Tip

Icinga Notifications comes with a Webhook channel plugin. Consider using this channel if your transport uses HTTP instead of writing a custom channel.

Tip

When developing custom channels, consider naming them with a unique prefix, as additional channels will be added to Icinga Notifications in the future. For example, name your channel x_irc or my_irc instead of irc.

The channels that ship with Icinga Notifications can only cover some use cases. Therefore, we encourage you to develop your own channels that cover your specific needs.

Writing Channel Plugins in Go¶

Since Icinga Notifications and all of its channels are written in the Go programming language, libraries already used internally can be reused. In particular, the Icinga Notifications Plugin and Icinga Notifications JSON-RPC packages provide a framework for writing channel plugins in Go. The jsonrpc package implements the JSON-RPC 2.0 protocol, while the plugin package provides types and functions for implementing channel plugins. Also, the plugin package provides a Run function that takes care of setting up a JSON-RPC server and dispatching requests to the appropriate methods of the channel plugin.

To respect the channel configuration logic described above, an implementation of the SetConfig method should start by calling PopulateDefaults. Also, note that Icinga Notifications might call SetConfig multiple times during the lifetime of a channel plugin, so the implementation should be able to handle multiple calls and update its configuration accordingly. This is further explained in the documentation of the SetConfig method.

Also, the jsonrpc implementation in the SDK is designed to allow for bidirectional communication, meaning that the channel plugin can also send requests to Icinga Notifications. Please refer to the Available Icinga Notifications Methods section for more information on the methods available to channel plugins.

To debug the JSON-RPC communication, build the channel plugin with the DebugJsonRpc build tag, which will log all requests and responses to stderr.

For concrete examples, there are the implemented channels in the Icinga Notifications repository at ./cmd/channels.