Automation: Data processing of imported data using property modifiers in Icinga Director

The raw data imported from the external sources (CSV, SQL, REST API, LDAP, etc.) is usually not in the right format. Hence, they have to be processed or converted before they are used to modify objects using synchronization rules. To do this, Icinga Director provides different property modifiers. There are many property modifiers provided by Icinga Director.

Here, I introduce some of these simple property modifiers for the import source using a sample data as shown below.

[
  {
    "fqdn": " web01.example.com ",
    "ip": " 192.168.1.10 ",
    "roles": "web,frontend,primary",
    "location": "new york",
    "os": "Ubuntu",
    "department": "it-operations",
    "env": "PROD",
    "contact": "admin@example.com"
  },
  {
    "fqdn": "DB01.staging.local",
    "ip": " 192.168.1.20",
    "roles": "db,primary",
    "location": "berlin",
    "os": "Windows Server 2019",
    "department": "data-services",
    "env": "stage",
    "contact": "dba@example.com"
  }
]

In my opinion, data processing in Icinga Director involves using simple property modifiers or a combination of modifiers to:

Clean or normalize values

• Trimming accidental white spaces introduced in the beginning and end of the property values (ip, fqdn)

• Lowercasing property values to maintain consistency (fqdn, env)

• Replacing some values with another values to be more descriptive or to map different values to one value (env: prod -> production; stage -> staging)

• Capitalize first character of each word in the values (location, env)

Transform values

• Strip domain names from to obtain host names (fqdn, host_name)

• Extracting specific values from the property using regex (os)

• Transforming a string with separators into an array (roles)

Enrich values

• Combining multiple properties into a single property (host_name and location to get display_name)

Applying multiple property modifiers on a single property

Additionally, you could also apply multiple property modifiers on a property, but in this case the order matters. In Icinga Director, we follow top-to-bottom order. That is, the topmost modifier is applied first followed by the ones below it. Here, the modifications applied on the property fqdn can be considered an example of this.

Once these modifiers are applied, check the modified data in the preview for any errors. If the data looks correct and in right structure, you could go ahead and run the import and trigger the corresponding synchronization rules.

Before:

After:

Conclusion

There are many more property modifiers introduced in Icinga Director for different use cases, one of which I have already introduced in this blogpost. In general, property modifiers are an important during automation to eliminate inconsistencies and errors, transform the raw data to desired structure or enrich the values before they are used to modify the objects. Also, learn more on automation in Icinga Director from its docs.