User Guide

Contents

Preparation

For information regarding the installation of the Grafana extension please refer to: Installation of Grafana

Details regarding the configuration of the extension may be found here: Configuration of Grafana

Grafana Data Sources

The Grafana Extension provides two data source endpoints for Grafana:

• Standard data source: A generic data source for all time series tasks. In panels and all queries around Grafana Metrics HAKOM time series are queried / listed

• Trees data source: A node attribute aggregation data source, specifically designed for WebTSM Services Trees. In panels and all queries around Grafana Metrics HAKOM Trees are queried / listed. Node attributes are rolled out flat, i.e. main nodes and all sub nodes are listed as single entries.

Working with Grafana and HAKOM TSM

Graphics (Panels)

Graphics (time series) are arranged in panels. Each panel can display several time series from the same data source.

Adding Time Series

Standard Data Source

Time series for panels can be added in the following simple steps:

1. Select Query data source (e.g. JSON_DOCKER_POSGRE)

2. Add Query

3. Format as Time series or Table (NOTE: with Grafana 8.0 / SimPod 0.3.0 this setting does not exist. The system automatically uses "table", to use "timeseries" instead, add Additional JSON Data / Payload Parameter "targetType":"timeseries").

4. Metric:

   1. Select the desired time series name (the list shows the first 100 hits for the respective input (an extension to several hits can be configured via HAKOM.Config LimitSearchItems)

   2. - Custom Query - select as time series name (this allows you to define a time series search using the "Additional JSON Data" or Payload field (see search, or timeseriesNames)

Now the time series is already displayed in the graph if it has data.

Subsequently, control functions can be defined via Additional JSON Data Field. See also Additional JSON Data Parameter / Payload

NOTE: With Grafana 8.0 / SimPod 0.3.0 a new field "Payload" is used instead of Additional JSON Data to transfer additional data to the service.
When migrating dashboards to 8.0, the Additional JSON Data content must be transferred to the new Payload field!

Trees Data Source

Node aggregates in trees data source can be added in the following simple steps:

1. Select Query data source (e.g. JSON_DOCKER_POSGRE)

2. Add Query

3. Format as Time series or Table

4. Metric: Select the desired tree node and set unit via Additional JSON Data if necessary.

Subsequently, further control functions can be added via Additional JSON Data.

Naming Conventions for Table Format

Column names are assigned according to the following rules, but can be renamed to Table Panel:

• Time series keep their respective names (or a formulaAlias is applied as name accordingly)

• Attribute columns:

  • If only one time series was selected, and its attributes are displayed: the attribute columns are given the respective attribute name

  • If attributes of several time series are displayed, the attribute columns are named according to the following pattern: Time series name:attribute name

  • If attributes are loaded from a time series specified in the background (sourceTimeseries) and noTimeseriesNameInAttributeHeader=false, the attribute columns are named according to the following pattern: sourceTimeseries:attribute name, otherwise only attribute names

• Flag columns:

  • If only one time series is displayed, the column is called "Flag

  • If multiple time series are displayed, the column names will follow the following pattern: Time series name:Flag

Additional JSON Data Parameter / Payload

The following parameters can be added to the Additional JSON Data (or wih Grafana 8.0+ "Payload") field in a bracketed format.

• Column "Format" shows for which Metric Types the parameters are supported

• Column "Path" shows in which nodes a parameter is located

  • path root: the parameter is entered via \{"Parameter":value,... }

  • Path search or columns: the parameter must be specified in the respective node. For example \{"columns":{"allAttributes":true}}

Parameter Examples

Retrieve time series in unit MWh with exact audit date 2019-11-14T23:00:00Z as total aggregate

{"unit":"MWh", "audit": "2019-11-14T23:00:00Z", "exactAudit": true, "aggregation": "Sum"}

Retrieve time series in 15 minute intervals exactly at 0-15-30-45 minute grid points

{"interval":"Minute", "intervalMultiplier":15, "exactInterval":true}

Retrieve Ad Hoc formula (time series*-1) with unit "MWh" in 15 minutes interval exactly on the 0-15-30-45 minutes grid points

{"formula":"[this,0,MWh]*-1","formulaAlias":"My Inverse Time Series","interval":"Minute", "intervalMultiplier":15, "exactInterval":true}

Define time series names as a list. (For queries using timeseriesNames, you must select Metric - Custom Query -)

{"timeseriesNames": ["MyTimeSeries1", "MyTimeSeries2"]}

Time series from time series search with limit and attribute search. (For queries using search, you must select Metric - Custom Query -)

{"search": {"name": "MyTestTimeS%", "type": "Begin", "interval": "Hour","description": "My time series description","unit": "MWh",
			"attributes": ["MyAttribute1", "MyAttribute2|Value"],"limit": 100,"offset": 0}}

DisplayName Examples

Simple query with display name

{"interval": "Hour", "displayName": "[Name]+ (+[Description]+)"}

Time series search combined with display name

{"search":{"name": "Consumption%"}, "displayName": "[Name]+ (+[Description]+)"}

TreeNodeSum including time series combined with display name

{"displayName": "[Name]+ (+[ID]+)", "treeNodeSum": {"name": "$rootNodes", "includeTimeSeries": true, "includeSum": false, "extendByAdhocFilter": "$extendByAdhocFilter"}, "exactInterval":"$controlExactInterval", "unit": "MWh"}

TreeNodeSum Examples - Standard Data Source

Loads a TreeNodeSum aggregate (that is, the sum of the selected TreeNode):

{"unit":"KWh", "treeNodeSum": {"name": "MyTree"}}

Loads a subnode aggregate (that is, the sum of the selected subnode ID):

{"unit":"KWh", "treeNodeSum": {"name": "MyTree", "nodeId":10}}

Loads all time series data in the selected TreeNode:

{"unit":"KWh", "treeNodeSum": {"name": "MyTree", "indludeSum": false, "includeTimeSeries": "true"}}

Loads all time series data in the selected subnode:

{"unit":"KWh", "treeNodeSum": {"name": "MyTree", "nodeId":10, "indludeSum": false, "includeTimeSeries": "true"}}

Loads all sub-subnode totals inside the selected subnodes:

{"unit":"KWh", "treeNodeSum": {"name": "MyTree", "nodeId":10, "indludeSum": false, "includeSubNodes": "true"}}

Loads both the subnode total and all time series data and sub-subnodes inside the selected subnodes:

{"unit":"KWh", "treeNodeSum": {"name": "MyTree", "nodeId":10, "includeTimeSeries": true, "includeSubNodes": "true"}}

Loads a TreeNodeSum aggregate, which can be switched to a subnode via Grafana AdHocFilter Variable.

{"unit":"KWh", "treeNodeSum": {"name": "MyTree", "extendByAdhocFilter": true}}

Loads 5 time series data and skips 10 time series in the selected TreeNode.

{"unit": "KWh", "treeNodeSum": {"name": "MyTree", "indludeSum": false, "includeTimeSeries": "true", "limitTimeSeries": 5, "offsetTimeSeries": 10}}

TreeNodeSum Examples - Trees Data Source

With Trees data source the parameters "name" and "nodeId" are overridden by the selected Grafana Metric, so that these parameters are ignored there. All other parameters apply as described above.

Table Examples

Display time series with flags (as flag name)

{"columns": {"showFlag": "Name"}}

Display time series with all attributes and flags

{"columns": {"showFlag": "Name", "allAttributes": true}}

Define time series names as list, get flags as id (Result: for all listed time series the flags are displayed, in both line graphs and table panels. The flags are listed under "time series name:flag").
Note: to retrieve flags as name, use "showFlag": "name" instead. (For queries using timeseriesNames, you must select Metric - Custom Query -)

{"timeseriesNames": ["MyTimeSeries1", "MyTimeSeries2"], "columns": {"showFlag": "id"}}

Define time series names as list, retrieve only specific attributes (result: for both time series the listed attributes are loaded if any assigned)

{"timeseriesNames": ["MyTimeSeries1", "MyTimeSeries2"], 
 "columns": {"customList": ["MyAttribute1", "MyAttribute2"]}}

Define time series names as a list, retrieve only specific attributes from only one specific time series (result: data is loaded for both time series, attributes are loaded from the designated sourceTimeseries)

{"timeseriesNames": ["MyTimeSeries1", "MyTimeSeries2"], 
 "columns": {"customList": ["MyAttribute1", "MyAttribute2"], "sourceTimeseries": "MyTimeSeriesWithAttribute"}}

Retrieve time series from time series search with limit and attribute search and specific attributes defined (result: all time series found within the specified limit and the specified attributes, if assigned, are loaded)

{"search": {"name": "MyTestTimeS%", "type": "Begin", "interval": "Hour","description": "My time series description","unit": "MWh",
			"attributes": ["MyAttribute1", "MyAttribute2|Value"],"limit": 100,"offset": 0},
 "columns": {"customList": ["MyAttribute1", "MyAttribute2"]}}

Annotations

Time-related information with textual context, such as certain events in Grafana, can be recorded and displayed. Such events can also be retrieved from an external JSON data source via /annotations path. Individual annotations can be visualized as individual points in time or time ranges (region).

In HAKOM TSM such events can be displayed via attributes and time slices or directly via time series containing category values. 

NOTE: with Grafana 8.0 / SimPod 0.3.0 the definition of annotations and the interface behind has been changed.

Query / Payload

Annotations Before Grafana 8.0

Define Annotation From TSM Attribute

Grafana - Dashboard - Settings - Annotations - New

1. Select a name in the Annotations dialog under General

2. Select a JSON data source in Data source

3. Define, whether the annotation is displayed and with which display color to select

4. Under Query, define the search parameter for the attribute using following patterns:

1. 1. Retrieve attribute without filtering by specific attribute value:

      XML

      {"timeseries":"timeseries_name","attribute":{"name":"attribute_name"}}
      

      
      

   2. Retrieve attribute with filtering by a specific attribute value: 

      XML

      {"timeseries":"timeseries_name","attribute":{"name":"attribute_name","value":"attribute_value"}}
      

      
      

   3. Retrieve attribute without filtering by specific attribute value displayed as region (colored time span while the attribute value is valid): 

      XML

      {"timeseries":"timeseries_name","isRegion":true,"attribute":{"name":"attribute_name"}}
      

      
      

   4. Retrieve attribute with filtering by a specific attribute value displayed as region (colored time span while the attribute value is valid): 

      XML

      {"timeseries":"timeseries_name","isRegion":true,"attribute":{"name":"attribute_name","value":"attribute_value"}}
      

      
      

Define Annotation From TSM TIme Series

Grafana - Dashboard - Settings - Annotations - New

1. Select a name in the Annotations dialog under General

2. Select a JSON data source in Data source

3. Define, whether the annotation is displayed and with which display color to select

4. Under Query, define the search parameter for the time series using following patterns:

   1. Retrieve time series values without filtering by specific time series value use the following pattern:

      XML

      {"timeseries":"timeseries_name", "exactInterval": true}
      

      
      

   2. Retrieve time series values with filtering by a specific time series value: 

      XML

      {"timeseries":"timeseries_name","value":1, "exactInterval": true}
      

      
      

   3. Retrieve time series values without filtering by specific time series value displayed as region (colored time span while the time series value is unchanged): 

      XML

      {"timeseries":"timeseries_name","isRegion":true, "exactInterval": true}
      

      
      

   4. Retrieve time series values with filtering by a specific time series value displayed as region (coloured time span while the time series value is unchanged): 

      XML

      {"timeseries":"timeseries_name","isRegion":true,"value":1, "exactInterval": true}
      

      
      

   5. Retrieve time series values mapped to specific text values:

      XML

      {"timeseries":"timeseries_name", "valueMap":{"1":"Mapping Text", "2":"Mapping Text 2"}}
      

      
      

   6. Retreive time series values of an ad hoc formula and mapped values to specific text (e.g. to display flags as annotation colours)

      XML

      {"timeseries":"timeseries_name", 
      	"formula":{"formula":"[this].Flag", "interval":"Hour", "intervalMultiplier":1}, 
      	"valueMap":{"9":"Valid", "19":"Missing"}}
      

      
      

Annotations Grafana 8.0 and Higher

Annotations can now access time series directly. This makes the configuration of both annotations from time series attributes and annotations from time series more intuitive.

Define Annotation From TSM Attribute

Grafana - Dashboard - Settings - Annotations - New

1. Select a name in the Annotations dialog under Name

2. Select a JSON data source in Data source

3. Define, whether the annotation is displayed and with which display color to select

4. Under Query - Metric select a time series

5. Finally add additional parameter in Payload field:

   1. Retrieve attribute without filtering by specific attribute value:

      XML

      {"attribute":{"name":"attribute_name"}}
      

      NOTE: the entry "timeseries": "timeseries_name" is no longer necessary, since a time series has been selected. If this parameter is set, the system will apply its value instead of the time series selected under Metric.
      
      

   2. Retrieve attribute with filtering by a specific attribute value: 

      XML

      {"attribute":{"name":"attribute_name","value":"attribute_value"}}
      

      
      

   3. Retrieve attribute without filtering by specific attribute value displayed as region (colored time span while the attribute value is valid): 

      XML

      {"isRegion":true,"attribute":{"name":"attribute_name"}}
      

      
      

   4. Retrieve attribute with filtering by a specific attribute value displayed as region (colored time span while the attribute value is valid): 

      XML

      {"isRegion":true,"attribute":{"name":"attribute_name","value":"attribute_value"}}
      

      
      

Define Annotation From TSM Time Series

Grafana - Dashboard - Settings - Annotations - New

1. Select a name in the Annotations dialog under Name

2. Select a JSON data source in Data source

3. Define, whether the annotation is displayed and with which display color to select

4. Under Query - Metric select a time series

5. Finally add additional parameter in Payload field:

   1. Retrieve time series values without filtering by specific time series value use the following pattern:

      XML

      {"exactInterval": true}
      

      NOTE: the entry "timeseries": "timeseries_name" is no longer necessary, since a time series has been selected. If this parameter is set, the system will apply its value instead of the time series selected under Metric.
      
      

   2. Retrieve time series values with filtering by a specific time series value: 

      XML

      {"value":1, "exactInterval": true}
      

      
      

   3. Retrieve time series values without filtering by specific time series value displayed as region (colored time span while the time series value is unchanged): 

      XML

      {"isRegion":true, "exactInterval": true}
      

      
      

   4. Retrieve time series values with filtering by a specific time series value displayed as region (coloured time span while the time series value is unchanged): 

      XML

      {"isRegion":true,"value":1, "exactInterval": true}
      

      
      

   5. Retrieve time series values mapped to specific text values:

      XML

      {"valueMap":{"1":"Mapping Text", "2":"Mapping Text 2"}}
      

      
      

   6. Retreive time series values of an ad hoc formula and mapped values to specific text (e.g. to display flags as annotation colours)

      XML

      {"formula":{"formula":"[this].Flag", "interval":"Hour", "intervalMultiplier":1}, 
       "valueMap":{"9":"Valid", "19":"Missing"}}
      

      
      

Grafana Variables

Using Tags and the Ad Hoc Filter Variables

To use tags and ad hoc filters in Grafana, a Grafana variable of the type "Ad hoc filters" must be defined (for each data source). This can be done in the Dashboard Settings - Variables:

An AdHocFilters control is then displayed on the dashboard, where the list of available tags and, next to the comparison operator, the list of available tag values (i.e. distinct values from all time series attribute assignments) are listed:

Note: AdHocFilters are applied to all existing data. If time series searches defined in Additional JSON Data are to be extended by further attributes selected in the AdHocFilter, the extendByAdhocFilter:true parameter in the according search section must be configured.

Using Interval Variable

In Grafana, intervals can be predefined to group time periods according to the selected interval across dashboards.

To use Grafana Interval variables a Grafana variable of type "Interval" with the name "Interval" (or with the name specified in HAKOM.Config "IntervalVariableName") must be defined. If another text is to be displayed instead of "Interval", this can be achieved by using the "Label" field:

The following interval identifiers are supported:

• ms: Milliseconds - Example: 1ms

• s: seconds - Example: 1s

• m: minute - example: 15m

• h: hour - example: 1h

• d: Days - Example: 1d

• w: weeks - Example: 1w

• M: months - Example: 1M

• q: Quarters - Example: 1q

• y: years - Example: 1y

After configuring the variable, it appears on the dashboard and can be used accordingly:

Note: Interval variable overrides the interval calculated by Grafana, but it does not override intervals configured in Additional JSON Data, unless it is explicitely enabled via overrirdeWithGreaterInterval: true.

Using Custom Variable to Simulate Interval Variable

In Grafana 7.x with SimPod 0.2.3, the Interval variable is currently not transmitted via the API. Instead, the variable of type Custom can be used: 

For this, a Grafana variable of type "Custom" with the name "Interval" (or with the name specified in HAKOM.Config "IntervalVariableName") must be defined. If another text is to be displayed instead of "Interval", this can be achieved with the help of the "Label" field:

In Cusom Options the same interval names as described above under Interval Variable can be used:

• ms: Milliseconds - Example: 1ms

• s: seconds - Example: 1s

• m: minute - example: 15m

• h: hour - example: 1h

• d: Days - Example: 1d

• w: weeks - Example: 1w

• M: months - Example: 1M

• q: Quarters - Example: 1q

• y: years - Example: 1y

Additionally, the option "auto" can be added (as shown above). This corresponds somewhat restricted to the "Auto Option" of the Interval variable, i.e. if this is selected, the interval suggested by Grafana depending on the selected period is used.

After configuring the variable, it appears on the dashboard and can be used accordingly:

Note: the variable overrides the interval calculated by Grafana, but it does not override intervals configured in Additional JSON Data, unless it is explicitely enabled via overrirdeWithGreaterInterval: true.

Using Text Box and Custom Variable Types in Additional JSON Data

To display additional variables on the dashboard that can be mapped to Additional JSON Data, it is also possible to use text box or custom variable types.

The variable name must currently be inserted between quotes as value in the according Additional JSON Data entry using following pattern: "$VariableName".

Example

To perform a time series search with name, description, etc. via Additional JSON Data entry "search", we create a variable for each time series search parameter:

• SearchName - Type: Text Box

• SearchType - Type: Text Box

• SearchInterval - Type: Text Box

• SearchDescription - Type: Text Box

• SearchUnit - Type: Text Box

• SearchAttributes - Type: Text Box

• SearchAdvancedAttributeFilter - Type: Text Box

• SearchextendByAdhocFilter - Type: Custom (without multiple selection), List: true,false

• ControlExactInterval - Type: Custom (without multiple selection), List: true,false

The variable names can be referenced in Additional JSON Data using quotation marks, each starting with a $ character:

{"exactInterval":"$ControlExactInterval",
 "search": {"name": "$SearchName", "type": "$SearchType", 
			"interval": "$SearchInterval","description": "$SearchDescription","unit": "$SearchUnit",
            "attributes": "$SearchAttributes","advancedAttributeFilter":"$SearchAdvancedAttributeFilter",
			"limit": 25,"offset": 0, "extendByAdhocFilter": "$SearchextendByAdhocFilter"}}

Similarly, a TreeNodeSum parameter (both in Standard and Trees Datasource) could also be filled:

{"exactInterval":"$ControlExactInterval",
 "treeNodeSum": {"timeSeriesName": "$SearchName", "attributes": "$SearchAttributes",
			"includeTimeSeries": true
			"limitTimeSeries": 25,"offsetTimeSeries": 0, "extendByAdhocFilter": "$SearchextendByAdhocFilter"}}

Note: for TreeNodeSum the search/filter function of WebTSM Services does not support the properties "interval", "description" and "unit".

Using the filters in the GUI:

• Text: can be entered as simple text (e.g.: time series name1)

• Integer: can be entered as simple text (e.g.: 25)

• Boolean: can be entered as simple text (e.g. true)

• Lists: can be entered as comma separated values (e.g. time series name1,time series name2) without space

• Attributes: see lists, attribute|value pairs can be entered using the pipe '|' separator (e.g.: attribute1|value1,attribute2|value2,attribute3)

• AdvancedAttributFilter: see lists, attribute+operator+value combinations can easily be written together (e.g. attribute1=value1,attribute2!=value3)

Using Query Variables in Additional JSON Data

Queries allow you to select one or more time series and use them in Additional JSON Data (best in the parameter "timeSeriesNames"). If required, the list can be filtered using regex patterns.

Defining a Query Variable:

Gereral:

• Name: any

• Type: Query

• Label: any

Query Options:

• Data source: one JSON Data Source

• Refresh: either "On Dashboard Load" or "On Time Range Change"

• Query: not supported / used yet

• Regex: to hide the Custom Query entry ("- Custom Query -") from the time series names query, use the following Regex pattern:

  /^(?!- Custom Query -$).*$/

  
  

Selection Options:

• Multi-Value: if several time series are to be selected simultaneously: yes (Attention, multiple selection variables require with JSON Datasource 0.1.7 the HAKOM.Config entry "UseJsonDataSourceQueryFix" = true being enabled, otherwise the interface cannot process the request)

• Include All option: due to a Grafana bug in some Grafana versions (e.g. 6.4.4) Multi-Value Lists can not be emptied from the dashboard again to no selection. As such clear selection leads to the first entry being selected. If Multi-Value is activated, this option should be activated as well, so that "All" is being displayed as first item, and as such the selection jumps to "All" when the list is emptied. Note: please also read "Custom all value" option below.

• Custom all value: In combination with "Multi-Value" and "Include All", you can define what the variable should transmit when "All" is selected. By default the complete time series list available from /Search path (i.e. the first 100 entries) is selected. This makes only limited sense. We recommend to use the "Custom all value" to solve the problem described issue regarding unable to empty multi-value list selections. To do this, enter a blank entry, i.e. " " (including quotation marks) here. Because HAKOM TSM does not allow time series with only one " " space, these are automatically filtered out in the search, so that a space entry (contrary to All) does not apply any filtering. If the user wants to list all entries, an empty query can be forced to list all time series with the search parameter "executeEmptySearch".

The query variable is best used in "timeSeriesNames" parameter:

{"exactInterval":"$ControlExactInterval","timeSeriesNames":"$Query",
 "search": {"name": "$SearchName", "type": "$SearchType", 
			"interval": "$SearchInterval","description": "$SearchDescription","unit": "$SearchUnit",
            "attributes": "$SearchAttributes","advancedAttributeFilter":"$SearchAdvancedAttributeFilter",
			"limit": 25,"offset": 0, "extendByAdhocFilter": "$SearchextendByAdhocFilter"}}