Send syslogs to remote endpoint
Example 1: Configure Rsyslog on Linux Ubuntu to forward syslogs to a remote server.
Add the following line to the syslog daemon configuration file /etc/rsyslog.conf (UDP protocol):
*.* @<OpenTelemetry Collector host IP>:54527
*.* - instructs the daemon to forward all messages to the specified OpenTelemetry Collector instance listening on port 54527 and <OpenTelemetry Collector host IP> needs to point to the IP address of OpenTelemetry Collector.
@@ - if you are using TCP, type two @ symbols
Example 2: Configuring the F5 BIG-IP system to log to a remote syslog server (11.x - 17.x)
Refer to F5 BIG-IP documentation for instructions regarding remote Syslog configuration.
Decorate syslogs with meaningful attributes
The Dynatrace software intelligence platform and its Davis AI engine depend on context-rich, high-quality data. You can provide the context for your data ingested via
Generic log ingest API that supports a set of keys and semantic attributes. You can also provide custom attributes that don't require indexing in Dynatrace Grail data lakehouse.
Apply syslog template to your f5 BIG_IP to always include host and IP
This option can be used as an alternative to OpenTelemetry collector receiver configurations described in Example 3.
modify syslog {
include "
options {
proto-template(t_isostamp);
long_hostnames(yes); use_dns(no); use_fqdn(no);
};
template t_isostamp {
template(\"$ISODATE $HOST 1xx.xx.xx.xx9 $MSGHDR$MSG\\n\");
};
destination remote_server {
tcp(\"1xx.xx.xx.xx1\"
port (54526)
template(t_isostamp)
persist-name(r1)
);
};
filter f_alllogs {
level (debug...emerg);
};
log {
source(s_syslog_pipe);
filter(f_alllogs);
destination(remote_server);
};
"
}
Example 3: OpenTelemetry Collector configuration:
- To successfully configure your Collector instance, you must configure each component (receiver, optional processor, and exporter) individually in a YAML file and enable them via pipelines. Follow below examples or Dynatrace documentation
Receiver configuration
receivers:
syslog/f5:
tcp:
listen_address: "0.0.0.0:54526"
protocol: rfc5424
operators:
- type: add
field: attributes.log.source
value: syslog
- type: add
field: attributes.dt.ip_addresses
value: "1xx.xx.xx.xx1"
- type: add
field: attributes.instance.name
value: "ip-1xx-xx-x-xx9.ec2.internal"
- type: add
field: attributes.device.type
value: "f5bigip"
syslog/host:
tcp:
listen_address: "0.0.0.0:54527"
protocol: rfc5424
operators:
- type: add
field: attributes.log.source
value: syslog
- type: add
field: attributes.device.type
value: "ubuntu-syslog"
Exporter configuration:
exporters:
logging:
verbosity: detailed
otlphttp/tenant_1:
endpoint: "https://{your-tenant}.live.dynatrace.com/api/v2/otlp"
headers:
Authorization: "Api-Token {your-api-token}"
Processor configuration
processors:
batch:
Service configuration
service:
pipelines:
logs:
receivers: [syslog/f5, syslog/host]
processors: [batch]
exporters: [logging, otlphttp/tenant_1]
- Dynatrace log processing - parse syslogs ingested to Dynatrace to improve readability
Example 4: Dynatrace processing: Syslog attributes extraction:
In the above configuration yaml file /etc/otelcol-contrib/config.yamlwe explicitly added `attributes.instance.name' attribute that can add additional configuration effort in case of change.
If syslog message contains instance name it can be extracted automatically.
In this example we assume hostname is always available and added after timestamp. Refer to F5 BIG-IP documentation for template instructions on how to include FQDN of the system by default to all syslogs.
Log sample
{
"content":"2023-10-18T06:55:01-07:00-07:00 ip-1xx-xx-x-xx9.ec2.internal 1xx.xx.x.xx9 info systemd[1]: Removed slice user-0.slice.",
"device.type": "f5bigip"
}
Add log processing rule
- In Dynatrace UI go to Settings>Log Monitoring>Processing and select Add rule
- Name a new rule
- Add matcher: device.type == "f5bigip"
- Add processor definition: PARSE(content,"NSPACE SPACE NSPACE:instance.name")
Test result:
{
"content": "2023-10-18T06:55:01-07:00-07:00 ip-1xx-xx-x-xx9.ec2.internal 1xx.xx.x.xx9 info systemd[1]: Removed slice user-0.slice.",
"device.type": "f5bigip",
"instance.name": "ip-1xx-xx-x-xx9.ec2.internal"
}
