Logfileimport per Logimporter¶

The QuantumCast Logimporter is a dedicated software tool, distributed as executable binary, that can be used to transmit log data in a secure and GDPR safe way to the QuantumCast big data platform. It can be used to send a log file at once or (preferred) to continously send new log entries in realtime by “tailing” files.

The latest version can be downloaded here:

Linux 64Bit: https://streamabc-sw.s3.eu-central-1.amazonaws.com/logimporter/logimporter-linux-amd64
Windows 64Bit: https://streamabc-sw.s3.eu-central-1.amazonaws.com/logimporter/logimporter.exe
MacOS Intel 64Bit: https://streamabc-sw.s3.eu-central-1.amazonaws.com/logimporter/logimporter-mac

All binaries are built static and have no further system requirements and are used as a command line tool.

Logimporter supports several import formats, log format types and output types. The configuration is provided using command line values and flags (or environment variables).

The configuration is divided into 3 parts:

input: defines what input type should be used v
parser: defines what log file type is provided and wich parser should be used
output: defines output of the parsed data and specifies an origin as identifier that you get from QuantumCast

The usage in general:

./logimporter input:file –path="./logs/access.log" parser:icecast output:amqp –origin=xxx

Hint

The value xxx in the examples as in -origin=xxx has to be substituted by the value provided by QuantumCast.

The -help flag shows a help note and can be used for every sub-command (input, parser, output)

./logimporter -help

Input Plugins for Logfiles¶

The input plugin is configured by the prefix “input:” followed by the wanted input type.

One of the following input types can be used:

input:file –path=/path/to/file

The provided file will be read, parsed and sent to QuantumCast. After this, the programs exits. You need to provide a path to an existing file. It is possible to use Gzip compressed files with the suffix .gz.

input:fileglob –path=/path/to/files*.log

All files that match the provided glob pattern are read, parsed and sent to QuantumCast. After all files are processed, the program exists. It is possible to use Gzip compressed files with the suffix .gz.

input:tail –path=/path/to/file

Continuously reading of new data in a file. This works like the “tail” command and keeps reading new lines until the program is stopped manually. It supports log rotation if the new file gets the same name. This is the preferred way to use in production environments to get realtime logs.

Additional flags for input:tail:
-whence=end or start
Start reading the file from end or beginning/start of the file. Default is end.
-listen=127.0.0.1:8008
Opens a HTTP server and binds to to provided address and port that can be used for health checks. In this example a call to http://127.0.0.1:8008 either returns a status 200 if everything is ok or a higher status if something does not work.
-polling
User polling instead of fsnotify for getting notified of new lines. Preferred is the default fsnotify.
-scanheader
Can be used to read the field definitions in W3C compatible AIS log files.

Hint

The Healthcheck can be use to monitor the input.

Parser Plugins for Streaming-Server Logs¶

The desired parser can be configured with the prefix “parser:”. You need the right parser for the log file type you use.

parser:icecast

Parser for Icecast logs.

parser:ais

AIS 7 session log format.

-version=ais8

AIS 8 and above session format. It also tries to guess the right fields by reading the file header. Please note that the AIS session log has to be used, not the access log.

Output Plugins for data transmission¶

The output plugin is configured usingthe prefix “output:”.

The available plugins are the following:

output:amqp

Logs are sent as AMQP messages to a QuantumCast message queue. This is the default and should only be changed if requested. You need to open port 5672 for outgoing TCP.

Additional Flags for output:amqp:
-origin=xx
Mandantory field. The value will be provided by QuantumCast and is used to assign incoming log data to the right customer.
-streamwatch
Optional. Should only be used if requested by QuantumCast.

output:noop

Test output “dry run”. No data is sent and it can be used to check if everything is working.

Additional flags for output:noop:
-output
Output parsed log data to stdout.

output:http

Send data using HTTPS to an ingest endpoint. Can be used if AMQP is not possible due to port restrictions and is requested by QuantumCast. Please note that this is less reliable as AMQP and can lead to loss of some log data under certain circumstances.

Additional flags for output:http:
-origin=xx
Mandantory field. The value will be provided by QuantumCast and is used to assign incoming log data to the right customer.
-streamwatch
Optional. Should only be used if requested by QuantumCast.

If “tail” mode is used the program runs indefinitely. In this case it is adviced to use a start-up script that manages the process. For instance on modern Linux systems you can use a systemd unit file.

Example for a SystemD unit file

[Unit]
Description=QuantumCast Logimporter
Wants=network-online.target
After=network-online.target

[Service]
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/logimporter-linux-amd64 input:tail –path=/var/log/icecast/access.log parser:icecast output:amqp –origin=xxx
User=icecast
KillMode=process
KillSignal=SIGINT
LimitNOFILE=infinity
LimitNPROC=infinity
Restart=on-failure
RestartSec=2
StartLimitBurst=3
StartLimitIntervalSec=10
TasksMax=infinity

[Install]
WantedBy=multi-user.target

The following parameters need to be changed and adapted to your needs:

Path to the program itself (here /usr/local/bin/logimporter-linux-amd64)
Path to the log file (here /var/log/icecast/access.log)
User (here icecast)

Examples for starting the Logimporter

./logimporter input:tail -path=/path/to/logfile parser:ais output:amqp -origin=xxx

./logimporter input:file -path=/path/to/logfile parser:ais output:amqp -origin=xxx

./logimporter input:fileglob -path=/path/to/logfiles* parser:ais output:amqp -origin=xxx

Re-delivery of missing logs¶

If some logs have not beend sent in tail mode because of errors or network issues, it is possible to use the normal file or fileglob mode to re-delivery the missing data. Since version v3.1.0 there are new flags to restrict the logs to a specific time range: -after="YYYY-MM-DD HH:mm:ss and -before="YYYY-MM-DD HH:mm:ss can be provided to restrict to data before and after the given dates.

The flags can be used individually or in combination. Both are global flags and need to go before the input defintion.

If you know the exakt time when for instance an error occured and data is missing, you can use -after to send all logs after this date regardless of the timespan the log covers.

If you know the point in time since when everything worked fine again, use this time with -before.

The full call could be something like this:

logimporter -after="2022-03-04 19:15:25" input:file –path="./logs/access-2022-03-04.log.gz" parser:icecast output:amqp –origin=xxx

Healthcheck¶

To start the HTTP server and enable /health endpoint, use the -listen flag.

Example:

-listen=127.0.0.1:8080 starts the HTTP server and connects it to IP 127.0.0.1 and port 8080. http://127.0.0.1:8080/health can be used to call the health check. It returns HTTP status 200 if everything is OK. If not enough new log lines were processed via input plugins for logfiles in the interval “-alertInterval 300” (default 300s), status 500 is returned.

If the IP is omitted for -listen, the health check uses all IPs that are configured -listen :8080