Logfileimport per Logimporter

The QuantumCast Logimporter is a dedicated software tool, distributed as executable binary, that can be used to transmit log data in a secure and GDPR safe way to the QuantumCast big data platform. It can be used to send a log file at once or (preferred) to continously send new log entries in realtime by “tailing” files.

The latest version can be downloaded here:

All binaries are built static and have no further system requirements and are used as a command line tool.

Logimporter supports several import formats, log format types and output types. The configuration is provided using command line values and flags (or environment variables).

The configuration is divided into 3 parts:

  • input: defines what input type should be used v

  • parser: defines what log file type is provided and wich parser should be used

  • output: defines output of the parsed data and specifies an origin as identifier that you get from QuantumCast

The usage in general:

./logimporter input:file –path="./logs/access.log" parser:icecast output:amqp –origin=xxx


The value xxx in the examples as in -origin=xxx has to be substituted by the value provided by QuantumCast.

The -help flag shows a help note and can be used for every sub-command (input, parser, output)

./logimporter -help

Input Plugins for Logfiles

The input plugin is configured by the prefix “input:” followed by the wanted input type.

One of the following input types can be used:

input:file –path=/path/to/file

The provided file will be read, parsed and sent to QuantumCast. After this, the programs exits. You need to provide a path to an existing file. It is possible to use Gzip compressed files with the suffix .gz.

input:fileglob –path=/path/to/files*.log

All files that match the provided glob pattern are read, parsed and sent to QuantumCast. After all files are processed, the program exists. It is possible to use Gzip compressed files with the suffix .gz.

input:tail –path=/path/to/file

Continuously reading of new data in a file. This works like the “tail” command and keeps reading new lines until the program is stopped manually. It supports log rotation if the new file gets the same name. This is the preferred way to use in production environments to get realtime logs.

Additional flags for input:tail:

-whence=end or start

Start reading the file from end or beginning/start of the file. Default is end.


Opens a HTTP server and binds to to provided address and port that can be used for health checks. In this example a call to either returns a status 200 if everything is ok or a higher status if something does not work.


User polling instead of fsnotify for getting notified of new lines. Preferred is the default fsnotify.


Can be used to read the field definitions in W3C compatible AIS log files.


The Healthcheck can be use to monitor the input.

Parser Plugins for Streaming-Server Logs

The desired parser can be configured with the prefix “parser:”. You need the right parser for the log file type you use.


Parser for Icecast logs.


AIS 7 session log format.


AIS 8 and above session format. It also tries to guess the right fields by reading the file header. Please note that the AIS session log has to be used, not the access log.

Output Plugins for data transmission

The output plugin is configured usingthe prefix “output:”.

The available plugins are the following:


Logs are sent as AMQP messages to a QuantumCast message queue. This is the default and should only be changed if requested. You need to open port 5672 for outgoing TCP.

Additional Flags for output:amqp:


Mandantory field. The value will be provided by QuantumCast and is used to assign incoming log data to the right customer.


Optional. Should only be used if requested by QuantumCast.


Test output “dry run”. No data is sent and it can be used to check if everything is working.

Additional flags for output:noop:


Output parsed log data to stdout.


Send data using HTTPS to an ingest endpoint. Can be used if AMQP is not possible due to port restrictions and is requested by QuantumCast. Please note that this is less reliable as AMQP and can lead to loss of some log data under certain circumstances.

Additional flags for output:http:


Mandantory field. The value will be provided by QuantumCast and is used to assign incoming log data to the right customer.


Optional. Should only be used if requested by QuantumCast.

If “tail” mode is used the program runs indefinitely. In this case it is adviced to use a start-up script that manages the process. For instance on modern Linux systems you can use a systemd unit file.

Example for a SystemD unit file

Description=QuantumCast Logimporter

ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/logimporter-linux-amd64 input:tail –path=/var/log/icecast/access.log parser:icecast output:amqp –origin=xxx


The following parameters need to be changed and adapted to your needs:

  • Path to the program itself (here /usr/local/bin/logimporter-linux-amd64)

  • Path to the log file (here /var/log/icecast/access.log)

  • User (here icecast)

Examples for starting the Logimporter

./logimporter input:tail -path=/path/to/logfile parser:ais output:amqp -origin=xxx

./logimporter input:file -path=/path/to/logfile parser:ais output:amqp -origin=xxx

./logimporter input:fileglob -path=/path/to/logfiles* parser:ais output:amqp -origin=xxx

Re-delivery of missing logs

If some logs have not beend sent in tail mode because of errors or network issues, it is possible to use the normal file or fileglob mode to re-delivery the missing data. Since version v3.1.0 there are new flags to restrict the logs to a specific time range: -after="YYYY-MM-DD HH:mm:ss and -before="YYYY-MM-DD HH:mm:ss can be provided to restrict to data before and after the given dates.

The flags can be used individually or in combination. Both are global flags and need to go before the input defintion.

If you know the exakt time when for instance an error occured and data is missing, you can use -after to send all logs after this date regardless of the timespan the log covers.

If you know the point in time since when everything worked fine again, use this time with -before.

The full call could be something like this:

logimporter -after="2022-03-04 19:15:25" input:file –path="./logs/access-2022-03-04.log.gz" parser:icecast output:amqp –origin=xxx


To start the HTTP server and enable /health endpoint, use the -listen flag.


-listen= starts the HTTP server and connects it to IP and port 8080. can be used to call the health check. It returns HTTP status 200 if everything is OK. If not enough new log lines were processed via input plugins for logfiles in the interval “-alertInterval 300” (default 300s), status 500 is returned.

If the IP is omitted for -listen, the health check uses all IPs that are configured -listen :8080