Install and configure Fluentd

Fluentd is a data collection and processing tool that supports multiple input sources, data formats, and output destinations. Fluentd streams and consolidates logs from different applications into one unified format and sends the data to Gcore Managed Logging.

Install Fluentd

Download Fluentd from the official Fluentd website and install it using your preferred method.

Configure Fluentd

A standard Fluentd configuration consists of three sections: input, filter, and output. The input and filter sections depend on the sources of logs.

To send logs to Gcore Managed Logging, configure Fluentd with the Kafka Integration Plugin in your Fluentd installation.

1. Configure Fluentd with Kafka output by adding the following data to the fluent.conf file:

<match pattern>
  @type kafka2


  # list of seed brokers
  brokers laas-example.gcore.com:443
  username: username
  password: password
  scram_mechanism: "sha512"
  default_topic: namespace_username.topic_name
  sasl_over_ssl: true


  # buffer settings
  <buffer>
    @type memory
    ## OPTIONAL PARAMS
    timekey: 30s
    timekey_wait: 10s
    timekey_use_utc: true
    chunk_limit_size: 4MB
    total_limit_size: 64MB
    chunk_full_threshold: "0.9"
    queued_chunks_limit_size: 12
    flush_thread_count: 12
  </buffer>


  # data type settings
  <format>
    @type json
  </format>
</match>

2. Replace the placeholder values:

• brokers: Kafka servers to which logs will be exported

• username: Username from the Managed Logging page in the Gcore Customer Portal

• password: Password from the Managed Logging page

• default_topic: Username from the Managed Logging page and topic name, separated by a dot (.)

Username, password, and topic name are available in the Gcore Customer Portal on the Managed Logging page. For configuration details, see Configure logging and view logs.

To explore more settings, check out the official Fluentd documentation.

Descriptions of the 'output' parameters

• brokers: Kafka servers to which Fluentd logs will be sent.
• scram_mechanism: Authentication mechanism used to verify the username and password for the logs storage.
• default_topic: Kafka topic for log export, specified as username.topic_name (username and topic name from the Managed Logging page, separated by a dot).
• sasl_over_ssl: Enables SASL for authentication and SSL for secure communication when set to true.

3. Save the changes in the Fluentd configuration file.

4. Restart Fluentd. Fluentd will then start sending logs to Gcore Managed Logging.