Introduction

Apache Kafka is a distributed event streaming platform capable of handling high volumes of data and real-time streams. It is designed to be scalable, fault-tolerant, and highly available, making it suitable for building real-time data pipelines and streaming applications.

This integration supports real-time monitoring of data ingestion, detailed analysis of streaming metrics, and proactive issue detection, enabling organizations to manage data flows efficiently, ensure data integrity, and respond promptly to operational events in their distributed systems.
This section provides instructions on integrating Kafka application.

Configure custom integration

You can set up log configuration for Kafka from the INGESTION tab.

To set up log configuration:

  1. Click Kafka under default section.
    The LOG CONFIGURATION page is displayed.
    You can perform any of the following actions from the LOG CONFIGURATION page.
    • Default Configuration - This page provides the predefined configurations that describe how logs are collected from various sources and subsequently analyzed for insights and patterns.
    • +ADD - This page refers to the process of adding new settings or parameters to the default configuration based on requirements or preferences.

Default Configuration

  1. Select Default Configuration to make the changes to the default template.
    You can make changes to the template by providing values against the data provided in the template.
  kafka:
  type: 
  source: 
  include:
  parser_type: 
  regex: 
  multiline:
    line_start_pattern:
  timestamp:
    layout_type: 
    layout: 
kafka_gc:
  type: 
  source: 
  include:
  parser_type: 
  regex: 
  multiline:
    line_start_pattern: 
  timestamp:
    layout_type:
    layout:

See Default configuration attributes explained for detailed information on the supported attributes.

  1. By default, the Enable Configuration checkbox is checked. If you want to disable the log ingestion for the app, you can uncheck the Enable Configuration checkbox.
  2. Click SAVE once you have made the changes.
    The data is now saved, and logs will be ingested based on the configuration settings.

Add Configuration

  1. In the Add Custom Configuration page:
    • Enter the name for the configuration.
    • Select the priority value from the Priority field. The priorities are assigned with a numerical value, the configuration with the highest value will be considered first.
    • To assign resources to the log configuration, click FILTER.
      The RESOURCES page is displayed.
    • You can filter the resources using LOGQL attributes. Once you select the resources, click DONE.
    • You can make changes to the template under LOG CONFIGURATION YAML by providing values against the data provided in the template.
      Note: See Default configuration for more details on the available configuration attributes.
  2. Click SAVE.
    The new log configurations have been added and will be displayed on the LOG CONFIGURATIONS page.