AWS Metadata
Last updated
Was this helpful?
Last updated
Was this helpful?
The AWS Filter enriches logs with AWS Metadata. The plugin adds the EC2 instance ID and availability zone to log records. To use this plugin, you must be running in EC2 and have the .
The plugin supports the following configuration parameters:
imds_version
Specify which version of the instance metadata service to use. Valid values are v1 and v2.
v2
az
true
ec2_instance_id
The EC2 instance ID.
true
ec2_instance_type
The EC2 instance type.
false
private_ip
The EC2 instance private IP.
false
ami_id
The EC2 instance image ID.
false
account_id
The account ID for the current EC2 instance.
false
hostname
The hostname for the current EC2 instance.
false
vpc_id
The VPC ID for the current EC2 instance.
false
tags_enabled
false
tags_include
Defines a list of specific EC2 tag keys to inject into the logs. Tag keys must be comma-separated (,). Tags not included in this list will be ignored. Example: Name,tag1,tag2.
none
tags_exclude
Defines a list of specific EC2 tag keys not to inject into the logs. Tag keys must be comma-separated (,). Tags not included in this list will be injected into the logs. If both tags_include and tags_exclude are specified, the configuration is invalid and the plugin fails. Example: Name,tag1,tag2
none
retry_interval_s
Defines minimum duration between retries for fetching EC2 instance tags.
300
If you run Fluent Bit in a container, you might need to use instance metadata v1. The plugin behaves the same regardless of which version is used.
Run Fluent Bit from the command line:
You should see results like this:
The following is an example of a configuration file:
EC2 Tags let you label and organize your EC2 instances by creating custom-defined key-value pairs. These tags are commonly used for resource management, cost allocation, and automation. Including them in the Fluent Bit-generated logs is almost essential.
To achieve this, AWS Filter can be configured with tags_enabled true to enable the tagging of logs with the relevant EC2 instance tags. This setup ensures that logs are appropriately tagged, making it easier to manage and analyze them based on specific criteria.
tags_includeAssume the EC2 instance has many tags, some of which have lengthy values that are irrelevant to the logs you want to collect. Only two tags, department and project, are valuable for your purpose. The following configuration reflects this requirement:
If you run Fluent Bit logs might look like the following:
tags_excludeSuppose the EC2 instance has three tags: Name:fluent-bit-docs-example, project:fluentbit, and department:it. In this example, the department tag is redundant and will be excluded. All of the projects belong to the it department, and you don't want to waste storage space on redundant labels.
Here is an example configuration that achieves this:
The resulting logs might look like this:
The , such as us-east-1a.
Specifies whether to attach EC2 instance tags. The EC2 instance must have the option enabled, which is disabled by default.
To use the tags_enabled true feature in Fluent Bit, the option must be enabled on the EC2 instance where Fluent Bit is running. Without this option enabled, Fluent Bit won't be able to retrieve the tags associated with the EC2 instance. However, this doesn't mean that Fluent Bit will fail or stop working altogether. Instead, if option isn't enabled, Fluent Bit will continue to operate normally and capture other values, such as the EC2 instance ID or availability zone, based on its configuration.