Hi users!
We have released v1.17.0 on 2024-04-30. ChangeLog is here.
This release is a new release of v1.17 series. In this release, we added some new features for some plugins and fixed bugs of Parser.
Enhancement
in_tail: Add glob_policy option for expanding glob capability of path and exclude_path
In this release, we added a new option glob_policy for in_tail plugin.
In previous versions, we can use only * in glob patterns for path and exclude_path option.
Example:
path /path/to/*
exclude_path ["/path/to/*.gz", "/path/to/*.zip"]
From this version, we can also use [], ?, and {} in glob patterns depending on the glob_policy option.
Example:
path "[0-1][2-3].log"
glob_policy extended
Please see the document and #4401 for more information.
out_http: Support AWS Signature Version 4 authentication
In this release, we added a new option aws_sigv4 for the method setting of out_http plugin.
By using this option, out_http can use AWS Signature Version 4.
For example, this allows out_http to write to Amazon OpenSearch Ingestion.
Please see the document and #4459 for more information.
out_http: Add option to reuse connections
In this release, we add a new option reuse_connections for out_http plugin.
This option will improve throughput of out_http.
Please see the document and #4330 for more information.
in_http: Recognize CSP reports as JSON data
In this release, we make the data type of the request where the Content-Type is application/csp-report be considered JSON by default.
Now, in_http can receive Content Security Policy's report by default.
Please see #4282 for more information.
Bug Fixes
Make sure parser returns hash
The record data in an event of Fluentd must be a hash object.
However, in the previous versions, some parser plugins could return a non-hash object, such as an array. It could cause errors in subsequent processing.
In this release, the following parser plugins have been fixed.
The changes are as follows:
- Make sure to return a hash record.
- Make sure to accept only a hash or an array of hash.
Here are the details for each case.
Example of changed behavior
Config:
<source>
@type tcp
tag test.tcp
<parse>
@type json
</parse>
</source>
<match test.**>
@type stdout
</match>
Send an array data:
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
The result before this version:
{datetime} test.tcp: [{"k":"v"},{"k2":"v2"}]
The result after this version:
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
Example of resolved error
Config:
<source>
@type tcp
tag test.tcp
<parse>
@type json
null_empty_string
</parse>
</source>
<match test.**>
@type stdout
</match>
Send an array data:
$ netcat 0.0.0.0 5170
[{"k":"v"}, {"k2":"v2"}]
The result before this version:
{datetime} [error]: #0 unexpected error on reading data host="xxx" port=xxx error_class=NoMethodError error="undefined method `each_key' for [{\"k\"=>\"v\"}, {\"k2\"=>\"v2\"}]:Array"
The result after this version:
{datetime} test.tcp: {"k":"v"}
{datetime} test.tcp: {"k2":"v2"}
Remaining problem: filter_parser
In the previous versions, filter_parser could return an array record based on this wrong behavior.
From this release, it can not return multiple parsed results anymore and Fluentd outputs a warning log in this case.
This behavior should improve in the future.
Here is an example.
<source>
@type sample
tag test.array
sample {"message": "[{\"k\":\"v\"}, {\"k2\":\"v2\"}]"}
</source>
<filter test.**>
@type parser
key_name message
<parse>
@type json
</parse>
</filter>
<match test.**>
@type stdout
</match>
The result before this version:
{datetime} test.array: [{"k":"v"},{"k2":"v2"}]
The result after this version:
{datetime} [warn]: #0 dump an error event: error_class=Fluent::Plugin::Parser::ParserError error="Could not emit the event. The parser returned multiple results, but currently filter_parser plugin only returns the first parsed result. Raw data: '[{\"k\":\"v\"}, {\"k2\":\"v2\"}]'" location=nil tag="test.array" time=xxx record={"k2"=>"v2"}
{datetime} test.array: {"k":"v"}
These are the major changes for this release.
In addition, some performance improvements have been included! Please see ChangeLog for details!
Enjoy logging!
