file

file

file source用于日志采集。

Example

sources:
- type: file
  name: accesslog

Tips

如果你使用logconfig/clusterlogconfig采集容器日志，file source里还增加了额外的字段，请参考这里。

paths

Example

需要采集的目标文件：

/tmp/loggie/service/order/access.log
/tmp/loggie/service/order/access.log.2022-04-11
/tmp/loggie/service/pay/access.log
/tmp/loggie/service/pay/access.log.2022-04-11

对应配置：

sources:
- type: file
  paths:
  - /tmp/loggie/**/access.log{,.[2-9][0-9][0-9][0-9]-[01][0-9]-[0123][0-9]}

excludeFiles

Example

sources:
- type: file
  paths:
  - /tmp/*.log
  excludeFiles:
  - \.gz$

ignoreOlder

addonMeta

event示例

{
  "body": "this is test",
  "state": {
    "pipeline": "local",
    "source": "demo",
    "filename": "/var/log/a.log",
    "timestamp": "2006-01-02T15:04:05.000Z",
    "offset": 1024,
    "bytes": 4096,
    "hostname": "node-1"
  }
}

state含义解释：

• pipeline: 所在的pipeline名称
• source: 所在的source名称
• filename: 采集的文件名称
• timestamp: 采集时刻的时间戳
• offset: 采集的数据在文件的offset偏移量
• bytes: 采集的数据字节数
• hostname: 所在节点名称

multi

多行采集相关配置

Example

sources:
- type: file
  name: accesslog
  multi:
    active: true
    pattern: '^\d{4}-\d{2}-\d{2}'

active

pattern

Example

假设有多行日志如下所示：

2023-05-11 14:30:15 ERROR Exception in thread "main" java.lang.NullPointerException
     at com.example.MyClass.myMethod(MyClass.java:25)
     at com.example.MyClass.main(MyClass.java:10)

配置pattern正则：^\d{4}-\d{2}-\d{2}
会将日志变成一行。这样在日志查询的时候，不会产生上面的多行异常日志堆栈乱序等问题。

maxLines

maxBytes

timeout

readFromTail

cleanFiles

清理文件相关配置。过期且已经采集完成的文件将会直接从磁盘删除以释放磁盘空间

maxHistoryDays

cleanUnfinished

fdHoldTimeoutWhenInactive

fdHoldTimeoutWhenRemove

workerCount

readBufferSize

maxContinueRead

maxContinueReadTimeout

inactiveTimeout

ignoreSymlink

firstNBytesForIdentifier

charset

编码转换，用于将不同的编码转换为utf8，当下支持的编码转换格式.

Example

    sources:
      - type: file
        name: demo
        paths:
          - /tmp/log/*.log
        fields:
          topic: "loggie"
        charset: "gbk"
``` `字段`| `类型`| `是否必填`| `默认值`| `含义`  
---|---|---|---|---  
charset| string| 否| utf-8| 提取字段的匹配模型  

当前支持的转换为utf-8的编码格式有

  * `nop`
  * `plain`
  * `utf-8`
  * `gbk`
  * `big5`
  * `euc-jp`
  * `iso2022-jp`
  * `shift-jis`
  * `euc-kr`
  * `iso8859-6e`
  * `iso8859-6i`
  * `iso8859-8e`
  * `iso8859-8i`
  * `iso8859-1`
  * `iso8859-2`
  * `iso8859-3`
  * `iso8859-4`
  * `iso8859-5`
  * `iso8859-6`
  * `iso8859-7`
  * `iso8859-8`
  * `iso8859-9`
  * `iso8859-10`
  * `iso8859-13`
  * `iso8859-14`
  * `iso8859-15`
  * `iso8859-16`
  * `cp437`
  * `cp850`
  * `cp852`
  * `cp855`
  * `cp858`
  * `cp860`
  * `cp862`
  * `cp863`
  * `cp865`
  * `cp866`
  * `ebcdic-037`
  * `ebcdic-1040`
  * `ebcdic-1047`
  * `koi8r`
  * `koi8u`
  * `macintosh`
  * `macintosh-cyrillic`
  * `windows1250`
  * `windows1251`
  * `windows1252`
  * `windows1253`
  * `windows1254`
  * `windows1255`
  * `windows1256`
  * `windows1257`
  * `windows1258`
  * `windows874`
  * `utf-16be-bom`
  * `utf-16le-bom`

## lineDelimiter

换行符相关配置

Example

sources: - type: file name: demo lineDelimiter: type: carriage_return_line_feed value: "\r\n" charset: gbk

### type

`字段`| `类型`| `是否必填`| `默认值`| `含义`  
---|---|---|---|---  
type| bool| 非必填| auto| 只有在type是custome时候value才会有效  

当前支持的type有

  * `auto`
  * `line_feed`
  * `vertical_tab`
  * `form_feed`
  * `carriage_return`
  * `carriage_return_line_feed`
  * `next_line`
  * `line_separator`
  * `paragraph_separator`
  * `null_terminator`

对应的换行符为：

auto: {'\u000A'}, line_feed: {'\u000A'}, vertical_tab: {'\u000B'}, form_feed: {'\u000C'}, carriage_return: {'\u000D'}, carriage_return_line_feed: []byte("\u000D\u000A"), next_line: {'\u0085'}, line_separator: []byte("\u2028"), paragraph_separator: []byte("\u2029"), null_terminator: {'\u0000'},

``` ```

### value

<table><thead><tr><th><code>字段</code></th><th><code>类型</code></th><th><code>是否必填</code></th><th><code>默认值</code></th><th><code>含义</code></th></tr></thead><tbody><tr><td>value</td><td>string</td><td>非必填</td><td>\n</td><td>换行符的内容</td></tr></tbody></table>

### charset

<table><thead><tr><th><code>字段</code></th><th><code>类型</code></th><th><code>是否必填</code></th><th><code>默认值</code></th><th><code>含义</code></th></tr></thead><tbody><tr><td>charset</td><td>string</td><td>非必填</td><td>utf-8</td><td>换行符编码</td></tr></tbody></table>

## ack

source的确认机制相关配置。如果需确保要`at least once`，需要开启ack机制，但是会有一定性能顺耗

Caution

该配置只能配置在defaults中

Example

defaults: sources:

- type: file
  ack:
    enable: true
``` ```

### enable

<table><thead><tr><th><code>字段</code></th><th><code>类型</code></th><th><code>是否必填</code></th><th><code>默认值</code></th><th><code>含义</code></th></tr></thead><tbody><tr><td>enable</td><td>bool</td><td>非必填</td><td>true</td><td>是否开启确认机制</td></tr></tbody></table>

### maintenanceInterval

<table><thead><tr><th><code>字段</code></th><th><code>类型</code></th><th><code>是否必填</code></th><th><code>默认值</code></th><th><code>含义</code></th></tr></thead><tbody><tr><td>maintenanceInterval</td><td>time.Duration</td><td>非必填</td><td>20h</td><td>维护周期。用来定时清理过期的确认文件数据(例如不再采集的文件的ack信息)</td></tr></tbody></table>

## watcher

监控文件变化的相关配置

Caution

该配置只能配置在defaults中

Example

defaults: sources:

- type: file
  watcher:
    maxOpenFds: 8000

```

scanTimeInterval

maintenanceInterval

maxOpenFds

maxEofCount

cleanWhenRemoved

taskStopTimeout

db

v1.5版本及以后已删除，请使用全局配置的db替换。