Kraag22

☰

Logstash pipeline with S3 output starts tens of minutes

Nov 27, 2019

What to do when your logstash pipelines starts tens of minutes

When your logstash pipelines starts tens of minutes, you can have these problems:

  • Not enought entropy - logstash doesn’t have enought random numbers
  • You have a lot S3 outputs (doesn’t matter if one bucket or many of them Github issue)

S3 output plugin

Try to limit number of outputs. If you want to save logs to different paths in you S3 bucket. You can use mutate filter and add path to event. And then use only one S3 output.

Second option is to disable validation and metrics, that helps to speed start time.

validate_credentials_on_root_bucket=false
enable_metrics=false