As of version 1.7.4, osquery can log results directly to Amazon AWS Kinesis Streams and Kinesis Firehose. For users of these services, osqueryd can eliminate the need for a separate log forwarding daemon running in your deployments.

Configuration

The Kinesis Streams and Kinesis Firehose logger plugins are named aws_kinesis and aws_firehose respectively. They can be enabled as with other logger plugins using the config flag logger_plugin.

Some configuration is shared between the two plugins:

  1. --aws_access_key_id VALUE               AWS access key ID override
  2. --aws_profile_name VALUE                AWS config profile to use for auth and region config
  3. --aws_region VALUE                      AWS region override
  4. --aws_secret_access_key VALUE           AWS secret access key override
  5. --aws_sts_arn_role VALUE                AWS STS assume role ARN
  6. --aws_sts_region VALUE                  AWS STS assume role region
  7. --aws_sts_session_name VALUE            AWS STS session name
  8. --aws_sts_timeout VALUE                 AWS STS temporary credential timeout period in seconds (900-3600)
  9. --aws_enable_proxy VALUE                Enable proxying of HTTP/HTTPS requests in AWS client config (true or false)
  10. --aws_proxy_scheme VALUE                Proxy HTTP scheme for use in AWS client config (http or https)
  11. --aws_proxy_host VALUE                  Proxy host for use in AWS client config
  12. --aws_proxy_port VALUE                  Proxy port for use in AWS client config
  13. --aws_proxy_username VALUE              Proxy username for use in AWS client config
  14. --aws_proxy_password VALUE              Proxy password for use in AWS client config

When working with AWS, osquery will look for credentials and region configuration in the following order:

  • Configuration flags
  • Profile from the AWS config files (only if —aws_profile_name is specified)
  • Environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
  • default profile in the AWS config files
  • Profile from the EC2 Instance Metadata ServiceAll of the STS configuration flags are optional. However, if aws_sts_arn_role is set, you can utilize temporary credentials via assume role with the AWS Security Token Service.

Kinesis Streams

When logging to Kinesis Streams, the stream name must be specified with aws_kinesis_stream, and the log flushing period can be configured with aws_kinesis_period.

Setting aws_kinesis_random_partition_key to true will use random partition keys when sending data to Kinesis. Using random values will load balance over stream shards if you are using multiple shards in a stream. Note that using this setting will result in the logs of each host distributed across shards, so do not use it if you need logs from each host to be processed by a consistent shard. The default for this setting is "false".

Kinesis Firehose

Similarly for Kinesis Firehose delivery streams, the stream name must be specified with aws_firehose_stream, and the period can be configured with aws_firehose_period.

Sample Config File

  1. {
  2.   "options": {
  3.     "host_identifier": "hostname",
  4.     "schedule_splay_percent": 10,
  5.     "logger_plugin": "aws_kinesis,aws_firehose",
  6.     "aws_kinesis_stream": "foo_stream",
  7.     "aws_firehose_stream": "bar_delivery_stream",
  8.     "aws_access_key_id": "ACCESS_KEY",
  9.     "aws_secret_access_key": "SECRET_KEY",
  10.     "aws_region": "us-east-1"
  11.   },
  12.   "schedule": {
  13.     "time": {
  14.       "query": "SELECT * FROM time;",
  15.       "interval": 2,
  16.       "removed": false
  17.     }
  18.   }
  19. }

Note: Kinesis services have a maximum 1MB record size. Result logs bigger than this will not be forwarded by osqueryd as they will be rejected by the Kinesis services.