Skip to main content

Cotool + AWS S3/SQS Logs

Ingest CloudTrail, GuardDuty, Security Hub, VPC flow, WAF, and other AWS logs into Cotool from S3 through a dedicated SQS queue and a cross-account role.

SIEM & Data Lake
Updated

The short answer

The AWS S3/SQS integration reads the log files AWS services already write to S3 and indexes their events into Cotool Logs, where agents search them and detections run on them. CloudTrail, GuardDuty, Security Hub, VPC flow logs, load balancer, CloudFront, WAF, S3 access, and Route 53 Resolver logs each get their own table.[1]

How it connects: Each new object is announced on an SQS queue dedicated to Cotool. Cotool assumes a role in your account that trusts it only with your external ID, receives the notification, reads the object, and deletes the message once the events are indexed.[1][5]

What Cotool uses from AWS S3/SQS Logs

CloudTrail events
Cotool unwraps CloudTrail's Records envelope into the aws_cloudtrail table and maps actor, action, service, resource, source IP, outcome, account, and region into columns shared with GCP audit logs. CloudTrail's own records carry fields such as eventName, userIdentity, and sourceIPAddress.[1][6]
GuardDuty and Security Hub findings
Exported GuardDuty findings and ASFF Security Hub findings become tables with finding type, severity, resource, actor, and remote IP. Each update of a finding is kept as its own version.[1][10]
Network and edge logs
VPC flow logs, Application Load Balancer, CloudFront, and WAF logs, and Route 53 Resolver query logs are parsed into per-request or per-flow rows.[1]
Generic JSON
Other JSON logs are stored with their original payload and top-level fields as a searchable map, without normalization or implied detection coverage.[1]

Detections, hunts, and triage

Detect risky CloudTrail activityDetection
Python detections run on the aws_cloudtrail table, for example on access key creation, logging changes, or console sign-ins without MFA, and every alert they raise is investigated by an agent.[1][3]
Triage a GuardDuty findingTriage
An agent reads the finding, pulls the same principal's CloudTrail activity around it, checks the remote IP, and records whether the activity was expected.[1][4]
Hunt across AWS and GCP audit logs at onceHunt
Because CloudTrail and GCP audit logs share actor, action, and resource columns, one hunt query can follow an identity across both clouds.[1]

Example query

Access keys created in the last daysql
SELECT event_time, actor, actor_type, ip_address, account_id, region, outcome
FROM aws_cloudtrail
WHERE action = 'CreateAccessKey'
  AND event_time > now() - INTERVAL 1 DAY
ORDER BY event_time DESC
LIMIT 100

Cotool Logs uses ClickHouse SQL. Check the column names in Log Search before reusing this, and keep a time filter and a LIMIT on exploratory queries.[1][2]

Setup and permissions

Cotool generates Terraform, an AWS CLI script, or a CloudFormation template that creates the queue, a dead-letter queue, and an IAM role named CotoolLogIngest-<suffix> that trusts the Cotool principal with your external ID. AWS designed the external ID to prevent the confused deputy problem.[1][5]

PermissionAccess
sqs:ReceiveMessageReads object-created notifications from the dedicated queue, along with GetQueueAttributes.[1][8]Read
sqs:DeleteMessageRemoves a message after its object's events are indexed, along with ChangeMessageVisibility for retries.[1][8]Write
s3:GetObjectReads the announced log objects, with s3:GetObjectVersion, on each bucket prefix you list.[1][9]Read
kms:DecryptOnly when objects are encrypted with customer-managed KMS keys.[1][10]Read
  • Use a queue dedicated to Cotool. Cotool deletes messages once their objects are ingested, so a shared queue would take messages from another SIEM. Fan out with SNS if another tool also reads the bucket's notifications.[1]
  • S3 publishes s3:ObjectCreated:* events to SQS queues, and only notifies queues in its own region, so deploy in the buckets' account and region.[1][7]

How Cotool works with AWS S3/SQS Logs

  • Before you enable a source, Cotool assumes the role, samples up to ten messages, and reads one announced object. Sampled messages go back on the queue.[1]
  • Redelivered notifications don't create duplicates: CloudTrail events are keyed by eventID, and findings by finding ID and update time.[1]
  • Temporary failures keep the message on the queue and retry with backoff of up to 15 minutes, for about a day with the template's dead-letter settings.[1]

FAQ

Can Cotool share an SQS queue with my current SIEM?
No. Cotool deletes messages after ingesting their objects, which would take them from the other consumer. Have the bucket notify an SNS topic and subscribe both queues to it.[1]
Which file formats does Cotool read from S3?
JSON, JSON arrays, newline-delimited JSON, AWS's text formats for flow, load balancer, CloudFront, and S3 access logs, and gzip-compressed versions of each. Parquet and ZIP objects are reported as errors on the source.[1]

Sources

  1. AWS S3/SQS Log Ingestion (Cotool docs) · accessed November 16, 2026
  2. Search Logs (Cotool docs) · accessed November 16, 2026
  3. Detections are code (Cotool Detect) · accessed November 16, 2026
  4. Respond to every signal (Cotool Respond) · accessed November 16, 2026
  5. Access to AWS accounts owned by third parties (AWS IAM docs) · accessed November 16, 2026
  6. CloudTrail record contents (AWS docs) · accessed November 16, 2026
  7. Enabling and configuring event notifications (Amazon S3 docs) · accessed November 16, 2026
  8. Amazon SQS API permissions reference (AWS docs) · accessed November 16, 2026
  9. Required permissions for Amazon S3 API operations (AWS docs) · accessed November 16, 2026
  10. Exporting GuardDuty findings to Amazon S3 (AWS docs) · accessed November 16, 2026

Related integrations

All integrations

Use cases with AWS S3/SQS Logs

Related reading

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo