Cotool + AWS S3/SQS Logs
Ingest CloudTrail, GuardDuty, Security Hub, VPC flow, WAF, and other AWS logs into Cotool from S3 through a dedicated SQS queue and a cross-account role.
The short answer
The AWS S3/SQS integration reads the log files AWS services already write to S3 and indexes their events into Cotool Logs, where agents search them and detections run on them. CloudTrail, GuardDuty, Security Hub, VPC flow logs, load balancer, CloudFront, WAF, S3 access, and Route 53 Resolver logs each get their own table.[1]
How it connects: Each new object is announced on an SQS queue dedicated to Cotool. Cotool assumes a role in your account that trusts it only with your external ID, receives the notification, reads the object, and deletes the message once the events are indexed.[1][5]
What Cotool uses from AWS S3/SQS Logs
- CloudTrail events
- Cotool unwraps CloudTrail's
Recordsenvelope into theaws_cloudtrailtable and maps actor, action, service, resource, source IP, outcome, account, and region into columns shared with GCP audit logs. CloudTrail's own records carry fields such aseventName,userIdentity, andsourceIPAddress.[1][6] - GuardDuty and Security Hub findings
- Exported GuardDuty findings and ASFF Security Hub findings become tables with finding type, severity, resource, actor, and remote IP. Each update of a finding is kept as its own version.[1][10]
- Network and edge logs
- VPC flow logs, Application Load Balancer, CloudFront, and WAF logs, and Route 53 Resolver query logs are parsed into per-request or per-flow rows.[1]
- Generic JSON
- Other JSON logs are stored with their original payload and top-level fields as a searchable map, without normalization or implied detection coverage.[1]
Detections, hunts, and triage
- Detect risky CloudTrail activityDetection
- Python detections run on the
aws_cloudtrailtable, for example on access key creation, logging changes, or console sign-ins without MFA, and every alert they raise is investigated by an agent.[1][3] - Triage a GuardDuty findingTriage
- An agent reads the finding, pulls the same principal's CloudTrail activity around it, checks the remote IP, and records whether the activity was expected.[1][4]
- Hunt across AWS and GCP audit logs at onceHunt
- Because CloudTrail and GCP audit logs share actor, action, and resource columns, one hunt query can follow an identity across both clouds.[1]
Example query
SELECT event_time, actor, actor_type, ip_address, account_id, region, outcome
FROM aws_cloudtrail
WHERE action = 'CreateAccessKey'
AND event_time > now() - INTERVAL 1 DAY
ORDER BY event_time DESC
LIMIT 100Cotool Logs uses ClickHouse SQL. Check the column names in Log Search before reusing this, and keep a time filter and a LIMIT on exploratory queries.[1][2]
Setup and permissions
Cotool generates Terraform, an AWS CLI script, or a CloudFormation template that creates the queue, a dead-letter queue, and an IAM role named CotoolLogIngest-<suffix> that trusts the Cotool principal with your external ID. AWS designed the external ID to prevent the confused deputy problem.[1][5]
| Permission | Access | Why |
|---|---|---|
sqs:ReceiveMessageReads object-created notifications from the dedicated queue, along with GetQueueAttributes.[1][8] | Read | Reads object-created notifications from the dedicated queue, along with GetQueueAttributes.[1][8] |
sqs:DeleteMessageRemoves a message after its object's events are indexed, along with ChangeMessageVisibility for retries.[1][8] | Write | Removes a message after its object's events are indexed, along with ChangeMessageVisibility for retries.[1][8] |
s3:GetObjectReads the announced log objects, with s3:GetObjectVersion, on each bucket prefix you list.[1][9] | Read | Reads the announced log objects, with s3:GetObjectVersion, on each bucket prefix you list.[1][9] |
kms:DecryptOnly when objects are encrypted with customer-managed KMS keys.[1][10] | Read | Only when objects are encrypted with customer-managed KMS keys.[1][10] |
- Use a queue dedicated to Cotool. Cotool deletes messages once their objects are ingested, so a shared queue would take messages from another SIEM. Fan out with SNS if another tool also reads the bucket's notifications.[1]
- S3 publishes
s3:ObjectCreated:*events to SQS queues, and only notifies queues in its own region, so deploy in the buckets' account and region.[1][7]
How Cotool works with AWS S3/SQS Logs
- Before you enable a source, Cotool assumes the role, samples up to ten messages, and reads one announced object. Sampled messages go back on the queue.[1]
- Redelivered notifications don't create duplicates: CloudTrail events are keyed by
eventID, and findings by finding ID and update time.[1] - Temporary failures keep the message on the queue and retry with backoff of up to 15 minutes, for about a day with the template's dead-letter settings.[1]
FAQ
- Can Cotool share an SQS queue with my current SIEM?
- No. Cotool deletes messages after ingesting their objects, which would take them from the other consumer. Have the bucket notify an SNS topic and subscribe both queues to it.[1]
- Which file formats does Cotool read from S3?
- JSON, JSON arrays, newline-delimited JSON, AWS's text formats for flow, load balancer, CloudFront, and S3 access logs, and gzip-compressed versions of each. Parquet and ZIP objects are reported as errors on the source.[1]
Sources
- AWS S3/SQS Log Ingestion (Cotool docs) · accessed November 16, 2026
- Search Logs (Cotool docs) · accessed November 16, 2026
- Detections are code (Cotool Detect) · accessed November 16, 2026
- Respond to every signal (Cotool Respond) · accessed November 16, 2026
- Access to AWS accounts owned by third parties (AWS IAM docs) · accessed November 16, 2026
- CloudTrail record contents (AWS docs) · accessed November 16, 2026
- Enabling and configuring event notifications (Amazon S3 docs) · accessed November 16, 2026
- Amazon SQS API permissions reference (AWS docs) · accessed November 16, 2026
- Required permissions for Amazon S3 API operations (AWS docs) · accessed November 16, 2026
- Exporting GuardDuty findings to Amazon S3 (AWS docs) · accessed November 16, 2026

