Skip to main content
Dithered dot illustration of an antique wooden ice box with brass hinges, one door slightly open, dots scattering off one edge

What security log data should you store, and where?

Store identity, EDR, cloud control plane, DNS, VPN, and SaaS audit logs where detections can read them fast; query the rest in place. Retention rules included.

Guide
Updated 12 min read

Summary

Store the logs your detections read, which are identity provider, EDR, cloud control plane, DNS, VPN, and SaaS admin audit logs, in fast, searchable storage, and keep the rest in cheaper storage you can query where it lives. Retention floors come from your regulators: PCI DSS 10.5.1 asks for 12 months with the latest three immediately available, and OMB M-21-31 sets 12 months active plus 18 months cold storage for federal agencies. Decide the tier for each source by how often something reads it, not by how important it sounds.

Key takeaways

  1. Tier each source by use: if a scheduled detection or anomaly baseline reads it, keep it hot; if people only read it during an investigation, federate it; if only an auditor reads it, archive it.
  2. Make the hot window at least as long as your longest detection lookback or baseline, and no shorter than any "immediately available" rule you fall under, such as the three months in PCI DSS 10.5.1.
  3. Export identity and SaaS audit logs before vendor defaults expire them: 7 days on Entra ID Free, 90 days in Okta, 180 days in Audit (Standard).
  4. Land everything else in object storage you own, in an open columnar format partitioned by date, so federated queries scan less and you can change tools later.
  5. Alert when a source goes quiet; a stopped pipeline looks the same as a quiet network.

The short version

Most teams decide what to store by asking which logs matter. A more useful question is which logs get read, by what, and how often. A detection that runs every 15 minutes over 30 days of Okta events needs that data indexed, cheap to keep, and fast to aggregate. A firewall allow log that someone opens twice a year during an investigation doesn't. Paying hot-storage prices for both is where most security logging budgets go.

That gives three tiers:

  • Hot (detection tier). Anything a scheduled detection, correlation, or anomaly baseline reads. It needs low query latency, analytical queries (counts, baselines, joins), and a price low enough that you don't drop sources to stay in budget.
  • Federated. Data that stays where it already lives, in an existing SIEM, a data lake, an object store, or a tool's own API, and gets queried when an investigation or a low-frequency detection needs it.
  • Cold archive. Data kept to satisfy a retention rule, restored when an auditor or a forensic investigation asks for it.

The joint 2025 guidance from Australia's ASD's ACSC, CISA, NSA, and partners says the same thing in policy language: the agencies "strongly discourage using a SIEM as the central repository for all logs," and a SIEM should centralize only specific security logs according to your risk profile.

Which sources matter for detection

Start from the sources that catch the intrusions you actually see. CISA's implementation guidance for M-21-31 ranks them for agencies that can't collect everything at once: identity, credential, and privileged access events first; then operating system events such as process creation, PowerShell, and scheduled task changes; then network infrastructure (DNS, DHCP, firewall); then cloud logs, including any use of break-glass accounts, AWS CloudTrail, Azure AD and Azure activity logs, the Microsoft 365 unified audit log, and GCP admin audit. The 2025 ASD-led guidance orders its list loosely, starting with EDR logs and network devices, then domain controllers, Windows endpoints, and virtualization, with cloud platforms eighth.

SourceWhy detections need itSuggested tier
Identity provider (Okta, Entra ID)Account takeover, MFA fatigue, new admin grantsHot
EDR telemetryProcess trees, persistence, command linesHot for event types your rules read
Cloud control plane (CloudTrail management events, Azure Activity, GCP Admin Activity)IAM changes, new access keys, logging disabledHot
Domain controller security eventsKerberos abuse, group changes, log clearing (1102)Hot
DNS and DHCPC2 lookups, mapping IPs to hostsHot if rules use it; otherwise federated
VPN and remote accessImpossible travel, session hijackingHot
SaaS admin audit (Microsoft 365, Google Workspace, GitHub)OAuth grants, external sharing, repo visibility changesHot for admin and sharing events
Firewall allow logs, VPC flow logs, proxy logsEgress volume, beaconing, investigation contextFederated, with hot aggregates
CloudTrail data events (S3 object reads)Exfiltration scopingFederated
HR system (Workday, Rippling)Leavers who still have accessFederated through the API
Full packet capture, debug and application logsForensics, rarely detectionCold

Two adjustments apply to almost everyone. First, high-volume sources are where the money goes: the ASD guidance warns that firewall or DNS volume "may overshadow the importance of the information received" and suggests querying them for timing anomalies or matching them against threat intelligence rather than storing them whole. Second, cloud logs often aren't on by default. CloudTrail trails log management events but not data events, network activity events, or Insights events until you add them.

How long to keep it

Retention has two parts: how long data must exist, and how much of it must be searchable right now. The rules you're most likely to meet treat them separately.

PCI DSS v4.0.1. Requirement 10.5.1 says to "retain audit log history for at least 12 months, with at least the most recent three months immediately available for analysis." The standard's examples of immediately available include storing logs online, archiving them, or restoring them quickly from backups, so a fast, tested restore counts. Requirement 10.4.1 adds daily review of security events and logs from critical systems, and 10.4.1.1, required since 31 March 2025, says that review must use automated mechanisms.

OMB M-21-31 (US federal agencies). The memo defines four event logging tiers, EL0 (not effective) through EL3 (advanced), and gives each log category a criticality from 0 (most useful for detection) to 3. Appendix C sets a minimum of 12 months in active storage and 18 months in cold storage for most categories, and 72 hours for full packet capture. The National Archives schedule that backs it combines those into 30 months, and notes one shorter exception: a cloud log type with 24 months total. Even if you aren't an agency, the memo's tables are a detailed list of which fields each log should carry.

HIPAA. The audit controls standard in 45 CFR 164.312(b) requires mechanisms that record and examine activity in systems holding electronic protected health information, but it sets no retention period. The six-year rule people quote is in 164.316 and covers policies, procedures, and documentation.

Your own dwell time. Retention also has to cover how long attackers go unnoticed. Mandiant's M-Trends 2026 reports a global median dwell time of 14 days for 2025 intrusions, but 122 days for cyber espionage and North Korean IT worker cases. If your searchable window is 30 days, the slow intrusions will start before it.

Vendor defaults are much shorter than any of these, so the clock usually starts at the source:

SourceDefault retention at the source
AWS CloudTrail event history90 days of management events
Okta System Log90-day sliding window
Microsoft Entra ID audit and sign-in logs7 days (Free), 30 days (P1, P2)
Microsoft 365 audit, Audit (Standard)180 days
Microsoft 365 audit, Audit (Premium)1 year for Exchange, SharePoint, OneDrive, Entra; up to 10 years with add-on
Google Workspace log eventsGenerally 6 months
GitHub Enterprise Cloud audit log180 days; Git events 7 days

Hot, federated, or cold: matching storage to use

The tiers differ less in where the data sits than in what can read it. Microsoft Sentinel and Splunk both offer a cheap tier for search and keep frequent detection on the expensive one.

Microsoft Sentinel's billing makes the gap concrete. As of November 2026, the analytics tier lists at $4.30 per GB on pay-as-you-go in East US. The data lake tier charges $0.05 per GB for ingestion plus $0.10 per GB for processing, $0.026 per GB per month for storage (billed at 6:1 compression), and $0.005 per GB scanned by queries, and it can keep data for up to 12 years. Microsoft describes the analytics tier as the one that supports alerting and real-time processing; the lake tier is for long-term retention, KQL exploration, notebooks, and jobs that promote data back to analytics. The Microsoft Sentinel pricing guide walks through the meters.

Splunk's Federated Search for Amazon S3 draws the same line in its own documentation: it "is not intended for high frequency and real-time searches," and high-frequency searches "will likely incur higher costs than natively ingesting." It's licensed in Data Scan Units of 10 TB scanned each, it can't read objects in S3 Intelligent-Tiering or Glacier storage classes, and each search stops at 10 TB unless support raises the limit.

In both platforms, the documented path for frequent detection runs through the ingested tier, and federated or lake data is mainly something you search. That's a reasonable default, but it pulls into the hot tier every source that even one hourly detection reads.

Cotool is built around moving that line. Its SIEM FAQ puts the split plainly: "Store the sources that need fast search and long retention. Agents query and detect on the rest where it already lives." Logs stored in Cotool get the hot-tier job, with aggregations and baselines for anomaly detection and no per-seat or per-query fees. Logs can also stay in open formats in your own cloud bucket, which Cotool searches in place. The difference is on the federated side: a Cotool detection is Python that can run against logs in Cotool or call the tools where the data lives. Its published example, leaver_still_active.py, has no log source at all. Every hour it asks Rippling for terminated workers and Okta for active users, and alerts on anyone in both. The same federated queries reach an existing Splunk, a Snowflake lake, or the CrowdStrike API during an investigation.

For a storage plan, that means a source no longer has to be ingested just because one detection reads it. HR records, EDR lookups, and an existing SIEM you're winding down can stay where they are and still be covered, while identity, cloud control plane, and EDR telemetry that rules scan constantly go in the hot tier. Legacy SIEM vs modern SIEM covers the broader design change.

Step by step

  1. List every source with its daily volume. Note which ones have detections today, which feed dashboards, and which exist only for compliance.
  2. Record the source's own retention. Use the defaults table; anything with a 7- or 90-day window needs exporting before you lose it.
  3. Write down your retention floors. For each regulation you're under, record both numbers: total retention and how much must be immediately searchable.
  4. Tier each source by use. Read by a scheduled detection or baseline: hot. Read during investigations: federated. Read only by auditors: cold. Split sources when you can, such as VPN connections hot and firewall allow logs federated.
  5. Size the hot window. Make it at least as long as your longest detection lookback and baseline (a 30-day per-user baseline needs 30 days hot), and at least as long as any "immediately available" rule, such as three months under PCI DSS or 12 months active under M-21-31. Detection testing needs the same history. Cotool replays every new detection and every tune against your history before it deploys, and its example mfa_fatigue.py backtest covers 30 days, which only works if those 30 days are fast to query.
  6. Land the rest in storage you own. Write it to object storage in a columnar format such as Parquet, partitioned by source and date. Splunk's guidance for S3 federated search recommends date partitions and objects over 128 MB, because both cut the data each query scans. If several tools will read the lake, a shared schema such as OCSF saves rewriting each query per source.
  7. Test the slow path. Run a federated query and an archive restore against a real investigation question, and time both. PCI DSS 10.5.1 testing procedures ask assessors to observe that the latest three months are immediately available.
  8. Watch source health. Alert when a source's volume drops to zero. Cotool, for example, raises an alert when a source goes quiet, such as an Okta System Log with no events for 47 minutes.

Common mistakes

  • Treating vendor defaults as your retention. Entra ID Free keeps sign-ins for seven days. If nobody exports them, a 14-day median intrusion outlives its evidence.
  • Sending everything to the SIEM. It inflates the bill, and the agencies writing logging guidance advise against it. Keep the SIEM for what detections read.
  • Archiving everything cold. Twelve months in an archive with no tested restore fails the "immediately available" half of PCI DSS 10.5.1.
  • Federating what detections poll. A rule that queries federated data every five minutes pays scan costs every five minutes; Splunk says that pattern costs more than ingesting.
  • Archiving to storage your query tool can't read. Splunk's S3 federated search can't search Glacier or Intelligent-Tiering objects, so lifecycle rules can quietly break investigations.
  • Assuming cloud logging is complete. CloudTrail doesn't log data events by default, and GitHub keeps Git events for only seven days.
  • Dropping fields at ingest to save money. M-21-31 lists the minimum fields each event should carry, including source and destination IP, user, command executed, and a unique event ID. A detection written next year will need one you dropped.

Checklist

  • Inventory of sources with volume, owner, and the detections that read each one
  • Identity provider, EDR, cloud control plane, domain controller, VPN, and SaaS admin audit logs in the hot tier
  • CloudTrail data events and Insights enabled where you need them
  • Exports in place for every source with a default window under your retention floor
  • Hot window at least as long as the longest detection lookback and any "immediately available" rule
  • Federated and archived data in an open, date-partitioned format in storage you own
  • A timed federated query and a timed restore, with the results kept as evidence
  • Volume alerts on every source
  • A review whenever a new detection reads a source that isn't hot, so you decide to promote it or query it in place

FAQ

How long should security logs be retained?
Security logs should be retained at least as long as your strictest applicable rule and your longest investigation lookback. PCI DSS 10.5.1 requires 12 months with the most recent three months immediately available, and OMB M-21-31 requires federal agencies to keep most log types 12 months in active storage and 18 months in cold storage. Mandiant's M-Trends 2026 put global median dwell time at 14 days, but 122 days for espionage and North Korean IT worker intrusions, which argues for keeping months, not weeks, of searchable history.
What logs should a SIEM collect first?
A SIEM should collect identity and privileged access logs, endpoint and EDR telemetry, cloud control plane logs, and DNS, DHCP, and firewall logs first. CISA's M-21-31 implementation guidance lists identity and credential management events first, then operating system events, then network infrastructure, then cloud logs such as AWS CloudTrail, Azure activity, the Microsoft 365 unified audit log, and GCP admin audit. The 2025 joint guidance led by ASD's ACSC starts its list with EDR logs.
Should you send all logs to your SIEM?
Sending all logs to your SIEM is discouraged by the agencies that publish logging guidance. The 2025 priority logs guidance from ASD's ACSC, CISA, NSA, and partners says they strongly discourage using a SIEM as the central repository for all logs and that a SIEM should centralize only specific security logs according to the organization's risk profile. Keep the rest in a log store or data lake and query it when you need it.
What is the difference between hot and cold log storage?
Hot log storage keeps data indexed and immediately searchable for detections and investigations, while cold log storage keeps it at lower cost with slower or restore-based access. OMB M-21-31 defines active storage as data kept for frequent use and ease of access, and cold storage as storage that minimizes cost while still allowing some access. PCI DSS counts quick restores from backup or archive as immediately available, but you have to be able to show it.
Does HIPAA require six years of log retention?
HIPAA's six-year rule applies to required policies, procedures, and documentation under 45 CFR 164.316, not explicitly to audit logs. The audit controls standard in 45 CFR 164.312(b) requires mechanisms that record and examine activity in systems with electronic protected health information but sets no retention period. Set an audit log retention period in your own policy and confirm it with counsel.

Sources

  1. Security Log Storage and Federated Search for Agents - Cotool · accessed November 16, 2026
  2. AI-Native SIEM Built for Agents - Cotool · accessed November 16, 2026
  3. AI Detection Engineering with Python Detections - Cotool · accessed November 16, 2026
  4. Security Integrations: SIEM, EDR, Identity, Cloud - Cotool · accessed November 16, 2026
  5. OMB M-21-31: Improving the Federal Government's Investigative and Remediation Capabilities Related to Cybersecurity Incidents · accessed November 16, 2026
  6. CISA: Guidance for Implementing M-21-31 · accessed November 16, 2026
  7. Frequently Asked Questions about GRS 3.2, Information Systems Security Records - National Archives · accessed November 16, 2026
  8. Payment Card Industry Data Security Standard v4.0.1 (PCI SSC) · accessed November 16, 2026
  9. Priority logs for SIEM ingestion: Practitioner guidance (ASD's ACSC, CISA, NSA and partners) · accessed November 16, 2026
  10. Best Practices for Event Logging and Threat Detection - CISA · accessed November 16, 2026
  11. M-Trends 2026: Data, Insights, and Strategies From the Frontlines - Google Cloud Blog · accessed November 16, 2026
  12. 45 CFR 164.316 - Policies and procedures and documentation requirements · accessed November 16, 2026
  13. 45 CFR 164.312 - Technical safeguards · accessed November 16, 2026
  14. Working with CloudTrail event history - AWS CloudTrail · accessed November 16, 2026
  15. CloudTrail concepts - AWS CloudTrail · accessed November 16, 2026
  16. Monitor Okta - Okta Developer · accessed November 16, 2026
  17. Microsoft Entra data retention · accessed November 16, 2026
  18. Manage audit log retention policies - Microsoft Purview · accessed November 16, 2026
  19. Data retention and lag times - Google Workspace Admin Help · accessed November 16, 2026
  20. Audit log for an enterprise - GitHub Enterprise Cloud Docs · accessed November 16, 2026
  21. Plan costs and understand pricing and billing - Microsoft Sentinel · accessed November 16, 2026
  22. Microsoft Sentinel data lake overview · accessed November 16, 2026
  23. Azure Retail Prices API: Microsoft Sentinel, East US · accessed November 16, 2026
  24. Federated Search for Amazon S3 - Splunk Validated Architectures · accessed November 16, 2026
  25. About Federated Search for Amazon S3 - Splunk Cloud Platform · accessed November 16, 2026

Keep reading

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo