What is a security data lake?
A security data lake keeps security logs in open formats in object storage, so many tools query one copy. How it works, what it costs, and how it differs from a SIEM.
Definition
Security data lake
Also called Security lake, Cybersecurity data lake, SIEM data lake
A security data lake is a store of security logs and telemetry kept in low-cost object storage in open, columnar file formats, usually under a shared schema such as OCSF, so that several tools can query one copy of the data without each ingesting its own. It is built for long retention and investigation; scheduled detection and alerting usually run in a SIEM or another engine that reads from it.
How it works
A security data lake is four layers, and each is a separate choice: where the bytes sit, how they're encoded, how tables are described, and what reads them. Because storage and compute are separate, one copy of the data can serve several engines. Microsoft lists "separation of storage and compute" and a "single copy of data" as core to the Microsoft Sentinel data lake's design.
| Layer | What it does | Amazon Security Lake | Microsoft Sentinel data lake |
|---|---|---|---|
| Storage | Holds the objects | S3 buckets in your AWS account | Managed by Microsoft on Azure |
| File format | Compresses and columnizes records | Apache Parquet | Apache Parquet |
| Schema and tables | Gives fields shared names and types | OCSF; Iceberg tables; Glue Data Catalog | Sentinel tables |
| Query engines | Read the data | Athena, Redshift, subscribers | KQL, Jupyter notebooks, jobs |
Storage is object storage you can keep for years. Amazon Security Lake is backed by S3 buckets in your own account, one per Region, and AWS says you retain ownership of the data.
File format decides what a query costs. Parquet is columnar, so a query reads only the columns it touches. AWS's Athena pricing example takes 3 TB of raw logs, compresses them 3:1 into Parquet, and shows a single-column query scanning 0.25 TB instead of 3 TB.
Schema is what lets one query cover many sources. Amazon Security Lake converts natively supported AWS sources to OCSF and requires custom sources to arrive already in OCSF and Parquet. It creates a Lake Formation table per source in the AWS Glue Data Catalog and uses the Apache Iceberg table format.
Query engines are whatever you point at the tables. Security Lake subscribers either get notified of new objects or query the tables directly with Amazon Athena or Amazon Redshift.
Security data lake vs SIEM
A SIEM indexes data so detections can run against it all the time; a security data lake stores data cheaply so people and scheduled jobs can query it when they need to. The vendors that sell both put the line in the same place.
Microsoft Sentinel's analytics tier is the one "supporting advanced hunting, alerting, and incident management." Its data lake tier is for "long-term storage for querying and Python-based advanced analytics," with retention up to 12 years, and data reaches alerting by jobs that promote it back to the analytics tier. Splunk says Federated Search for Amazon S3 "is not intended for high frequency and real-time searches," and that high-frequency searches "will likely incur higher costs than natively ingesting."
In both designs, a source that even one frequent detection reads has to be ingested into the indexed tier, and the lake keeps what's left over. Cotool draws the line elsewhere. Its SIEM can detect on a source without ingesting it: detections are Python that runs against logs stored in Cotool or against the systems where data already lives, and log data can stay in open formats in your own cloud bucket, which Cotool searches in place. Logs stored in Cotool get aggregations and baselines for anomaly detection, with no per-seat or per-query fees. What to ingest then depends on how fast a source needs to be searched, not on whether a detection reads it.
Snowflake, Databricks, and ClickHouse
A security data lake can also run on a data platform the company already uses for analytics. Snowflake, Databricks, and ClickHouse now each sell a security layer on top of theirs, and they differ most in where detection runs.
Snowflake pitches its AI Data Cloud as a place to keep security logs next to enterprise data, with security applications deployed in your account "without moving your data." Data sits in native tables, billed at a flat rate per compressed TB, or in Apache Iceberg tables in S3, Google Cloud Storage, or Azure Storage that you manage. Queries run on virtual warehouses that bill credits per second with a 60-second minimum, whether or not a running warehouse is doing work. Detection comes from the applications you deploy, not from Snowflake itself.
Databricks announced Lakewatch, an agentic SIEM on its lakehouse, on March 24, 2026; its docs list it as Beta, and it needs a Databricks workspace on AWS, Azure, or GCP. Lakewatch ingests from cloud storage, from SaaS API connectors, or from an existing Unity Catalog Delta table, which it parses without copying. Detection rules are SQL that run on a SQL warehouse on a schedule of at least every 5 minutes, and matches become signals that Lakewatch groups into alerts.
ClickHouse is a columnar database whose Cloud service lists storage at $25.30 per TB per month of compressed data, with compute billed per unit-hour (one unit is 8 GiB of RAM and 2 vCPUs). It can query Iceberg tables in S3 directly. ClickHouse Security, built on RunReveal, entered private preview on November 9, 2026. It runs on your own ClickHouse Cloud service with scheduled SQL and streaming Sigma detections, and it's free during the preview apart from standard Cloud compute and storage.
Cotool works alongside these platforms rather than replacing them: its federated search lists a Snowflake data lake among the sources its agents query in place, so security data kept in Snowflake can stay there.
Why it matters for a SOC
Retention. AWS pitches Security Lake as a way to keep logs that were "previously unfeasible to store" and to extend storage beyond a SIEM's retention policy. Microsoft's data lake tier holds data for up to 12 years. Which sources deserve the lake and which deserve the indexed tier is covered in what security logs to store, and where.
Ownership. With Security Lake the objects are in your account in an open schema, so several tools can read them in parallel, each limited to the sources you grant (up to 10 per subscriber).
Query cost. Lakes are cheap to fill and keep, and billed when you read. Athena charges $5 per TB scanned, so AWS's 0.25 TB single-column query costs $1.25. Run that once during an investigation and it's nothing. Schedule it hourly as a detection and it's 720 runs and $900 a month for one rule. Microsoft's lake tier works the same way, at $0.005 per GB of uncompressed data scanned by KQL queries and jobs.
That last point is why detection placement matters as much as storage. A baseline detection that compares each user's day with the last 30 reads a month of history on every run. Cotool's personal_cloud_exfil.py is that shape: it computes a 30-day rolling median of uploads to personal cloud storage for each user from proxy logs, flags days more than 20 times the user's normal, and asks Rippling for each flagged user's last day. Cotool runs detections like it on its own stored logs, while sources read only occasionally stay in the lake.
AI agents raise the stakes on both cost and schema. An agent investigating an alert may issue dozens of queries, and each one is a scan. It also needs consistent field names to write them. Cotool exposes the same logs over an API and an MCP server, so cotool.query_logs("okta", days=30) in a notebook, or Claude Code calling search_logs, reads the data its own agents read.
What a lake leaves to you
A security data lake stores and serves data; the rest is assembly.
- Detection, alerting, and cases. Security Lake hands data to subscribers by notification or query access; the analytics come from whatever subscribes.
- The surrounding bill. Security Lake orchestrates AWS Glue, Amazon EventBridge, AWS Lambda, Amazon SQS, and Amazon SNS, each billed separately, plus S3 storage and your query costs. It requires a CloudTrail organization trail; in AWS's second pricing example, creating one alongside existing account trails adds $3,739.84 a month in CloudTrail charges.
- Schema work. Custom sources must arrive as OCSF Parquet, and Security Lake's version 2 AWS sources are labeled OCSF 1.1.0, so you track schema versions.
- File layout. Splunk recommends date partitions and merging objects under 128 MB, because both cut the data a search scans.
Cotool can take on the ingest and normalization side. Its S3 and SQS ingest has built-in parsers for AWS services and a "Custom format, normalized to OCSF" option for other sources. Its logging sinks copy what it ingests to Amazon S3, Google Cloud Storage, Azure Blob Storage, or S3-compatible stores, as Normalized Parquet or JSON Lines, or as Raw records exactly as the source sent them.
Examples
Seven ways to run security data in a lake, as of November 2026:
| Implementation | Where data lives | How it's queried | Published pricing |
|---|---|---|---|
| Amazon Security Lake | S3 in your AWS account | Athena, Redshift, subscribers | $0.25–$0.75/GB ingest, $0.035/GB normalization, plus S3 |
| Microsoft Sentinel data lake | Microsoft-managed, Parquet | KQL, notebooks, promotion jobs | $0.05/GB ingest, $0.10/GB processing, $0.026/GB-month |
| Self-built on S3 with Athena | S3 in your AWS account | Athena SQL over Glue tables | $5 per TB scanned, plus S3 |
| Snowflake | Native tables or Iceberg in your storage | Warehouse SQL, security apps | Credits per second; storage per TB |
| Databricks Lakewatch (Beta) | Unity Catalog Delta tables | Scheduled SQL rules, Genie | Not published for Lakewatch |
| ClickHouse Cloud | ClickHouse storage; reads Iceberg in S3 | SQL, ClickHouse Security (preview) | $25.30/TB-month; compute per unit-hour |
| Cotool with your own storage | Your cloud bucket, open formats | Agents, Python detections, API, MCP | Custom quote |
An Amazon Security Lake bill. AWS's first pricing example is one month in US East (N. Virginia): 512 GB of CloudTrail logs at $0.75 per GB is $384.00, 1,024 GB of other AWS logs at $0.25 per GB is $256.00, and 1,536 GB of normalization at $0.035 per GB is $53.76, for $693.76. S3 storage, the orchestrated services, and queries come on top.
Cotool searching your bucket. Cotool's published example is a bucket, s3://acme-security-logs, in the customer's AWS account, with okta/, crowdstrike-fdr/, and cloudtrail/ prefixes in Parquet, searched in place with nothing copied out. In Cotool's federated search example, one agent question covers firewall and VPC flow logs stored in Cotool, a Snowflake data lake, an existing Splunk, and the CrowdStrike API.
Related terms
- SIEM: the system that indexes security data for detection, alerting, and case management; often the tier that sits in front of a lake.
- OCSF: the open schema Amazon Security Lake stores data in, so one query can read events from many products.
- Log normalization: mapping each source's fields to a common schema, done at ingest for most lakes.
- Federated search: querying data where it lives instead of ingesting it first.
- Apache Iceberg: an open table format over Parquet files, used by Amazon Security Lake and readable by Snowflake and ClickHouse.
- Delta Lake: a table format that stores data as Parquet files; Lakewatch can parse existing Unity Catalog Delta tables in place.
- Modern SIEM: a SIEM design that separates storage from detection; see legacy SIEM vs modern SIEM.
FAQ
- What is the difference between a security data lake and a SIEM?
- A SIEM indexes security data so detections and alerts can run against it continuously, while a security data lake stores the data cheaply in open formats for long retention and on-demand queries. Microsoft Sentinel draws that line inside one product: its analytics tier supports alerting and incident management, and its data lake tier holds data for up to 12 years for querying and notebooks. Many teams run both, with the lake holding the history and the high-volume sources.
- What is Amazon Security Lake?
- Amazon Security Lake is AWS's managed security data lake service. It stores data in Amazon S3 buckets in your own AWS account, one per Region, converts natively supported AWS sources to OCSF in Apache Parquet, and lets subscribers either receive new-object notifications or query the tables with services such as Amazon Athena. Custom sources must already be written as OCSF in Parquet.
- How much does a security data lake cost?
- A security data lake's cost has three parts: getting data in, storing it, and scanning it at query time. As of November 2026, Amazon Security Lake charges $0.75 per GB for CloudTrail logs, $0.25 per GB for other AWS logs, and $0.035 per GB for normalization, plus S3 storage and query charges, and Amazon Athena charges $5 per TB scanned. Microsoft Sentinel's data lake tier lists $0.05 per GB for ingestion, $0.10 per GB for processing, $0.026 per GB per month for storage, and $0.005 per GB scanned in East US.
- Can you run detections on a security data lake?
- You can run detections on a security data lake, but most lake designs bill every query by the data it scans, so a detection that runs every hour pays for its scan every hour. Splunk says Federated Search for Amazon S3 is not intended for high-frequency searches and that such searches will likely cost more than ingesting. Teams usually run frequent detections on indexed data and use the lake for hunting, investigations, and scheduled jobs. Cotool detections are Python that can run against logs stored in Cotool or query systems where data already lives, including logs kept in open formats in your own cloud bucket.
- Can Snowflake, Databricks, or ClickHouse be a security data lake?
- Snowflake, Databricks, and ClickHouse can each be a security data lake, and each now sells a security layer on top. Snowflake positions its AI Data Cloud for security data and runs security applications inside your account. Databricks offers Lakewatch, an agentic SIEM on the lakehouse that is in Beta as of November 2026. ClickHouse offers ClickHouse Security, built on RunReveal, which entered private preview on November 9, 2026.
- What file format does a security data lake use?
- A security data lake usually stores data as Apache Parquet, a compressed columnar format, so queries read only the columns they need. Amazon Security Lake and Microsoft Sentinel data lake both store Parquet, and Amazon Security Lake also uses the Apache Iceberg table format and the OCSF schema. AWS's Athena pricing example shows a single-column query scanning 0.25 TB of Parquet instead of 3 TB of raw text.
Sources
- What is Amazon Security Lake? - Amazon Security Lake · accessed November 18, 2026
- Amazon Security Lake Pricing · accessed November 18, 2026
- Open Cybersecurity Schema Framework (OCSF) in Security Lake - Amazon Security Lake · accessed November 18, 2026
- Subscriber management in Security Lake - Amazon Security Lake · accessed November 18, 2026
- Amazon Athena Pricing · accessed November 18, 2026
- Microsoft Sentinel data lake overview · accessed November 18, 2026
- Plan costs and understand pricing and billing - Microsoft Sentinel · accessed November 18, 2026
- Azure Retail Prices API: Microsoft Sentinel, East US · accessed November 18, 2026
- Federated Search for Amazon S3 - Splunk Validated Architectures · accessed November 18, 2026
- AI Data Cloud for Cybersecurity | Snowflake · accessed November 18, 2026
- Apache Iceberg tables | Snowflake Documentation · accessed November 18, 2026
- Introduction to external tables | Snowflake Documentation · accessed November 18, 2026
- Understanding compute cost | Snowflake Documentation · accessed November 18, 2026
- Understanding storage cost | Snowflake Documentation · accessed November 18, 2026
- Databricks Lakewatch: the agentic SIEM for security teams · accessed November 18, 2026
- Databricks Enters Security Market with Launch of Lakewatch: New, Agentic SIEM · accessed November 18, 2026
- Introduction to Lakewatch | Databricks on AWS · accessed November 18, 2026
- Create and manage datasources | Databricks on AWS · accessed November 18, 2026
- Create and manage detection rules | Databricks on AWS · accessed November 18, 2026
- Introducing ClickHouse Security in private preview, built on RunReveal · accessed November 18, 2026
- Real-time security analytics on ClickHouse · accessed November 18, 2026
- Pricing | ClickHouse · accessed November 18, 2026
- Iceberg table engine - ClickHouse Documentation · accessed November 18, 2026
- AI-Native SIEM Built for Agents - Cotool · accessed November 18, 2026
- Security Log Storage and Federated Search for Agents - Cotool · accessed November 18, 2026
- AI Detection Engineering with Python Detections - Cotool · accessed November 18, 2026
- Export Logs to Your Storage - Cotool docs · accessed November 18, 2026
- AWS S3/SQS Log Ingestion - Cotool docs · accessed November 18, 2026





