Skip to main content

Cotool + Databricks

Connect Databricks to Cotool so agents discover Unity Catalog tables and run read-only SQL over your security lakehouse for triage, hunts, and detections.

SIEM & Data Lake
Updated

The short answer

The Databricks integration lets Cotool agents treat your lakehouse as a security data source: they browse Unity Catalog catalogs, schemas, and tables, inspect columns and sample rows, and run read-only Databricks SQL through a SQL warehouse for investigations, hunts, and code detections.[1]

How it connects: Cotool calls the SQL Statement Execution and Unity Catalog REST APIs as a service principal with OAuth machine-to-machine credentials. A personal access token works as a fallback. Databricks SQL has no read-only session mode, so the service principal's grants keep Cotool read-only.[1][6][7]

What Cotool uses from Databricks

Unity Catalog metadata
Agents list catalogs, then schemas, then tables, and describe only the tables they need. The metadata APIs don't wake the SQL warehouse; sampling rows does.[1]
SQL statements
Queries run through the Statement Execution API on a SQL warehouse, not an all-purpose cluster. Cotool returns 500 rows by default and at most 10,000, with a 4 MiB result cap and a truncated flag when more rows matched.[1][7]
Partition and clustering columns
For log tables, agents see the event time column and the partition or clustering columns, and are told to filter on them first so scans stay small.[1]
Legacy Hive metastore tables
Tables in hive_metastore also work. Listing them uses SHOW statements, which briefly start the warehouse.[1]

Detections, hunts, and triage

Hunt across lakehouse log tablesHunt
An agent starts from Cotool's map of the workspace, picks the tables that hold the relevant log source, and runs time-bounded SQL. Table and column comments such as "AWS CloudTrail management events" help it pick the right table.[1][4]
Run Python detections on lakehouse dataDetection
Cotool detections are Python and can query data where it lives, so a detection can read a Databricks table on a schedule without copying it into another SIEM.[1][3]
Pull history for an alertTriage
A response agent looks up the user, host, or IP from an alert across months of lakehouse history and attaches the rows it used to its verdict.[1][5]

Example query

Failed Okta sign-ins from a lakehouse tablesql
SELECT actor.alternateId AS user,
       client.ipAddress AS ip,
       COUNT(*) AS failures
FROM security.okta.system_log
WHERE event_date >= current_date() - INTERVAL 1 DAY
  AND outcome.result = 'FAILURE'
GROUP BY 1, 2
ORDER BY failures DESC
LIMIT 100

The table name is an example. The first filter is on the partition column, which is what Cotool tells agents to do on large log tables. Replace it with your own catalog, schema, and date column.[1]

Setup and permissions

Create a service principal, give it the Databricks SQL access entitlement, and generate an OAuth secret on it with a lifetime of up to 730 days. Databricks shows the secret value once. On Azure Databricks, the secret must come from Databricks, not from an Entra ID app registration.[1][6]

PermissionAccess
Databricks SQL accessEntitlement that grants access to Databricks SQL features, including SQL warehouses. Without it the principal can't use any warehouse.[1][9]Read
CAN USEPermission on the SQL warehouse Cotool queries through. Cotool recommends a dedicated serverless warehouse.[1][10]Read
USE CATALOG, USE SCHEMA, SELECTUnity Catalog grants on the catalog, schemas, or tables that hold your security data.[1][8]Read
sql and unity-catalog scopesScopes on the OAuth secret. A scoped secret can't mint tokens beyond its scopes, so Cotool doesn't need all-apis.[1][6]Read
  • Don't grant MODIFY, ALL PRIVILEGES, or ownership. To keep Cotool from seeing a catalog at all, revoke the service principal's USE CATALOG on it.[1]
  • A secret created without scopes keeps all-apis access. Databricks recommends binding each secret to only the scopes it needs.[6]

How Cotool works with Databricks

  • After you connect, and weekly after that, Cotool maps every catalog, schema, and table the service principal can read, each with a one-sentence summary. Cotool writes the summaries from names, comments, and column names and never reads rows to build the map.[1]
  • Each statement has a timeout of 300 seconds by default and at most 30 minutes, and Cotool cancels statements that exceed it so abandoned queries don't keep running.[1]
  • Cotool rebuilt the Databricks integration in release 0.73.0 with faster read-only queries, service principal authentication, and step-by-step table discovery.[2]

FAQ

Can Cotool use an all-purpose Databricks cluster?
No. Cotool queries through a SQL warehouse. If your HTTP path starts with sql/protocolv1/o/, it belongs to a cluster; create or pick a SQL warehouse instead.[1]
Why does Cotool recommend a serverless warehouse?
Serverless warehouses start in seconds, while classic and pro warehouses can take several minutes after auto-stop, and Cotool's first queries after an idle period may time out while they start.[1]

Sources

  1. Databricks Integration (Cotool docs) · accessed November 16, 2026
  2. Changelog (Cotool docs) · accessed November 16, 2026
  3. Detections are code (Cotool Detect) · accessed November 16, 2026
  4. Hunt threats continuously (Cotool Hunt) · accessed November 16, 2026
  5. Respond to every signal (Cotool Respond) · accessed November 16, 2026
  6. Authorize service principal access with OAuth (Databricks docs) · accessed November 16, 2026
  7. Statement Execution API: Run SQL on warehouses (Databricks docs) · accessed November 16, 2026
  8. Manage privileges in Unity Catalog (Databricks docs) · accessed November 16, 2026
  9. Manage entitlements (Databricks docs) · accessed November 16, 2026
  10. Access control lists (Databricks docs) · accessed November 16, 2026

Related integrations

All integrations

Use cases with Databricks

Related reading

Attackers are scaling with tokens

Cotool helps defenders operate at machine speed. See how security teams are scaling Detection & Response.

Book a demo