Connect Databricks
Databricks spend and resources appear in CloudYali after CloudYali support connects a workspace with a service principal you create. CloudYali reads Databricks system tables through a SQL warehouse, using OAuth machine-to-machine credentials. The Integrations page lists the connection but has no form to add one yet, so the last step goes through support@cloudyali.io.
Before you start
- A Databricks workspace with Unity Catalog, on AWS, Microsoft Azure or Google Cloud.
- The system schemas listed below enabled for your account. An account admin enables them.
- A SQL warehouse the service principal can use. CloudYali runs its queries on it once a day.
- Someone who can create a service principal and grant access to system tables, usually a workspace admin and a metastore admin.
What CloudYali can read
CloudYali reads only the system tables below. It does not read notebooks, query text or table data.
| Table | What CloudYali uses it for | Needed |
|---|---|---|
system.billing.usage | Usage and DBUs per SKU, product, workspace and tag | Yes |
system.billing.list_prices | The list price that turns usage into cost | Yes |
system.access.workspaces_latest | Workspace names | Yes |
system.compute.clusters, system.compute.warehouses | Clusters and SQL warehouses in Resource Inventory | Yes |
system.lakeflow.jobs, system.lakeflow.job_tasks, system.lakeflow.pipelines | Jobs and pipelines in Resource Inventory | Yes |
system.serving.served_entities | Models and endpoints served by Model Serving, in Resource Inventory | Yes |
system.mlflow.experiments_latest | MLflow experiments in Resource Inventory | Yes |
system.query.history | Who ran queries and for how long, for per-user attribution | Optional |
system.access.audit | Commands run on clusters, for per-user attribution | Optional |
system.billing.usage holds usage for every workspace in your Databricks account, so one connection
brings in the whole account's spend. The console does not show per-user attribution yet, so you can
leave out the two optional tables.
Connect Databricks
- Create the service principal, its OAuth secret and its grants, as described in Set up a Databricks service principal.
- Email support@cloudyali.io with the workspace name, the workspace URL
(
https://…), the SQL warehouse ID and the service principal's client ID. Do not put the client secret in the email: support arranges how to share it. - Support connects the workspace. CloudYali checks that it can sign in and run a query on the warehouse before it saves the connection.
Check that it worked
The workspace appears on the Connected tab of Integrations, with its name, workspace ID, host and the date it was added.
- Cost arrives with the next daily collection, at about 14:40 UTC. The first collection reads six months of usage, for your first Databricks workspace. Last sync fills in when a collection succeeds.
- Resources arrive with the inventory collection, at about 14:50 UTC.
Databricks then shows in:
| Where | What you see |
|---|---|
| Cost Explorer and the Dashboard | Spend by Workspace (the workspace name), service (the Databricks product) and usage type (the SKU). Region shows as Global |
| Budgets | Budgets scoped to Databricks, narrowed by workspace, service or tag |
| Tag Governance | Tagged and untagged spend from your Databricks custom tags |
| Resource Inventory | Clusters, SQL warehouses, jobs, job tasks, pipelines, served models, MLflow experiments and the workspace. See Supported resources |
Limits
- List price, not your contract price. CloudYali prices usage at Databricks' list price from
system.billing.list_prices. Committed-use and negotiated discounts are not applied, so CloudYali can read higher than your invoice. - Not in every feature. Databricks spend does not reach Cost Savings, Cost Anomalies or Cost Views.
- No connect form or Remove button. Support connects, updates and removes Databricks workspaces.
- Errors after connecting are not shown. If a grant is missing, the daily collection fails or skips that data, and no console screen says so. If Last sync stays empty a day after connecting, email support.
Replace a key or disconnect
Email support@cloudyali.io to rotate the client secret or to remove a workspace. Removing it stops collection; the cost already collected stays. To connect a workspace again after removing it, ask support, because adding it again the usual way fails.
Troubleshooting
| What support tells you | Cause | Fix |
|---|---|---|
| Failed to validate credentials with Databricks | The workspace URL, client ID or client secret is wrong, the warehouse ID is wrong, or the service principal cannot use the warehouse | Check the values, and give the service principal Can use on the warehouse |
| host_url must start with https:// | The workspace URL is missing https:// | Send the full URL from your browser's address bar |
| An account for workspace '…' is already connected | The workspace is already connected to your CloudYali account | Find it on the Connected tab |
| Last sync stays empty | A grant on a system table is missing, or a system schema is not enabled | Run the grants in Set up a Databricks service principal, then email support |