Set up a Databricks service principal for CloudYali
CloudYali reads Databricks through a service principal: an OAuth client ID and secret, Can use on
one SQL warehouse, and SELECT on the system tables it needs. This page creates all three. Then
Connect Databricks sends them to CloudYali support.
Before you start
- Workspace admin rights, to add the service principal and set warehouse permissions.
- A metastore admin, to grant access to the
systemcatalog. - An account admin, if a system schema below is not enabled yet.
1. Create the service principal and its secret
- In the workspace, open Settings → Identity and access → Service principals, and add a new
service principal, for example
cloudyali-reader. - Open it and, on the Secrets tab, generate a secret.
- Copy the Client ID and the Secret. Databricks shows the secret only once. Note when it expires: CloudYali stops collecting after that, until support has the new secret.
2. Give it a SQL warehouse
- Open SQL Warehouses and choose the warehouse CloudYali will use. A small serverless warehouse is enough: CloudYali runs its queries once a day.
- On Permissions, add the service principal with Can use.
- On Connection details, copy the HTTP path. The warehouse ID is its last part, after
/sql/1.0/warehouses/. CloudYali needs the ID, not the whole path.
3. Check the system schemas are enabled
CloudYali reads the billing, access, compute, lakeflow, serving and mlflow schemas of the
system catalog. An account admin can list the enabled schemas with the Databricks CLI, and enable a
missing one. Replace METASTORE_ID with your metastore's ID and SCHEMA_NAME with the schema:
databricks system-schemas list METASTORE_ID
databricks system-schemas enable METASTORE_ID SCHEMA_NAME
4. Grant access to the system tables
A metastore admin runs these grants in the SQL editor. Replace CLIENT_ID with the service
principal's client (application) ID, and keep the backticks:
GRANT USE CATALOG ON CATALOG system TO `CLIENT_ID`;
GRANT USE SCHEMA ON SCHEMA system.billing TO `CLIENT_ID`;
GRANT USE SCHEMA ON SCHEMA system.access TO `CLIENT_ID`;
GRANT USE SCHEMA ON SCHEMA system.compute TO `CLIENT_ID`;
GRANT USE SCHEMA ON SCHEMA system.lakeflow TO `CLIENT_ID`;
GRANT USE SCHEMA ON SCHEMA system.serving TO `CLIENT_ID`;
GRANT USE SCHEMA ON SCHEMA system.mlflow TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.billing.usage TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.billing.list_prices TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.access.workspaces_latest TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.compute.clusters TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.compute.warehouses TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.lakeflow.jobs TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.lakeflow.job_tasks TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.lakeflow.pipelines TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.serving.served_entities TO `CLIENT_ID`;
GRANT SELECT ON TABLE system.mlflow.experiments_latest TO `CLIENT_ID`;
Cost needs the two billing tables. system.access.workspaces_latest adds workspace names; without it,
cost shows under the workspace ID. The others fill Resource Inventory; without one, that resource type
is skipped and cost still arrives.
To check a grant, run SHOW GRANTS for the service principal on a table:
SHOW GRANTS `CLIENT_ID` ON TABLE system.billing.usage;
CloudYali can also read system.query.history and system.access.audit to attribute cost to the people
who ran queries and commands. The console does not show this attribution yet, so leave these grants out
unless support asks for them. system.access.audit also needs verbose audit logs turned on.
5. Send the details to support
Email support@cloudyali.io with:
| Detail | Where you find it |
|---|---|
| Workspace name | The name you want CloudYali to show for this workspace |
| Workspace URL | Your browser's address bar, starting with https:// |
| SQL warehouse ID | Step 2 |
| Client ID | Step 1 |
Do not put the client secret in the email: support arranges how to share it. Then follow Check that it worked.