Databricks

Databricks is a cloud-based data engineering tool used for processing and transforming massive quantities of data and exploring the data through machine learning models. Developed by the creators of Apache Spark, it provides a unified, open platform for all your data.

You can connect Databricks to Relyance AI in two ways. Pick one under Authentication Method on the connection wizard's Authentication step:

• OAuth (service principal), recommended. A Databricks service principal authenticates with OAuth machine-to-machine credentials. It gets only the privileges you grant it, and nothing depends on an individual user's personal token.
• Custom, with a Databricks Instance URL and a Personal Access Token. This is the original method and keeps working for existing connections.

#Connect with a service principal (recommended)

#In Databricks

A workspace admin does these steps.

1. Note the workspace URL. This is the Databricks Instance URL, for example https://adb-1234567890123456.7.azuredatabricks.net.
2. Create a service principal. In the workspace, open Settings → Identity and access → Service principals, click Add service principal, and give it a name such as relyance-scanner.
3. Generate an OAuth secret. Open the service principal, go to its Secrets tab, and click Generate secret. Copy the Client ID and the Secret right away; the secret is shown only once. Databricks OAuth secrets expire, so note the expiry date.
4. Grant access to the data you want scanned. In Catalog Explorer, grant the service principal USE CATALOG on each catalog, USE SCHEMA on each schema, and SELECT on the tables to scan. Granting at the catalog level covers everything below it.
5. Make it a workspace admin (add it to the workspace admins group). Relyance needs this to list notebooks and the workspace's users, groups and service principals.
6. Optional: give it a SQL warehouse to sample rows. Grant the service principal CAN USE on a SQL warehouse and copy the warehouse's ID from its Connection details. Without a warehouse, Relyance scans metadata only (catalogs, schemas, tables, columns and grants) and reads no row data.

#In the Relyance AI application

1. Log in to your Relyance AI account.
2. Open Settings (bottom left) and select Integrations.
3. Find the Databricks integration card and click it, then click Add Connection.
4. Follow the wizard. On the Authentication step, choose OAuth (service principal) and fill in:
   • Databricks Instance URL: the workspace URL from step 1.
   • Client ID and Client Secret: from step 3.
   • SQL Warehouse ID (optional): from step 6. Letters and digits only, for example 1234567890abcdef. Leave it blank for a metadata-only scan.
   • Config Options (optional): which catalogs, schemas and objects to scan. See Limit what is scanned below. Leave the default to scan everything the service principal can see.
5. Click Authenticate and finish the wizard. Relyance checks each step separately and tells you which one failed: getting a token, listing catalogs, and reaching the SQL warehouse if you set one.

#Limit what is scanned

Config Options takes an allow list and a deny list of Unity Catalog names:

{
  "allow_list": [
    "prod",
    "prod.analytics.*"
  ],
  "deny_list": [
    "prod.hr"
  ]
}

• Patterns have one to three dot-separated parts: catalog, catalog.schema or catalog.schema.object. Each part may use * (any characters) and ? (one character). Matching is case-insensitive, and missing trailing parts match everything, so prod means every schema and object in the prod catalog.
• An object is scanned when the allow list is empty or any allow pattern matches it, and no deny pattern matches it. Deny wins.
• Scope: the lists apply to catalogs, schemas, tables, columns, volumes, functions, registered models and their grants. Notebooks, jobs, pipelines, workspace identities, model serving endpoints, Genie spaces, apps and experiments are not filtered.
• Invalid lists are rejected. Unknown keys such as a misspelled denylist, a value that isn't a list of strings, a pattern containing whitespace, an empty part, or more than three parts all fail validation. Relyance never falls back to scanning everything.
• Excluded objects are removed from inventory. Objects that fall outside the lists stop appearing in your inventory after the next scan. Widen the lists again and they come back on the following scan.

Examples:

#Connect with a personal access token

Token based authentication must be enabled by the administrator.

In Databricks:

1. The Databricks instance URL will be the custom Databricks URL for your organization.
2. Click Settings in the lower left corner of your Databricks workspace.
3. Click User Settings.
4. Go to the Access Tokens tab.
5. Click the Generate New Token button.
6. Click the Generate button and copy the generated token.

In the Relyance AI application:

1. Login to your Relyance account.
2. Navigate to the Settings Menu in the bottom left-hand side.
3. Select Integrations.
4. Find the Databricks integration card and click on it.
5. In the Databricks integration page, click Add Connection.
6. Follow the wizard, adjusting any defaults as needed.
7. Choose Custom for the Authentication Method.
8. Paste the Databricks Instance URL and Personal Access Token into their respective fields. You can also specify the Clusters and SQL Warehouses to scan. For Asset Discovery its important and necessary to provide the warehouse or cluster name.

9. Click Authenticate and finish the wizard.
10. At this point, you should see the following result on the Vendor Integrations page:
11. Congratulations, you are now connected to Databricks.

#If the connection reports Connected but returns nothing

These are the ways this integration comes back empty without reporting an error. Generated from the integration catalog, so it tracks what the connection actually asks for.

1. A credential rotated at the vendor is not picked up here. Personal Access Token and Client Secret are stored when you save the connection, so regenerating the value at the vendor breaks the next scan until it is re-pasted here. Recording the expiry on the connection means Relyance warns you before it lapses.
2. Check the address fields before suspecting the credentials. Databricks Instance URL identifies which tenant, region or host to talk to. A wrong value there fails authentication and looks exactly like a bad secret.

#Authentication methods and fields

Pick one of these under Authentication Method on the connection wizard's Authentication step. This table is generated from the integration catalog, so it always matches what the form actually asks for.

#Manage this integration with Terraform

Connections for this integration can be managed as code with the Relyance Terraform provider. Non-secret fields go in auth.params; secret fields go in auth.secrets_wo, which is write-only — never stored in Terraform state. Rotate secrets by bumping auth.secrets_wo_version.

#API token

resource "relyance_integration_connection" "databricks_0" {
  vendor = "databricks"
  name   = "<your connection name>"

  auth = {
    method = "api-token"
    params = {
      instance_url = "<instance_url>"
      clusters = jsonencode({
        cluster_names = [
          ""
        ]
      })
      sql_warehouses = jsonencode({
        sql_warehouse_names = [
          ""
        ]
      })
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      api_token = var.databricks_api_token
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}

#Client ID & secret

resource "relyance_integration_connection" "databricks_1" {
  vendor = "databricks"
  name   = "<your connection name>"

  auth = {
    method = "client-credentials"
    params = {
      instance_url = "<instance_url>"
      client_id = "<client_id>"
      data_storage_location = "us"
    }
    # Secret fields are write-only: sent to Relyance, never stored in state.
    secrets_wo = {
      client_secret = var.databricks_client_secret
    }
    secrets_wo_version = 1
  }

  scans = { "data-inspection" = { enabled = true } }
}