Snapshot

Snapshot build of the main branch. Updated automatically on each commit.

v0.232.1

This patch release fixes the following error observed when deploying to /Shared root folder:

Error: Path (/Shared/.bundle/.../resources) doesn't exist

Bundles:

• Fixed adding /Workspace prefix for resource paths (#1866).

v0.232.0

New features for Databricks Asset Bundles:

This release adds support for managing AI/BI dashboards as part of your bundle configuration. The bundle generate command is updated to support producing dashboard bundle configuration as well as a serialized JSON representation of the dashboard.

You can find an example configuration and walkthrough at https://github.com/databricks/bundle-examples/tree/main/knowledge_base/dashboard_nyc_taxi

CLI:

• Add privacy notice to README (#1841).

Bundles:

• Add support for AI/BI dashboards (#1743).
• Added validator for folder permissions (#1824).
• Add bundle generate variant for dashboards (#1847).
• Use SetPermissions instead of UpdatePermissions when setting folder permissions based on top-level ones (#1822).

Internal:

• Attempt to reduce test flakiness on Windows (#1845).
• Reuse resource resolution code for the run command (#1858).
• [Internal] Automatically trigger integration tests on PR (#1857).
• [Internal] Add test instructions for external contributors (#1863).
• Add libs/dyn/jsonsaver (#1862).

Dependency updates:

• Bump github.com/fatih/color from 1.17.0 to 1.18.0 (#1861).

v0.231.0

CLI:

• Added JSON input validation for CLI commands (#1771).
• Support Git worktrees for sync (#1831).

Bundles:

• Add bundle summary to display URLs for deployed resources (#1731).
• Added a warning when incorrect permissions used for /Workspace/Shared bundle root (#1821).
• Show actionable errors for collaborative deployment scenarios (#1386).
• Fix path to repository-wide exclude file (#1837).
• Fixed typo in converting cluster permissions (#1826).
• Ignore metastore permission error during template generation (#1819).
• Handle normalization of dyn.KindTime into an any type (#1836).
• Added support for pip options in environment dependencies (#1842).
• Fix race condition when restarting continuous jobs (#1849).
• Fix pipeline in default-python template not working for certain workspaces (#1854).
• Add "output" flag to the bundle sync command (#1853).

Internal:

• Move utility functions dealing with IAM to libs/iamutil (#1820).
• Remove unused IS_OWNER constant (#1823).
• Assert SDK version is consistent in the CLI generation process (#1814).
• Fixed unmarshalling json input into interface{} type (#1832).
• Fix TestAccFsMkdirWhenFileExistsAtPath in isolated Azure environments (#1833).
• Add behavioral tests for examples from the YAML spec (#1835).
• Remove Terraform conversion function that's no longer used (#1840).
• Encode assumptions about the dashboards API in a test (#1839).
• Add script to make testing of code on branches easier (#1844).

API Changes:

• Added databricks disable-legacy-dbfs command group.

OpenAPI commit cf9c61453990df0f9453670f2fe68e1b128647a2 (2024-10-14)
Dependency updates:

• Upgrade TF provider to 1.54.0 (#1852).
• Bump github.com/databricks/databricks-sdk-go from 0.48.0 to 0.49.0 (#1843).

v0.230.0

Notable changes for Databricks Asset Bundles:

Workspace paths are automatically prefixed with /Workspace. In addition, all usage of path strings such as /Workspace/${workspace.root_path}/... in bundle configuration is automatically replaced with ${workspace.root_path}/... and generates a warning as part of bundle validate.

More details can be found here: https://docs.databricks.com/en/release-notes/dev-tools/bundles.html#workspace-paths

Bundles:

• Add an error if state files grow bigger than the export limit (#1795).
• Always prepend bundle remote paths with /Workspace (#1724).
• Add resource path field to bundle workspace configuration (#1800).
• Add validation for files with a .(resource-name).yml extension (#1780).

Internal:

• Remove deprecated or readonly fields from the bundle schema (#1809).

API Changes:

• Changed databricks git-credentials create, databricks git-credentials delete, databricks git-credentials get, databricks git-credentials list, databricks git-credentials update commands .
• Changed databricks repos create, databricks repos delete, databricks repos get, databricks repos update command .

OpenAPI commit 0c86ea6dbd9a730c24ff0d4e509603e476955ac5 (2024-10-02)
Dependency updates:

• Upgrade TF provider to 1.53.0 (#1815).
• Bump golang.org/x/term from 0.24.0 to 0.25.0 (#1811).
• Bump golang.org/x/text from 0.18.0 to 0.19.0 (#1812).
• Bump github.com/databricks/databricks-sdk-go from 0.47.0 to 0.48.0 (#1810).

v0.229.0

Bundles:

• Added support for creating all-purpose clusters (#1698).
• Reduce time until the prompt is shown for bundle run (#1727).
• Use Unity Catalog for pipelines in the default-python template (#1766).
• Add verbose flag to the "bundle deploy" command (#1774).
• Fixed full variable override detection (#1787).
• Add sub-extension to resource files in built-in templates (#1777).
• Fix panic in apply_presets.go (#1796).

Internal:

• Assert tokens are redacted in origin URL when username is not specified (#1785).
• Refactor jobs path translation (#1782).
• Add JobTaskClusterSpec validate mutator (#1784).
• Pin Go toolchain to 1.22.7 (#1790).
• Modify SetLocation test utility to take full locations as argument (#1788).
• Simplified isFullVariableOverrideDef implementation (#1791).
• Sort tasks by task_key before generating the Terraform configuration (#1776).
• Trim trailing whitespace (#1794).
• Move trampoline code into trampoline package (#1793).
• Rename RootPath -> BundleRootPath (#1792).

API Changes:

• Changed databricks apps delete command to return .
• Changed databricks apps deploy command with new required argument order.
• Changed databricks apps start command to return .
• Changed databricks apps stop command to return .
• Added databricks temporary-table-credentials command group.
• Added databricks serving-endpoints put-ai-gateway command.
• Added databricks disable-legacy-access command group.
• Added databricks account disable-legacy-features command group.

OpenAPI commit 6f6b1371e640f2dfeba72d365ac566368656f6b6 (2024-09-19)
Dependency updates:

• Upgrade to Go SDK 0.47.0 (#1799).
• Upgrade to TF provider 1.52 (#1781).
• Bump golang.org/x/mod from 0.20.0 to 0.21.0 (#1758).
• Bump github.com/hashicorp/hc-install from 0.7.0 to 0.9.0 (#1772).

v0.228.1

Bundles:

• Added listing cluster filtering for cluster lookups (#1754).
• Expand library globs relative to the sync root (#1756).
• Fixed generated YAML missing 'default' for empty values (#1765).
• Use periodic triggers in all templates (#1739).
• Use the friendly name of service principals when shortening their name (#1770).
• Fixed detecting full syntax variable override which includes type field (#1775).

Internal:

• Pass copy of dyn.Path to callback function (#1747).
• Make bundle JSON schema modular with $defs (#1700).
• Alias variables block in the Target struct (#1748).
• Add end to end integration tests for bundle JSON schema (#1726).
• Fix artifact upload integration tests (#1767).

API Changes:

• Added databricks quality-monitors regenerate-dashboard command.

OpenAPI commit d05898328669a3f8ab0c2ecee37db2673d3ea3f7 (2024-09-04)
Dependency updates:

• Bump golang.org/x/term from 0.23.0 to 0.24.0 (#1757).
• Bump golang.org/x/oauth2 from 0.22.0 to 0.23.0 (#1761).
• Bump golang.org/x/text from 0.17.0 to 0.18.0 (#1759).
• Bump github.com/databricks/databricks-sdk-go from 0.45.0 to 0.46.0 (#1760).

v0.228.0

CLI:

• Do not error if we cannot prompt for a profile in auth login (#1745).

Bundles:

As of this release, the CLI will show a prompt if there are configuration changes that lead to DLT pipeline recreation.
Users can skip the prompt by specifying the --auto-approve flag.

• Pass along $AZURE_CONFIG_FILE to Terraform process (#1734).
• Add prompt when a pipeline recreation happens (#1672).
• Use materialized views in the default-sql template (#1709).
• Update templates to latest LTS DBR (#1715).
• Make lock optional in the JSON schema (#1738).
• Do not suppress normalisation diagnostics for resolving variables (#1740).
• Include a permissions section in all templates (#1713).
• Fixed complex variables are not being correctly merged from include files (#1746).
• Fixed variable override in target with full variable syntax (#1749).

Internal:

• Consider serverless clusters as compatible for Python wheel tasks (#1733).
• PythonMutator: explain missing package error (#1736).
• Add dyn.Time to box a timestamp with its original string value (#1732).
• Fix streaming of stdout, stdin, stderr in cobra test runner (#1742).

Dependency updates:

• Bump github.com/Masterminds/semver/v3 from 3.2.1 to 3.3.0 (#1741).

v0.227.1

CLI:

• Disable prompt for storage-credentials get command (#1723).

Bundles:

• Do not treat empty path as a local path (#1717).
• Correctly mark PyPI package name specs with multiple specifiers as remote libraries (#1725).
• Improve error handling for /Volumes paths in mode: development (#1716).

Internal:

• Ignore CLI version check on development builds of the CLI (#1714).

API Changes:

• Added databricks resource-quotas command group.
• Added databricks policy-compliance-for-clusters command group.
• Added databricks policy-compliance-for-jobs command group.

OpenAPI commit 3eae49b444cac5a0118a3503e5b7ecef7f96527a (2024-08-21)
Dependency updates:

• Bump github.com/databricks/databricks-sdk-go from 0.44.0 to 0.45.0 (#1719).
• Revert hc-install version to 0.7.0 (#1711).

v0.227.0

CLI:

• Added filtering flags for cluster list commands (#1703).

Bundles:

• Allow users to configure paths (including outside of the bundle root) to synchronize to the workspace. (#1694).
• Add configurable presets for name prefixes, tags, etc. (#1490).
• Add support for requirements libraries in Job Tasks (#1543).
• Remove reference to "dbt" in the default-sql template (#1696).
• Pause continuous pipelines when 'mode: development' is used (#1590).
• Report all empty resources present in error diagnostic (#1685).
• Improves detection of PyPI package names in environment dependencies (#1699).

Internal:

• Add import option for PyDABs (#1693).
• Make fileset take optional list of paths to list (#1684).
• Pass through paths argument to libs/sync (#1689).
• Correctly mark package names with versions as remote libraries (#1697).
• Share test initializer in common helper function (#1695).
• Make pydabs/venv_path optional (#1687).
• Use API mocks for duplicate path errors in workspace files extensions client (#1690).
• Fix prefix preset used for UC schemas (#1704).