Releases: dataform-co/dataform
3.0.0-beta.6
What's Changed
- Fix and re-enable integration tests by @Ekrekr in #1729
- Make example tests run off of the CLI, tidy testing/ by @Ekrekr in #1730
- Bump semver from 5.6.0 to 7.6.0 by @dependabot in #1724
- Some project config improvements by @Ekrekr in #1732
- Make proto object verification give more descriptive error messages by @Ekrekr in #1733
- Add tests for locking in SQLX config options validity by @Ekrekr in #1734
- Add tests for compilation errors where they're missing by @Ekrekr in #1735
- Fix bigquery dryrun by @Ekrekr in #1742
- Make formatting in the CLI use the same SQL formatter as BigQuery by @Ekrekr in #1741
Full Changelog: 3.0.0-beta.5...3.0.0-beta.6
3.0.0-beta.5
What's Changed
Introduced new easier ways to add assertions from dependency as dependencies.
dependOnDependencyAssertions
in config can be used to add assertions from all dependencies of the action as dependencies.
config {
type: "view",
dependOnDependencyAssertions: true,
dependencies: ["some_table"]
}
select test from ${ref("some_other_table")}
includeDependentAssertions
parameter can be used when setting individual dependencies either in config.dependencies or in ref() to add assertions for these dependencies as the dependencies for current action.
config {
type: "view",
dependencies: [{name: "some_table", includeDependentAssertions: true}]
}
select test from ${ref({name: "some_other_table", includeDependentAssertions: true})}
- Update to using GitHub actions for tests by @Ekrekr in #1708
- Re-add cloud build while a new GCP project is set up for GitHub actions by @Ekrekr in #1710
- Update contributing tests guide to just test core by @Ekrekr in #1711
- Bump tar from 6.1.11 to 6.2.1 by @dependabot in #1715
- Bump semver from 5.6.0 to 7.6.0 by @dependabot in #1716
- Bump protobufjs from 7.2.4 to 7.2.5 by @dependabot in #1714
- Update cloudbuild file to default name, make yaml by @Ekrekr in #1722
- Make cloud build for dataform-open-source work for pull requests by @Ekrekr in #1728
- Feature: dependent assertions as dependencies by @Tuseeq1 in #1720
New Contributors
Full Changelog: 3.0.0-beta.4...3.0.0-beta.5
3.0.0-beta4
What's Changed
- Expose
defaultLocation
indataform.projectConfig
. by @BenBirt in #1705 - Publish 3.0.0-beta.4. by @BenBirt in #1706
Full Changelog: 3.0.0-beta.3...3.0.0-beta.4
3.0.0-beta.3
What's Changed
- Add notebook runtime options to workflow settings by @Ekrekr in #1686
- Add a configs reference page, and tidy up other readmes by @Ekrekr in #1688
- Bump follow-redirects from 1.15.4 to 1.15.6 by @dependabot in #1690
- Remove rules docker by @Ekrekr in #1689
- Fix credentials path resolution when projectDir is specified by @Ekrekr in #1694
- Fix CLI tags, allow comma delisted tags, add test by @Ekrekr in #1695
- Prevent confusion around JSON printing of the execution graph by @Ekrekr in #1697
- Extend dryrun to do dry runs against BigQuery by @Ekrekr in #1698
- Add bool argument to disable, default true to keep the API the same by @Ekrekr in #1700
- Throw errors when dots are in fields in targets by @Ekrekr in #1699
- Normalize paths during compilation by @Ekrekr in #1702
Full Changelog: 3.0.0-beta.1...3.0.0-beta.3
3.0.0-beta1
What's Changed
- Remove all analytics from the dataform CLI by @GJMcGowan in #1683
- Remove a bunch of random packages we don't use by @GJMcGowan in #1684
- Add notebook options to configs proto by @Ekrekr in #1685
Full Changelog: 3.0.0-beta.0...3.0.0-beta.1
3.0.0-beta.0
TL;DR of What's Changed Since 2.9.0
dataform.json
-> workflow_settings.yaml
dataform.json
is being deprecated in favor of workflow_settings.yaml
. This means that:
- Workflow settings are now strictly typed, in Protobuf format.
- The Dataform Core version can be specified directly in the
workflow_settings.yaml
file. Note: to have more than just @dataform/core as a dependency, apackage.json
must still be used.
Example conversion of workflow_settings.yaml
:
defaultProject: dataform-demos
defaultLocation: us
defaultDataset: dataform
defaultAssertionDataset: dataform_assertions
version: 3.0.0-beta.0
The above is equivalent to the dataform.json
file:
{
"warehouse": "bigquery",
"defaultDatabase": "dataform-demos",
"defaultLocation": "us",
"defaultSchema": "dataform",
"assertionSchema": "dataform_assertions"
}
Notebooks Actions and actions.yaml
Notebooks as Dataform actions are on their way - but not quite yet! They're part of the compiled graph, and soon they'll be executable.
A new way of configuring action configs through actions.yaml
has been implemented to support this.
An example of loading a notebook in Dataform can be seen at https://github.com/dataform-co/dataform/tree/main/examples/extreme_weather_programming.
Stateless Package Installation by @dataform/cli
Package installation by @dataform/cli is now stateless! The CLI will install NPM packages during compilation if version
is defined in the workflow_settings.yaml
file.
This means no node_modules folder has to be seen in the project, and Dataform users no longer need to be familiar with NPM.
Compilation Output is Now Warehouse Agnostic
Previously the output of compilation results from @dataform/core would insert warehouse specific SQL into the compiled graph. Where possible, this has been removed - transferring the responsibility of inserting warehouse specific SQL into whichever execution engine is running Dataform.
Additionally, support for non-BigQuery warehouses has been dropped. We're in discussions with Datashell for them to provide a warehouse-agnostic CLI execution engine based off of Dataform compiled graphs. In the meantime however, if you need support for a non-BigQuery warehouse, please continue using the latest version starting with 2.x.x!
What's Changed
- Remove non bigquery warehouses by @Ekrekr in #1550
- Remove dynamic warehouse inference, and more clearly differentiate core and api's SQL adapters by @Ekrekr in #1554
- Consolidate cli package by @Ekrekr in #1557
- Merge main to v3 by @Ekrekr in #1563
- Deprecate support for old type
.sql
files by @Ekrekr in #1564 - Add initial action config proto definitions by @Ekrekr in #1576
- Merge main to main_v3 by @Ekrekr in #1579
- Remove gen index in favor of main by @Ekrekr in #1571
- Read workflow_settings.yaml by @Ekrekr in #1580
- Validate workflow settings fields by @Ekrekr in #1581
- Remove deprecated run cache and gcloud project ID fields by @Ekrekr in #1584
- Move concurrency and action retries from project config proto to the CLI by @Ekrekr in #1585
- Merge main to main_v3 3 by @Ekrekr in #1590
- Populate warehouse as BigQuery by default by @Ekrekr in #1591
- Update v3 to use the apache 2.0 license by @Ekrekr in #1594
- Stop using ProtobufJS's verify method (it doesn't do much) by @Ekrekr in #1596
- Clean up action proto constructors in prep for proto config definitions by @Ekrekr in #1597
- Add basic actions.yaml and notebook support by @Ekrekr in #1595
- Move custom variables test to the new testing structure by @Ekrekr in #1601
- Add auto assertion database override to core test by @Ekrekr in #1602
- Move core/tasks.ts to CLI, and minor cleanup by @Ekrekr in #1598
- Add more test coverage of variables and project config overrides for Dataform Core by @Ekrekr in #1603
- Add licenses as prefixes to output bundles by @Ekrekr in #1605
- Throw a more interpretable error when a =v3 Core version by @Ekrekr in #1607
- Action builders and path utils fixes by @Ekrekr in #1606
- Prevent non bq warehouse from being set in workflow settings by @Ekrekr in #1609
- Strip notebook cell outputs by @Ekrekr in #1613
- Add version to workflow settings by @Ekrekr in #1610
- Move common target constructor methods to the action builder by @Ekrekr in #1626
- Add basic support for actions.yaml reading SQL files as operations by @Ekrekr in #1628
- Add support for declarations in action config files by @Ekrekr in #1630
- Add support for Tables to action config files by @Ekrekr in #1631
- Add action config support for the remaining SQL based action types by @Ekrekr in #1633
- Improve proto validation errors where possible by @Ekrekr in #1635
- Make filenames defined in action config files be treated as relative to the action config file by @Ekrekr in #1636
- Better errors for invalid declarations, add more test coverage by @Ekrekr in #1637
- Add an example of a workflow containing notebooks/SQL scripts, with tests by @Ekrekr in #1644
- Make the CLI use and default to workflow_settings.yaml by @Ekrekr in #1648
- Update Dataform CLI npm installs to best-effort by @Ekrekr in #1649
- Merge to v3 4 by @Ekrekr in #1654
- Remove dead resolve code by @Ekrekr in #1652
- Make main_test more concise by making filenames optional by @Ekrekr in #1651
- Remove more dead code, including navigator column descriptors and tools by @Ekrekr in #1655
- Remove warehouse option from the CLI, more tidying by @Ekrekr in #1653
- Make v3 JS API act the same as v2 by @Ekrekr in #1657
- Remove inline tables functionality by @Ekrekr in #1658
- Update proto to use a separate compiled graph, with UX review changes by @Ekrekr in #1656
- Core path tidying by @Ekrekr in #1659
- Export configs proto, bump version by @Ekrekr in #1661
- Migrate core sqlx syntax and assertions context functions tests to _test style by @Ekrekr in #1665
- Better cli errors during compilation by @Ekrekr in #1669
- Replace cli spec tests with _test format by @Ekrekr in #1670
- Improve generated workflow_settings.yaml field order by @Ekrekr in #1671
- Fix config targets by @Ekrekr in #1673
- Remove JS context from SQL files by @Ekrekr in #1672
- Lazy / stateless installation by @Ekrekr in #1675
- Don't check we are on main branch for pre-releases by @lewish in #1664
- Merge main to main_v3 5 by @Ekrekr in #1678
- Bump ip from 1.1.5 to 1.1.9 by @dependabot in #1679
- Merge v3 into main by @Ekrekr in #1680
- Bump v3 to beta by @Ekrekr in #1682
Full Changelog: 2.9.0...3.0.0-beta.0
2.9.0
What's Changed
- Bump follow-redirects from 1.15.3 to 1.15.4 by @dependabot in #1645
- Include type declarations with @dataform/core by @bmagyarkuti in #1647
- Allow 'partitionBy' and 'clusterBy' params for bigquery materialized views by @vlad-ogol in #1660
New Contributors
- @vlad-ogol made their first contribution in #1660
Full Changelog: 2.8.4...2.9.0
2.8.4
2.8.3
What's Changed
- Make Vscode Buildable Again by @GJMcGowan in #1632
- VSCode: avoid errors upon completion requests by @barna-emarsys in #1627
- VSCode: Add ability to configure compile command, add option to skip compilation on save by @bmagyarkuti in #1639
- Escape backslashes and backticks in multiline string literals by @diasdauletov in #1641
New Contributors
- @barna-emarsys made their first contribution in #1627
- @bmagyarkuti made their first contribution in #1639
Full Changelog: 2.8.2...2.8.3