Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Dataform CLI npm installs to best-effort #1649

Merged
merged 4 commits into from
Jan 16, 2024

Conversation

Ekrekr
Copy link
Contributor

@Ekrekr Ekrekr commented Jan 15, 2024

General UX:

dataformCoreVersion present no dataformCoreVersion present
package.json present ✅ Install from dataformCoreVersion in workflow_settings.yaml ✅ Install from @dataform/core version in package.json
no package.json ✅ Install from dataformCoreVersion in workflow_settings.yaml ❌ Throw error about no version definition present

@Ekrekr Ekrekr requested a review from BenBirt January 15, 2024 16:05
@BenBirt
Copy link
Collaborator

BenBirt commented Jan 15, 2024

I think the top-left of the table has incorrect semantics. I think that if both files exist, workflow_settings.yaml should not contain a version. Otherwise things get way too confusing.

@Ekrekr
Copy link
Contributor Author

Ekrekr commented Jan 15, 2024

I think the top-left of the table has incorrect semantics. I think that if both files exist, workflow_settings.yaml should not contain a version. Otherwise things get way too confusing.

This issue with that in the CLI is that compilation isn't stateless, so we don't delete package.json after an install. So the first time installing works if there's dataformCoreVersion in workflow settings, but then subsequent installs will fail. From playing around with it, doing this seems to work just fine!

A workaround could be making it stateless, and updating the package manager to pnpm instead.

Let's decide on this in the UX review though, to avoid doing work that might not be needed.

@BenBirt
Copy link
Collaborator

BenBirt commented Jan 16, 2024

OK, but we did already discuss this point!

Comment on lines +25 to +26
// If there are other packages already in the package.json, specifying a specific package to
// install will trigger the other packages to be installed too.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you sure about this? it seems very surprising to me, and it does not seem to be in the docs: https://docs.npmjs.com/cli/v10/commands/npm-install

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was surprised too, but it happened when I tried it!

@Ekrekr
Copy link
Contributor Author

Ekrekr commented Jan 16, 2024

OK, but we did already discuss this point!

In closed source, the Dataform Core version can be specified in either the package.json or the workflow_settings.yaml file.

It's true - but it's a pretty terrible UX when using it in practice.

In any case, the code fix if we do decide to do this is trivial.

@Ekrekr Ekrekr requested a review from BenBirt January 16, 2024 11:23
@BenBirt
Copy link
Collaborator

BenBirt commented Jan 16, 2024

OK - I'll say:

  1. The UX nastiness is pretty resolvable. For one, it's a once-off thing (when migrating from workflow_settings.yaml to package.json). But if even that is too much, we can always make the CLI help you out with that migration.
  2. Even if we do want to allow the version to be configured in two places, I would strongly argue that the preferred location should be package.json, not least because that's what all other NPM tooling would actually look at.

@Ekrekr
Copy link
Contributor Author

Ekrekr commented Jan 16, 2024

I would strongly argue that the preferred location should be package.json

I agree with this, extending it even to be the only location that open source packages configure versions - it's a bit weird having two ways of doing things.

once-off thing

I'm not sure that's true, because it's fairly common to run the Dataform CLI as part of CI/CD in a GitHub action from PRs - which gets run on the version of code on Google Cloud.

I'll have a think about the different user journeys, but atm this is the most versatile option!

@Ekrekr Ekrekr merged commit bc49654 into main_v3 Jan 16, 2024
4 checks passed
@Ekrekr Ekrekr deleted the cli-install-workflow-settings branch January 16, 2024 13:45
@BenBirt
Copy link
Collaborator

BenBirt commented Jan 16, 2024

But GCP should follow the same logic: i.e. if both files provide versions, it should throw. Therefore: in all environments, either you are setting it in one place or the other, but never both.

moker-spaghetti pushed a commit to moker-spaghetti/dataform that referenced this pull request May 26, 2024
* Update Dataform CLI npm installs to best-effort

* Tidy logic

* Prevent errors when package.json already present

* Fix lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants