Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Bigtables Garbage Collection Policy is not applicable #300

Closed
wesam80 opened this issue Nov 5, 2020 · 10 comments
Closed

Bigtables Garbage Collection Policy is not applicable #300

wesam80 opened this issue Nov 5, 2020 · 10 comments
Labels
bug Something isn't working

Comments

@wesam80
Copy link

wesam80 commented Nov 5, 2020 •
edited

Describe the bug
When applying a simple BigtableGCPolicy like the one here in the documentation, the objects are created properly, but the policy is not really applied.

image

If the policy is updated using the cli cbt it works fine

ConfigConnector Version
1.27.1

To Reproduce
Steps to reproduce the behavior:

YAML snippets:

apiVersion: bigtable.cnrm.cloud.google.com/v1beta1
kind: BigtableGCPolicy
metadata:
  name: bigtablegcpolicy-sample
spec:
  tableRef:
    name: bigtablename
  columnFamily: quote
  instanceRef:
    name: clustername
  mode: INTERSECTION
  maxAge:
  - days: 30
  maxVersion:
  - number: 10

Then check the GC policies on the field using cbt command line, it will show <never>

If the policy is update manually cbt setgcpolicy ... it will work fine

Debugging:
kubectl describe BigtableGCPolicy -n datastores-wes-sit

Name:         datastores-wes-sit-trade-gcp
Namespace:    datastores-wes-sit
Labels:       <none>
Annotations:  cnrm.cloud.google.com/management-conflict-prevention-policy: none
              cnrm.cloud.google.com/project-id: qt-msa-nonprod-6f
              kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"bigtable.cnrm.cloud.google.com/v1beta1","kind":"BigtableGCPolicy","metadata":{"annotations":{},"name":"datastores-wes-sit-t...
API Version:  bigtable.cnrm.cloud.google.com/v1beta1
Kind:         BigtableGCPolicy
Metadata:
  Creation Timestamp:  2020-11-05T19:11:11Z
  Finalizers:
    cnrm.cloud.google.com/finalizer
    cnrm.cloud.google.com/deletion-defender
  Generation:  2
  Managed Fields:
.....................
.....................
    Manager:         cnrm-controller-manager
    Operation:       Update
    Time:            2020-11-05T19:11:11Z
  Resource Version:  168755889
  Self Link:         /apis/bigtable.cnrm.cloud.google.com/v1beta1/namespaces/datastores-wes-sit/bigtablegcpolicies/datastores-wes-sit-trade-gcp
  UID:               ffd50825-f3e6-43da-b048-6e3fbb956019
Spec:
  Column Family:  trade
  Instance Ref:
    Name:  shared-nonprod
  Max Age:
    Days:  10
  Max Version:
    Number:  10
  Mode:      INTERSECTION
  Table Ref:
    Name:  datastores-wes-sit
Status:
  Conditions:
    Last Transition Time:  2020-11-05T19:11:11Z
    Message:               The resource is up to date
    Reason:                UpToDate
    Status:                True
    Type:                  Ready
Events:
  Type    Reason    Age   From                         Message
  ----    ------    ----  ----                         -------
  Normal  UpToDate  94s   bigtablegcpolicy-controller  The resource is up to date
@wesam80 wesam80 added the bug Something isn't working label Nov 5, 2020
@jcanseco
Copy link
Member

Hi @wesam80, thanks, I've confirmed that this is a bug on our side. We'll work on fixing the issue and let you know when we have an update.

@wesam80
Copy link
Author

wesam80 commented Nov 10, 2020

Thanks @jcanseco for confirming. Any determined time frame? Our implementation depends on this and we are going to production soon. It is great to know the estimated time so we plan accordingly.

@jcanseco
Copy link
Member

Hi @wesam80, I believe we theoretically should be able to support the creation case by late Nov or early Dec (i.e. the case of setting a GC policy when there is none).

However, it'll take us awhile longer to support the update case (i.e. the case of updating a GC policy that is already there). In theory, you should still be able to perform an "update" once we fix the creation case, but you'd have to do so by deleting and then re-creating the GC policy.

The problem behind this issue is unfortunately a bit tricky, so these are the best estimates I can give you for now.

What do you think? Would the fix for the creation case be sufficient for your use-case, or would you rather just wait for both the creation and update cases to be fixed given your production timelines?

@wesam80
Copy link
Author

wesam80 commented Nov 12, 2020

Yes @jcanseco, supporting the creation by the end of November would help us at the short term, and definitly we will be happy to have both create and update later on :)

@jcanseco
Copy link
Member

Great, thanks for confirming @wesam80. We'll let you know once we have more updates.

@maqiuyujoyce
Copy link
Collaborator

Hi @wesam80 , we just released Config Connector v1.32.0, which fixes the creation case for BigtableGCPolicy. Please let us know if it works.

@wesam80
Copy link
Author

wesam80 commented Dec 3, 2020

Thank you @maqiuyujoyce
I can confirm that the the GCP are created successfully after upgrading to Config Connector v1.32.0,

As you know, people get greedy :)

Any timeline for supporting updates as well without the need for deleting GCPs and creating them again?

Thx

@maqiuyujoyce
Copy link
Collaborator

maqiuyujoyce commented Dec 4, 2020 •
edited

Hi @wesam80 , thank you for confirming that the creation works!

In regard to supporting updates, currently we don't have a timeline yet. It's a bigger change compared to supporting creations, so we are still looking into how to better address it. Will let you know when we have any update.

@AndreLSnyk AndreLSnyk mentioned this issue Mar 7, 2021
Open
@Matthelonianxl Matthelonianxl mentioned this issue Mar 8, 2021
Open
@snyk-bot snyk-bot mentioned this issue Apr 7, 2022
Open
@kevinsi4508
Copy link

This should be resolved by using gc_rules. Currently we are waiting for KCC to update to the latest Google Terraform Provider version.

@mbzomowski
Copy link

spec.gcRules has been added to BigTableGCPolicy as of Config Connector v1.97.0. We don't recommend using maxAge, maxVersion or mode for defining a
BigTableGCPolicy as these fields have known drift detection issues. Please use gcRules instead.

@AndreLSnyk AndreLSnyk mentioned this issue Apr 30, 2023
Open
@AndreLSnyk AndreLSnyk mentioned this issue Dec 14, 2023
Open
@Matthelonianxl Matthelonianxl mentioned this issue Dec 14, 2023
Open
@AndreLSnyk AndreLSnyk mentioned this issue Jan 1, 2024
Open
@Matthelonianxl Matthelonianxl mentioned this issue Jan 2, 2024
Open
@AndreLSnyk AndreLSnyk mentioned this issue Jan 5, 2024
Open
@Matthelonianxl Matthelonianxl mentioned this issue Jan 6, 2024
Open
@AndreLSnyk AndreLSnyk mentioned this issue Mar 15, 2024
Open
@Matthelonianxl Matthelonianxl mentioned this issue Mar 16, 2024
Open
@AndreLSnyk AndreLSnyk mentioned this issue May 13, 2024
Open
@Matthelonianxl Matthelonianxl mentioned this issue May 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@jcanseco @kevinsi4508 @mbzomowski @maqiuyujoyce