Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add soft delete feature #2403

Merged
merged 30 commits into from
Mar 14, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
26326a4
feat: add soft delete feature
JesseLovelace Feb 8, 2024
407af36
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 8, 2024
22ae88d
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 8, 2024
bd4bb67
Merge branch 'softdel' of https://github.com/googleapis/java-storage …
gcf-owl-bot[bot] Feb 8, 2024
d77aa58
add softDeleteTime and hardDeleteTime object fields
JesseLovelace Feb 9, 2024
a888e80
Merge branch 'softdel' of github.com:googleapis/java-storage into sof…
JesseLovelace Feb 9, 2024
2a85a65
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 9, 2024
4f1b6de
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 9, 2024
fd22008
add new fields to field tests
JesseLovelace Feb 9, 2024
7142bcb
Merge branch 'softdel' of github.com:googleapis/java-storage into sof…
JesseLovelace Feb 9, 2024
0f837d3
clirr ignore
JesseLovelace Feb 9, 2024
c2884aa
remove debug comments
JesseLovelace Feb 14, 2024
ac6022e
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 14, 2024
1c363ff
Merge branch 'main' into softdel
JesseLovelace Feb 28, 2024
dbe0d92
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 28, 2024
fb88608
add softdeletetime and harddeletetime to grpc codec
JesseLovelace Feb 28, 2024
04975fc
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 28, 2024
1de29b8
fix read mask test
JesseLovelace Feb 29, 2024
c98d8c7
Merge branch 'softdel' of github.com:googleapis/java-storage into sof…
JesseLovelace Feb 29, 2024
3a4a00f
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 29, 2024
b3d70db
fix style issues
JesseLovelace Feb 29, 2024
7986f9c
Merge branch 'softdel' of github.com:googleapis/java-storage into sof…
JesseLovelace Feb 29, 2024
a93e162
Merge branch 'main' into softdel
JesseLovelace Mar 11, 2024
5a744ca
updates to apiary library
JesseLovelace Mar 11, 2024
0b8fe3f
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 11, 2024
be3e975
Merge branch 'main' into softdel
JesseLovelace Mar 14, 2024
5e65c40
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 14, 2024
938e00c
fix read mask test
JesseLovelace Mar 14, 2024
8f38221
Merge branch 'softdel' of github.com:googleapis/java-storage into sof…
JesseLovelace Mar 14, 2024
68ce8c0
fix typo
JesseLovelace Mar 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,13 +57,13 @@ implementation 'com.google.cloud:google-cloud-storage'
If you are using Gradle without BOM, add this to your dependencies:

```Groovy
implementation 'com.google.cloud:google-cloud-storage:2.32.1'
implementation 'com.google.cloud:google-cloud-storage:2.33.0'
```

If you are using SBT, add this to your dependencies:

```Scala
libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "2.32.1"
libraryDependencies += "com.google.cloud" % "google-cloud-storage" % "2.33.0"
```
<!-- {x-version-update-end} -->

Expand Down Expand Up @@ -428,7 +428,7 @@ Java is a registered trademark of Oracle and/or its affiliates.
[kokoro-badge-link-5]: http://storage.googleapis.com/cloud-devrel-public/java/badges/java-storage/java11.html
[stability-image]: https://img.shields.io/badge/stability-stable-green
[maven-version-image]: https://img.shields.io/maven-central/v/com.google.cloud/google-cloud-storage.svg
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-storage/2.32.1
[maven-version-link]: https://central.sonatype.com/artifact/com.google.cloud/google-cloud-storage/2.33.0
[authentication]: https://github.com/googleapis/google-cloud-java#authentication
[auth-scopes]: https://developers.google.com/identity/protocols/oauth2/scopes
[predefined-iam-roles]: https://cloud.google.com/iam/docs/understanding-roles#predefined_roles
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1090,9 +1090,10 @@ public Blob get(String blob, BlobGetOption... options) {
}

/**
* Returns the requested blob in this bucket of a specific generation or {@code null} if not found.
* Returns the requested blob in this bucket of a specific generation or {@code null} if not
* found.
*
* <p>Example of getting a blob of a specific in the bucket.
* <p>Example of getting a blob of a specific in the bucket.
*
* <pre>{@code
* String blobName = "my_blob_name";
Expand All @@ -1106,7 +1107,7 @@ public Blob get(String blob, BlobGetOption... options) {
* @throws StorageException upon failure
*/
@TransportCompatibility({Transport.HTTP, Transport.GRPC})
public Blob get(String blob, Long generation, BlobGetOption...options) {
public Blob get(String blob, Long generation, BlobGetOption... options) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC this is needed because you can't specify generation on Bucket.get() that exists today; Would recommend using BlobId instead of adding Long generation overload.

Do you think this is hard requirement to support this feature since there's storage.get(BlobId..)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would feel pretty weird that soft deleted would become the only state an object can be in where you can get it from Storage but can't get it from the Bucket that the object lives in.

I also think using BlobId here would feel weird, because you're already calling from a Bucket, so it's weird to have to supply the same Bucket name into the BlobId. There are no other methods in Bucket.java that have a BlobId in the signature

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bleh, missed that bucket name is also part of BlobId. thanks.
We haven't had generation in Bucket.get() at all which is required for version enabled buckets, that's why i asked this question as well.

For the sake of discussion, WDYT about having generation under BlobGetOption's? It's a query paramter just like IfGenerationMatch etc.

return storage.get(BlobId.of(getName(), blob, generation), options);
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -371,19 +371,20 @@ public boolean equals(Object o) {
}
SoftDeletePolicy that = (SoftDeletePolicy) o;
return Objects.equals(retentionDuration, that.retentionDuration)
&& Objects.equals(effectiveTime, that.effectiveTime);
&& Objects.equals(effectiveTime, that.effectiveTime);
}

@Override
public int hashCode() {
return Objects.hash(retentionDuration, effectiveTime);
}

@Override public String toString() {
@Override
public String toString() {
return MoreObjects.toStringHelper(this)
.add("retentionDuration", retentionDuration)
.add("effectiveTime", effectiveTime)
.toString();
.add("retentionDuration", retentionDuration)
.add("effectiveTime", effectiveTime)
.toString();
}

public static Builder newBuilder() {
Expand Down Expand Up @@ -413,17 +414,15 @@ public static final class Builder {
private Duration retentionDuration;
private OffsetDateTime effectiveTime;

/**
* Sets the length of time to retain soft-deleted objects for, expressed as a Duration
*/
/** Sets the length of time to retain soft-deleted objects for, expressed as a Duration */
public Builder setRetentionDuration(Duration retentionDuration) {
this.retentionDuration = retentionDuration;
return this;
}

/**
* Sets the time from which this soft-delete policy is effective.
* This is package-private because it can only be set by the backend.
* Sets the time from which this soft-delete policy is effective. This is package-private
* because it can only be set by the backend.
*/
Builder setEffectiveTime(OffsetDateTime effectiveTime) {
this.effectiveTime = effectiveTime;
Expand Down Expand Up @@ -2281,7 +2280,7 @@ Builder setObjectRetention(ObjectRetention objectRetention) {

@Override
public Builder setSoftDeletePolicy(SoftDeletePolicy softDeletePolicy) {
if(!Objects.equals(this.softDeletePolicy, softDeletePolicy)) {
if (!Objects.equals(this.softDeletePolicy, softDeletePolicy)) {
modifiedFields.add(BucketField.SOFT_DELETE_POLICY);
}
this.softDeletePolicy = softDeletePolicy;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@

package com.google.cloud.storage;

import com.google.api.client.util.Data;
import static com.google.cloud.storage.Storage.BucketField.SOFT_DELETE_POLICY;
import static com.google.cloud.storage.Utils.*;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert com.google.cloud.storage.Utils.*;


import com.google.api.pathtemplate.PathTemplate;
import com.google.cloud.Binding;
import com.google.cloud.Condition;
Expand Down Expand Up @@ -65,9 +67,6 @@
import java.util.Map;
import java.util.function.Function;

import static com.google.cloud.storage.Storage.BucketField.SOFT_DELETE_POLICY;
import static com.google.cloud.storage.Utils.*;

final class GrpcConversions {
static final GrpcConversions INSTANCE = new GrpcConversions();

Expand All @@ -90,7 +89,7 @@ final class GrpcConversions {
Codec.of(this::autoclassEncode, this::autoclassDecode);

private final Codec<BucketInfo.SoftDeletePolicy, Bucket.SoftDeletePolicy> softDeletePolicyCodec =
Codec.of(this::softDeletePolicyEncode, this::softDeletePolicyDecode);
Codec.of(this::softDeletePolicyEncode, this::softDeletePolicyDecode);
private final Codec<BucketInfo.LifecycleRule, Bucket.Lifecycle.Rule> lifecycleRuleCodec =
Codec.of(this::lifecycleRuleEncode, this::lifecycleRuleDecode);
private final Codec<BucketInfo, Bucket> bucketInfoCodec =
Expand Down Expand Up @@ -291,7 +290,7 @@ private BucketInfo bucketInfoDecode(Bucket from) {
if (from.hasAutoclass()) {
to.setAutoclass(autoclassCodec.decode(from.getAutoclass()));
}
if(from.hasSoftDeletePolicy()) {
if (from.hasSoftDeletePolicy()) {
to.setSoftDeletePolicy(softDeletePolicyCodec.decode(from.getSoftDeletePolicy()));
}
if (from.hasCustomPlacementConfig()) {
Expand Down Expand Up @@ -380,7 +379,8 @@ private Bucket bucketInfoEncode(BucketInfo from) {
ifNonNull(from.getIamConfiguration(), iamConfigurationCodec::encode, to::setIamConfig);
ifNonNull(from.getAutoclass(), autoclassCodec::encode, to::setAutoclass);
ifNonNull(from.getSoftDeletePolicy(), softDeletePolicyCodec::encode, to::setSoftDeletePolicy);
if(from.getModifiedFields().contains(SOFT_DELETE_POLICY) && from.getSoftDeletePolicy() == null) {
if (from.getModifiedFields().contains(SOFT_DELETE_POLICY)
&& from.getSoftDeletePolicy() == null) {
System.out.println("this is happening");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

debug code

System.out.println(Bucket.SoftDeletePolicy.getDefaultInstance().toString());
to.clearSoftDeletePolicy();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -412,22 +412,22 @@ public Blob restore(BlobId blob, BlobRestoreOption... options) {
private Blob internalObjectRestore(BlobId blobId, Opts<ObjectSourceOpt> opts) {
Opts<ObjectSourceOpt> finalOpts = opts.prepend(defaultOpts).prepend(ALL_BLOB_FIELDS);
GrpcCallContext grpcCallContext =
finalOpts.grpcMetadataMapper().apply(GrpcCallContext.createDefault());
finalOpts.grpcMetadataMapper().apply(GrpcCallContext.createDefault());
RestoreObjectRequest.Builder builder =
RestoreObjectRequest.newBuilder()
.setBucket(bucketNameCodec.encode(blobId.getBucket()))
.setObject(blobId.getName());
RestoreObjectRequest.newBuilder()
.setBucket(bucketNameCodec.encode(blobId.getBucket()))
.setObject(blobId.getName());
ifNonNull(blobId.getGeneration(), builder::setGeneration);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the API failure helpful when generation isn't supplied?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it says "you must specify a generation"

RestoreObjectRequest req = finalOpts.restoreObjectRequest().apply(builder).build();
GrpcCallContext merge = Utils.merge(grpcCallContext, Retrying.newCallContext());
return Retrying.run(
getOptions(),
retryAlgorithmManager.getFor(req),
() -> storageClient.restoreObjectCallable().call(req, merge),
resp -> {
BlobInfo tmp = codecs.blobInfo().decode(resp);
return finalOpts.clearBlobFields().decode(tmp).asBlob(this);
});
getOptions(),
retryAlgorithmManager.getFor(req),
() -> storageClient.restoreObjectCallable().call(req, merge),
resp -> {
BlobInfo tmp = codecs.blobInfo().decode(resp);
return finalOpts.clearBlobFields().decode(tmp).asBlob(this);
});
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,8 @@ public ResultRetryAlgorithm<?> getForObjectsGet(
return retryStrategy.getIdempotentHandler();
}

public ResultRetryAlgorithm<?> getForObjectsRestore(StorageObject pb, Map<StorageRpc.Option, ?> optionsMap) {
public ResultRetryAlgorithm<?> getForObjectsRestore(
StorageObject pb, Map<StorageRpc.Option, ?> optionsMap) {
return retryStrategy.getIdempotentHandler();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missed this on first pass; ObjectRestore is idempotent because it can only succeed once which is similar to Bucket; is that your understanding as well?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, restore has the same idempotency as Copy and similar operations

}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,8 +66,8 @@
import com.google.cloud.storage.BucketInfo.LifecycleRule.SetStorageClassLifecycleAction;
import com.google.cloud.storage.BucketInfo.Logging;
import com.google.cloud.storage.BucketInfo.ObjectRetention;
import com.google.cloud.storage.BucketInfo.SoftDeletePolicy;
import com.google.cloud.storage.BucketInfo.PublicAccessPrevention;
import com.google.cloud.storage.BucketInfo.SoftDeletePolicy;
import com.google.cloud.storage.Conversions.Codec;
import com.google.cloud.storage.Cors.Origin;
import com.google.cloud.storage.HmacKey.HmacKeyMetadata;
Expand Down Expand Up @@ -123,7 +123,7 @@ final class JsonConversions {
Codec.of(this::objectRetentionEncode, this::objectRetentionDecode);

private final Codec<SoftDeletePolicy, Bucket.SoftDeletePolicy> softDeletePolicyCodec =
Codec.of(this::softDeletePolicyEncode, this::softDeletePolicyDecode);
Codec.of(this::softDeletePolicyEncode, this::softDeletePolicyDecode);
private final Codec<LifecycleRule, Rule> lifecycleRuleCodec =
Codec.of(this::lifecycleRuleEncode, this::lifecycleRuleDecode);
private final Codec<LifecycleCondition, Condition> lifecycleConditionCodec =
Expand Down Expand Up @@ -379,13 +379,15 @@ private Retention retentionDecode(StorageObject.Retention from) {

private Bucket.SoftDeletePolicy softDeletePolicyEncode(SoftDeletePolicy from) {
Bucket.SoftDeletePolicy to = new Bucket.SoftDeletePolicy();
ifNonNull(from.getRetentionDuration(), durationSecondsCodec::encode, to::setRetentionDurationSeconds);
ifNonNull(
from.getRetentionDuration(), durationSecondsCodec::encode, to::setRetentionDurationSeconds);
return to;
}

private SoftDeletePolicy softDeletePolicyDecode(Bucket.SoftDeletePolicy from) {
SoftDeletePolicy.Builder to = SoftDeletePolicy.newBuilder();
ifNonNull(from.getRetentionDurationSeconds(), durationSecondsCodec::decode, to::setRetentionDuration);
ifNonNull(
from.getRetentionDurationSeconds(), durationSecondsCodec::decode, to::setRetentionDuration);
ifNonNull(from.getEffectiveTime(), dateTimeCodec::decode, to::setEffectiveTime);
return to.build();
}
Expand Down Expand Up @@ -461,7 +463,8 @@ private Bucket bucketInfoEncode(BucketInfo from) {
to::setCustomPlacementConfig);
ifNonNull(from.getObjectRetention(), this::objectRetentionEncode, to::setObjectRetention);
ifNonNull(from.getSoftDeletePolicy(), this::softDeletePolicyEncode, to::setSoftDeletePolicy);
if(from.getSoftDeletePolicy() == null && from.getModifiedFields().contains(SOFT_DELETE_POLICY)) {
if (from.getSoftDeletePolicy() == null
&& from.getModifiedFields().contains(SOFT_DELETE_POLICY)) {
to.setSoftDeletePolicy(Data.nullOf(Bucket.SoftDeletePolicy.class));
}
return to;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1580,8 +1580,8 @@ public static BlobGetOption shouldReturnRawInputStream(boolean shouldReturnRawIn

/**
* Returns an option for whether the request should return a soft-deleted object. If an object
* has been soft-deleted (Deleted while a Soft Delete Policy) is in place, this must be true
* or the request will return null.
* has been soft-deleted (Deleted while a Soft Delete Policy) is in place, this must be true or
* the request will return null.
*/
@TransportCompatibility({Transport.HTTP, Transport.GRPC})
public static BlobGetOption softDeleted(boolean softDeleted) {
Expand Down Expand Up @@ -1623,7 +1623,7 @@ public static BlobGetOption[] dedupe(BlobGetOption[] array, BlobGetOption... os)
}
}

/** Class for specifying blob restore options **/
/** Class for specifying blob restore options * */
class BlobRestoreOption extends Option<ObjectSourceOpt> {

private static final long serialVersionUID = 1922118465380110958L;
Expand All @@ -1632,7 +1632,6 @@ class BlobRestoreOption extends Option<ObjectSourceOpt> {
super(opt);
}


/**
* Returns an option for blob's data generation match. If this option is used the request will
* fail if generation does not match.
Expand Down Expand Up @@ -1670,7 +1669,8 @@ public static BlobRestoreOption metagenerationNotMatch(long generation) {
}

/**
* Returns an option for whether the restored object should copy the access controls of the source object.
* Returns an option for whether the restored object should copy the access controls of the
* source object.
*/
@TransportCompatibility({Transport.HTTP, Transport.GRPC})
public static BlobRestoreOption copySourceAcl(boolean copySourceAcl) {
Expand Down Expand Up @@ -1901,9 +1901,7 @@ public static BlobListOption fields(BlobField... fields) {
return new BlobListOption(UnifiedOpts.fields(set));
}

/**
* Returns an option for whether the list result should include soft-deleted objects.
*/
/** Returns an option for whether the list result should include soft-deleted objects. */
@TransportCompatibility({Transport.HTTP, Transport.GRPC})
public static BlobListOption softDeleted(boolean softDeleted) {
return new BlobListOption(UnifiedOpts.softDeleted(softDeleted));
Expand Down Expand Up @@ -3115,8 +3113,8 @@ Blob createFrom(
Blob get(BlobId blob);

/**
* Restores a soft-deleted object to full object status and returns the object.
* Note that you must specify a generation to use this method.
* Restores a soft-deleted object to full object status and returns the object. Note that you must
* specify a generation to use this method.
*
* <p>Example of restoring an object.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -340,23 +340,23 @@ public Blob get(BlobId blob, BlobGetOption... options) {
public Blob get(BlobId blob) {
return get(blob, new BlobGetOption[0]);
}

@Override
public Blob restore(BlobId blob, BlobRestoreOption... options) {
ImmutableMap<StorageRpc.Option, ?> optionsMap =
Opts.unwrap(options).resolveFrom(blob).getRpcOptions();
Opts.unwrap(options).resolveFrom(blob).getRpcOptions();

StorageObject obj = codecs.blobId().encode(blob);

ResultRetryAlgorithm<?> algorithm =
retryAlgorithmManager.getForObjectsRestore(obj, optionsMap);
ResultRetryAlgorithm<?> algorithm = retryAlgorithmManager.getForObjectsRestore(obj, optionsMap);

return run(
algorithm,
() -> storageRpc.restore(obj, optionsMap),
(x) -> {
BlobInfo info = Conversions.json().blobInfo().decode(x);
return info.asBlob(this);
});
algorithm,
() -> storageRpc.restore(obj, optionsMap),
(x) -> {
BlobInfo info = Conversions.json().blobInfo().decode(x);
return info.asBlob(this);
});
}

private static class BucketPageFetcher implements NextPageFetcher<Bucket> {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -481,9 +481,11 @@ static Projection projection(@NonNull String projection) {
static SoftDeleted softDeleted(boolean softDeleted) {
return new SoftDeleted(softDeleted);
}

static CopySourceAcl copySourceAcl(boolean copySourceAcl) {
return new CopySourceAcl(copySourceAcl);
}

static RequestedPolicyVersion requestedPolicyVersion(long l) {
return new RequestedPolicyVersion(l);
}
Expand Down Expand Up @@ -655,7 +657,8 @@ public Mapper<ListObjectsRequest.Builder> listObjects() {
}
}

static final class SoftDeleted extends RpcOptVal<Boolean> implements ObjectListOpt, ObjectSourceOpt {
static final class SoftDeleted extends RpcOptVal<Boolean>
implements ObjectListOpt, ObjectSourceOpt {

private static final long serialVersionUID = -8526951678111463350L;

Expand Down Expand Up @@ -1030,8 +1033,8 @@ public Mapper<GetObjectRequest.Builder> getObject() {
}

@Override
public Mapper <RestoreObjectRequest.Builder> restoreObject() {
return b-> b.setIfGenerationMatch(val);
public Mapper<RestoreObjectRequest.Builder> restoreObject() {
return b -> b.setIfGenerationMatch(val);
}

@Override
Expand Down