Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add Custom Part Metadata Decorator to ParallelCompositeUploadConfig #2434

Merged
merged 24 commits into from
Mar 15, 2024
Merged

feat: Add Custom Part Metadata Decorator to ParallelCompositeUploadConfig #2434

Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
33122ad
feat: Custom Time Part Metadata Strategy
sydney-munro Mar 4, 2024
fe168d6
set customtime directly instead of via metadata builder
sydney-munro Mar 4, 2024
d14d141
lint and change null to IllegalStateException
sydney-munro Mar 4, 2024
9f12759
Add in version to javadoc
sydney-munro Mar 5, 2024
71ec5cf
removing unnecessary override
sydney-munro Mar 5, 2024
8b64aa6
removing unnecessary tag
sydney-munro Mar 5, 2024
15a29dc
address pr comments
sydney-munro Mar 13, 2024
b1bc015
add in ability to set a specific clock
sydney-munro Mar 13, 2024
19297a0
chore: refactor and separate lifecycle of config from customer and wh…
BenWhitehead Mar 13, 2024
4a51be1
chore: add pool size validation
BenWhitehead Mar 13, 2024
2a98f6c
IT test to ensure customtime is updated
sydney-munro Mar 14, 2024
746b581
lint
sydney-munro Mar 14, 2024
4acfa8c
Adding in more tests
sydney-munro Mar 14, 2024
f4e7fa8
lint
sydney-munro Mar 14, 2024
e88574b
more tests
sydney-munro Mar 14, 2024
57597dd
lint
sydney-munro Mar 14, 2024
80c747d
add missing null check
sydney-munro Mar 14, 2024
2adc0e5
address pr comments
sydney-munro Mar 14, 2024
6678349
unit test for offset calculations
sydney-munro Mar 15, 2024
7d39672
fixing docs
sydney-munro Mar 15, 2024
f7c4358
lint
sydney-munro Mar 15, 2024
6762593
Update google-cloud-storage/src/main/java/com/google/cloud/storage/Pa…
sydney-munro Mar 15, 2024
4f51449
Apply suggestions from code review
sydney-munro Mar 15, 2024
4861c85
lint
sydney-munro Mar 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Next Next commit
lint
  • Loading branch information
@sydney-munro
sydney-munro committed Mar 15, 2024
commit 57597dd14c904f008b3811bddd514d231b6e4fc5
47 changes: 22 additions & 25 deletions ...rc/test/java/com/google/cloud/storage/ParallelCompositeUploadWritableByteChannelTest.java
View file
Open in desktop
Expand Up @@ -25,11 +25,9 @@
import com.google.api.gax.grpc.GrpcStatusCode;
import com.google.api.gax.rpc.ApiExceptionFactory;
import com.google.api.gax.rpc.ApiExceptions;
import com.google.cloud.storage.BlobWriteSessionConfig.WriterFactory;
import com.google.cloud.storage.BufferHandlePool.PooledBuffer;
import com.google.cloud.storage.Crc32cValue.Crc32cLengthKnown;
import com.google.cloud.storage.MetadataField.PartRange;
import com.google.cloud.storage.ParallelCompositeUploadBlobWriteSessionConfig.BufferAllocationStrategy;
import com.google.cloud.storage.ParallelCompositeUploadBlobWriteSessionConfig.PartCleanupStrategy;
import com.google.cloud.storage.ParallelCompositeUploadBlobWriteSessionConfig.PartMetadataFieldDecorator;
import com.google.cloud.storage.ParallelCompositeUploadBlobWriteSessionConfig.PartMetadataFieldDecoratorInstance;
Expand All @@ -55,7 +53,6 @@
import java.nio.ByteBuffer;
import java.nio.channels.AsynchronousCloseException;
import java.nio.channels.ClosedChannelException;
import java.nio.channels.WritableByteChannel;
import java.time.Duration;
import java.time.Instant;
import java.time.OffsetDateTime;
Expand Down Expand Up @@ -768,21 +765,20 @@ public BlobInfo internalObjectGet(BlobId blobId, Opts<ObjectSourceOpt> opts) {
.containsExactly(p1, p2, p3, expectedId),
() -> assertThat(storageInternal.deleteRequests).containsExactly(p1, p2, p3));
}

@Test
public void partMetadataFieldDecoratorUsesCustomTime()
throws IOException {
public void partMetadataFieldDecoratorUsesCustomTime() throws IOException {
TestClock clock = TestClock.tickBy(Instant.EPOCH, Duration.ofSeconds(1));
OffsetDateTime rangeBegin = OffsetDateTime.from(
Instant.EPOCH.plus(Duration.ofSeconds(29)).atZone(
ZoneId.of("Z")));
OffsetDateTime rangeEnd = OffsetDateTime.from(Instant.EPOCH.plus(Duration.ofMinutes(2)).atZone(
ZoneId.of("Z")));
OffsetDateTime rangeBegin =
OffsetDateTime.from(Instant.EPOCH.plus(Duration.ofSeconds(29)).atZone(ZoneId.of("Z")));
OffsetDateTime rangeEnd =
OffsetDateTime.from(Instant.EPOCH.plus(Duration.ofMinutes(2)).atZone(ZoneId.of("Z")));

FakeStorageInternal storageInternal =
new FakeStorageInternal() {
@Override
public BlobInfo internalDirectUpload(BlobInfo info, Opts<ObjectTargetOpt> opts,
ByteBuffer buf) {
public BlobInfo internalDirectUpload(
BlobInfo info, Opts<ObjectTargetOpt> opts, ByteBuffer buf) {
if (info.getBlobId().getName().endsWith(".part")) {
// Kinda hacky but since we are creating multiple parts we will use a range
// to ensure the customTimes are being calculated appropriately
Expand All @@ -793,24 +789,25 @@ public BlobInfo internalDirectUpload(BlobInfo info, Opts<ObjectTargetOpt> opts,
}
return super.internalDirectUpload(info, opts, buf);
}

};
ParallelCompositeUploadWritableByteChannel pcu = new ParallelCompositeUploadWritableByteChannel(
bufferHandlePool,
MoreExecutors.directExecutor(),
partNamingStrategy,
PartCleanupStrategy.always(),
10,
PartMetadataFieldDecorator.setCustomTimeInFuture(Duration.ofSeconds(30)).newInstance(clock),
finalObject,
storageInternal,
info,
opts);
ParallelCompositeUploadWritableByteChannel pcu =
new ParallelCompositeUploadWritableByteChannel(
bufferHandlePool,
MoreExecutors.directExecutor(),
partNamingStrategy,
PartCleanupStrategy.always(),
10,
PartMetadataFieldDecorator.setCustomTimeInFuture(Duration.ofSeconds(30))
.newInstance(clock),
finalObject,
storageInternal,
info,
opts);
byte[] bytes = DataGenerator.base64Characters().genBytes(bufferCapacity * 3 - 1);
pcu.write(ByteBuffer.wrap(bytes));

pcu.close();
}
}

sydney-munro marked this conversation as resolved.
Show resolved Hide resolved
@NonNull
private ParallelCompositeUploadWritableByteChannel defaultPcu(int maxElementsPerCompact) {
Expand Down