Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

perf(storage): remove protobuf's copy of data on unmarshalling #9526

Merged
merged 13 commits into from
Mar 19, 2024
+405 −12
Merged

perf(storage): remove protobuf's copy of data on unmarshalling #9526

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
58f5314
draft(storage): read optimization proof of concept
BrennaEpp Feb 21, 2024
112bae1
go mod tidy, nits
BrennaEpp Feb 21, 2024
d592214
add custom decoding for the first msg
BrennaEpp Mar 8, 2024
7f11278
change marshalling, add fields to err msg, add crc32c parsing
BrennaEpp Mar 14, 2024
427ad59
clarify comment
BrennaEpp Mar 14, 2024
5ca2d2f
Merge branch 'main' into protobuf-1
BrennaEpp Mar 14, 2024
8ef71fb
update comment
BrennaEpp Mar 14, 2024
d1a9f00
add test + fix empty struct decoding
BrennaEpp Mar 15, 2024
85335d0
update headrer
BrennaEpp Mar 15, 2024
50f8286
Merge branch 'main' into protobuf-1
BrennaEpp Mar 15, 2024
5b2a0f0
Merge branch 'main' into protobuf-1
BrennaEpp Mar 18, 2024
07e6b4f
loop over bytes instead of reusing function
BrennaEpp Mar 19, 2024
b06366b
Merge branch 'main' into protobuf-1
BrennaEpp Mar 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Next Next commit
go mod tidy, nits
  • Loading branch information
@BrennaEpp
BrennaEpp committed Feb 21, 2024
commit 112bae14f1701207f84ada8eba1d59eadf9b4129
2 changes: 1 addition & 1 deletion storage/go.mod
View file
Open in desktop
Expand Up @@ -8,6 +8,7 @@ require (
cloud.google.com/go v0.112.0
cloud.google.com/go/compute/metadata v0.2.3
cloud.google.com/go/iam v1.1.6
github.com/golang/protobuf v1.5.3
github.com/google/go-cmp v0.6.0
github.com/google/uuid v1.6.0
github.com/googleapis/gax-go/v2 v2.12.0
Expand All @@ -26,7 +27,6 @@ require (
github.com/go-logr/logr v1.4.1 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.3 // indirect
github.com/google/martian/v3 v3.3.2 // indirect
github.com/google/s2a-go v0.1.7 // indirect
github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
Expand Down
9 changes: 4 additions & 5 deletions storage/grpc_client.go
View file
Open in desktop
Expand Up @@ -27,6 +27,7 @@
"cloud.google.com/go/internal/trace"
gapic "cloud.google.com/go/storage/internal/apiv2"
"cloud.google.com/go/storage/internal/apiv2/storagepb"
"github.com/golang/protobuf/proto"
"github.com/googleapis/gax-go/v2"
"google.golang.org/api/googleapi"
"google.golang.org/api/iterator"
Expand All @@ -39,8 +40,6 @@
"google.golang.org/grpc/status"
"google.golang.org/protobuf/encoding/protowire"
fieldmaskpb "google.golang.org/protobuf/types/known/fieldmaskpb"

"github.com/golang/protobuf/proto"
)

const (
Expand Down Expand Up @@ -926,8 +925,8 @@
func (bytesCodec) Unmarshal(data []byte, v any) error {
switch v := v.(type) {
case *[]byte:
// gRPC can recycle the data []byte after unmarshaling,
// so we need to make a copy here.
// If gRPC could recycle the data []byte after unmarshaling (through
// buffer pools), we would need to make a copy here.
*v = data
return nil
case proto.Message:
Expand All @@ -949,7 +948,7 @@

s := callSettings(c.settings, opts...)

s.gax = append(s.gax[:len(s.gax)], gax.WithGRPCOptions(
s.gax = append(s.gax, gax.WithGRPCOptions(
grpc.ForceCodec(bytesCodec{}),
))

Expand Down Expand Up @@ -1558,8 +1557,8 @@
// ReadObjectResponse message.
func readObjectResponseContent(b []byte) ([]byte, error) {
const (
readObjectResponse_checksummedData = protowire.Number(1)

Check failure on line 1560 in storage/grpc_client.go

View workflow job for this annotation

GitHub Actions / vet 

don't use underscores in Go names; const readObjectResponse_checksummedData should be readObjectResponseChecksummedData
BrennaEpp marked this conversation as resolved.
Show resolved Hide resolved
checksummedData_content = protowire.Number(1)

Check failure on line 1561 in storage/grpc_client.go

View workflow job for this annotation

GitHub Actions / vet 

don't use underscores in Go names; const checksummedData_content should be checksummedDataContent
)
checksummedData := readProtoBytes(b, readObjectResponse_checksummedData)
content := readProtoBytes(checksummedData, checksummedData_content)
Expand Down