Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(datastore): Adding BeginLater transaction option implementation #8972

Merged
merged 11 commits into from
Apr 23, 2024

Conversation

bhshkh
Copy link
Contributor

@bhshkh bhshkh commented Nov 3, 2023

What is new_transaction ?
ReadOptions are the options shared by Datastore read requests.
new_transaction is a consistency type in ReadOptions. When new_transaction is used in a read request, a new transaction is begun for the request.

This PR adds following:

  1. BeginLater transaction option implementation
  2. Mutex for state in the transaction struct

Currently, when user calls NewTransaction method, client library makes a BeginTransaction rpc call and obtains transaction id. Reads (RunQuery / RunAggregationQuery / LookupRequest), Commit and Rollback use this transaction id in ReadOptions of rpc request.

Without BeginLater transaction option: the above behavior remains unchanged.

With BeginLater transaction option:

  1. Calling NewTransaction method does not make a BeginTransaction rpc call.
  2. The first read rpc passes new_transaction as consistency type to obtain transaction id.
  3. If there is no read called before commit / rollback, begin transaction is called to obtain transaction id before commit / rollback rpc.
    Since there is one less round trip to the Datastore service in point 1 above, user may see lesser latencies when using BeginLater transaction option.

@bhshkh bhshkh requested review from a team as code owners November 3, 2023 15:50
@product-auto-label product-auto-label bot added the size: xl Pull request size is extra large. label Nov 3, 2023
@bhshkh bhshkh requested a review from enocom November 3, 2023 15:50
@product-auto-label product-auto-label bot added the api: datastore Issues related to the Datastore API. label Nov 3, 2023
@bhshkh bhshkh enabled auto-merge (squash) November 3, 2023 15:50
@bhshkh bhshkh disabled auto-merge November 3, 2023 15:50
@bhshkh bhshkh changed the title feat(datastore): Adding BeginLater transaction option to improve performance feat(datastore): Adding BeginLater transaction option Nov 3, 2023
@bhshkh bhshkh marked this pull request as draft November 7, 2023 21:41
@product-auto-label product-auto-label bot added the stale: old Pull request is old and needs attention. label Dec 4, 2023
@product-auto-label product-auto-label bot added stale: extraold Pull request is critically old and needs prioritization. and removed stale: old Pull request is old and needs attention. labels Jan 3, 2024
@bhshkh bhshkh marked this pull request as ready for review January 26, 2024 18:44
@bhshkh bhshkh requested a review from a team as a code owner January 26, 2024 18:44
@bhshkh bhshkh marked this pull request as draft January 26, 2024 18:44
@product-auto-label product-auto-label bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels Feb 5, 2024
@bhshkh bhshkh marked this pull request as ready for review February 5, 2024 23:57
@bhshkh bhshkh enabled auto-merge (squash) February 5, 2024 23:57
@bhshkh bhshkh changed the title feat(datastore): Adding BeginLater transaction option feat(datastore): Adding BeginLater transaction option implementation Feb 5, 2024
Copy link

@danieljbruce danieljbruce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initial review with a few comments and questions

datastore/query.go Show resolved Hide resolved
datastore/transaction.go Outdated Show resolved Hide resolved
datastore/transaction.go Outdated Show resolved Hide resolved
@bhshkh bhshkh added the do not merge Indicates a pull request not ready for merge, due to either quality or timing. label Mar 29, 2024
@bhshkh
Copy link
Contributor Author

bhshkh commented Mar 29, 2024

Do not merge until release freeze ends in mid April

@danieljbruce
Copy link

Does this client support retry transactions? Like before any of these changes, if the user provides an id when creating a transaction then will calling a read function make a grpc read request with that transaction id like it does in Node Datastore?

Copy link

@danieljbruce danieljbruce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added one question about retry transactions to make sure there is parity with the Node client.

@bhshkh
Copy link
Contributor Author

bhshkh commented Apr 5, 2024

Does this client support retry transactions? Like before any of these changes, if the user provides an id when creating a transaction then will calling a read function make a grpc read request with that transaction id like it does in Node Datastore?

before and after any of these changes, no. Go client did not have retry transaction support

Copy link

@danieljbruce danieljbruce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall. Just a few nits and questions.

return func() {}
}

func (t *Transaction) startProgress(id []byte) {

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe change the function name to setToInProgress or something like that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #10032

settings: &transactionSettings{},
},
{
desc: "[RunAggregationQuery, Get, Put, Commit] BeginLater. RunAggregationQuery passes new_transaction",

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this test verifies that runAggregationQuery will pass in new transaction options. Just for my own knowledge though, what change in this PR allows newTransaction options to be passed in for the runAggregationQuery call? Does runAggregationQuery make a call to the new parseReadOptions()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@@ -88,3 +88,398 @@ func TestNewTransaction(t *testing.T) {
}
}
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Other tests to consider if necessary:

  1. Make sure readTime is passed along for read only transactions
  2. Make sure prevId is passed along and used for read/write transactions

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bhshkh bhshkh merged commit 4067f4e into googleapis:main Apr 23, 2024
8 of 9 checks passed
@bhshkh bhshkh deleted the feature/new-transaction branch April 23, 2024 20:50
@bhshkh bhshkh requested a review from gkevinzheng April 23, 2024 20:52
@@ -889,6 +886,8 @@ func TestAggregationQueryIsNil(t *testing.T) {
}

func TestValidateReadOptions(t *testing.T) {
eventualInTxnErr := errors.New("datastore: cannot use EventualConsistency query in a transaction")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Since you're using this exact error in testing as you are in the code, would it be worth it to have this variable be defined in the code so you can import it in the test package, similar to errExpiredTransaction?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #10063

return nil
}

// Acquire state lock if transaction has not started
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add to the comment for acquireLock more about what it is, what its return value signifies, and how it's used in other parts of the PR? It does a lot more than simply acquiring the state lock.

Also because this function does a lot more than just acquiring the state lock, could you rename the function to reflect that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Resolved in #10063

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: datastore Issues related to the Datastore API. do not merge Indicates a pull request not ready for merge, due to either quality or timing. size: l Pull request size is large. stale: extraold Pull request is critically old and needs prioritization.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants