Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: include context on batch log errors #650

Merged
merged 10 commits into from
Oct 18, 2022
Merged

feat: include context on batch log errors #650

merged 10 commits into from
Oct 18, 2022

Conversation

daniel-sanche
Copy link
Contributor

Sometimes, a log may fail as part of a larger batch of logs (eg, due to payload size). These exceptions can be hard to track down, because batches happen asynchronously on a background thread

This PR attempts to help resolve the issue by adding a truncated copy of the failing log to the batch log exception message

  File "router.py", line 86, in pubsub_callback
    found_func(**kwargs)
  File "/app/snippets.py", line 81, in batch_large_log
    batch.commit()
  File "/app/python-logging/google/cloud/logging_v2/logger.py", line 470, in commit
    raise e
  File "/app/python-logging/google/cloud/logging_v2/logger.py", line 465, in commit
    client.logging_api.write_entries(entries, partial_success=True, **kwargs)
  File "/app/python-logging/google/cloud/logging_v2/_gapic.py", line 160, in write_entries
    self._gapic_api.write_log_entries(request=request)
  File "/app/python-logging/google/cloud/logging_v2/services/logging_service_v2/client.py", line 725, in write_log_entries
    metadata=metadata,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 154, in __call__
    return wrapped_func(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 288, in retry_wrapped_func
    on_error=on_error,
  File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 190, in retry_target
    return target()
  File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 74, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.InvalidArgument: 400 Log entry with size 263.8K exceeds maximum size of 256.0K: StructEntry(log_name=None, labels=None, insert_id=None, severity='DEFAULT', http_request=None, timestamp=None, resource=Resource(type='global', labels={}), trace=None, span_id=None, trace_sampled=None, source_location=None, operation=None, logger=None, payload={'message': 'simple_log', 'buffer': '0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000... [type_url: "type.googleapis.com/google.logging.v2.WriteLogEntriesPartialErrors"
value: "\nA\010\007\022=\010\003\0229Log entry with size 263.8K exceeds maximum size of 256.0K"
, detail: "[ORIGINAL ERROR] generic::invalid_argument: Log entry with size 263.8K exceeds maximum size of 256.0K [google.rpc.error_details_ext] { message: \"Log entry with size 263.8K exceeds maximum size of 256.0K\" details { [type.googleapis.com/google.logging.v2.WriteLogEntriesPartialErrors] { log_entry_errors { key: 7 value { code: 3 message: \"Log entry with size 263.8K exceeds maximum size of 256.0K\" } } } } }"
]

Fixes: #53

@product-auto-label product-auto-label bot added size: s Pull request size is small. api: logging Issues related to the googleapis/python-logging API. labels Oct 12, 2022
losalex
losalex reviewed Oct 13, 2022
View reviewed changes
Copy link
Contributor

@losalex losalex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets some comments

google/cloud/logging_v2/logger.py Outdated Show resolved Hide resolved
google/cloud/logging_v2/logger.py Show resolved Hide resolved
@product-auto-label product-auto-label bot added size: m Pull request size is medium. and removed size: s Pull request size is small. labels Oct 13, 2022
@daniel-sanche daniel-sanche changed the title [DRAFT] feat: include context on batch log errors feat: include context on batch log errors Oct 17, 2022
@daniel-sanche daniel-sanche marked this pull request as ready for review October 17, 2022 17:38
@daniel-sanche daniel-sanche requested review from a team as code owners October 17, 2022 17:38
@daniel-sanche daniel-sanche mentioned this pull request Oct 17, 2022
Closed
@daniel-sanche daniel-sanche added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Oct 17, 2022
@yoshi-kokoro yoshi-kokoro removed kokoro:force-run Add this label to force Kokoro to re-run the tests. labels Oct 17, 2022
@daniel-sanche daniel-sanche merged commit d08be9a into main Oct 18, 2022
@daniel-sanche daniel-sanche deleted the detailed_large_log_errors branch October 18, 2022 18:55
@release-please release-please bot mentioned this pull request Oct 18, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@losalex losalex losalex approved these changes

Assignees
No one assigned
Labels
api: logging Issues related to the googleapis/python-logging API. size: m Pull request size is medium.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improve handling of "Log entry with size X exceeds maximum size"
3 participants
@daniel-sanche @losalex @yoshi-kokoro