Skip to content

models - litellm - capture usage #73

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 21, 2025
Merged

Conversation

pgrayy
Copy link
Member

@pgrayy pgrayy commented May 21, 2025

Description

Certain models executed under LiteLLM will return additional payloads inbetween the finish_reason message and the usage message. Here is an example of an inbetween payload:

ModelResponseStream(id='chatcmpl-3cdd4223-9c65-4ce0-9456-0a41d4a70b86', created=1747836095, model='gpt-4.1-mini-2025-04-14', object='chat.completion.chunk', system_fingerprint='fp_c8225066ea', choices=[StreamingChoices(finish_reason=None, index=0, delta=Delta(provider_specific_fields=None, content=None, role=None, function_call=None, tool_calls=None, audio=None), logprobs=None)], provider_specific_fields=None, stream_options={'include_usage': True})

We don't have any use for these inbetween messages and so for now, we will skip over them so we can get to the final usage message for reporting.

Related Issues

#47

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Other (please describe):

Testing

  • hatch fmt --linter
  • hatch fmt --formatter
  • hatch test --all: Relying on existing unit tests and integration tests. They still cover these changes.
  • Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

Checklist

  • I have read the CONTRIBUTING document
  • I have added tests that prove my fix is effective or my feature works
  • I have updated the documentation accordingly
  • I have added an appropriate example to the documentation to outline the feature
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@pgrayy pgrayy requested a review from a team as a code owner May 21, 2025 14:29
zastrowm
zastrowm previously approved these changes May 21, 2025
@pgrayy pgrayy merged commit d0a57a9 into strands-agents:main May 21, 2025
10 checks passed
@pgrayy pgrayy deleted the litellm-usage branch May 27, 2025 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants