async models #306

pgrayy · 2025-06-27T20:48:07Z

Description

We are currently working on support for an iterative async stream method on the agent class (#83). As part of this work, we need to convert any component that yields model events into async generator.

Related Issues

#83

Type of Change

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

I ran hatch run prepare

Checklist

I have read the CONTRIBUTING document
I have added any necessary tests that prove my fix is effective or my feature works
I have updated the documentation accordingly
I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
My changes generate no new warnings
Any dependent changes have been merged and published

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

pgrayy · 2025-06-27T20:49:18Z

src/strands/agent/agent.py

@@ -497,14 +500,16 @@ def _run_loop(
            self.messages.append(new_message)

            # Execute the event loop cycle with retry logic for context limits
-            yield from self._execute_event_loop_cycle(callback_handler, kwargs)


For some reason you can't yield from in an async generator. Consequently, we have to convert to individual yields.

zastrowm · 2025-06-27T21:02:05Z

tests/strands/event_loop/test_event_loop.py

@@ -484,7 +521,7 @@ def test_event_loop_cycle_stop(
        tool_execution_handler=tool_execution_handler,
        request_state={"stop_event_loop": True},
    )
-    event = list(stream)[-1]
+    event = [event async for event in stream][-1]


Can we have a helper to do this - presumedly we'll need to collect events a lot? (maybe not though)

Done. I defined this helper in conftest.py and named it alist. I renamed my existing alist helper to agenerator.

zastrowm · 2025-06-27T21:05:05Z

src/strands/agent/agent.py

            events = self._run_loop(callback_handler, prompt, kwargs)
-            for event in events:
+            async for event in events:


Is it possible to call stream_async here or unify it so that they're both just plumbing to the same place?

And with #289 we'll now removing the callback_handler override, so I think this gets simpler overall. Maybe coordinate with @Unshure to see what else simplifies down

zastrowm · 2025-06-27T21:06:57Z

src/strands/agent/agent.py

-            stop_reason, message, metrics, state = event["stop"]
-            result = AgentResult(stop_reason, message, metrics, state)
-
+            result = AgentResult(*event["stop"])


💭 I wonder if the Async Iterator should be emitting something like this anyway (allowing us to unify the __call__ method even more)

See revision.

zastrowm · 2025-06-27T21:08:25Z

tests/strands/agent/test_agent.py

@@ -161,6 +161,15 @@ def agent(
    return agent


+@pytest.fixture
+def alist():
+    async def gen(items):


where does items come from here? is this python magic?

This fixture is returning a function and so items is just the argument you pass to the call. For example:

def test_me(alist): await alist([1, 2, 3])

[1, 2, 3] is items. Note though, this has been renamed to agenerator in my revision.

tests/strands/event_loop/test_event_loop.py

src/strands/models/anthropic.py

…nto async-components

pgrayy · 2025-06-30T15:55:44Z

src/strands/agent/agent.py

-        except Exception as e:
-            self._end_agent_trace_span(error=e)
-            raise
+        return asyncio.run(acall())


Note that there is overhead in running asyncio.run. It creates an event loop on every invocation. However, this overhead should be negligible compared to the underlying LLM calls.

Additional call out, you cannot call asyncio.run in an async context and so this could technically be a breaking change for some customers. As an example:

async def func_a(): return 1 def func_b(): asyncio.run(func_a()) async def func_c(): return func_b() asyncio.run(func_c())

The above will result in the following exception:

RuntimeError: asyncio.run() cannot be called from a running event loop

As a fix, we just ask our async callers to use stream_async instead of __call__. Added a note about this for #253.

As a fix, we just ask our async callers to use stream_async instead of call. Added a note about this for #253.

is a big lift as they do different things. I think instead we need a call_async which does the same thing

Additional call out, you cannot call asyncio.run in an async context and so this could technically be a breaking change for some customers. As an example:

I think we should explore threading in this case. Otherwise it breaks Jupyter notebooks, for instance

Good call out. I just tested the notebook use case and it will break without a thread. I'll add threading and expose a new invoke_call method to replace acall.

pgrayy · 2025-06-30T16:00:01Z

src/strands/agent/agent.py


-            self._end_agent_trace_span(response=result)
+        async def acall() -> AgentResult:


I wanted to avoid having to define a helper function to invoke stream_async. Unfortunately, asyncio.run cannot call async generators. There is no equivalent method for this either in asyncio. Consequently, I define acall to return (instead of yield) the agent result.

I think ultimately we might want an async version of __call__; should we just expose acall as call_async?

To mirror bedrock naming, I'll create an invoke_async method. So we will have stream_async and invoke_async.

…nto async-components

zastrowm

Let's make sure this does not go out today if we're releasing.

I'd like to bug bash on this before it shipped a bit

zastrowm · 2025-07-01T19:38:13Z

src/strands/agent/agent.py

-        except Exception as e:
-            self._end_agent_trace_span(error=e)
-            raise
+        return asyncio.run(acall())


As a fix, we just ask our async callers to use stream_async instead of call. Added a note about this for #253.

is a big lift as they do different things. I think instead we need a call_async which does the same thing

zastrowm · 2025-07-01T20:11:32Z

src/strands/agent/agent.py


-            self._end_agent_trace_span(response=result)
+        async def acall() -> AgentResult:


I think ultimately we might want an async version of __call__; should we just expose acall as call_async?

zastrowm · 2025-07-01T20:12:20Z

src/strands/agent/agent.py

-        except Exception as e:
-            self._end_agent_trace_span(error=e)
-            raise
+        return asyncio.run(acall())


Additional call out, you cannot call asyncio.run in an async context and so this could technically be a breaking change for some customers. As an example:

I think we should explore threading in this case. Otherwise it breaks Jupyter notebooks, for instance

src/strands/models/anthropic.py

async models

01f7d1b

pgrayy temporarily deployed to auto-approve June 27, 2025 20:48 — with GitHub Actions Inactive

pgrayy commented Jun 27, 2025

View reviewed changes

zastrowm reviewed Jun 27, 2025

View reviewed changes

ryanycoleman added this to Strands Agents Roadmap Jun 27, 2025

ryanycoleman moved this to We're Working On It in Strands Agents Roadmap Jun 27, 2025

pgrayy marked this pull request as draft June 30, 2025 13:35

pgrayy added 3 commits June 30, 2025 13:38

Merge branch 'main' of https://github.com/strands-agents/sdk-python i…

0fd2671

…nto async-components

lint

ab2c088

tests

3fd243e

pgrayy temporarily deployed to auto-approve June 30, 2025 13:43 — with GitHub Actions Inactive

pgrayy added 11 commits June 30, 2025 14:43

agent - asyncio.run stream_async in call

d9deb93

tests - agenerator helper

b864b65

tests - agent - stream async result

bd2adff

lint

7ea90e9

agent - stream async - result

3462e1d

typing

8063c08

tests - anext

f8e58a1

tests - alist

5bb0620

lint

760fcfb

tests - async utilities - scope session

61bb44d

tests integ - conftest

e47567f

pgrayy temporarily deployed to auto-approve June 30, 2025 15:49 — with GitHub Actions Inactive

pgrayy commented Jun 30, 2025

View reviewed changes

pgrayy marked this pull request as ready for review June 30, 2025 16:04

dbschmigelski mentioned this pull request Jun 30, 2025

feat: Implement the core system of typed hooks & callbacks #304

Open

7 tasks

pgrayy mentioned this pull request Jun 30, 2025

[FEATURE] Implement native async iterator support #83

Open

pgrayy added 3 commits July 1, 2025 17:03

Merge branch 'main' of https://github.com/strands-agents/sdk-python i…

90aaa47

…nto async-components

lint

ccc44d9

tests - async mock model provider

e8c7bda

lint

17b24b3

pgrayy temporarily deployed to auto-approve July 1, 2025 19:23 — with GitHub Actions Inactive

zastrowm reviewed Jul 1, 2025

View reviewed changes


		self._end_agent_trace_span(response=result)
		async def acall() -> AgentResult:

async models #306

Are you sure you want to change the base?

async models #306

Uh oh!

Conversation

pgrayy commented Jun 27, 2025

Description

Related Issues

Type of Change

Testing

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgrayy Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pgrayy Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zastrowm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pgrayy Jun 30, 2025 •

edited

Loading

pgrayy Jul 1, 2025 •

edited

Loading