Python: Fix for AzureAIAgent streaming func call invocations. Other fixes/improvements for tool call content types, and logging. (#12335)

### Motivation and Context

The AzureAIAgent streaming function call invocation is broken when
handling more than one required function call from the service. There is
nested logic to handle passing the tool calls back to the service
because it requires one to pass in a handler.

In this PR, we're now controlling everything from the main loop and are
passing back the handler as the event stream for further processing.
This does require some logic to determine if we have an initial async
context manager (based on the first call to create the run stream) or if
we're now using the handler from submitting tool calls. We can now see
that when we have multiple functions involved, we are able to get the
results as we expect:

```
Sample Output:

# User: 'What is the price of the special drink and then special food item added together?'
Function Call:> MenuPlugin-get_specials with arguments: {}
Function Result:> 
        Special Soup: Clam Chowder
        Special Salad: Cobb Salad
        Special Drink: Chai Tea
        for function: MenuPlugin-get_specials
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Chai Tea"}
Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Clam Chowder"}
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Result:> $9.99 for function: MenuPlugin-get_item_price
Function Call:> MathPlugin-Add with arguments: {"input":9.99,"amount":9.99}
Function Result:> 19.98 for function: MathPlugin-Add
# AuthorRole.ASSISTANT: The price of the special drink, Chai Tea, is $9.99 and the price of the special food 
    item, Clam Chowder, is $9.99. Added together, the total price is $19.98.
```

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Fix handling multiple function calls for the streaming AzureAIAgent.
- Fixes a missing `quote` from the url citation for streaming annotation
content
- Shows the correct way to retrieve Bing Grounding tool calls in the two
concept samples.
- Adds handling for tool content content for OpenAPI plugins.
- Adds logging for streaming agent invocation tool calls.
- Relaxes the handling of the `FunctionResultContent` id attribute to be
`str | None` similar to that of `FunctionCallContent`. The type hints
for `id` have always been `str | None`, but the actual type hint on the
attribute has been `str`. We don't always get a `FunctionResultContent`
id from the Azure Agent Service during streaming invocations - message
addition via the overide `__add__` method still work properly.
- Move the AzureAIAgent pkg dependencies to be installed by default,
instead of requiring the `azure` extra.
- Closes #12312
- Closes #12328
- Closes #12331
- Closes #12324

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄

Author: moonbox3

python/pyproject.toml

@@ -23,6 +23,9 @@ classifiers = [
   "Typing :: Typed",
 ]
 dependencies = [
+    # azure agents
+    "azure-ai-projects >= 1.0.0b11",
+    "azure-ai-agents >= 1.0.0",
     "aiohttp ~= 3.8",
     "cloudevents ~=1.0",
     "pydantic >=2.0,!=2.10.0,!=2.10.1,!=2.10.2,!=2.10.3,<2.12",
@@ -67,10 +70,7 @@ autogen = [
 aws = [
     "boto3>=1.36.4,<1.39.0",
 ]
-# This is temporary to get the PR reviewed, and will be replaced when their new version is public.
 azure = [
-    "azure-ai-projects >= 1.0.0b11",
-    "azure-ai-agents >= 1.0.0",
     "azure-ai-inference >= 1.0.0b6",
     "azure-core-tracing-opentelemetry >= 1.0.0b11",
     "azure-search-documents >= 11.6.0b4",
@@ -132,7 +132,7 @@ qdrant = [
     "qdrant-client ~= 1.9"
 ]
 redis = [
-    "redis[hiredis] >= 5,< 7",
+    "redis[hiredis] >= 5,< 6",
     "types-redis ~= 4.6.0.20240425",
     "redisvl ~= 0.4"
 ]

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_bing_grounding.py

@@ -6,7 +6,12 @@
 from azure.identity.aio import DefaultAzureCredential
 
 from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
-from semantic_kernel.contents import AnnotationContent
+from semantic_kernel.contents import (
+    AnnotationContent,
+    ChatMessageContent,
+    FunctionCallContent,
+    FunctionResultContent,
+)
 
 """
 The following sample demonstrates how to create an Azure AI agent that
@@ -20,13 +25,23 @@
 TASK = "Which team won the 2025 NCAA basketball championship?"
 
 
+async def handle_intermediate_steps(message: ChatMessageContent) -> None:
+    for item in message.items or []:
+        if isinstance(item, FunctionResultContent):
+            print(f"Function Result:> {item.result} for function: {item.name}")
+        elif isinstance(item, FunctionCallContent):
+            print(f"Function Call:> {item.name} with arguments: {item.arguments}")
+        else:
+            print(f"{item}")
+
+
 async def main() -> None:
     async with (
         DefaultAzureCredential() as creds,
         AzureAIAgent.create_client(credential=creds) as client,
     ):
         # 1. Enter your Bing Grounding Connection Name
-        bing_connection = await client.connections.get(connection_name="<your-bing-grounding-connection-name>")
+        bing_connection = await client.connections.get(name="<your-bing-grounding-connection-name>")
         conn_id = bing_connection.id
 
         # 2. Initialize agent bing tool and add the connection id
@@ -54,7 +69,9 @@ async def main() -> None:
         try:
             print(f"# User: '{TASK}'")
             # 6. Invoke the agent for the specified thread for response
-            async for response in agent.invoke(messages=TASK, thread=thread):
+            async for response in agent.invoke(
+                messages=TASK, thread=thread, on_intermediate_message=handle_intermediate_steps
+            ):
                 print(f"# {response.name}: {response}")
                 thread = response.thread
 
@@ -75,10 +92,17 @@ async def main() -> None:
         Sample Output:
         
         # User: 'Which team won the 2025 NCAA basketball championship?'
-        # BingGroundingAgent: The Florida Gators won the 2025 NCAA basketball championship, defeating the Houston Cougars 65-63 in the final to secure their third national title【3:5†source】【3:6†source】【3:9†source】.
-        Annotation :> https://www.usatoday.com/story/sports/ncaab/2025/04/07/houston-florida-live-updates-national-championship-score/82982004007/, source=【3:5†source】, with start_index=147 and end_index=159
-        Annotation :> https://bleacherreport.com/articles/25182096-winners-and-losers-2025-mens-ncaa-tournament, source=【3:6†source】, with start_index=159 and end_index=171
-        Annotation :> https://wtop.com/ncaa-basketball/2025/04/ncaa-basketball-champions/, source=【3:9†source】, with start_index=171 and end_index=183
+        Function Call:> bing_grounding with arguments: 
+            {
+                'requesturl': 'https://api.bing.microsoft.com/v7.0/search?q=search(query:2025 NCAA basketball championship winner)', 
+                'response_metadata': "{'market': 'en-US', 'num_docs_retrieved': 5, 'num_docs_actually_used': 5}"
+            }
+        # BingGroundingAgent: The team that won the 2025 NCAA men's basketball championship was the Florida Gators. They defeated the Houston Cougars with a final score of 65-63. 
+            The championship game took place in San Antonio, Texas, and the Florida team was coached by Todd Golden. This victory made Florida the national champion for the 2024-25 
+            NCAA Division I men's basketball season【3:0†source】【3:1†source】【3:2†source】.
+        Annotation :> https://en.wikipedia.org/wiki/2025_NCAA_Division_I_men%27s_basketball_championship_game, source=【3:0†source】, with start_index=357 and end_index=369
+        Annotation :> https://www.ncaa.com/history/basketball-men/d1, source=【3:1†source】, with start_index=369 and end_index=381
+        Annotation :> https://sports.yahoo.com/article/won-march-madness-2025-ncaa-100551421.html, source=【3:2†source】, with start_index=381 and end_index=393
         """  # noqa: E501
 
 

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_bing_grounding_streaming_with_message_callback.py

@@ -1,7 +1,6 @@
 # Copyright (c) Microsoft. All rights reserved.
 
 import asyncio
-from functools import reduce
 
 from azure.ai.agents.models import BingGroundingTool
 from azure.identity.aio import DefaultAzureCredential
@@ -10,8 +9,8 @@
 from semantic_kernel.contents import (
     ChatMessageContent,
     FunctionCallContent,
+    FunctionResultContent,
     StreamingAnnotationContent,
-    StreamingChatMessageContent,
 )
 
 """
@@ -32,7 +31,13 @@
 
 
 async def handle_streaming_intermediate_steps(message: ChatMessageContent) -> None:
-    intermediate_steps.append(message)
+    for item in message.items or []:
+        if isinstance(item, FunctionResultContent):
+            print(f"Function Result:> {item.result} for function: {item.name}")
+        elif isinstance(item, FunctionCallContent):
+            print(f"Function Call:> {item.name} with arguments: {item.arguments}")
+        else:
+            print(f"{item}")
 
 
 async def main() -> None:
@@ -41,8 +46,7 @@ async def main() -> None:
         AzureAIAgent.create_client(credential=creds) as client,
     ):
         # 1. Enter your Bing Grounding Connection Name
-        # <your-bing-grounding-connection-name>
-        bing_connection = await client.connections.get(connection_name="skbinggrounding")
+        bing_connection = await client.connections.get(name="<your-bing-grounding-connection-name>")
         conn_id = bing_connection.id
 
         # 2. Initialize agent bing tool and add the connection id
@@ -94,41 +98,18 @@ async def main() -> None:
             await thread.delete() if thread else None
             await client.agents.delete_agent(agent.id)
 
-        print("====================================================")
-        print("\nResponse complete:\n")
-        # Combine the intermediate `StreamingChatMessageContent` chunks into a single message
-        filtered_steps = [step for step in intermediate_steps if isinstance(step, StreamingChatMessageContent)]
-        streaming_full_completion: StreamingChatMessageContent = reduce(lambda x, y: x + y, filtered_steps)
-        # Grab the other messages that are not `StreamingChatMessageContent`
-        other_steps = [s for s in intermediate_steps if not isinstance(s, StreamingChatMessageContent)]
-        final_msgs = [streaming_full_completion] + other_steps
-        for msg in final_msgs:
-            if any(isinstance(item, FunctionCallContent) for item in msg.items):
-                for item in msg.items:
-                    if isinstance(item, FunctionCallContent):
-                        # Note: the AI Projects SDK is not returning a `requesturl` for streaming events
-                        # The issue was raised with the AI Projects team
-                        print(f"Function call: {item.function_name} with arguments: {item.arguments}")
-
-            print(f"{msg.content}")
-
         """
         Sample Output:
         
         # User: 'Which team won the 2025 NCAA basketball championship?'
-        # BingGroundingAgent: The Florida Gators won the 2025 NCAA men's basketball championship, defeating the Houston Cougars 65-63. It marked Florida's third national title and their first since back-to-back wins in 2006-2007【5:0†source】
-        Annotation :> https://www.usatoday.com/story/sports/ncaab/2025/04/07/houston-florida-live-updates-national-championship-score/82982004007/, source=Florida vs Houston final score: Gators win 2025 NCAA championship, with start_index=198 and end_index=210
-        【5:5†source】
-        Annotation :> https://www.nbcsports.com/mens-college-basketball/live/florida-vs-houston-live-score-updates-game-news-stats-highlights-for-2025-ncaa-march-madness-mens-national-championship, source=Houston vs. Florida RECAP: Highlights, stats, box score, results as ..., with start_index=210 and end_index=222
-        .
-        ====================================================
-
-        Response complete:
-
-        Function call: bing_grounding with arguments: None
-        Function call: bing_grounding with arguments: None
-
-        The Florida Gators won the 2025 NCAA men's basketball championship, defeating the Houston Cougars 65-63. It marked Florida's third national title and their first since back-to-back wins in 2006-2007【5:0†source】【5:5†source】.
+        Function Call:> bing_grounding with arguments: {'requesturl': 'https://api.bing.microsoft.com/v7.0/search?q=search(query: 2025 NCAA basketball championship winner)'}
+        Function Call:> bing_grounding with arguments: {'response_metadata': "{'market': 'en-US', 'num_docs_retrieved': 5, 'num_docs_actually_used': 5}"}
+        # BingGroundingAgent: The Florida Gators won the 2025 NCAA men's basketball championship. They defeated the Houston Cougars with a close score of 65-63 in the championship game held in San Antonio, Texas. This victory marked their third national title. Florida overcame a 12-point deficit during the game to claim the championship【3:0†source】
+        Annotation :> https://en.wikipedia.org/wiki/2025_NCAA_Division_I_men%27s_basketball_championship_game, source=None, with start_index=308 and end_index=320
+        【3:1†source】
+        Annotation :> https://www.ncaa.com/history/basketball-men/d1, source=None, with start_index=320 and end_index=332
+        【3:2†source】
+        Annotation :> https://sports.yahoo.com/article/florida-gators-win-2025-ncaa-034021303.html, source=None, with start_index=332 and end_index=344.
         """  # noqa: E501
 
 

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_message_callback_streaming.py

@@ -1,13 +1,14 @@
 # Copyright (c) Microsoft. All rights reserved.
 
 import asyncio
+import logging
 from typing import Annotated
 
 from azure.identity.aio import DefaultAzureCredential
 
 from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
-from semantic_kernel.contents import FunctionCallContent, FunctionResultContent
-from semantic_kernel.contents.chat_message_content import ChatMessageContent
+from semantic_kernel.contents import ChatMessageContent, FunctionCallContent, FunctionResultContent
+from semantic_kernel.core_plugins import MathPlugin
 from semantic_kernel.functions import kernel_function
 
 """
@@ -22,6 +23,8 @@
 while assistant replies stream back incrementally through the main response loop.
 """
 
+logging.basicConfig(level=logging.DEBUG)
+
 
 # Define a sample plugin for the sample
 class MenuPlugin:
@@ -67,14 +70,14 @@ async def main() -> None:
         agent_definition = await client.agents.create_agent(
             model=ai_agent_settings.model_deployment_name,
             name="Host",
-            instructions="Answer questions about the menu.",
+            instructions="Answer questions from the user using your provided functions. You must invoke multiple functions to answer the user's questions. ",  # noqa: E501
         )
 
         # Create the AzureAI Agent
         agent = AzureAIAgent(
             client=client,
             definition=agent_definition,
-            plugins=[MenuPlugin()],  # add the sample plugin to the agent
+            plugins=[MenuPlugin(), MathPlugin()],
         )
 
         # Create a thread for the agent
@@ -83,10 +86,7 @@ async def main() -> None:
         thread: AzureAIAgentThread = None
 
         user_inputs = [
-            "Hello",
-            "What is the special soup?",
-            "How much does that cost?",
-            "Thank you",
+            "What is the price of the special drink and the special food item added together?",
         ]
 
         try:
@@ -112,23 +112,21 @@ async def main() -> None:
         """
         Sample Output:
 
-        # User: 'Hello'
-        # AuthorRole.ASSISTANT: Hello! How can I assist you today?
-        # User: 'What is the special soup?'
+        # User: 'What is the price of the special drink and then special food item added together?'
         Function Call:> MenuPlugin-get_specials with arguments: {}
         Function Result:> 
                 Special Soup: Clam Chowder
                 Special Salad: Cobb Salad
                 Special Drink: Chai Tea
                 for function: MenuPlugin-get_specials
-        # AuthorRole.ASSISTANT: The special soup is Clam Chowder. Would you like to know more about it or anything 
-            else from the menu?
-        # User: 'How much does that cost?'
-        Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item":"Clam Chowder"}
+        Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Chai Tea"}
+        Function Call:> MenuPlugin-get_item_price with arguments: {"menu_item": "Clam Chowder"}
+        Function Result:> $9.99 for function: MenuPlugin-get_item_price
         Function Result:> $9.99 for function: MenuPlugin-get_item_price
-        # AuthorRole.ASSISTANT: The Clam Chowder costs $9.99. Would you like to order it?
-        # User: 'Thank you'
-        # AuthorRole.ASSISTANT: You're welcome! Let me know if you need anything else. Enjoy your day! 😊
+        Function Call:> MathPlugin-Add with arguments: {"input":9.99,"amount":9.99}
+        Function Result:> 19.98 for function: MathPlugin-Add
+        # AuthorRole.ASSISTANT: The price of the special drink, Chai Tea, is $9.99 and the price of the special food 
+            item, Clam Chowder, is $9.99. Added together, the total price is $19.98.
         """
 
 

python/samples/getting_started_with_agents/azure_ai_agent/README.md

@@ -7,7 +7,7 @@ To set up the required resources, follow the "Quickstart: Create a new agent" gu
 You will need to install the optional Semantic Kernel `azure` dependencies if you haven't already via:
 
 ```bash
-pip install semantic-kernel[azure]
+pip install semantic-kernel
 ```
 
 Before running an Azure AI Agent, modify your .env file to include:

python/samples/getting_started_with_agents/azure_ai_agent/step6_azure_ai_agent_openapi.py

@@ -8,7 +8,7 @@
 from azure.identity.aio import DefaultAzureCredential
 
 from semantic_kernel.agents import AzureAIAgent, AzureAIAgentSettings, AzureAIAgentThread
-from semantic_kernel.contents import AuthorRole
+from semantic_kernel.contents import ChatMessageContent, FunctionCallContent, FunctionResultContent
 
 """
 The following sample demonstrates how to create a simple, Azure AI agent that
@@ -23,6 +23,16 @@
 ]
 
 
+async def handle_streaming_intermediate_steps(message: ChatMessageContent) -> None:
+    for item in message.items or []:
+        if isinstance(item, FunctionResultContent):
+            print(f"Function Result:> {item.result} for function: {item.name}")
+        elif isinstance(item, FunctionCallContent):
+            print(f"Function Call:> {item.name} with arguments: {item.arguments}")
+        else:
+            print(f"{item}")
+
+
 async def main() -> None:
     async with (
         DefaultAzureCredential() as creds,
@@ -76,12 +86,11 @@ async def main() -> None:
                 print(f"# User: '{user_input}'")
                 # 7. Invoke the agent for the specified thread for response
                 async for response in agent.invoke(messages=user_input, thread=thread):
-                    if response.role != AuthorRole.TOOL:
-                        print(f"# Agent: {response}")
+                    print(f"# Agent: {response}")
                     thread = response.thread
         finally:
             # 8. Cleanup: Delete the thread and agent
-            await client.agents.threads.delete(thread.id)
+            await client.agents.threads.delete(thread.id) if thread else None
             await client.agents.delete_agent(agent.id)
 
         """

python/samples/getting_started_with_agents/azure_ai_agent/step7_azure_ai_agent_retrieval.py

@@ -5,6 +5,7 @@
 from azure.identity.aio import DefaultAzureCredential
 
 from semantic_kernel.agents import AzureAIAgent, AzureAIAgentThread
+from semantic_kernel.contents import ChatMessageContent, FunctionCallContent, FunctionResultContent
 
 """
 The following sample demonstrates how to use an already existing
@@ -16,10 +17,20 @@
 
 # Simulate a conversation with the agent
 USER_INPUTS = [
-    "Why is the sky blue?",
+    "Using the provided doc, tell me about the evolution of RAG.",
 ]
 
 
+async def handle_streaming_intermediate_steps(message: ChatMessageContent) -> None:
+    for item in message.items or []:
+        if isinstance(item, FunctionResultContent):
+            print(f"Function Result:> {item.result} for function: {item.name}")
+        elif isinstance(item, FunctionCallContent):
+            print(f"Function Call:> {item.name} with arguments: {item.arguments}")
+        else:
+            print(f"{item}")
+
+
 async def main() -> None:
     async with (
         DefaultAzureCredential() as creds,
@@ -29,7 +40,7 @@ async def main() -> None:
         # Replace the "your-agent-id" with the actual agent ID
         # you want to use.
         agent_definition = await client.agents.get_agent(
-            agent_id="your-agent-id",
+            agent_id="<your-agent-id>",
         )
 
         # 2. Create a Semantic Kernel agent for the Azure AI agent
@@ -47,8 +58,15 @@ async def main() -> None:
             for user_input in USER_INPUTS:
                 print(f"# User: '{user_input}'")
                 # 4. Invoke the agent for the specified thread for response
-                response = await agent.get_response(messages=user_input, thread=thread)
-                print(f"# {response.name}: {response}")
+                async for response in agent.invoke_stream(
+                    messages=user_input,
+                    thread=thread,
+                    on_intermediate_message=handle_streaming_intermediate_steps,
+                ):
+                    # Print the agent's response
+                    print(f"{response}", end="", flush=True)
+                    # Update the thread for subsequent messages
+                    thread = response.thread
         finally:
             # 5. Cleanup: Delete the thread and agent
             await thread.delete() if thread else None