Skip to content

Commit f08cf3c

Browse files
authoredOct 2, 2024
Python: Add support for streaming OpenAI assistants (microsoft#9055)
### Motivation and Context OpenAI assistants were introduced several versions ago; however, there was open work to support streaming for OpenAI assistants v2. Work to support streaming assistants was also needed in Agent Group Chat scenarios. <!-- Thank you for your contribution to the semantic-kernel repo! Please help reviewers and future users, providing the following information: 1. Why is this change required? 2. What problem does it solve? 3. What scenario does it contribute to? 4. If it fixes an open issue, please link to the issue here. --> ### Description This PR introduces: - Support for streaming with (Azure) OpenAI assistants - Support for Agent Chat Scenarios that can now leverage a streaming invoke instead of just a sync invoke. - Unit tests to increase the test coverage to near 100% for the newly added code - Adds a few more OpenAI assistant agent samples - Closes microsoft#7267 <!-- Describe your changes, the overall approach, the underlying design. These notes will help understanding how your code works. Thanks! --> ### Contribution Checklist <!-- Before submitting this PR, please make sure: --> - [X] The code builds clean without any errors or warnings - [X] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [X] All unit tests pass, and I have added new tests where possible - [X] I didn't break anyone 😄
1 parent ca17571 commit f08cf3c

33 files changed

+1791
-56
lines changed
 

‎docs/PLANNERS.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,4 @@
22

33
This document has been moved to the Semantic Kernel Documentation site. You can find it by navigating to the [Automatically orchestrate AI with planner](https://learn.microsoft.com/en-us/semantic-kernel/ai-orchestration/planner) page.
44

5-
To make an update on the page, file a PR on the [docs repo.](https://github.com/MicrosoftDocs/semantic-kernel-docs/blob/main/semantic-kernel/ai-orchestration/planner.md)
5+
To make an update on the page, file a PR on the [docs repo.](https://github.com/MicrosoftDocs/semantic-kernel-docs/blob/main/semantic-kernel/concepts/planning.md)

‎python/samples/concepts/README.md

+6
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,12 @@ In Semantic Kernel for Python, we leverage Pydantic Settings to manage configura
4141
3. **Direct Constructor Input:**
4242
- As an alternative to environment variables and `.env` files, you can pass the required settings directly through the constructor of the AI Connector or Memory Connector.
4343

44+
## Microsoft Entra Token Authentication
45+
46+
To authenticate to your Azure resources using a Microsoft Entra Authentication Token, the `AzureChatCompletion` AI Service connector now supports this as a built-in feature. If you do not provide an API key -- either through an environment variable, a `.env` file, or the constructor -- and you also do not provide a custom `AsyncAzureOpenAI` client, an `ad_token`, or an `ad_token_provider`, the `AzureChatCompletion` connector will attempt to retrieve a token using the [`DefaultAzureCredential`](https://learn.microsoft.com/en-us/python/api/azure-identity/azure.identity.defaultazurecredential?view=azure-python).
47+
48+
To successfully retrieve and use the Entra Auth Token, you need the `Cognitive Services OpenAI Contributor` role assigned to your Azure OpenAI resource. By default, the `https://cognitiveservices.azure.com` token endpoint is used. You can override this endpoint by setting an environment variable `.env` variable as `AZURE_OPENAI_TOKEN_ENDPOINT` or by passing a new value to the `AzureChatCompletion` constructor as part of the `AzureOpenAISettings`.
49+
4450
## Best Practices
4551

4652
- **.env File Placement:** We highly recommend placing the `.env` file in the `semantic-kernel/python` root directory. This is a common practice when developing in the Semantic Kernel repository.
+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Semantic Kernel: Agent concept examples
2+
3+
This project contains a step by step guide to get started with _Semantic Kernel Agents_ in Python.
4+
5+
#### PyPI:
6+
- For the use of Chat Completion agents, the minimum allowed Semantic Kernel pypi version is 1.3.0.
7+
- For the use of OpenAI Assistant agents, the minimum allowed Semantic Kernel pypi version is 1.4.0.
8+
- For the use of Agent Group Chat, the minimum allowed Semantic kernel pypi version is 1.6.0.
9+
- For the use of Streaming OpenAI Assistant agents, the minimum allowed Semantic Kernel pypi version is 1.11.0
10+
11+
#### Source
12+
13+
- [Semantic Kernel Agent Framework](../../../semantic_kernel/agents/)
14+
15+
## Examples
16+
17+
The concept agents examples are grouped by prefix:
18+
19+
Prefix|Description
20+
---|---
21+
assistant|How to use agents based on the [Open AI Assistant API](https://platform.openai.com/docs/assistants).
22+
chat_completion|How to use Semantic Kernel Chat Completion agents.
23+
mixed_chat|How to combine different agent types.
24+
complex_chat|**Coming Soon**
25+
26+
*Note: As we strive for parity with .NET, more getting_started_with_agent samples will be added. The current steps and names may be revised to further align with our .NET counterpart.*
27+
28+
## Configuring the Kernel
29+
30+
Similar to the Semantic Kernel Python concept samples, it is necessary to configure the secrets
31+
and keys used by the kernel. See the follow "Configuring the Kernel" [guide](../README.md#configuring-the-kernel) for
32+
more information.
33+
34+
## Running Concept Samples
35+
36+
Concept samples can be run in an IDE or via the command line. After setting up the required api key or token authentication
37+
for your AI connector, the samples run without any extra command line arguments.

‎python/samples/concepts/agents/assistant_agent_chart_maker.py

+26-8
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@
44
from semantic_kernel.agents.open_ai import AzureAssistantAgent, OpenAIAssistantAgent
55
from semantic_kernel.contents.chat_message_content import ChatMessageContent
66
from semantic_kernel.contents.file_reference_content import FileReferenceContent
7+
from semantic_kernel.contents.streaming_file_reference_content import StreamingFileReferenceContent
78
from semantic_kernel.contents.utils.author_role import AuthorRole
89
from semantic_kernel.kernel import Kernel
910

@@ -19,6 +20,8 @@
1920
# Note: you may toggle this to switch between AzureOpenAI and OpenAI
2021
use_azure_openai = True
2122

23+
streaming = True
24+
2225

2326
# A helper method to invoke the agent with the user input
2427
async def invoke_agent(agent: OpenAIAssistantAgent, thread_id: str, input: str) -> None:
@@ -27,14 +30,29 @@ async def invoke_agent(agent: OpenAIAssistantAgent, thread_id: str, input: str)
2730

2831
print(f"# {AuthorRole.USER}: '{input}'")
2932

30-
async for message in agent.invoke(thread_id=thread_id):
31-
if message.content:
32-
print(f"# {message.role}: {message.content}")
33-
34-
if len(message.items) > 0:
35-
for item in message.items:
36-
if isinstance(item, FileReferenceContent):
37-
print(f"\n`{message.role}` => {item.file_id}")
33+
if streaming:
34+
first_chunk = True
35+
async for message in agent.invoke_stream(thread_id=thread_id):
36+
if message.content:
37+
if first_chunk:
38+
print(f"# {message.role}: ", end="", flush=True)
39+
first_chunk = False
40+
print(message.content, end="", flush=True)
41+
42+
if len(message.items) > 0:
43+
for item in message.items:
44+
if isinstance(item, StreamingFileReferenceContent):
45+
print(f"\n# {message.role} => {item.file_id}")
46+
print()
47+
else:
48+
async for message in agent.invoke(thread_id=thread_id):
49+
if message.content:
50+
print(f"# {message.role}: {message.content}")
51+
52+
if len(message.items) > 0:
53+
for item in message.items:
54+
if isinstance(item, FileReferenceContent):
55+
print(f"\n`{message.role}` => {item.file_id}")
3856

3957

4058
async def main():

‎python/samples/concepts/agents/assistant_agent_retrieval.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
AGENT_INSTRUCTIONS = "You are a funny comedian who loves telling G-rated jokes."
1818

1919
# Note: you may toggle this to switch between AzureOpenAI and OpenAI
20-
use_azure_openai = False
20+
use_azure_openai = True
2121

2222

2323
# A helper method to invoke the agent with the user input
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
import asyncio
3+
from typing import Annotated
4+
5+
from semantic_kernel.agents.open_ai import AzureAssistantAgent, OpenAIAssistantAgent
6+
from semantic_kernel.contents.chat_message_content import ChatMessageContent
7+
from semantic_kernel.contents.utils.author_role import AuthorRole
8+
from semantic_kernel.functions.kernel_function_decorator import kernel_function
9+
from semantic_kernel.kernel import Kernel
10+
11+
#####################################################################
12+
# The following sample demonstrates how to create an OpenAI #
13+
# assistant using either Azure OpenAI or OpenAI. OpenAI Assistants #
14+
# allow for function calling, the use of file search and a #
15+
# code interpreter. Assistant Threads are used to manage the #
16+
# conversation state, similar to a Semantic Kernel Chat History. #
17+
# This sample also demonstrates the Assistants Streaming #
18+
# capability and how to manage an Assistants chat history. #
19+
#####################################################################
20+
21+
HOST_NAME = "Host"
22+
HOST_INSTRUCTIONS = "Answer questions about the menu."
23+
24+
# Note: you may toggle this to switch between AzureOpenAI and OpenAI
25+
use_azure_openai = True
26+
27+
28+
# Define a sample plugin for the sample
29+
class MenuPlugin:
30+
"""A sample Menu Plugin used for the concept sample."""
31+
32+
@kernel_function(description="Provides a list of specials from the menu.")
33+
def get_specials(self) -> Annotated[str, "Returns the specials from the menu."]:
34+
return """
35+
Special Soup: Clam Chowder
36+
Special Salad: Cobb Salad
37+
Special Drink: Chai Tea
38+
"""
39+
40+
@kernel_function(description="Provides the price of the requested menu item.")
41+
def get_item_price(
42+
self, menu_item: Annotated[str, "The name of the menu item."]
43+
) -> Annotated[str, "Returns the price of the menu item."]:
44+
return "$9.99"
45+
46+
47+
# A helper method to invoke the agent with the user input
48+
async def invoke_agent(
49+
agent: OpenAIAssistantAgent, thread_id: str, input: str, history: list[ChatMessageContent]
50+
) -> None:
51+
"""Invoke the agent with the user input."""
52+
message = ChatMessageContent(role=AuthorRole.USER, content=input)
53+
await agent.add_chat_message(thread_id=thread_id, message=message)
54+
55+
# Add the user message to the history
56+
history.append(message)
57+
58+
print(f"# {AuthorRole.USER}: '{input}'")
59+
60+
first_chunk = True
61+
async for content in agent.invoke_stream(thread_id=thread_id, messages=history):
62+
if content.role != AuthorRole.TOOL:
63+
if first_chunk:
64+
print(f"# {content.role}: ", end="", flush=True)
65+
first_chunk = False
66+
print(content.content, end="", flush=True)
67+
print()
68+
69+
70+
async def main():
71+
# Create the instance of the Kernel
72+
kernel = Kernel()
73+
74+
# Add the sample plugin to the kernel
75+
kernel.add_plugin(plugin=MenuPlugin(), plugin_name="menu")
76+
77+
# Create the OpenAI Assistant Agent
78+
service_id = "agent"
79+
if use_azure_openai:
80+
agent = await AzureAssistantAgent.create(
81+
kernel=kernel, service_id=service_id, name=HOST_NAME, instructions=HOST_INSTRUCTIONS
82+
)
83+
else:
84+
agent = await OpenAIAssistantAgent.create(
85+
kernel=kernel, service_id=service_id, name=HOST_NAME, instructions=HOST_INSTRUCTIONS
86+
)
87+
88+
thread_id = await agent.create_thread()
89+
90+
history: list[ChatMessageContent] = []
91+
92+
try:
93+
await invoke_agent(agent, thread_id=thread_id, input="Hello", history=history)
94+
await invoke_agent(agent, thread_id=thread_id, input="What is the special soup?", history=history)
95+
await invoke_agent(agent, thread_id=thread_id, input="What is the special drink?", history=history)
96+
await invoke_agent(agent, thread_id=thread_id, input="Thank you", history=history)
97+
finally:
98+
await agent.delete_thread(thread_id)
99+
await agent.delete()
100+
101+
# You may then view the conversation history
102+
print("========= Conversation History =========")
103+
for content in history:
104+
if content.role != AuthorRole.TOOL:
105+
print(f"# {content.role}: {content.content}")
106+
print("========= End of Conversation History =========")
107+
108+
109+
if __name__ == "__main__":
110+
asyncio.run(main())
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
# Copyright (c) Microsoft. All rights reserved.
2+
3+
import asyncio
4+
5+
from semantic_kernel.agents import AgentGroupChat, ChatCompletionAgent
6+
from semantic_kernel.agents.open_ai import OpenAIAssistantAgent
7+
from semantic_kernel.agents.strategies.termination.termination_strategy import TerminationStrategy
8+
from semantic_kernel.connectors.ai.open_ai import AzureChatCompletion
9+
from semantic_kernel.contents.chat_message_content import ChatMessageContent
10+
from semantic_kernel.contents.utils.author_role import AuthorRole
11+
from semantic_kernel.kernel import Kernel
12+
13+
#####################################################################
14+
# The following sample demonstrates how to create an OpenAI #
15+
# assistant using either Azure OpenAI or OpenAI, a chat completion #
16+
# agent and have them participate in a group chat to work towards #
17+
# the user's requirement. #
18+
#####################################################################
19+
20+
21+
class ApprovalTerminationStrategy(TerminationStrategy):
22+
"""A strategy for determining when an agent should terminate."""
23+
24+
async def should_agent_terminate(self, agent, history):
25+
"""Check if the agent should terminate."""
26+
return "approved" in history[-1].content.lower()
27+
28+
29+
REVIEWER_NAME = "ArtDirector"
30+
REVIEWER_INSTRUCTIONS = """
31+
You are an art director who has opinions about copywriting born of a love for David Ogilvy.
32+
The goal is to determine if the given copy is acceptable to print.
33+
If so, state that it is approved. Only include the word "approved" if it is so.
34+
If not, provide insight on how to refine suggested copy without example.
35+
"""
36+
37+
COPYWRITER_NAME = "CopyWriter"
38+
COPYWRITER_INSTRUCTIONS = """
39+
You are a copywriter with ten years of experience and are known for brevity and a dry humor.
40+
The goal is to refine and decide on the single best copy as an expert in the field.
41+
Only provide a single proposal per response.
42+
You're laser focused on the goal at hand.
43+
Don't waste time with chit chat.
44+
Consider suggestions when refining an idea.
45+
"""
46+
47+
48+
def _create_kernel_with_chat_completion(service_id: str) -> Kernel:
49+
kernel = Kernel()
50+
kernel.add_service(AzureChatCompletion(service_id=service_id))
51+
return kernel
52+
53+
54+
async def main():
55+
try:
56+
agent_reviewer = ChatCompletionAgent(
57+
service_id="artdirector",
58+
kernel=_create_kernel_with_chat_completion("artdirector"),
59+
name=REVIEWER_NAME,
60+
instructions=REVIEWER_INSTRUCTIONS,
61+
)
62+
63+
agent_writer = await OpenAIAssistantAgent.create(
64+
service_id="copywriter",
65+
kernel=Kernel(),
66+
name=COPYWRITER_NAME,
67+
instructions=COPYWRITER_INSTRUCTIONS,
68+
)
69+
70+
chat = AgentGroupChat(
71+
agents=[agent_writer, agent_reviewer],
72+
termination_strategy=ApprovalTerminationStrategy(agents=[agent_reviewer], maximum_iterations=10),
73+
)
74+
75+
input = "a slogan for a new line of electric cars."
76+
77+
await chat.add_chat_message(ChatMessageContent(role=AuthorRole.USER, content=input))
78+
print(f"# {AuthorRole.USER}: '{input}'")
79+
80+
last_agent = None
81+
async for message in chat.invoke_stream():
82+
if message.content is not None:
83+
if last_agent != message.name:
84+
print(f"\n# {message.name}: ", end="", flush=True)
85+
last_agent = message.name
86+
print(f"{message.content}", end="", flush=True)
87+
88+
print()
89+
print(f"# IS COMPLETE: {chat.is_complete}")
90+
finally:
91+
await agent_writer.delete()
92+
93+
94+
if __name__ == "__main__":
95+
asyncio.run(main())

‎python/samples/getting_started/CONFIGURING_THE_KERNEL.md

+4
Original file line numberDiff line numberDiff line change
@@ -61,3 +61,7 @@ chat_completion = AzureChatCompletion(service_id="test", env_file_path=env_file_
6161

6262
- Manually configure the `api_key` or required parameters on either the `OpenAIChatCompletion` or `AzureChatCompletion` constructor with keyword arguments.
6363
- This requires the user to manage their own keys/secrets as they aren't relying on the underlying environment variables or `.env` file.
64+
65+
### 4. Microsoft Entra Authentication
66+
67+
To learn how to use a Microsoft Entra Authentication token to authenticate to your Azure OpenAI resource, please navigate to the following [guide](../concepts/README.md#microsoft-entra-token-authentication).

‎python/samples/getting_started_with_agents/README.md

+1
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ This project contains a step by step guide to get started with _Semantic Kernel
66
- For the use of Chat Completion agents, the minimum allowed Semantic Kernel pypi version is 1.3.0.
77
- For the use of OpenAI Assistant agents, the minimum allowed Semantic Kernel pypi version is 1.4.0.
88
- For the use of Agent Group Chat, the minimum allowed Semantic kernel pypi version is 1.6.0.
9+
- For the use of Streaming OpenAI Assistant agents, the minimum allowed Semantic Kernel pypi version is 1.11.0
910

1011
#### Source
1112

‎python/samples/getting_started_with_agents/step8_assistant_vision.py

+16-3
Original file line numberDiff line numberDiff line change
@@ -35,16 +35,29 @@ def create_message_with_image_reference(input: str, file_id: str) -> ChatMessage
3535
)
3636

3737

38+
streaming = False
39+
40+
3841
# A helper method to invoke the agent with the user input
3942
async def invoke_agent(agent: OpenAIAssistantAgent, thread_id: str, message: ChatMessageContent) -> None:
4043
"""Invoke the agent with the user input."""
4144
await agent.add_chat_message(thread_id=thread_id, message=message)
4245

4346
print(f"# {AuthorRole.USER}: '{message.items[0].text}'")
4447

45-
async for content in agent.invoke(thread_id=thread_id):
46-
if content.role != AuthorRole.TOOL:
47-
print(f"# {content.role}: {content.content}")
48+
if streaming:
49+
first_chunk = True
50+
async for content in agent.invoke_stream(thread_id=thread_id):
51+
if content.role != AuthorRole.TOOL:
52+
if first_chunk:
53+
print(f"# {content.role}: ", end="", flush=True)
54+
first_chunk = False
55+
print(content.content, end="", flush=True)
56+
print()
57+
else:
58+
async for content in agent.invoke(thread_id=thread_id):
59+
if content.role != AuthorRole.TOOL:
60+
print(f"# {content.role}: {content.content}")
4861

4962

5063
async def main():

0 commit comments

Comments
 (0)
Please sign in to comment.