Python: Support AzureAIAgent agent_id instead of assistant_id (microsoft#10908)

### Motivation and Context

The Azure Python SDK made an underlying change to move away from
`assistant_id` keyword arguments to now support `agent_id` keyword
arguments. Additionally, they now support a
`MessageTextUrlCitationAnnotation` and
`MessageDeltaTextUrlCitationAnnotation` for annotations.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

Update the `AzureAIAgent` to pass in `agent_id` in place of
`assistant_id`, and bring in support for the new annotation types.
- Updates samples to simpler patterns where needed.
- Updates unit tests.
- Closes microsoft#10852

Validated that our [AzureAIAgent + Azure AI Search
sample](https://github.com/microsoft/semantic-kernel/blob/main/python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_azure_ai_search.py)
works properly:

---

```markdown
# User: 'Which hotels are available with full-sized kitchens in Nashville, TN?'

# Agent: In Nashville, TN, here are some hotel options that feature full-sized kitchens:

1. **Extended Stay Suites** - All suites include full-sized kitchens stocked with cookware, separate living and sleeping areas, and sofa beds. Some larger rooms have fireplaces and patios or balconies【3:0†source】.

2. **New Premium Rooms** - These newly furnished rooms are fully equipped with premium amenities, including a full kitchen, washer & dryer, and access to a fitness center. Units also include fireplaces and small outdoor balconies【3:1†source】.

3. **Conveniently Located Extended Stay** - This option offers a full kitchen along with a washer & dryer, and amenities such as free WiFi and a fitness center, located near downtown Nashville【3:3†source】.

4. **Inn with Functional Kitchens** - This inn provides living room suites and functional kitchens, situated conveniently near a lake and welcomes pets【3:4†source】.

These options provide a range of amenities for travelers looking for the convenience of a full-sized kitchen during their stay.

# User: 'Fun hotels with free WiFi.'

# Agent: Here are some fun hotels that offer free WiFi:

1. **Hotel with Rooftop Plaza** - Offers complimentary continental breakfast in the lobby and free Wi-Fi throughout the hotel. Its penthouse suites provide stunning views【7:0†source】.

2. **Lakeside Hotel** - Features newly redesigned rooms, an airport shuttle, and lakeside amenities including a resort-style pool. The hotel provides stylish new guestrooms with Internet TVs and free WiFi【7:3†source】.

3. **Downtown Hotel** - Located near the vibrant downtown area and the business district. This hotel offers free WiFi, local transportation, and a special treat of milk and cookies for guests【7:4†source】.

These hotels combine fun experiences with the convenience of free internet access for guests.
```

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄

Author: moonbox3

python/pyproject.toml

@@ -57,7 +57,7 @@ autogen = [
 ]
 azure = [
     "azure-ai-inference >= 1.0.0b6",
-    "azure-ai-projects == 1.0.0b5",
+    "azure-ai-projects >= 1.0.0b7",
     "azure-core-tracing-opentelemetry >= 1.0.0b11",
     "azure-search-documents >= 11.6.0b4",
     "azure-identity ~= 1.13",

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_azure_ai_search.py

@@ -7,8 +7,6 @@
 from azure.identity.aio import DefaultAzureCredential
 
 from semantic_kernel.agents.azure_ai import AzureAIAgent, AzureAIAgentSettings
-from semantic_kernel.contents.chat_message_content import ChatMessageContent
-from semantic_kernel.contents.utils.author_role import AuthorRole
 
 logging.basicConfig(level=logging.WARNING)
 
@@ -84,13 +82,12 @@ async def main() -> None:
                 # Add the user input as a chat message
                 await agent.add_chat_message(
                     thread_id=thread.id,
-                    message=ChatMessageContent(role=AuthorRole.USER, content=user_input),
+                    message=user_input,
                 )
                 print(f"# User: '{user_input}'\n")
                 # Invoke the agent for the specified thread
                 async for content in agent.invoke(thread_id=thread.id):
-                    if content.role != AuthorRole.TOOL:
-                        print(f"# Agent: {content.content}\n")
+                    print(f"# Agent: {content.content}\n")
         finally:
             await client.agents.delete_thread(thread.id)
             await client.agents.delete_agent(agent.id)

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_file_manipulation.py

@@ -8,14 +8,13 @@
 
 from semantic_kernel.agents.azure_ai import AzureAIAgent, AzureAIAgentSettings
 from semantic_kernel.contents.annotation_content import AnnotationContent
-from semantic_kernel.contents.chat_message_content import ChatMessageContent
 from semantic_kernel.contents.utils.author_role import AuthorRole
 
-###################################################################
-# The following sample demonstrates how to create a simple,       #
-# Azure AI agent that uses the code interpreter tool to answer    #
-# a coding question.                                              #
-###################################################################
+"""
+The following sample demonstrates how to create a simple,
+Azure AI agent that uses the code interpreter tool to answer
+a coding question.
+"""
 
 
 async def main() -> None:
@@ -66,7 +65,7 @@ async def main() -> None:
                 # Add the user input as a chat message
                 await agent.add_chat_message(
                     thread_id=thread.id,
-                    message=ChatMessageContent(role=AuthorRole.USER, content=user_input),
+                    message=user_input,
                 )
                 print(f"# User: '{user_input}'")
                 # Invoke the agent for the specified thread

python/samples/concepts/agents/azure_ai_agent/azure_ai_agent_streaming.py

@@ -6,9 +6,7 @@
 from azure.identity.aio import DefaultAzureCredential
 
 from semantic_kernel.agents.azure_ai import AzureAIAgent, AzureAIAgentSettings
-from semantic_kernel.contents.chat_message_content import ChatMessageContent
-from semantic_kernel.contents.utils.author_role import AuthorRole
-from semantic_kernel.functions.kernel_function_decorator import kernel_function
+from semantic_kernel.functions import kernel_function
 
 
 # Define a sample plugin for the sample
@@ -54,11 +52,9 @@ async def main() -> None:
         agent = AzureAIAgent(
             client=client,
             definition=agent_definition,
+            plugins=[MenuPlugin()],  # add the sample plugin to the agent
         )
 
-        # Add the sample plugin to the kernel
-        agent.kernel.add_plugin(MenuPlugin(), plugin_name="menu")
-
         # Create a new thread
         thread = await client.agents.create_thread()
 
@@ -73,16 +69,16 @@ async def main() -> None:
             for user_input in user_inputs:
                 # Add the user input as a chat message
                 await agent.add_chat_message(
-                    thread_id=thread.id, message=ChatMessageContent(role=AuthorRole.USER, content=user_input)
+                    thread_id=thread.id,
+                    message=user_input,
                 )
                 print(f"# User: '{user_input}'")
                 first_chunk = True
                 async for content in agent.invoke_stream(thread_id=thread.id):
-                    if content.role != AuthorRole.TOOL:
-                        if first_chunk:
-                            print(f"# {content.role}: ", end="", flush=True)
-                            first_chunk = False
-                        print(content.content, end="", flush=True)
+                    if first_chunk:
+                        print(f"# {content.role}: ", end="", flush=True)
+                        first_chunk = False
+                    print(content.content, end="", flush=True)
                 print()
         finally:
             await client.agents.delete_thread(thread.id)

python/semantic_kernel/agents/azure_ai/agent_content_generation.py

@@ -8,10 +8,12 @@
     MessageDeltaTextContent,
     MessageDeltaTextFileCitationAnnotation,
     MessageDeltaTextFilePathAnnotation,
+    MessageDeltaTextUrlCitationAnnotation,
     MessageImageFileContent,
     MessageTextContent,
     MessageTextFileCitationAnnotation,
     MessageTextFilePathAnnotation,
+    MessageTextUrlCitationAnnotation,
     RunStep,
     RunStepDeltaCodeInterpreterDetailItemObject,
     RunStepDeltaCodeInterpreterImageOutput,
@@ -44,11 +46,11 @@
         RunStepDeltaToolCallObject,
     )
 
-###################################################################
-# The methods in this file are used with Azure AI Agent           #
-# related code. They are used to invoke, create chat messages,    #
-# or generate message content.                                    #
-###################################################################
+"""
+The methods in this file are used with Azure AI Agent
+related code. They are used to invoke, create chat messages,
+or generate message content.
+"""
 
 
 @experimental
@@ -109,7 +111,7 @@ def generate_message_content(
             "step_id": completed_step.id,
             "run_id": completed_step.run_id,
             "thread_id": completed_step.thread_id,
-            "assistant_id": completed_step.assistant_id,
+            "agent_id": completed_step.agent_id,
             "usage": completed_step.usage,
         }
         if completed_step is not None
@@ -173,6 +175,7 @@ def generate_streaming_message_content(
                             (
                                 MessageDeltaTextFileCitationAnnotation,
                                 MessageDeltaTextFilePathAnnotation,
+                                MessageDeltaTextUrlCitationAnnotation,
                             ),
                         ):
                             items.append(generate_streaming_annotation_content(annotation))
@@ -399,37 +402,51 @@ def generate_streaming_code_interpreter_content(
 
 @experimental
 def generate_annotation_content(
-    annotation: MessageTextFilePathAnnotation | MessageTextFileCitationAnnotation,
+    annotation: MessageTextFilePathAnnotation | MessageTextFileCitationAnnotation | MessageTextUrlCitationAnnotation,
 ) -> AnnotationContent:
     """Generate annotation content."""
     file_id = None
+    url = None
     if isinstance(annotation, MessageTextFilePathAnnotation) and annotation.file_path is not None:
         file_id = annotation.file_path.file_id
     elif isinstance(annotation, MessageTextFileCitationAnnotation) and annotation.file_citation is not None:
         file_id = annotation.file_citation.file_id
+    elif isinstance(annotation, MessageTextUrlCitationAnnotation) and annotation.url_citation is not None:
+        url = annotation.url_citation.url if annotation.url_citation.url else None
 
     return AnnotationContent(
         file_id=file_id,
         quote=annotation.text,
         start_index=annotation.start_index if annotation.start_index is not None else None,
         end_index=annotation.end_index if annotation.end_index is not None else None,
+        url=url,
     )
 
 
 @experimental
 def generate_streaming_annotation_content(
-    annotation: MessageDeltaTextFilePathAnnotation | MessageDeltaTextFileCitationAnnotation,
+    annotation: MessageDeltaTextFilePathAnnotation
+    | MessageDeltaTextFileCitationAnnotation
+    | MessageDeltaTextUrlCitationAnnotation,
 ) -> StreamingAnnotationContent:
     """Generate streaming annotation content."""
     file_id = None
+    url = None
+    quote = None
     if isinstance(annotation, MessageDeltaTextFilePathAnnotation) and annotation.file_path:
         file_id = annotation.file_path.file_id if annotation.file_path.file_id else None
+        quote = annotation.text if annotation.text else None
     elif isinstance(annotation, MessageDeltaTextFileCitationAnnotation) and annotation.file_citation:
         file_id = annotation.file_citation.file_id if annotation.file_citation.file_id else None
+        quote = annotation.text if annotation.text else None
+    elif isinstance(annotation, MessageDeltaTextUrlCitationAnnotation) and annotation.url_citation:
+        url = annotation.url_citation.url if annotation.url_citation.url else None
+        quote = annotation.url_citation.title if annotation.url_citation.title else None
 
     return StreamingAnnotationContent(
         file_id=file_id,
-        quote=annotation.text,
+        quote=quote,
         start_index=annotation.start_index if annotation.start_index is not None else None,
         end_index=annotation.end_index if annotation.end_index is not None else None,
+        url=url,
     )

python/semantic_kernel/agents/azure_ai/agent_thread_actions.py

@@ -162,7 +162,7 @@ async def invoke(
         run_options = {k: v for k, v in run_options.items() if v is not None}
 
         run: ThreadRun = await agent.client.agents.create_run(
-            assistant_id=agent.id,
+            agent_id=agent.id,
             thread_id=thread_id,
             instructions=merged_instructions or agent.instructions,
             tools=tools,
@@ -388,7 +388,7 @@ async def invoke_stream(
         run_options = {k: v for k, v in run_options.items() if v is not None}
 
         stream: AsyncAgentRunStream = await agent.client.agents.create_stream(
-            assistant_id=agent.id,
+            agent_id=agent.id,
             thread_id=thread_id,
             instructions=merged_instructions or agent.instructions,
             tools=tools,
@@ -663,12 +663,12 @@ async def get_messages(
                 last_id = message.id
                 assistant_name: str | None = None
 
-                if message.assistant_id and message.assistant_id.strip() and message.assistant_id not in agent_names:
-                    assistant = await client.agents.get_agent(message.assistant_id)
+                if message.agent_id and message.agent_id.strip() and message.agent_id not in agent_names:
+                    assistant = await client.agents.get_agent(message.agent_id)
                     if assistant.name and assistant.name.strip():
                         agent_names[assistant.id] = assistant.name
 
-                assistant_name = agent_names.get(message.assistant_id) or message.assistant_id
+                assistant_name = agent_names.get(message.agent_id) or message.agent_id
 
                 content = generate_message_content(assistant_name, message)
 

python/semantic_kernel/contents/annotation_content.py

@@ -25,10 +25,11 @@ class AnnotationContent(KernelContent):
     quote: str | None = None
     start_index: int | None = None
     end_index: int | None = None
+    url: str | None = None
 
     def __str__(self) -> str:
         """Return the string representation of the annotation content."""
-        return f"AnnotationContent(file_id={self.file_id}, quote={self.quote}, start_index={self.start_index}, end_index={self.end_index})"  # noqa: E501
+        return f"AnnotationContent(file_id={self.file_id}, url={self.url}, quote={self.quote}, start_index={self.start_index}, end_index={self.end_index})"  # noqa: E501
 
     def to_element(self) -> Element:
         """Convert the annotation content to an Element."""
@@ -41,6 +42,8 @@ def to_element(self) -> Element:
             element.set("start_index", str(self.start_index))
         if self.end_index is not None:
             element.set("end_index", str(self.end_index))
+        if self.url is not None:
+            element.set("url", self.url)
         return element
 
     @classmethod
@@ -51,11 +54,12 @@ def from_element(cls: type[_T], element: Element) -> _T:
             quote=element.get("quote"),
             start_index=int(element.get("start_index")) if element.get("start_index") else None,  # type: ignore
             end_index=int(element.get("end_index")) if element.get("end_index") else None,  # type: ignore
+            url=element.get("url") if element.get("url") else None,  # type: ignore
         )
 
     def to_dict(self) -> dict[str, Any]:
         """Convert the instance to a dictionary."""
         return {
             "type": "text",
-            "text": f"{self.file_id} {self.quote} (Start Index={self.start_index}->End Index={self.end_index})",
+            "text": f"{self.file_id or self.url} {self.quote} (Start Index={self.start_index}->End Index={self.end_index})",  # noqa: E501
         }

python/semantic_kernel/contents/streaming_annotation_content.py

@@ -28,10 +28,11 @@ class StreamingAnnotationContent(KernelContent):
     quote: str | None = None
     start_index: int | None = None
     end_index: int | None = None
+    url: str | None = None
 
     def __str__(self) -> str:
         """Return the string representation of the annotation content."""
-        return f"StreamingAnnotationContent(file_id={self.file_id}, quote={self.quote}, start_index={self.start_index}, end_index={self.end_index})"  # noqa: E501
+        return f"StreamingAnnotationContent(file_id={self.file_id}, url={self.url}, quote={self.quote}, start_index={self.start_index}, end_index={self.end_index})"  # noqa: E501
 
     def to_element(self) -> Element:
         """Convert the annotation content to an Element."""
@@ -44,6 +45,8 @@ def to_element(self) -> Element:
             element.set("start_index", str(self.start_index))
         if self.end_index is not None:
             element.set("end_index", str(self.end_index))
+        if self.url is not None:
+            element.set("url", self.url)
         return element
 
     @classmethod
@@ -54,11 +57,12 @@ def from_element(cls: type[_T], element: Element) -> _T:
             quote=element.get("quote"),
             start_index=int(element.get("start_index")) if element.get("start_index") else None,  # type: ignore
             end_index=int(element.get("end_index")) if element.get("end_index") else None,  # type: ignore
+            url=element.get("url") if element.get("url") else None,  # type: ignore
         )
 
     def to_dict(self) -> dict[str, Any]:
         """Convert the instance to a dictionary."""
         return {
             "type": "text",
-            "text": f"{self.file_id} {self.quote} (Start Index={self.start_index}->End Index={self.end_index})",
+            "text": f"{self.file_id or self.url} {self.quote} (Start Index={self.start_index}->End Index={self.end_index})",  # noqa: E501
         }

python/tests/unit/agents/azure_ai_agent/test_agent_content_generation.py

@@ -12,6 +12,8 @@
     MessageDeltaTextFileCitationAnnotationObject,
     MessageDeltaTextFilePathAnnotation,
     MessageDeltaTextFilePathAnnotationObject,
+    MessageDeltaTextUrlCitationAnnotation,
+    MessageDeltaTextUrlCitationDetails,
     MessageImageFileContent,
     MessageImageFileDetails,
     MessageTextContent,
@@ -20,6 +22,8 @@
     MessageTextFileCitationDetails,
     MessageTextFilePathAnnotation,
     MessageTextFilePathDetails,
+    MessageTextUrlCitationAnnotation,
+    MessageTextUrlCitationDetails,
     RunStep,
     RunStepDeltaFunction,
     RunStepDeltaFunctionToolCall,
@@ -90,18 +94,25 @@ def test_generate_message_content_text_and_image():
                     start_index=1,
                     end_index=10,
                 ),
+                MessageTextUrlCitationAnnotation(
+                    text="text",
+                    url_citation=MessageTextUrlCitationDetails(title="some title", url="http://example.com"),
+                    start_index=1,
+                    end_index=10,
+                ),
             ],
         )
     )
 
     thread_msg.content = [image, text]
-    step = RunStep(id="step_id", run_id="run_id", thread_id="thread_id", assistant_id="assistant_id")
+    step = RunStep(id="step_id", run_id="run_id", thread_id="thread_id", agent_id="assistant_id")
     out = generate_message_content("assistant", thread_msg, step)
-    assert len(out.items) == 4
+    assert len(out.items) == 5
     assert isinstance(out.items[0], FileReferenceContent)
     assert isinstance(out.items[1], TextContent)
     assert isinstance(out.items[2], AnnotationContent)
     assert isinstance(out.items[3], AnnotationContent)
+    assert isinstance(out.items[4], AnnotationContent)
 
     assert out.items[0].file_id == "test_file_id"
 
@@ -117,6 +128,11 @@ def test_generate_message_content_text_and_image():
     assert out.items[3].start_index == 1
     assert out.items[3].end_index == 10
 
+    assert out.items[4].url == "http://example.com"
+    assert out.items[4].quote == "text"
+    assert out.items[4].start_index == 1
+    assert out.items[4].end_index == 10
+
     assert out.metadata["step_id"] == "step_id"
     assert out.role == AuthorRole.USER
 
@@ -171,6 +187,15 @@ def test_generate_streaming_message_content_text_annotations():
                     end_index=9,
                     text="some text",
                 ),
+                MessageDeltaTextUrlCitationAnnotation(
+                    index=0,
+                    url_citation=MessageDeltaTextUrlCitationDetails(
+                        title="some title",
+                        url="http://example.com",
+                    ),
+                    start_index=0,
+                    end_index=9,
+                ),
             ],
         ),
     )
@@ -183,7 +208,7 @@ def test_generate_streaming_message_content_text_annotations():
     out = generate_streaming_message_content("assistant", delta)
     assert out is not None
     assert out.content == "some text"
-    assert len(out.items) == 4
+    assert len(out.items) == 5
     assert out.items[0].file_id == "image_file"
     assert isinstance(out.items[0], StreamingFileReferenceContent)
     assert isinstance(out.items[1], StreamingTextContent)
@@ -200,6 +225,12 @@ def test_generate_streaming_message_content_text_annotations():
     assert out.items[3].start_index == 0
     assert out.items[3].end_index == 9
 
+    assert isinstance(out.items[4], StreamingAnnotationContent)
+    assert out.items[4].url == "http://example.com"
+    assert out.items[4].quote == "some title"
+    assert out.items[4].start_index == 0
+    assert out.items[4].end_index == 9
+
 
 def test_generate_streaming_function_content_with_function():
     step_details = RunStepDeltaToolCallObject(

python/tests/unit/agents/azure_ai_agent/test_agent_thread_actions.py

@@ -94,7 +94,7 @@ async def test_agent_thread_actions_invoke(ai_project_client, ai_agent_definitio
                 created_at=int(datetime.now(timezone.utc).timestamp()),
                 completed_at=int(datetime.now(timezone.utc).timestamp()),
                 status="completed",
-                assistant_id="assistant123",
+                agent_id="agent123",
                 step_details=RunStepMessageCreationDetails(
                     message_creation=RunStepMessageCreationReference(
                         message_id="msg123",
@@ -113,7 +113,7 @@ async def test_agent_thread_actions_invoke(ai_project_client, ai_agent_definitio
         created_at=int(datetime.now(timezone.utc).timestamp()),
         completed_at=int(datetime.now(timezone.utc).timestamp()),
         status="completed",
-        assistant_id="assistant123",
+        agent_id="agent123",
         role="assistant",
         content=[MessageTextContent(text=MessageTextDetails(value="some message", annotations=[]))],
     )
@@ -180,7 +180,7 @@ def mock_get_function_call_contents(run: ThreadRun, function_steps: dict):
         created_at=int(datetime.now(timezone.utc).timestamp()),
         completed_at=int(datetime.now(timezone.utc).timestamp()),
         status="completed",
-        assistant_id="assistant123",
+        agent_id="agent123",
         step_details=RunStepToolCallDetails(
             tool_calls=[
                 RunStepCodeInterpreterToolCall(
@@ -209,7 +209,7 @@ def mock_get_function_call_contents(run: ThreadRun, function_steps: dict):
         created_at=int(datetime.now(timezone.utc).timestamp()),
         completed_at=int(datetime.now(timezone.utc).timestamp()),
         status="completed",
-        assistant_id="assistant123",
+        agent_id="agent123",
         step_details=RunStepMessageCreationDetails(
             message_creation=RunStepMessageCreationReference(message_id="msg123")
         ),
@@ -225,7 +225,7 @@ def mock_get_function_call_contents(run: ThreadRun, function_steps: dict):
         created_at=int(datetime.now(timezone.utc).timestamp()),
         completed_at=int(datetime.now(timezone.utc).timestamp()),
         status="completed",
-        assistant_id="assistant123",
+        agent_id="agent123",
         role="assistant",
         content=[MessageTextContent(text=MessageTextDetails(value="some message", annotations=[]))],
     )

python/tests/unit/contents/test_annotation_content.py

@@ -70,7 +70,10 @@ def test_update_indices():
 
 def test_to_str():
     annotation = AnnotationContent(file_id="12345", quote="This is a quote.", start_index=5, end_index=20)
-    assert str(annotation) == "AnnotationContent(file_id=12345, quote=This is a quote., start_index=5, end_index=20)"
+    assert (
+        str(annotation)
+        == "AnnotationContent(file_id=12345, url=None, quote=This is a quote., start_index=5, end_index=20)"
+    )  # noqa: E501
 
 
 def test_to_element():

python/tests/unit/contents/test_streaming_annotation_content.py

@@ -73,7 +73,7 @@ def test_to_str():
     annotation = StreamingAnnotationContent(file_id="12345", quote="This is a quote.", start_index=5, end_index=20)
     assert (
         str(annotation)
-        == "StreamingAnnotationContent(file_id=12345, quote=This is a quote., start_index=5, end_index=20)"
+        == "StreamingAnnotationContent(file_id=12345, url=None, quote=This is a quote., start_index=5, end_index=20)"
     )