Add test LLM provider to support testing (#764)

## Description
<!-- Note: The pull request title will be included in the CHANGELOG. -->
<!-- Provide a standalone description of changes in this PR. -->
<!-- Reference any issues closed by this PR with "closes #1234". All PRs
should have an issue they close-->
Closes [AIQ-439](https://jirasw.nvidia.com/browse/AIQ-439)

* Added a deterministic testing LLM (nat_test_llm) in the
nvidia_nat_test plugin to validate workflows without real API calls.
Supports a cycling `response_seq` and artificial latency via `delay_ms`.
* Made the testing LLM adaptable to all five core frameworks
(`LangChain`, `LlamaIndex`, `CrewAI`, `SemanticKernel`, `Agno`)
* Added related tutorial and documents.
* Added the test LLM to the package loader
(packages/nvidia_nat_test/src/nat/test/register.py).
* Added unit tests for `nat_test_llm` behavior (cycling, latency, and
stream yielding a single chunk).
* Fixed a deprecation warning for `model_fields` and a small typo.


## By Submitting this PR I confirm:
- I am familiar with the [Contributing
Guidelines](https://github.com/NVIDIA/NeMo-Agent-Toolkit/blob/develop/docs/source/resources/contributing.md).
- We require that all contributors "sign-off" on their commits. This
certifies that the contribution is your original work, or you have
rights to submit it under the same license, or a compatible license.
- Any contribution which contains commits that are not Signed-Off will
not be accepted.
- When the PR is ready for review, new or existing tests cover these
changes.
- When the PR is ready for review, the documentation is up to date with
these changes.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added a deterministic test LLM provider for development/CI that cycles
predefined responses with optional per-call delay and integrates with
multiple LLM frameworks.

- **Documentation**
- New tutorial and provider docs with install, YAML examples, CLI and
programmatic usage, and links to LLM configuration guidance.

- **Tests**
- Comprehensive cross-framework tests validating response cycling,
delays, special characters, independent configs, and persistence.

- **Refactor**
  - Improved dependency-graph field handling.

- **Style**
  - Minor documentation typo fix.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Daniel Wang <[email protected]>

Author: zhongxuanwang-nv

docs/source/tutorials/index.md

@@ -24,4 +24,5 @@ limitations under the License.
 ./add-tools-to-a-workflow.md
 ./create-a-new-workflow.md
 ./build-a-demo-agent-workflow-using-cursor-rules.md
+./testing-with-nat-test-llm.md
 ```

docs/source/tutorials/testing-with-nat-test-llm.md

@@ -0,0 +1,72 @@
+<!--
+SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+SPDX-License-Identifier: Apache-2.0
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+
+# Testing with `nat_test_llm`
+
+Use `nat_test_llm` to quickly validate workflows during development and CI. It yields deterministic, cycling responses and avoids real API calls. It is not intended for production use.
+
+## Prerequisites
+
+- Install the testing plugin package:
+
+```bash
+uv pip install nvidia-nat-test
+```
+
+## Minimal YAML
+
+The following YAML config defines a testing LLM and a simple `chat_completion` workflow that uses it.
+
+```yaml
+llms:
+  main:
+    _type: nat_test_llm
+    response_seq: [alpha, beta, gamma]
+    delay_ms: 0
+workflow:
+  _type: chat_completion
+  llm_name: main
+  system_prompt: "Say only the answer."
+```
+
+Save this as `config.yml`.
+
+## Run from the CLI
+
+```bash
+nat run --config_file config.yml --input "What is 1 + 2?"
+```
+
+You should see a response corresponding to the first item in `response_seq` (for example, `alpha`). Repeated runs will cycle through the sequence (`alpha`, `beta`, `gamma`, then repeat).
+
+## Run programmatically
+
+```python
+from nat.runtime.loader import load_workflow
+
+async def main():
+    async with load_workflow("config.yml") as workflow:
+        async with workflow.run("What is 1 + 2?") as runner:
+            result = await runner.result()
+            print(result)
+```
+
+## Notes
+
+- `nat_test_llm` is for development and CI only. Do not use it in production.
+- To implement your own provider, see: [Adding an LLM Provider](../extend/adding-an-llm-provider.md).
+- For more about configuring LLMs, see: [LLMs](../workflows/llms/index.md).

docs/source/workflows/llms/index.md

@@ -19,7 +19,7 @@ limitations under the License.
 
 ## Supported LLM Providers
 
-NeMo Agent toolkit supports the following LLM providers:
+NVIDIA NeMo Agent toolkit supports the following LLM providers:
 | Provider | Type | Description |
 |----------|------|-------------|
 | [NVIDIA NIM](https://build.nvidia.com) | `nim` | NVIDIA Inference Microservice (NIM) |
@@ -128,6 +128,32 @@ The Azure OpenAI LLM provider is defined by the {py:class}`~nat.llm.azure_openai
 `temperature` is model-gated and may not be supported by all models. See [Gated Fields](../../extend/gated-fields.md) for details.
 :::
 
+## Testing Provider
+### `nat_test_llm`
+`nat_test_llm` is a development and testing provider intended for examples and CI. It is not intended for production use.
+
+* Installation: `uv pip install nvidia-nat-test`
+* Purpose: Deterministic cycling responses for quick validation
+* Not for production
+
+Minimal YAML example with `chat_completion`:
+
+```yaml
+llms:
+  main:
+    _type: nat_test_llm
+    response_seq: [alpha, beta, gamma]
+    delay_ms: 0
+workflow:
+  _type: chat_completion
+  llm_name: main
+  system_prompt: "Say only the answer."
+```
+
+* Learn how to add your own LLM provider: [Adding an LLM Provider](../../extend/adding-an-llm-provider.md)
+<!-- vale off -->
+* See a short tutorial using YAML and `nat_test_llm`: [Testing with nat_test_llm](../../tutorials/testing-with-nat-test-llm.md)
+<!-- vale on -->
 
 ```{toctree}
 :caption: LLMs

packages/nvidia_nat_test/src/nat/test/llm.py

@@ -0,0 +1,205 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# pylint: disable=unused-argument,missing-class-docstring,missing-function-docstring,import-outside-toplevel
+# pylint: disable=too-few-public-methods
+
+import asyncio
+import time
+from collections.abc import AsyncGenerator
+from collections.abc import Iterator
+from itertools import cycle as iter_cycle
+from typing import Any
+
+from pydantic import Field
+
+from nat.builder.builder import Builder
+from nat.builder.framework_enum import LLMFrameworkEnum
+from nat.builder.llm import LLMProviderInfo
+from nat.cli.register_workflow import register_llm_client
+from nat.cli.register_workflow import register_llm_provider
+from nat.data_models.llm import LLMBaseConfig
+
+
+class TestLLMConfig(LLMBaseConfig, name="nat_test_llm"):
+    """Test LLM configuration."""
+    __test__ = False
+    response_seq: list[str] = Field(
+        default=[],
+        description="Returns the next element in order (wraps)",
+    )
+    delay_ms: int = Field(default=0, ge=0, description="Artificial per-call delay in milliseconds to mimic latency")
+
+
+class _ResponseChooser:
+    """
+    Helper class to choose the next response according to config using itertools.cycle and provide synchronous and
+    asynchronous sleep functions.
+    """
+
+    def __init__(self, response_seq: list[str], delay_ms: int):
+        self._cycler = iter_cycle(response_seq) if response_seq else None
+        self._delay_ms = delay_ms
+
+    def next_response(self) -> str:
+        """Return the next response in the cycle, or an empty string if no responses are configured."""
+        if self._cycler is None:
+            return ""
+        return next(self._cycler)
+
+    def sync_sleep(self) -> None:
+        time.sleep(self._delay_ms / 1000.0)
+
+    async def async_sleep(self) -> None:
+        await asyncio.sleep(self._delay_ms / 1000.0)
+
+
+@register_llm_provider(config_type=TestLLMConfig)
+async def test_llm_provider(config: TestLLMConfig, builder: Builder):
+    """Register the `nat_test_llm` provider for the NAT registry."""
+    yield LLMProviderInfo(config=config, description="Test LLM provider")
+
+
+@register_llm_client(config_type=TestLLMConfig, wrapper_type=LLMFrameworkEnum.LANGCHAIN)
+async def test_llm_langchain(config: TestLLMConfig, builder: Builder):
+    """LLM client for LangChain."""
+
+    chooser = _ResponseChooser(response_seq=config.response_seq, delay_ms=config.delay_ms)
+
+    class LangChainTestLLM:
+
+        def invoke(self, messages: Any, **_kwargs: Any) -> str:
+            chooser.sync_sleep()
+            return chooser.next_response()
+
+        async def ainvoke(self, messages: Any, **_kwargs: Any) -> str:
+            await chooser.async_sleep()
+            return chooser.next_response()
+
+        def stream(self, messages: Any, **_kwargs: Any) -> Iterator[str]:
+            chooser.sync_sleep()
+            yield chooser.next_response()
+
+        async def astream(self, messages: Any, **_kwargs: Any) -> AsyncGenerator[str]:
+            await chooser.async_sleep()
+            yield chooser.next_response()
+
+    yield LangChainTestLLM()
+
+
+@register_llm_client(config_type=TestLLMConfig, wrapper_type=LLMFrameworkEnum.LLAMA_INDEX)
+async def test_llm_llama_index(config: TestLLMConfig, builder: Builder):
+
+    try:
+        from llama_index.core.base.llms.types import ChatMessage
+        from llama_index.core.base.llms.types import ChatResponse
+    except ImportError as exc:
+        raise ImportError("llama_index is required for using the test_llm with llama_index. "
+                          "Please install the `nvidia-nat-llama-index` package. ") from exc
+
+    chooser = _ResponseChooser(response_seq=config.response_seq, delay_ms=config.delay_ms)
+
+    class LITestLLM:
+
+        def chat(self, messages: list[Any] | None = None, **_kwargs: Any) -> ChatResponse:
+            chooser.sync_sleep()
+            return ChatResponse(message=ChatMessage(chooser.next_response()))
+
+        async def achat(self, messages: list[Any] | None = None, **_kwargs: Any) -> ChatResponse:
+            await chooser.async_sleep()
+            return ChatResponse(message=ChatMessage(chooser.next_response()))
+
+        def stream_chat(self, messages: list[Any] | None = None, **_kwargs: Any) -> Iterator[ChatResponse]:
+            chooser.sync_sleep()
+            yield ChatResponse(message=ChatMessage(chooser.next_response()))
+
+        async def astream_chat(self,
+                               messages: list[Any] | None = None,
+                               **_kwargs: Any) -> AsyncGenerator[ChatResponse, None]:
+            await chooser.async_sleep()
+            yield ChatResponse(message=ChatMessage(chooser.next_response()))
+
+    yield LITestLLM()
+
+
+@register_llm_client(config_type=TestLLMConfig, wrapper_type=LLMFrameworkEnum.CREWAI)
+async def test_llm_crewai(config: TestLLMConfig, builder: Builder):
+    """LLM client for CrewAI."""
+
+    chooser = _ResponseChooser(response_seq=config.response_seq, delay_ms=config.delay_ms)
+
+    class CrewAITestLLM:
+
+        def call(self, messages: list[dict[str, str]] | None = None, **kwargs: Any) -> str:
+            chooser.sync_sleep()
+            return chooser.next_response()
+
+    yield CrewAITestLLM()
+
+
+@register_llm_client(config_type=TestLLMConfig, wrapper_type=LLMFrameworkEnum.SEMANTIC_KERNEL)
+async def test_llm_semantic_kernel(config: TestLLMConfig, builder: Builder):
+    """LLM client for SemanticKernel."""
+
+    try:
+        from semantic_kernel.contents.chat_message_content import ChatMessageContent
+        from semantic_kernel.contents.utils.author_role import AuthorRole
+    except ImportError as exc:
+        raise ImportError("Semantic Kernel is required for using the test_llm with semantic_kernel. "
+                          "Please install the `nvidia-nat-semantic-kernel` package. ") from exc
+
+    chooser = _ResponseChooser(response_seq=config.response_seq, delay_ms=config.delay_ms)
+
+    class SKTestLLM:
+
+        async def get_chat_message_contents(self, chat_history: Any, **_kwargs: Any) -> list[ChatMessageContent]:
+            await chooser.async_sleep()
+            text = chooser.next_response()
+            return [ChatMessageContent(role=AuthorRole.ASSISTANT, content=text)]
+
+        async def get_streaming_chat_message_contents(self, chat_history: Any,
+                                                      **_kwargs: Any) -> AsyncGenerator[ChatMessageContent, None]:
+            await chooser.async_sleep()
+            text = chooser.next_response()
+            yield ChatMessageContent(role=AuthorRole.ASSISTANT, content=text)
+
+    yield SKTestLLM()
+
+
+@register_llm_client(config_type=TestLLMConfig, wrapper_type=LLMFrameworkEnum.AGNO)
+async def test_llm_agno(config: TestLLMConfig, builder: Builder):
+    """LLM client for agno."""
+
+    chooser = _ResponseChooser(response_seq=config.response_seq, delay_ms=config.delay_ms)
+
+    class AgnoTestLLM:
+
+        def invoke(self, messages: Any | None = None, **_kwargs: Any) -> str:
+            chooser.sync_sleep()
+            return chooser.next_response()
+
+        async def ainvoke(self, messages: Any | None = None, **_kwargs: Any) -> str:
+            await chooser.async_sleep()
+            return chooser.next_response()
+
+        def invoke_stream(self, messages: Any | None = None, **_kwargs: Any) -> Iterator[str]:
+            chooser.sync_sleep()
+            yield chooser.next_response()
+
+        async def ainvoke_stream(self, messages: Any | None = None, **_kwargs: Any) -> AsyncGenerator[str, None]:
+            await chooser.async_sleep()
+            yield chooser.next_response()
+
+    yield AgnoTestLLM()

packages/nvidia_nat_test/src/nat/test/register.py

@@ -21,3 +21,4 @@
 from . import embedder
 from . import functions
 from . import memory
+from . import llm