Python: .Net - Expand chat prompt parser to handle audio, doc, dox, pdf, and other binary files

[glorious-beard]

Motivation and Context

Why is this change required?

This template parsers like the YAML parser to embed content types other than just text and images for LLMs that support additional content types, like PDFs for OpenAI and DOCXs for Claude. Without this capability, functions with prompts that have attachments would have to manually build it's chat history in code.

What problem does it solve?

See above

What scenario does it contribute to?

Usage additional content types beyond visuals and audio for user messages

Open Issues Addressed

• Expanding ChatPromptParser to handle other content types #11044

Description

Chat Prompt Parser

To preserve backward compatibility, rather than consolidating binary content types, I chose to go with adding additional content types so that LLM chat service providers could opt-in to new content types. It also reduces the chances of breaking existing code.

3 new content types are created:

• PdfContent for PDF files. Uses the tag "<pdf>". Allows for Base64 data URIs or standard URIs, similar to ImageContent.
• DocContent for MS Word .doc files. Uses the tag "<doc>". Allows for Base64 data URIs or standard URIs, similar to ImageContent.
• DocxContent for MS Word .docx files. Uses the tag "<docx>". Allows for Base64 data URIs or standard URIs, similar to ImageContent.

(NOTE: DocContent and DocxContent are mainly separate because they have different MIME types and different content formats, though they could easily be consolidated into a single tag and just let the LLM provider handle distinguishing between "doc" and "docx" files. Alternately, I could also see the case for dropping ".doc" support and requiring the caller to only use ".docx".)

In addition, the following 2 contents are now parsed from the XML:

• AudioContent - Parses the tag "<audio>" with either Base64 data URIs or standard URIs, similar to ImageContent.
• BinaryContent - Parses the tag "<file>" with either Base64 data URIs or standard URIs, similar to ImageContent.

Here is a sample:

            
<message role='user'>
  This part will be discarded upon parsing
  <text>Make sense of this random assortment of stuff.</text>
  <image>https://fake-link-to-image/</image>
  <audio>data:audio/wav;base64,UklGRiQAAABXQVZFZm10IBAAAAABAAEAIlYAAACABAAZGF0YVgAAAAA</audio>
  <pdf>data:application/pdf;base64,JVBERi0xLjQKJeLjz9MKMyAwIG9iago8PC9UeXBlL1hSZWYvUGFnZXMgNiAwIFIKL1R5cGUvUGFnZS9NZWRpYUJveCBbMCAwIDQ4MCA1MF0KL0NvbnRlbnRzIDw8L0V4dEdTdGF0ZSA8PC9JRCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GMSA8PC9GMiA8PC9GMyA8PC9GNCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GNSA8PC9GNiA8PC9GNyBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GOCAvPj4KZW5kb2JqCjEwIDAgb2JqCjw8L1R5cGUvUGFnZS9NYWRlYUJveCBbMCAwIDQ4MCA1MF0KL0NvbnRlbnRzIDw8L0V4dEdTdGF0ZSA8PC9JRCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GMSA8PC9GMiA8PC9GMyA8PC9GNCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GNSA8PC9GNiA8PC9GNyBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GOCAvPj4KZW5kb2JqCjEwIDAgb2JqCjw8L1R5cGUvUGFnZS9NYWRlYUJveCBbMCAwIDQ4MCA1MF0KL0NvbnRlbnRzIDw8L0V4dEdTdGF0ZSA8PC9JRCBbPDwvTGVuZ3RoIDQ4XQovRm9udCA8PC9GMSA8PC9G</pdf>
  <pdf>https://fake-link-to-pdf/</pdf>  
 
 <doc>data:application/msword;base64,UEsDBBQAAAAIAI+Q1k5a2gAAABQAAAAIAAAAbmFtZS5kb2N4VVQJAAD9AAAACwAAAB4AAAAAA==</doc>
  <doc>https://fake-link-to-doc/</doc>
  <docx>data:application/vnd.openxmlformats-officedocument.wordprocessingml.document;base64,UEsDBBQAAAAIAI+Q1k5a2gAAABQAAAAIAAAAbmFtZS5kb2N4VVQJAAD9AAAACwAAAB4AAAAAA==</docx>
  <docx>https://fake-link-to-docx/</docx>
  <file>data:application/octet-stream;base64,UEsDBBQAAAAIAI+Q1k5a2gAAABQAAAAIAAAAbmFtZS5kb2N4VVQJAAD9AAAACwAAAB4AAAAAA==</file>
  <file>https://fake-link-to-binary/</file>
  This part will also be discarded upon parsing
</message>

Amazon Bedrock

Modified the Converse API request generator to handle the subset of binary content supported by Amazon Bedrock (PDF, DOC, DOCX, and Image), as documented here.

OpenAI

Modified the client to handle PDF content, audio content, and file references when generating a request to an OpenAI (or OpenAI compatible) client.

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[eavanvalkenburg]

@glorious-beard seems like you need to properly rebase your branch because there is a lot of unrelated changes in here. And I'm also a bit hesitant to add additional content types, on dotnet, they even removed Image and AudioContent in favor of just using BinaryContent with the right MimeType, wondering if that wouldn't be possible for this as well?

[moonbox3]

@glorious-beard please re-open the PR with only relevant files updated/added. The current state is not reviewable. Thanks.

[glorious-beard]

sorry for being out of contact @moonbox3 ; I've created a new PR #11919.