Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Advanced Paste] Add Semantic Kernel opt-in to allow chaining of paste actions #35901

Closed
drawbyperpetual opened this issue Nov 12, 2024 · 0 comments
Assignees
Labels
Product-Advanced Paste Refers to the Advanced Paste module Resolution-Already Fixed/Doesn't Apply A change in the product has made the issue obsolete.

Comments

@drawbyperpetual
Copy link
Collaborator

Description of the new feature / enhancement

Add new toggle "Enable advanced AI":

  • Off when "Enable Paste with AI" is already enabled (so no change for existing users).
  • On when explicitly toggled or when "Enable Paste with AI" is activated (so new users get the feature immediately).

This feature allows for the use of Semantic Kernel to determine a "chain" of paste-actions to be performed when using "Paste with AI." This means Paste with AI can now:

  • ...Work with non-text input like images.
  • ...Produce non-text output like files.
  • ...Execute multiple transformations (in a chain, e.g. Image to text -> Text to JSON text -> JSON text to TXT file).
  • ...Produce meaningful AI-generated error messages.

Example queries

  • Explicit steps: (with image on the clipboard) "Convert this image to text using OCR, translate the text to French and then save the result as a TXT file"
  • Implicit steps: (with image on the clipboard) "Translate to French and save as a TXT file"

Screenshots

Image
Image
Image

Scenario when this would be used?

Semantic Kernel is useful in any situation when multiple transformations are desirable. As the catalog of actions available in Advanced Paste grows, this feature will become more and more useful.

Currently, this feature is especially useful in:

  • ...Allowing text transformations to work on images containing text (e.g. "Translate to French" now works on text as well as on images).
  • ...Allowing intermediate transformations to first occur when saving data as files (e.g. "Translate to Spanish and save as a TXT file").

Supporting information

Further details:

  • Semantic Kernel currently uses OpenAI as its backend, reusing the user's credentials that are used for regular text -> text AI transformations.
  • Semantic Kernel uses gpt-4o. Regular text -> text AI transformations continue to use gpt-3.5-turbo-instruct for now.
  • Generally speaking, queries using Semantic Kernel consume more OpenAI API credits.
  • When Semantic Kernel is used for Custom Actions, the chain of actions executed is cached on disk, meaning subsequent executions of a custom action with the same available clipboard formats should be faster and not consume further OpenAI API credits (unless of course text -> text AI transformations are called as 'sub-routines').
@drawbyperpetual drawbyperpetual added the Needs-Triage For issues raised to be triaged and prioritized by internal Microsoft teams label Nov 12, 2024
@drawbyperpetual drawbyperpetual self-assigned this Nov 12, 2024
@drawbyperpetual drawbyperpetual added Product-Advanced Paste Refers to the Advanced Paste module Status-In progress This issue or work-item is under development and removed Needs-Triage For issues raised to be triaged and prioritized by internal Microsoft teams labels Nov 12, 2024
@drawbyperpetual drawbyperpetual added this to the PowerToys 0.87 milestone Nov 12, 2024
@jaimecbernardo jaimecbernardo added Resolution-Fix Committed Fix is checked in, but it might be 3-4 weeks until a release. and removed Status-In progress This issue or work-item is under development labels Dec 11, 2024
@cinnamon-msft cinnamon-msft added Resolution-Already Fixed/Doesn't Apply A change in the product has made the issue obsolete. and removed Resolution-Fix Committed Fix is checked in, but it might be 3-4 weeks until a release. labels Dec 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Product-Advanced Paste Refers to the Advanced Paste module Resolution-Already Fixed/Doesn't Apply A change in the product has made the issue obsolete.
Projects
None yet
Development

No branches or pull requests

3 participants