Fixes #760
/claim #760
What kind of change does this PR introduce?
Feature
Summary
This PR introduces the CursorReplayStrategy feature, which allows the AI model to suggest coordinates for a red dot on a screenshot representing the intended action target location. This new strategy provides the ability to visualize and self-correct initial suggestions, improving replay accuracy. It also includes an update to the screenshot capturing functionality to support capturing all monitors.
Checklist
- My code follows the style guidelines of OpenAdapt
- I have performed a self-review of my code
- If applicable, I have added tests to prove my fix is functional/effective
- I have linted my code locally prior to submission
- I have commented my code, particularly in hard-to-understand areas
- I have made corresponding changes to the documentation (e.g. README.md, requirements.txt)
- New and existing unit tests pass locally with my changes
How can your code be run and tested?
- Test the
CursorReplayStrategy by running the AI model and verifying the red dot is correctly suggested and visualized on the screenshot.
- Use the self-correction mechanism to adjust the red dot position and verify the corrections are applied accurately.
Other information
No additional context needed.