UI Actions - Core UI interaction tools for Android device control.
Core UI interaction tools for Android device control.
Initialize the AdbTools instance.
Arguments:
serial
- Device serial numberCreate an AdbTools instance.
Arguments:
serial
- Optional device serial number. If not provided, the first device found will be used.Returns:
AdbTools instance
Tap on a UI element by its index.
This function uses the cached clickable elements to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Tap on the device screen at specific coordinates.
Arguments:
x
- X coordinatey
- Y coordinateReturns:
Bool indicating success or failure
Tap on a UI element by its index.
This function uses the cached clickable elements from the last get_clickables call to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Performs a straight-line swipe gesture on the device screen. To perform a hold (long press), set the start and end coordinates to the same values and increase the duration as needed.
Arguments:
start_x
- Starting X coordinatestart_y
- Starting Y coordinateend_x
- Ending X coordinateend_y
- Ending Y coordinateduration_ms
- Duration of swipe in millisecondsReturns:
Bool indicating success or failure
Input text on the device. Always make sure that the Focused Element is not None before inputting text.
Arguments:
text
- Text to input. Can contain spaces, newlines, and special characters including non-ASCII.Returns:
Result message
Go back on the current view. This presses the Android back button.
Press a key on the Android device.
Common keycodes:
Arguments:
keycode
- Android keycode to pressStart an app on the device.
Arguments:
package
- Package name (e.g., “com.android.settings”)activity
- Optional activity nameInstall an app on the device.
Arguments:
apk_path
- Path to the APK filereinstall
- Whether to reinstall if app existsgrant_permissions
- Whether to grant all permissionsTake a screenshot of the device. This function captures the current screen and adds the screenshot to context in the next message. Also stores the screenshot in the screenshots list with timestamp for later GIF creation.
List installed packages on the device.
Arguments:
include_system_apps
- Whether to include system apps (default: False)Returns:
List of package names
Mark the task as finished.
Arguments:
success
- Indicates if the task was successful.reason
- Reason for failure/successStore important information to remember for future context.
This information will be extracted and included into your next steps to maintain context across interactions. Use this for critical facts, observations, or user preferences that should influence future decisions.
Arguments:
information
- The information to rememberReturns:
Confirmation message
Retrieve all stored memory items.
Returns:
List of stored memory items
Get both the a11y tree and phone state in a single call using the combined /state endpoint.
Arguments:
serial
- Optional device serial numberReturns:
Dictionary containing both ‘a11y_tree’ and ‘phone_state’ data
UI Actions - Core UI interaction tools for Android device control.
Core UI interaction tools for Android device control.
Initialize the AdbTools instance.
Arguments:
serial
- Device serial numberCreate an AdbTools instance.
Arguments:
serial
- Optional device serial number. If not provided, the first device found will be used.Returns:
AdbTools instance
Tap on a UI element by its index.
This function uses the cached clickable elements to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Tap on the device screen at specific coordinates.
Arguments:
x
- X coordinatey
- Y coordinateReturns:
Bool indicating success or failure
Tap on a UI element by its index.
This function uses the cached clickable elements from the last get_clickables call to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Performs a straight-line swipe gesture on the device screen. To perform a hold (long press), set the start and end coordinates to the same values and increase the duration as needed.
Arguments:
start_x
- Starting X coordinatestart_y
- Starting Y coordinateend_x
- Ending X coordinateend_y
- Ending Y coordinateduration_ms
- Duration of swipe in millisecondsReturns:
Bool indicating success or failure
Input text on the device. Always make sure that the Focused Element is not None before inputting text.
Arguments:
text
- Text to input. Can contain spaces, newlines, and special characters including non-ASCII.Returns:
Result message
Go back on the current view. This presses the Android back button.
Press a key on the Android device.
Common keycodes:
Arguments:
keycode
- Android keycode to pressStart an app on the device.
Arguments:
package
- Package name (e.g., “com.android.settings”)activity
- Optional activity nameInstall an app on the device.
Arguments:
apk_path
- Path to the APK filereinstall
- Whether to reinstall if app existsgrant_permissions
- Whether to grant all permissionsTake a screenshot of the device. This function captures the current screen and adds the screenshot to context in the next message. Also stores the screenshot in the screenshots list with timestamp for later GIF creation.
List installed packages on the device.
Arguments:
include_system_apps
- Whether to include system apps (default: False)Returns:
List of package names
Mark the task as finished.
Arguments:
success
- Indicates if the task was successful.reason
- Reason for failure/successStore important information to remember for future context.
This information will be extracted and included into your next steps to maintain context across interactions. Use this for critical facts, observations, or user preferences that should influence future decisions.
Arguments:
information
- The information to rememberReturns:
Confirmation message
Retrieve all stored memory items.
Returns:
List of stored memory items
Get both the a11y tree and phone state in a single call using the combined /state endpoint.
Arguments:
serial
- Optional device serial numberReturns:
Dictionary containing both ‘a11y_tree’ and ‘phone_state’ data