UI Actions - Core UI interaction tools for Android device control.

​
AdbTools

class AdbTools(Tools)
Core UI interaction tools for Android device control.

​
AdbTools.__init__

def __init__(
    serial: str | None = None,
    use_tcp: bool = False,
    tcp_port: int = 8080
) -> None
Initialize the AdbTools instance. Arguments:
  • serial - Device serial number
  • use_tcp - Whether to use TCP communication (default: False)
  • tcp_port - TCP port for communication (default: 8080)

​
AdbTools.setup_tcp_forward

def setup_tcp_forward() -> bool
Set up ADB TCP port forwarding for communication with the portal app. Returns:
  • bool - True if forwarding was set up successfully, False otherwise

​
AdbTools.teardown_tcp_forward

def teardown_tcp_forward() -> bool
Remove ADB TCP port forwarding. Returns:
  • bool - True if forwarding was removed successfully, False otherwise

​
AdbTools.__del__

def __del__()
Cleanup when the object is destroyed.

​
AdbTools.tap_by_index

def tap_by_index(index: int) -> str
Tap on a UI element by its index. This function uses the cached clickable elements to find the element with the given index and tap on its center coordinates. Arguments:
  • index - Index of the element to tap
Returns: Result message

​
AdbTools.tap_by_coordinates

def tap_by_coordinates(x: int, y: int) -> bool
Tap on the device screen at specific coordinates. Arguments:
  • x - X coordinate
  • y - Y coordinate
Returns: Bool indicating success or failure

​
AdbTools.tap

def tap(index: int) -> str
Tap on a UI element by its index. This function uses the cached clickable elements from the last get_clickables call to find the element with the given index and tap on its center coordinates. Arguments:
  • index - Index of the element to tap
Returns: Result message

​
AdbTools.swipe

def swipe(
    start_x: int,
    start_y: int,
    end_x: int,
    end_y: int,
    duration_ms: float = 300
) -> bool
Performs a straight-line swipe gesture on the device screen. To perform a hold (long press), set the start and end coordinates to the same values and increase the duration as needed. Arguments:
  • start_x - Starting X coordinate
  • start_y - Starting Y coordinate
  • end_x - Ending X coordinate
  • end_y - Ending Y coordinate
  • duration - Duration of swipe in seconds
Returns: Bool indicating success or failure

​
AdbTools.drag

def drag(
    start_x: int,
    start_y: int,
    end_x: int,
    end_y: int,
    duration: float = 3
) -> bool
Performs a straight-line drag and drop gesture on the device screen. Arguments:
  • start_x - Starting X coordinate
  • start_y - Starting Y coordinate
  • end_x - Ending X coordinate
  • end_y - Ending Y coordinate
  • duration - Duration of swipe in seconds
Returns: Bool indicating success or failure

​
AdbTools.input_text

def input_text(text: str) -> str
Input text on the device. Always make sure that the Focused Element is not None before inputting text. Arguments:
  • text - Text to input. Can contain spaces, newlines, and special characters including non-ASCII.
Returns: Result message

​
AdbTools.back

def back() -> str
Go back on the current view. This presses the Android back button.

​
AdbTools.press_key

def press_key(keycode: int) -> str
Press a key on the Android device. Common keycodes:
  • 3: HOME
  • 4: BACK
  • 66: ENTER
  • 67: DELETE
Arguments:
  • keycode - Android keycode to press

​
AdbTools.start_app

def start_app(package: str, activity: str | None = None) -> str
Start an app on the device. Arguments:
  • package - Package name (e.g., “com.android.settings”)
  • activity - Optional activity name

​
AdbTools.install_app

def install_app(
    apk_path: str,
    reinstall: bool = False,
    grant_permissions: bool = True
) -> str
Install an app on the device. Arguments:
  • apk_path - Path to the APK file
  • reinstall - Whether to reinstall if app exists
  • grant_permissions - Whether to grant all permissions

​
AdbTools.take_screenshot

def take_screenshot() -> Tuple[str, bytes]
Take a screenshot of the device. This function captures the current screen and adds the screenshot to context in the next message. Also stores the screenshot in the screenshots list with timestamp for later GIF creation.

​
AdbTools.list_packages

def list_packages(include_system_apps: bool = False) -> List[str]
List installed packages on the device. Arguments:
  • include_system_apps - Whether to include system apps (default: False)
Returns: List of package names

​
AdbTools.complete

def complete(success: bool, reason: str = "")
Mark the task as finished. Arguments:
  • success - Indicates if the task was successful.
  • reason - Reason for failure/success

​
AdbTools.remember

def remember(information: str) -> str
Store important information to remember for future context. This information will be extracted and included into your next steps to maintain context across interactions. Use this for critical facts, observations, or user preferences that should influence future decisions. Arguments:
  • information - The information to remember
Returns: Confirmation message

​
AdbTools.get_memory

def get_memory() -> List[str]
Retrieve all stored memory items. Returns: List of stored memory items

​
AdbTools.get_state

def get_state(serial: Optional[str] = None) -> Dict[str, Any]
Get both the a11y tree and phone state in a single call using the combined /state endpoint. Arguments:
  • serial - Optional device serial number
Returns: Dictionary containing both ‘a11y_tree’ and ‘phone_state’ data

​
AdbTools.get_a11y_tree

def get_a11y_tree() -> Dict[str, Any]
Get just the accessibility tree using the /a11y_tree endpoint. Returns: Dictionary containing accessibility tree data

​
AdbTools.get_phone_state

def get_phone_state() -> Dict[str, Any]
Get just the phone state using the /phone_state endpoint. Returns: Dictionary containing phone state data

​
AdbTools.ping

def ping() -> Dict[str, Any]
Test the TCP connection using the /ping endpoint. Returns: Dictionary with ping result