UI Actions - Core UI interaction tools for iOS device control.
Core UI interaction tools for iOS device control.
Initialize the IOSTools instance.
Arguments:
url
- iOS device URL. This is the URL of the iOS device. It is used to send requests to the iOS device.bundle_identifiers
- List of bundle identifiers to include in the list of packagesGet all clickable UI elements from the iOS device using accessibility API.
Arguments:
serial
- Optional device URL (not used for iOS, uses instance URL)Returns:
List of dictionaries containing UI elements extracted from the device screen
Tap on a UI element by its index.
This function uses the cached clickable elements to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Tap on a UI element by its index.
This function uses the cached clickable elements from the last get_clickables call to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Performs a straight-line swipe gesture on the device screen. To perform a hold (long press), set the start and end coordinates to the same values and increase the duration as needed.
Arguments:
start_x
- Starting X coordinatestart_y
- Starting Y coordinateend_x
- Ending X coordinateend_y
- Ending Y coordinateduration_ms
- Duration of swipe in milliseconds (not used in iOS API)Returns:
Bool indicating success or failure
Input text on the iOS device.
Arguments:
text
- Text to input. Can contain spaces, newlines, and special characters including non-ASCII.serial
- Optional device serial (not used for iOS, uses instance URL)Returns:
Result message
Press a key on the iOS device.
iOS Key codes:
Arguments:
keycode
- iOS keycode to pressStart an app on the iOS device.
Arguments:
package
- Bundle identifier (e.g., “com.apple.MobileSMS”)activity
- Optional activity name (not used on iOS)Take a screenshot of the iOS device. This function captures the current screen and adds the screenshot to context in the next message. Also stores the screenshot in the screenshots list with timestamp for later GIF creation.
Get the current phone state including current activity and keyboard visibility.
Arguments:
serial
- Optional device serial number (not used for iOS)Returns:
Dictionary with current phone state information
Store important information to remember for future context.
This information will be included in future LLM prompts to help maintain context across interactions. Use this for critical facts, observations, or user preferences that should influence future decisions.
Arguments:
information
- The information to rememberReturns:
Confirmation message
Retrieve all stored memory items.
Returns:
List of stored memory items
Mark the task as finished.
Arguments:
success
- Indicates if the task was successful.reason
- Reason for failure/successUI Actions - Core UI interaction tools for iOS device control.
Core UI interaction tools for iOS device control.
Initialize the IOSTools instance.
Arguments:
url
- iOS device URL. This is the URL of the iOS device. It is used to send requests to the iOS device.bundle_identifiers
- List of bundle identifiers to include in the list of packagesGet all clickable UI elements from the iOS device using accessibility API.
Arguments:
serial
- Optional device URL (not used for iOS, uses instance URL)Returns:
List of dictionaries containing UI elements extracted from the device screen
Tap on a UI element by its index.
This function uses the cached clickable elements to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Tap on a UI element by its index.
This function uses the cached clickable elements from the last get_clickables call to find the element with the given index and tap on its center coordinates.
Arguments:
index
- Index of the element to tapReturns:
Result message
Performs a straight-line swipe gesture on the device screen. To perform a hold (long press), set the start and end coordinates to the same values and increase the duration as needed.
Arguments:
start_x
- Starting X coordinatestart_y
- Starting Y coordinateend_x
- Ending X coordinateend_y
- Ending Y coordinateduration_ms
- Duration of swipe in milliseconds (not used in iOS API)Returns:
Bool indicating success or failure
Input text on the iOS device.
Arguments:
text
- Text to input. Can contain spaces, newlines, and special characters including non-ASCII.serial
- Optional device serial (not used for iOS, uses instance URL)Returns:
Result message
Press a key on the iOS device.
iOS Key codes:
Arguments:
keycode
- iOS keycode to pressStart an app on the iOS device.
Arguments:
package
- Bundle identifier (e.g., “com.apple.MobileSMS”)activity
- Optional activity name (not used on iOS)Take a screenshot of the iOS device. This function captures the current screen and adds the screenshot to context in the next message. Also stores the screenshot in the screenshots list with timestamp for later GIF creation.
Get the current phone state including current activity and keyboard visibility.
Arguments:
serial
- Optional device serial number (not used for iOS)Returns:
Dictionary with current phone state information
Store important information to remember for future context.
This information will be included in future LLM prompts to help maintain context across interactions. Use this for critical facts, observations, or user preferences that should influence future decisions.
Arguments:
information
- The information to rememberReturns:
Confirmation message
Retrieve all stored memory items.
Returns:
List of stored memory items
Mark the task as finished.
Arguments:
success
- Indicates if the task was successful.reason
- Reason for failure/success