DroidRun provides powerful capabilities for controlling and interacting with Android devices. This guide explains the available interactions and best practices.
Before controlling a device, you need to establish a connection:
1
Enable USB Debugging
On your Android device, go to Settings → About phone and tap Build number 7 times to enable Developer options. Then go to Settings → Developer options and enable USB debugging.
2
Connect via USB
Connect your device to your computer with a USB cable and authorize the computer on your device when prompted.
3
Verify Connection
Copy
Ask AI
# Check if your device is recognizeddroidrun devices
from droidrun.agent.react_agent import ReActAgent# Create a minimal instance of the agent with a specific device serialagent = ReActAgent( llm=llm, device_serial="device1", )
# Swipe from one point to another (e.g., to scroll)await agent.execute_tool("swipe", start_x=500, start_y=1500, end_x=500, end_y=500)# CLI equivalentdroidrun "Scroll down on the screen"
# Type text into the current fieldawait agent.execute_tool("input_text", text="Hello world")# CLI equivalentdroidrun "Type 'Hello world' in the search box"
# Capture the current screenresult = await agent.execute_tool("take_screenshot")# The screenshot is automatically analyzed by the LLM for future reasoning
# Get clickable elements on screenelements = await agent.execute_tool("get_clickables")# Elements include details like text, position, and element type
Before interacting with UI elements, verify the current screen:
Copy
Ask AI
# Take a screenshot to understand current stateawait agent.execute_tool("take_screenshot")# Then get clickable elementselements = await agent.execute_tool("get_clickables")
For complex scenarios, create custom Python scripts:
Copy
Ask AI
async def toggle_wifi(): """Toggle WiFi on/off.""" # Open settings await agent.execute_tool("start_app", package_name="com.android.settings") # Take screenshot to analyze UI await agent.execute_tool("take_screenshot") # Find and tap on Network & internet elements = await agent.execute_tool("get_clickables") # Find network settings element and tap it # ...