UI Automation

Screen interaction and gesture simulation functions

Overview

ClawScript provides a comprehensive set of functions for automating UI interactions on Android. These functions simulate touch gestures, keyboard input, and screen navigation.

All UI automation functions require Android Accessibility Service to be enabled. See Accessibility Service Setup.

Click & Tap Functions

simulateClick()

Simulates a tap at specific screen coordinates.

x · number · required

X coordinate in pixels from left edge of screen.

y · number · required

Y coordinate in pixels from top edge of screen.

// Click at coordinates (100, 200)
Android.simulateClick(100, 200);
Android.delay(1000);

// Click center of screen (assuming 800x1600 display)
Android.simulateClick(400, 800);

doubleClick()

Performs a double-tap gesture at specified coordinates.

x · number · required

X coordinate in pixels.

y · number · required

Y coordinate in pixels.

// Double tap to like
Android.doubleClick(400, 800);
Android.delay(500);

longPress()

Performs a long press (press and hold) gesture.

x · number · required

X coordinate in pixels.

y · number · required

Y coordinate in pixels.

// Long press to open context menu
Android.longPress(400, 600);
Android.delay(1500);

Content-Based Clicking

clickNodesByContentDescription()

Clicks elements by their accessibility content description.

description · string · required

Content description text to match (e.g., “Share”, “Like”, “Next”).

// Click by button label
Android.clickNodesByContentDescription("Next");
Android.delay(1000);

Android.clickNodesByContentDescription("Share");
Android.delay(1000);

Android.clickNodesByContentDescription("Send");

This is more reliable than coordinate-based clicking since UI layouts can vary across devices.

Typing & Text Input

simulateTypeInFirstEditableField()

Enters text in the first editable field on screen.

text · string · required

Text to type into the field.

// Type username
Android.simulateTypeInFirstEditableField("john_doe");
Android.delay(1000);

simulateTypeInSecondEditableField()

Enters text in the second editable field on screen.

text · string · required

Text to type into the field.

// Type password
Android.simulateTypeInSecondEditableField("myPassword123");
Android.delay(1000);

simulateType()

Types text in a field identified by View ID.

id · string · required

Android View ID of the target field.

text · string · required

Text to type.

// Type in specific field
Android.simulateType("com.example.app:id/username", "john_doe");
Android.delay(1000);

simulateTypeByClass()

Types text in a field by its class name.

className · string · required

Android class name (e.g., “android.widget.EditText”).

text · string · required

Text to type.

Android.simulateTypeByClass("android.widget.EditText", "Hello");

pressEnterKey()

Simulates pressing the Enter/Return key.

// Submit form
Android.simulateTypeInFirstEditableField("search query");
Android.delay(500);
Android.pressEnterKey();
Android.delay(2000);

Swipe & Scroll Functions

swipeUp()

Simulates an upward swipe gesture (scrolling down the page).

// Scroll feed
Android.swipeUp();
Android.delay(1000);
Android.swipeUp();
Android.delay(1000);

swipeDown()

Simulates a downward swipe gesture (scrolling up the page).

// Pull to refresh
Android.swipeDown();
Android.delay(2000);

swipeLeft()

Simulates a left swipe gesture.

// Navigate to next story
Android.swipeLeft();
Android.delay(800);

swipeRight()

Simulates a right swipe gesture.

// Navigate to previous story
Android.swipeRight();
Android.delay(800);

simulateScrollToBottom()

Scrolls to the bottom of a scrollable view.

// Scroll to end of page
Android.simulateScrollToBottom();
Android.delay(1500);

simulateScrollToTop()

Scrolls to the top of a scrollable view.

// Return to top
Android.simulateScrollToTop();
Android.delay(1500);

View ID Interactions

clickElementByViewId()

Clicks an element by its Android View ID.

viewId · string · required

Full View ID (e.g., “com.instagram.android:id/profile_tab”).

Android.clickElementByViewId("com.example.app:id/submit_button");
Android.delay(1000);

Complete Workflow Examples

Login Flow

function loginToApp(username, password) {
  Android.speakText("Starting login");
  
  // Click username field
  Android.simulateClick(400, 600);
  Android.delay(500);
  
  // Enter username
  Android.simulateTypeInFirstEditableField(username);
  Android.delay(1000);
  
  // Click password field
  Android.simulateClick(400, 800);
  Android.delay(500);
  
  // Enter password
  Android.simulateTypeInSecondEditableField(password);
  Android.delay(1000);
  
  // Submit
  Android.pressEnterKey();
  Android.delay(3000);
  
  Android.speakText("Login complete");
}

loginToApp("john_doe", "myPassword123");

Screen Detection

isTextPresentOnScreen()

Checks if specific text is visible on screen using AI vision.

text · string · required

Text or description to search for.

return · boolean

Returns true if text is found, false otherwise.

// Check for error message
if (Android.isTextPresentOnScreen("Error")) {
  Android.speakText("Error detected");
  // Handle error
}

// Check for success
if (Android.isTextPresentOnScreen("Success")) {
  Android.speakText("Operation successful");
}

// Check for specific element
if (Android.isTextPresentOnScreen("Login")) {
  Android.speakText("Need to log in");
  loginToApp("user", "pass");
}

This function uses AI vision analysis and may take 1-2 seconds to complete. Use sparingly for critical checks.

Best Practices

Always add delays after interactions

UI elements need time to respond. Add 500-1500ms delays after every action.

Android.simulateClick(100, 200);
Android.delay(1000); // Essential!

Use content descriptions over coordinates

Content description clicks are more reliable across devices.

// Good
Android.clickNodesByContentDescription("Share");

// Less reliable
Android.simulateClick(650, 1200);

Verify state before actions

Check screen state before performing actions.

if (Android.isTextPresentOnScreen("Login")) {
  // Perform login
} else {
  Android.speakText("Already logged in");
}

Handle errors gracefully

Wrap UI automation in try-catch blocks.

try {
  Android.simulateClick(100, 200);
  Android.delay(1000);
} catch (e) {
  Android.speakText("Click failed");
}

Limitations

• Requires Accessibility Service enabled
• Coordinates vary by device screen size and resolution
• Some apps block accessibility interactions
• Rapid interactions may be rate-limited
• Background execution may be restricted by Android

Related Functions

• delay() - Essential for timing UI interactions
• speakText() - Announce automation steps
• magicClicker() - AI-powered element detection
• magicScraper() - Extract screen information