Android CV Bot Template

You can visit my Android bot of Granblue Fantasy for more usage details and examples of how to use this framework: https://github.com/steve1316/granblue-automation-android

This template project serves as a starter point for Android botting that relies on Computer Vision template matching via OpenCV and executing gestures like tapping and scrolling to accomplish a automation goal. It uses MediaProjection Service to programmatically take screenshots and the Accessibility Service to execute gestures. The framework is well annotated with documentation to serve as explanations and usage hints.

Application.Preview.Demo.mp4

Provided Features

• A Home page that also houses the Message Log to allow the user to see informational logging messages.
• A Settings page that utilizes SharedPreferences to share data across the application and its associated Services.
• Floating overlay button to issue START/STOP signals to the bot process.
• Fleshed out template matching functions via OpenCV
• Notifications to alert users of various changes during workflow.
• Accessibility Service that will allow the bot process to execute gestures on the screen like tapping a specific point.
• Automatically checking for new app updates from your designated GitHub repository.

Instructions

1. Download the project and extract.
2. Download the Android version of OpenCV and extract its sdk folder contents into a newly created folder named opencv in the root of the project folder.
3. Clean Project and then Rebuild Project under the Build menu.
4. After building is complete, you can test the capability of this framework in the Android Studio's emulator.
5. After you familiarized yourself with what the framework can do, you can refactor the application's package name in various parts of the project and in settings.gradle.
6. If you want your application automatically check for the latest updates from your GitHub repo using AppUpdater, do the following:
   1. Upload a .xml file to your Github repo using the provided example app/update.xml with your updated version number, release notes, and link to the Releases page of your GitHub repo.
   2. Update the setUpdateXML() with the RAW link to your new update.xml.
   3. Now when a user has a lower version number compared to the latest version in your Releases page in your GitHub repo, they will be prompted with a dialog window like this:

Some things to note while developing

1. ImageUtils class reads in images in .webp format to reduce apk file size. You can change this if you wish.
2. All images are recommended to be categorized in separate folders inside the /assets/ folder. Be sure to update the folderName variables inside the various functions in ImageUtils. Or you could remove the need to organize them and just put all image assets into one place. Just make sure to update the code references to the folderName variables.
3. When working on a horizontal screen, the coordinate axis gets flipped as well. So if your vertical orientation dimensions is 1080x2400, then the horizontal orientation dimensions gets flipped to 2400x1080.
4. If you want to create nested Fragment Preference settings, there is an example provided to showcase how to do that in SettingsFragment.kt and mobile_navigation.xml.

Important Classes to be familiar with

BotService

• Facilitates the display and moving of the floating overlay button.
• Able to start/stop the bot process on a new Thread and notify users of bot state changes like Success or Exception encountered.

ImageUtils

• Able to template match for a single image or multiple image locations on the screen.
• Able to detect text via Google's ML Kit (Read up on the Google's documentation and my usage of it at https://github.com/steve1316/granblue-automation-android/blob/main/app/src/main/java/com/steve1316/granblueautomation_android/utils/ImageUtils.kt for a better understanding of it)

MediaProjectionService

• Starts up the MediaProjection Service to allow the ImageUtils class to programmatically grab screenshots to perform template matching on it.

MessageLog

• Sends informational logging messages from the Game class to the Home page of the application to quickly view what is going on.
• Automatically saves logs into text files when the bot stops.

MyAccessibilityService

• Starts up the Accessibility Service to allow the Game class to execute gestures at specified (x,y) coordinates on the screen.
  • Supported gestures are: tap, swipe, and scroll via AccessibilityAction.
• Note: If you encounter this Exception: kotlin.UninitializedPropertyAccessException: lateinit property instance has not been initialized, it means that you must have terminated the application via Android Studio. This causes the Accessibility Service to bug out. This does not happen in regular use without interference from Android Studio. To fix this, you can toggle on/off the Accessibility Service until the Toast message pops back up signalling that the Service is now running properly again.

NotificationUtils

• Allows the bot process to create and update Notifications to notify users of the status of the bot and whether or not the bot encounters an Exception.
• Sends a STOP signal to the bot process from the Notification's button via the StopServiceReceiver class.

Technologies used

1. MediaProjection - Used to obtain full screenshots
2. AccessibilityService - Used to dispatch gestures like tapping and scrolling
3. OpenCV Android 4.5.1 - Used to template match
4. Tesseract4Android 2.1.1 - For performing OCR on the screen
5. AppUpdater 2.7 - For automatically checking and notifying the user for new app updates