Using a TensorFlow Pretrained Model to Detect Everyday Objects
Teams have the option of using a custom TensorFlow object detection model to detect objects other than the current season's game elements. This tutorial demonstrates how use a pretrained model to look for and track everyday objects. This particular tutorial uses the OnBot Java programming tool, but teams can also use Android Studio or the Blocks programming tool to implement this example. This tutorial covers an advanced topic and assumes that the user has good programming knowledge and is familiar with Android devices and computer technology.
This tutorial also assumes that you have already completed the steps in a previous TensorFlow tutorial. If you have not yet completed the steps in the previous TensorFlow tutorial, then please do so before continuing with this tutorial.
The custom inference model must be in the form of a TensorFlow Lite (.tflite) file. For this example, we will use the same object detection model that is used in Google's example TensorFlow Object Detection Android app.
The model and its corresponding label map can be downloaded from this link.
The .zip archive contains a file called "detect.tflite". This TensorFlow Lite file is the inference model that TensorFlow will use to recognize the everyday objects. It is based on the mobilenet architecture, which was designed to provide low latency recognitions, while still maintaining reasonable recognition accuracy.
The .zip archive also contains a text file called "labelmap.txt". This text file contains a list of labels that correspond to the known objects in the "detect.tflite" model file.
Download the .zip archive to your laptop and uncompress its contents to a folder.
For this example, we want to transfer the .tflite and labelmap files to a directory on your robot controller.
If you are an advanced user and are familiar with using adb, then you can use adb to push the files to the directory "/sdcard/FIRST/tflitemodels" on your robot controller.
If you do not know how to use the Android Debug Bridge tool, then you can use the File Explorer on a Windows laptop to copy and paste the files to this directory.
If you are using an Android phone as your robot controller, then connect the phone to your laptop using a USB cable. Swipe down from the top of the phone's screen to display the menu. Look for an item that indicates the USB mode that the phone is currently in. By default, most phones will be in charging mode.
Tap on the "Android System" item in the menu to get additional details on the current USB mode.
Tap on the Android System item.
Tap where it says "Tap for more options" display an activity that will allow you to switch the phone into file transfer mode.
Tap to display the USB mode options.
Select the "Transfer files" mode and the phone should now appear as a browsable storage device in your computer's Windows Explorer.
Select the "Transfer files" option.
If you are using a REV Robotics Control Hub, you can connect the Control Hub (that is powered on by a fully charged 12V battery) to your laptop using a USB Type C cable and the Control Hub will automatically appear as a browsable storage device in your computer's Windows Explorer. You do not need to switch it to "Transfer file" mode since it is automatically always in this mode.
Use your Windows Explorer to locate and copy the "detect.tflite" and "labelmap.txt" files.
Copy the .tflite and .txt files.
Use Windows Explorer to browse the internal shared storage of your Android device and navigate to the FIRST->tflitemodels directory. Paste the "detect.tflite" and "labelmap.txt" files to this directory.
Navigate to FIRST->tflitemodels and paste the two files in this directory.
Now the files are where we want them to be for this example.
Use the OnBot Java editor to create a new op mode that is called "TFODEverydayObjects" and that is based on the "ConceptTensorFlowObjectDetection" sample op mode. Note that a copy of the full op mode that was used for this example (except for the Vuforia key) is included at the end of this tutorial.
Modify the annotations to change the name to avoid a "collision" with any other op modes on your robot controller that are based on the same sample op mode. Also comment at the @Disabled annotation to enable this op mode.
@TeleOp(name = "TFOD Everyday Objects", group = "Concept")
//@Disabled
public class TFODEverydayObjects extends LinearOpMode {
Before you can run your op mode, you must first make sure you have a valid Vuforia developer license key to initialize the Vuforia software. You can obtain a key for free from https://developer.vuforia.com/license-manager. Once you obtain your key, replace the VUFORIA_KEY static String with the actual license key so the Vuforia software will be able to initialize properly.
private static final String VUFORIA_KEY =
" -- YOUR NEW VUFORIA KEY GOES HERE --- ";
Modify the sample op mode so that you specify the paths to your model file ("detect.tflite") and to your label map file ("labelmap.txt").
public class TFODEverydayObjects extends LinearOpMode {
private static final String TFOD_MODEL_FILE = "/sdcard/FIRST/tflitemodels/detect.tflite";
private static final String TFOD_MODEL_LABELS = "/sdcard/FIRST/tflitemodels/labelmap.txt";
private String[] labels;
Your op mode will need the following additional import statements:
import java.io.BufferedReader;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
Create a method called readLabels() that will be used to read the label map file from the tflitemodels subdirectory:
/**
* Read the labels for the object detection model from a file.
*/
private void readLabels() {
ArrayList<String> labelList = new ArrayList<>();
// try to read in the the labels.
try (BufferedReader br = new BufferedReader(new FileReader(TFOD_MODEL_LABELS))) {
int index = 0;
while (br.ready()) {
// skip the first row of the labelmap.txt file.
// if you look at the TFOD Android example project (https://github.com/tensorflow/examples/tree/master/lite/examples/object_detection/android)
// you will see that the labels for the inference model are actually extracted (as metadata) from the .tflite model file
// instead of from the labelmap.txt file. if you build and run that example project, you'll see that
// the label list begins with the label "person" and does not include the first line of the labelmap.txt file ("???").
// i suspect that the first line of the labelmap.txt file might be reserved for some future metadata schema
// (or that the generated label map file is incorrect).
// for now, skip the first line of the label map text file so that your label list is in sync with the embedded label list in the .tflite model.
if(index == 0) {
// skip first line.
br.readLine();
} else {
labelList.add(br.readLine());
}
index++;
}
} catch (Exception e) {
telemetry.addData("Exception", e.getLocalizedMessage());
telemetry.update();
}
if (labelList.size() > 0) {
labels = getStringArray(labelList);
RobotLog.vv("readLabels()", "%d labels read.", labels.length);
for (String label : labels) {
RobotLog.vv("readLabels()", " " + label);
}
} else {
RobotLog.vv("readLabels()", "No labels read!");
}
}
Important note: The readLabels() method actually skips the first line of the "labelmap.txt" file. If you review Google's example TensorFlow Object Detection Android app carefully you will notice that the app actually extracts the label map as metadata from the .tflite file. If you build and run the app, you will see that when the the app extracts the labels from the .tflite file's metadata, the first label is "person". In order to ensure that your labels are in sync with the known objects of the sample .tflite model, the readLabels() method skips the first line of the label map file and starts with the second label ("person"). I suspect that the first line of the label map file might be reserved for future use (or it might be an error in the file).
You will also need to define the getStringArray() method which the readLabels() method uses to convert the ArrayList to a String array.
// Function to convert ArrayList<String> to String[]
private String[] getStringArray(ArrayList<String> arr)
{
// declaration and initialize String Array
String str[] = new String[arr.size()];
// Convert ArrayList to object array
Object[] objArr = arr.toArray();
// Iterating and converting to String
int i = 0;
for (Object obj : objArr) {
str[i++] = (String)obj;
}
return str;
}
When I tested this example op mode, I disabled the digital zoom factor because for my testing, the phone was relatively close to the target objects. If you would like to test the op mode using small targets at larger distances, you can use a zoom factor greater than one to magnify the target object and increase the detection reliability.
// Uncomment the following line if you want to adjust the magnification and/or the aspect ratio of the input images.
//tfod.setZoom(2.5, 1.78);
You can set the minimum result confidence level to a relatively lower value so that TensorFlow will identify a greater number of objects when you test your op mode. I tested my op mode with a value of 0.6.
tfodParameters.minResultConfidence = 0.6;
Call the readLabels() method to read the label map and generate the labels list. This list will be needed when TensorFlow attempts to load the custom model file.
public void runOpMode() {
// read the label map text files.
readLabels();
// The TFObjectDetector uses the camera frames from the VuforiaLocalizer, so we create that
// first.
initVuforia();
initTfod();
-
TensorFlow 2023-2024