15-494/694 Cognitive Robotics: Lab 1

I. Setup

Set up your laptop to run vex-aim-tools by following these installation instructions.

If you've previously installed vex-aim-tools, then cd to that directory and do "git pull" to update to the latest version. You should do this at the start of every lab.

II. Meet VEX AIM

1. Open your CogRob folder in a shell or terminal window and activate the Python virtual environment you set up in the installation step.

2. Get a robot and its box of toys from the cabinet.

3. Note the id number on the label on the back of the robot. On Windows, type the following in the terminal window:

   	$env:ROBOT=AIM-xxxxxxxx.wifi.local.cmu.edu
       

   On Linux or macOS, enter the following command in the shell:

   	export ROBOT=AIM-xxxxxxxx.wifi.local.cmu.edu
       

4. Power on the robot using the button on the back. Using the touchscreen, swipe left to to "Settings", then to "Radio", then to "Station". If you see a green checkmark below the Station icon, click on it. If you instead see a white curved arrow below the Station icon, do nothing.

5. Wait for the wifi symbol in the top left corner of the display to turn steady green, showing that the robot is connected to WiFi.

6. In your terminal window, type "simple_cli".

7. simple_cli should connect to the robot and pop up two windows: the camera viewer and the worldmap viewer. In addition, a new tab should open in your browser, where speech recognition will be done. If asked, click to allow your browser to access your microphone.

8. Disable speech recognition by clicking the red "Stop Listening" button in the browser tab.

9. Keep the robot away from the edge of the table. Type the following commands in simple_cli to manually operate the robot:

   	Forward(50).now()
   	Turn(90).now()
   	Sideways(50).now()
   	Say("Hello human").now()
   	Kick().now()
       

III. Object Detection and the World Map

1. Make sure the camera viewer and world map viewer windows are visible.

2. Take a blue barrel out of the box and place it on the table so the robot can see it. When detected, a box is drawn around the barrel in the camera viewer.

3. The barrel should also be visible in the world map viewer. If not, rotate the view using the arrow keys. Type "h" in the worldmap viewer for a list of keyboard commands.

4. Note: if the barrel is not detected in the camera viewer, or if it flickers on the world map, this means the camera is not getting enough light. Let your instructor know. As a temporary fix, you can use your cellphone in flashlight mode to better light the scene.

5. Slide the barrel away from the robot and watch the world map update.

6. Use your hand to block the robot's view, and notice that the barrel disappears from the world map.

7. Place an orange barrel next to the blue barrel and observe its appearance in the world map.

8. Tell the robot to turn by 90 degrees and observe that the barrels are no longer in view, so they are dimmed in the world map.

9. In simple_cli type "show objects" and observe the barrels' coordinates.

10. Try showing the robot some of the purple AprilTags.

IV. Meet Celeste

1. Exit simple_cli by typing control-\ on Linux or control-Z on Windows.

2. cd to the vex-aim-tools directory.

3. Run simple_cli again.

4. Type the following command in simple_cli:

   	runfsm('Celeste')
       

5. You are now talking to Celeste, an intelligent robot powered by GPT-4o. Try saying the following to Celeste. If speech recognition is not working because the lab is too noisy, you can turn off the listener (click the red "Stop listening" button in the browser window) and then type your inputs into simple_cli preceded by a period and a space, as in ". Who are you?"
   • Who are you?
   • What do you see? (Show it some barrels.)
   • Move forward 50 millimeters.
   • Turn left by 90 degrees.
   • Turn toward an orange barrel.
   • Pick it up.
   • Turn around.
   • Move sideways by 50 millimeters.
   • Drop the barrel.
   • Turn away from it.
   • How is your battery?
   • How many barrels are there?
   • Are there any green barrels?
   • How far away is the closest blue barrel?
   • How tall are you?
   • Please move in a square with sides of length 50 millimeters.

V. Clean Up

1. Quit Python by typing control-\ (Linux or macOS) or "exit" followed by control-Z (Windows).

2. Hold the robot's power button in until the robot shuts down.

3. Take proper care of your equipment. Put the robot and its toy box back in the cabinet in the correct cubbyhole, and plug the robot into the charger.

VI. Homework

Try asking Celeste to do various things or to tell you various things. What can she do? What does she fail at?

Write down five things you tried, and what the results were. Hand this in via Canvas. Points will be awarded for diverse and imaginative requests, or for interesting or surprising results. Be creative!