How to use the sandbox

Before you start: what's what on the screen

When you open the sandbox, it looks like this. Match the numbers with the list below.

1234567891011

1. View buttons: choose what to look at (All, 3D, Top, Side, End, Camera).
2. Drive: fly the robot by hand.
3. ▶ Play: the robot flies your route by itself.
4. The readout: where the robot is, which way it faces, and the nearby checkpoint's level.
5. The robot camera: what the robot sees right now.
6. 3D view: the whole of Kibo, the robot and your route.
7. The log: a short list of what happened.
8. Drive controls: keys and buttons (joysticks on a phone).
9. Status bar: whether the route is clear, the distance, the time and the estimated score.
10. Tabs: Route (the list of waypoints), Code and World.
11. Mission list: what's still left to do at each checkpoint.

1. Getting started

1. Look around. The green box is the Keep-In Zone: the robot must stay inside it. Blue squares are the AR markers, and the small pink dots are the checkpoints. Each checkpoint has a dashed line to its marker and an orange stick showing which way the top of the robot must point there.
2. Read the Mission list in the Route tab. It lists every checkpoint, the goal last, and says what is still wrong with each one.
3. Start from the starter route of four waypoints. Waypoint 1 is a worked example that passes CP 1. The next three are rough drafts that are a little wrong on purpose. Press Play to watch the robot visit them and read each AR marker, then fix the drafts and plan the rest yourself. Load the starter route in the ⋯ menu brings it back.

Your first ten minutes, step by step

1. Press 3D at the top. Drag with the mouse (or a finger on a phone) to turn the module: find the green box, the blue markers and the pink checkpoints.
2. Press ▶ Play. The robot flies the starter route, stopping at each waypoint to read the markers. Watch what the log says.
3. Press ✕ Exit play to come back. Look at the Mission list to see what's wrong at each checkpoint.
4. Fix one of the rough waypoints the way the Mission list suggests: change its x, y or z a little in the list, or pick "Face a target" in its orientation menu. See whether the level in the Mission list changes.
5. When every checkpoint passes, go to the Code tab and copy the code. Well done: your first route!

2. The views

Choose a view with the buttons at the top: All, 3D, Top, Side, End and Camera.

• 3D shows the whole module from the long side: the dock end on the left, the airlock end on the right, overhead at the top. With a mouse, drag to turn it and scroll to zoom toward the pointer. On a touch screen, drag one finger to turn it, pinch to zoom toward your fingers, drag two fingers to slide it, and double-tap to put it back. On a phone held upright (not full screen), dragging up or down scrolls the page as usual and dragging sideways turns the view; to roll the module about its long axis (airlock to dock), drag the ⟲ roll wheel on the left edge of the 3D view up or down. (A phone starts scrolling the page as soon as a finger moves up or down, so the roll has its own wheel.) Reset view puts every view back as it started: the 3D angle, the Top, Side and End framing, the camera picture and the small camera window.
• Top is seen from overhead, Side is the long side and End looks down the module from the dock end. The labels around each view name the walls (Dock end, Airlock end, SAIBO wall, EPS2 wall, Overhead, Deck), and the small arrows in the corner show which way x, y and z grow.
• Camera shows what the robot camera sees. See section 6.
• In 3D, a small camera window in the top right shows what the camera sees from the selected waypoint. Tap ⤢ to make it fill the view, then ⤡ Back to return.

3. Mission and scoring

Pass every checkpoint, then finish at the goal checkpoint, all within 5 minutes. A checkpoint is judged on three things, and its level (Lv1 to Lv3) is the lowest of the three:

• Level, at the top of the Mission list, sets the difficulty: Easy (6 checkpoints), Normal (8), Hard (10) or Very Hard (12). Picking one draws a new random layout; New layout draws another at the same level. The line under it names the AR markers that have no checkpoint this run: the real game uses 6 to 12 checkpoints, so some markers are always left out. The rules don't say how many checkpoints each level has, so these numbers are a guess.
• The Mission list says what to change for the next level, and which roll button to press (⟲ or ⟳).
• The goal must be the last checkpoint you pass. Passes after the 5-minute limit don't count.
• The estimated score is (checkpoint points + goal points) × a time bonus. It is a rough guide: the simulator's timing and disturbances differ.

4. Waypoints

• Add one with + Waypoint, by double-clicking (or double-tapping) in the Top, Side or End view, or by driving there (section 5). Go to… adds a rough pose near a checkpoint for you to fix.
• Move a waypoint by dragging it in the Top, Side or End view, or by typing its x, y and z. The readout under the pointer shows the coordinates and whether that spot is allowed.
• Aim it with the orientation menu: keep the previous direction, face the travel direction, Face a target (an AR marker), or type a custom quaternion. Under each waypoint you can see its heading, pitch and roll.
• Red moves leave the Keep-In Zone or hit a Keep-Out Zone; the X marks where. Press Hint to learn why, then move or add waypoints. Route around it and Fix blocked moves add via points for you.
• Use ↑ ↓ to reorder waypoints and ✕ to delete one. Undo is in the ⋯ menu, or Ctrl+Z.

5. Driving the robot

The sandbox opens in Drive, so you can fly the robot by hand straight away (▶ Play leaves it and Drive brings you back; pressing Drive while driving keeps you in Drive). Tap or click in the sandbox first: until then the keyboard keeps scrolling the page. The robot camera appears on the left and the 3D view on the right (camera on top on a phone held upright). The controls follow the camera picture, whichever way up the robot is.

Zero gravity (the Zero-g switch beside Reset, or G; off each time Drive starts) flies the robot the way it moves on the station: the controls fire thrusters, and when you let go it keeps drifting and turning. The drift fades only very slowly (to half in about 20 seconds), so the robot keeps going until you thrust the other way, press Halt or hit a wall. Thrust builds speed up to 1 m/s; turning is limited to 40° a second and settles in about 2 seconds; thrusting the other way brakes harder than it pushes. It flies almost the same on the Free flight page, which adds a little help to keep it under control. To stop at once, press Halt (shown beside the switch while zero gravity is on) or X; thrusting the other way brakes too. Bumping a wall stops the motion into it. The HUD shows the drift speed. Reset or turning Zero-g off stops the robot.

• The keys work anywhere on the page while Drive is on, except while you are typing in a box. On a computer you can also hold the on-screen keys with the mouse.
• The readout at the top shows the position, heading, pitch and roll, and near a checkpoint its level for position, centring and roll. It also says what the camera reads. On a computer it always takes two lines, so the camera picture stays put; a line too long to fit is cut short with …, and hovering over the readout shows all of it.
• The robot can't leave the Keep-In Zone: it slides along the walls.
• Driving always starts with the camera picture the right way up. At the dock the robot sits upside down, so it is rolled over first.
• The tool works out the quaternion of each waypoint you add, so you never have to type one.
• Driving works in the camera + 3D view. If you pick another view from the buttons at the top, driving pauses; press ⤡ Back to driving (or Esc) to carry on.
• ☰ Waypoints opens the waypoint list while you drive, so you can check or edit the numbers.

6. The robot camera and AR markers

The Camera view shows what the robot camera sees from the selected waypoint, or live while you drive or play. The AR markers are real 4×4 ArUco markers (IDs 1 to 12, 2 cm), so the sandbox reads them the way OpenCV does on the robot.

• A marker it reads gets a green outline, its id and its axes. An orange dashed outline means the marker is in the picture but can't be read, and the panel says why (too far away, too small, too edge-on, partly outside, or seen from behind).
• The robot has to be close to read a marker: within 1 m of the camera. Farther away, the marker shows but isn't read.
• The dashed rings around the centre are the centring levels: 10° (Lv3), 20° (Lv2) and 30° (Lv1).
• Below the picture: each marker's id, distance, how far it is off centre and how far it looks turned, the tvec and rvec your program would get, and the marker position worked back from the tvec. Copy reading copies it all.
• The camera numbers (resolution and focal length) are assumed. On the robot, read the real ones from /camera_main/camera_info.

7. Play and checking a route

• Play shows a short Initializing log, then flies the robot along your route; Pause holds it, and the speed menu next to it works during Play too. It flies the clear moves and stops before the first blocked one. When it ends, the robot stays where it stopped until you press Exit play, or Play again to fly it once more.
• While it plays you see the robot camera and the 3D view (camera on top on a phone held upright). At each waypoint the robot stops for a moment and reads the AR markers. Tap ⤢ Expand to make the camera fill the view.
• The log (the small dark panel between or under the views) lists what happens: each waypoint the robot heads to, what the camera reads there ([AR] ID 4 · 0.71 m · 0° off centre), whether the checkpoint passes, and the result at the end. While you drive it logs markers coming into and out of view, checkpoint levels and the waypoints you add. Tap ▾ to collapse it to the latest line and ▴ to open it again; on a phone in full screen it starts collapsed.
• Play starts at 2×. Pick 1× to watch more slowly, or 5× or 10× to go faster.
• The status bar shows whether the route is clear, the distance, the mission time against the 5-minute limit, and the estimated score.

8. Practising

• The simulator draws new checkpoints every run, so press New layout in the Mission list (or New random checkpoints in the World tab) and plan again. Back to the sample layout returns to the first one.
• To try a layout from your own simulator run, paste its /mission/get_checkpoints message into the World JSON box and press Apply.
• Add a Keep-Out Zone in the World tab and route around it.
• For a harder challenge, your teacher can open the sandbox in practice mode: an empty route and no helpers (no Go to…, Route around it, Fix blocked moves or starter route).

9. Copying the code

• Python is ready to paste between the USER EDIT AREA markers of the official template. It moves to each waypoint and reports each checkpoint as passed.
• C++ is the same route for the C++ template.
• Your final program must read the checkpoints from /mission/get_checkpoints, because they change every run. Use the sandbox to learn the moves, not to copy fixed numbers.

10. Saves and progress

• Your route, world and progress save automatically in this browser.
• Saves keeps named versions, downloads a backup file to move your work to another device, and loads it back.
• The Progress list ticks off milestones as you learn: your first waypoint, driving, passing a checkpoint, reading a marker, completing the mission and more.

11. Phones and full screen

• Full screen fills the screen with the views and turns a phone to landscape where the browser allows it. On an iPhone, turn the phone sideways yourself. Exit full screen (or Esc) leaves it.
• In full screen the tabs are hidden. ☰ Waypoints slides the waypoint list in from the side, with a ✕ to close it.
• When you drive on a phone, the joysticks also respond to touches near them, so your thumbs don't turn the 3D view by mistake.
• On a phone held upright, the view buttons (All, 3D, Top, Side, End, Camera) are folded into one button next to the name, which shows the current view; tap it to pick another. That leaves more room for the camera and 3D views.
• When you start steering (a joystick on a phone, a Drive key on a computer), the page scrolls so the views and their controls sit in the middle of the screen.
• Messages that pop up have a ✕ to close them, or they go away on their own.

Word list

waypoint

One stop on your route.

route

The path through your waypoints, in order.

checkpoint

A place where the robot must point its camera at a marker. (rules guide)

level

How well a checkpoint was passed: Lv1 to Lv3.

Keep-In / Keep-Out Zone

Where the robot must stay / where it must not go.

Drive

Flying the robot by hand.

zero gravity (Zero-g)

The robot floats like on the station; to stop, it has to push the other way.

HUD, the readout

The lines over the picture: where the robot is, which way it faces, and the nearby checkpoint.

log

A short diary of what happened.

quaternion

Four numbers that write an orientation. (Coordinates and rotations)

Now the last step on the learning path: try it yourself in the sandbox, or fly the robot round the whole station in Free flight.