Use your phone as a sensor in ROS2

While many projects exist to control your robot from your phone, this project is the other way around; your phone is the robot’ sensors ! It will send the camera feed, IMU and GPS so that you may integrate the phone onto a mobile base.

The particularity of this project is that it relies on the mobile browser instead of a specific app. A webpage is served from the ROS2 node, opening the page from a mobile client will ask for permissions in the browser. The data is transmitted between the server and client using websockets for a modern and fast communication, as opposed to making HTTP requests or streaming UDP.

This repository is inspired by a project I did with students as a TA called phone-imu.

Build

source /opt/ros/humble/setup.bash
rosdep install -i --from-path src --rosdistro humble -y --ignore-src
colcon build --packages-up-to phone_sensors_bridge
# Or build for development, but remember to clean and rebuild every time the JS files are changed
# colcon build --symlink-install  --packages-up-to phone_sensors_bridge_examples --event-handlers console_direct+

phone_sensors_bridge usage

Quickstart

Run the following in a different terminal than the one used for building, otherwise the server might start from inside the build folder which will not contain the template and static folders for the webpage.

source install/setup.bash

# Ubuntu IP
EXTRA_IP=$(ip route get 8.8.8.8 | grep -oP 'src \K[^ ]+')
echo "My IP is $EXTRA_IP"
# Generate SSL certificates for local webserver 
ros2 run phone_sensors_bridge generate_dev_certificates.sh $EXTRA_IP

# Start server
ros2 run phone_sensors_bridge server --ros-args -p video_width:=1280 -p video_height:=720

Open the webpage from your mobile device. The URL contains the server host IP where the node is running. It depends on your network, but most likely is https://<EXTRA_IP>:2000. If you have multiple network interfaces, favour the fastest such as ethernet over wifi.

The page will prompt for permissions, then display the chosen camera

Published topics

Parameters

A negative value for the time reference, IMU, GNSS frequencies or video FPS will disable sending the corresponding data from the client device. This allows conserving bandwidth and processing power when certain sensors are not needed.

Note: gnss_frequency only takes effect when useWatchPosition is set to false in geolocation.js. When watchPosition is used (the current default), the browser controls the GPS update rate and this parameter is ignored.

To find out the available camera1_device_label and camera2_device_label, open the video test page

File truncated at 100 lines see the full file