2023 03 25 Raven Status Update

A fair amount of progress as been made on Raven over the last few days.

I have list of TODO items that need to be tackled. One of those items has been on the list or months and this week it bubbled to the top. My proximity sensor values were being time stamped with old values.

In ROS (Robot Operating System), the whole software stack that tries to generate commands to move the robot somewhere interesting relies on a flood of data coming in, like SONAR readings, time of flight readings and LIDAR readings. Each of those readings comes with a timestamp indicating when the reading was taken.

On my robot, the readings are made on a custom PC board I made, and then sent to the main computer of the robot. The custom board needs to have the same sense of “now”-ness as the main PC — the two computers have to agree within a few milliseconds on what time it is now. It doesn’t do any good to take a SONAR reading and then have it show up a half second late to the main computer. The algorithms all have a small “tolerance” for time skew, and if sensor readings are too old, they are rejected out of hand as useless.

The software piece I use to transmit my sensor values from the custom board to the main computer is a package called “Micro ROS”. I use this software with some small customization as my custom board has a lot more capability to it that the usual, intended board using Micro ROS. 

One function built into Micro ROS is the ability to ask another computer what time it is, and set its own clock to match that other computer. But setting it just once doesn’t work. Each computer has a clock driven by a crystal-controlled oscillator, and crystals drift as they heat up. Worse, much worse, the CPU clock chip in my custom board seems to “burp” now and then when interrupt signals come in and my hardware generates a fair number of interrupt signals.

Another problem is that Micro ROS has baked into it an expectation that the main computer is using a particular version of communication software, and the expected version currently has bugs which cause my stereo cameras to not operate correctly. It took a bit of reading for me to realize that little factoid.

For the moment, I can ignore that, so I set my main computer back to using the buggy communication software. Also, when Micro ROS asks for the current time, it takes “quite a bit of time” to get the result, usually about 150 milliseconds, but sometimes as much as a second. So any answer it gets from the main PC will be inherently wrong right away.

My last few days of programming have been devoted to finding someway to make that all work with an allowable tolerance for errors in timing. I tried over and over, and I’m normally very good at finding creative solutions to tricky problems. Still my code got progressively worse the more I tried to fix the problem. And then, my robot buddy Ralph Hipps called for one of our at-least-daily robot chats and in the process of explaining the problem to him, it occurred to me what the root cause was. My custom board was attempting to do a complex operation during interrupt handling.

Interrupt handlers on a computer must be very quick. If your interrupt handler code takes more than about a millisecond, sometimes even only a few tens of microseconds, everything falls apart. And because I was explaining the symptoms to someone else, I finally realized that I was taking probably tens of milliseconds in my interrupt handler for the SONAR sensors.

Once I realized that, the fix was easy. The details aren’t too important, but I stashed the important information from the signal in a shared, global location and exited the interrupt handler. Then the normal, non-interrupt code picked up the stashed information for processing. Outside of the interrupt handler, you can largely take whatever time you want to process data. Sort of. ROS navigation depends heavily on high “frame rates” from sensors. If you can’t generate, say, LIDAR data 10 times a second or faster, you have to slow your robot way down, otherwise the robot would move faster than it could react to, say, the cat rearing up in front of your robot with nunchucks in hand.

The robot is now happily sending sensor data nearly three times faster than before, about 80 frames a second and rarely gets the time out of sync by very much. Now I can move on to the new problems that have shown up because I fixed this problem.

Below is an obligatory snap shot of a part of the visualization software showing my robot, in a room, with LIDAR, SONAR and time of flight sensors showing as well. No cat is in this picture.

Everything about robots is hard (TM)

Efficient multi-SONAR with Arduino-like systems

Handling multiple sensors on an Arduino-like processor, specifically a Teensy 4.1 in my case, can be a bit tricky if you want to get high loop rates.

In my robot, I’m using the Teensy to send data from 8 time-of-flight sensors, 4 SONAR sensors, 2 motor current sensors and 2 motor temperature sensors while also controlling a RoboClaw motor controller, an 8-channel relay board and a touch screen display. On top of this, it runs micro-ROS to send and receive data messages to the rest of the robot.

In this article I will talk only about the issue of using the HC-SR04, inexpensive SONAR sensors. Here is one article that gives you some background in what the sensor is and how it works.

For a robot that uses the ROS software stack, getting data at fast frame rates is crucial, otherwise the robot will have moved far along from where it was when the sensor data was gathered, so the data no longer represents the truth from the point where the robot is now.

It can take on order of 20 milliseconds to get a reading from the SONAR at the maximum distance. If you had a single sensor and read it once per loop, you could, in theory, send about 50 readings a second to your robot. With 4 sensors, however, if you sensed one SONAR after the other in a round-robin manner, the frame rate would drop down to about 50/4 or 12.5 frames per second, which may be too low a rate for your robot.

If your robot is moving at the rate of a slow walk for a person, about 1 mile per hour, that translates to 0.45 meters per second. At a frame rate of 12.5 frames per second, the robot will move 36 millimeters or about 1.4 inches between readings. If you are traveling along a narrow hallway or in a crowded room, that may not be good enough to avoid crashing into nearby obstacles. If you can get back up to nearly 50 frames per second, though, the robot will only travel about 9 millimeters or just over a third of an inch. Add in the other things your Arduino processor is doing besides reading SONAR sensors and frame rates could get much worse.

To get around this problem, you need to use interrupt handlers to control the SONAR sensors. Briefly, what I do is:

  • Use a timer to drive a state machine. The timer generates an interrupt every 20 microseconds.
  • The timer handler has a counter. Every 500 counts, or 500 * 20 microseconds which is 0.01 seconds, the next SONAR in the set of 4 begins a reading. I used this fixed spacing to solve the issue of multiple SONARs interfering with each other. Whenever a SONAR begins sending its ping sound, it’s possible for all the other SONAR sensors to receive the echo. I want to make it pretty likely that a SONAR hears only its own echo. Sound travels at about 340 meters per second. In 0.01 seconds, a SONAR ping can travel about 3.4 meters. Since SONAR measures round-trip time delay, the SONAR can detect objects up to about 1.7 meters in that 0.01 seconds. Practically, the HC-SR04 sensors are not so accurate or sensitive after about that distance, anyway.
  • Once a SONAR starts transmitting its ping, 20 microseconds later the transmitter is turned off. Then the next SONAR in the set of 4 will get the on/off sequence after 0.01 seconds.
  • When the transmitter stops for a SONAR, it begins listening for the echo. The echo is processed via an interrupt handler. The interrupt handler computes the time it took to hear the echo, uses the speed of sound to convert echo time to the distance to the obstacle. Then a ROS message is sent for the sensor reading.

On a Teensy4.1, my code is publishing about 25 frames per second for each of the 4 SONAR sensors, or 100 frames per second alltogether. The sensors are spaced out in time, the readings don’t come altogether at once, as described above. Also, the whole code is driven by my TModule code module, which I’ll describe in another article, but you can take this code by itself and combine the two loop and setup methods together and run it without my TModule module.

Here is tsonar.h

#pragma once

#include <Wire.h>
#include <stdint.h>

#include "tmodule.h"

class TSonar : TModule {
 public:
  // Which SONAR sensor.
  typedef enum Sonar {
    kFront,
    kRight,
    kBack,
    kLeft,
    kNumberSonars  // Number of SONAR sensors.
  } Sonar;

  // Get sensed range for the device. A value of < 0 => no sensor 'device'
  // detected during setup.
  int GetValueMm(Sonar device);

  // Get the average range for the device.
  float GetAverageValueM(Sonar device);

  // Singleton constructor.
  static TSonar& singleton();

 protected:
  // From TModule.‰
  void loop();

  // From TModule.‰
  const char* name() { return "Sonr"; }

  // From TModule.‰
  void setup();

 private:
  // Number of readings to average together to reduce noise.
  static const uint8_t kNumberReadingsToAverage = 4;

  // Private constructor.
  TSonar();

  // Common interrupt handler.
  static void CommonInterruptHandler(uint8_t pin, long& end_time,
                                     long& start_time, uint8_t& average_index,
                                     size_t sonar_index);

  // Interrupt handler for per device echo.
  static void Echo0InterruptHandler();
  static void Echo1InterruptHandler();
  static void Echo2InterruptHandler();
  static void Echo3InterruptHandler();
  static void TimerInterruptHandler();

  // Next device being handled in the round-robin processing.
  static uint8_t g_next_sensor_index_;

  // Singleton instance.
  static TSonar* g_singleton_;

  // Last captured sensed distance for each SONAR device.
  static int g_values_mm_[kNumberSonars];

  // List of last values, for computing an average.
  static int g_values_mm_history_[kNumberSonars][kNumberReadingsToAverage];

  // Last captured average distance for each SONAR device.
  static float g_average_value_m_[kNumberSonars];

  // GPIO pins for controlling the SONAR sensors.
  enum {
    kPinEcho0 = 35,
    kPinTrigger0 = 34,
    kPinEcho1 = 37,
    kPinTrigger1 = 36,
    kPinEcho2 = 41,
    kPinTrigger2 = 40,
    kPinEcho3 = 15,
    kPinTrigger3 = 14
  };

  // Microseconds per timer interrupt.
  static const uint16_t kTimerPeriodUSec = 20;

  // Desired milliseconds to wait between a low pulse and the next high pulse.
  static const uint16_t kTimerSamplingPeriodMSec = 10;

  // Timer interrupts to expire before a low pulse to high pulse transition is
  // taken..
  static const uint16_t kTimerCountsPerSamplingPeriod =
      (kTimerSamplingPeriodMSec * 1000) / kTimerPeriodUSec;

  // For converting SONAR echo time to millimeters.
  static const float g_time_to_mm_scaler_;
};

And here is tsonar.cpp

#include "tsonar.h"

#include <stdint.h>

#include "Arduino.h"
#include "TimerOne.h"
#include "Wire.h"
#include "tmicro_ros.h"

void TSonar::CommonInterruptHandler(uint8_t pin, long& end_time,
                                    long& start_time, uint8_t& average_index,
                                    size_t sonar_index) {
  switch (digitalRead(pin)) {
    case HIGH:
      // When the SONAR begins transmitting, the echo pin will
      // go HIGH. Capture the start time.
      end_time = 0;
      start_time = micros();
      break;

    case LOW:
      // When an echo is received, the echo pin will go LOW.
      // Compute the distance to the object and send a ROS message.
      end_time = micros();
      g_values_mm_[sonar_index] =
          (end_time - start_time) * g_time_to_mm_scaler_;
      g_values_mm_history_[sonar_index][average_index++] =
          g_values_mm_[sonar_index];
      if (average_index >= kNumberReadingsToAverage) {
        average_index = 0;
      }

      // Compute an average distance over the last few readings
      // to help reduce the sensor noise.
      int average_sum_mm = 0;
      for (size_t i = 0; i < kNumberReadingsToAverage; i++) {
        average_sum_mm += g_values_mm_history_[sonar_index][i];
      }

      g_average_value_m_[sonar_index] =
          (average_sum_mm * 0.001) / kNumberReadingsToAverage;

      // Publish the average distance for the sensor.
      TMicroRos::PublishSonar(sonar_index, g_average_value_m_[sonar_index]);
      break;
  }
}

float TSonar::GetAverageValueM(Sonar device) {
  return g_average_value_m_[device];
}

void TSonar::Echo0InterruptHandler() {
  // Each sensor has it's own distance timer and it's own
  // set of values used to compute an averate distance.
  static long end_time = 0;
  static long start_time = 0;
  static uint8_t average_index = 0;
  CommonInterruptHandler(kPinEcho0, end_time, start_time, average_index, 0);
}

void TSonar::Echo1InterruptHandler() {
  static long end_time = 0;
  static long start_time = 0;
  static uint8_t average_index = 0;
  CommonInterruptHandler(kPinEcho1, end_time, start_time, average_index, 1);
}

void TSonar::Echo2InterruptHandler() {
  static long end_time = 0;
  static long start_time = 0;
  static uint8_t average_index = 0;
  CommonInterruptHandler(kPinEcho2, end_time, start_time, average_index, 2);
}

void TSonar::Echo3InterruptHandler() {
  static long end_time = 0;
  static long start_time = 0;
  static uint8_t average_index = 0;
  CommonInterruptHandler(kPinEcho3, end_time, start_time, average_index, 3);
}

int TSonar::GetValueMm(Sonar device) {
  if (static_cast<int>(device) >= kNumberSonars) {
    return -1;
  } else {
    return g_values_mm_[static_cast<int>(device)];
  }
}

void TSonar::loop() {
  // Nothing needs to be done in the loop. Everything is drivven
  // by interrupts.
}

void TSonar::setup() {
  // Set up the Teensy pins to talk to the SONAR devices.
  pinMode(kPinEcho0, INPUT);
  pinMode(kPinTrigger0, OUTPUT);
  pinMode(kPinEcho1, INPUT);
  pinMode(kPinTrigger1, OUTPUT);
  pinMode(kPinEcho2, INPUT);
  pinMode(kPinTrigger2, OUTPUT);
  pinMode(kPinEcho3, INPUT);
  pinMode(kPinTrigger3, OUTPUT);

  // Setup a timer to drive the state machine.
  Timer1.initialize(kTimerPeriodUSec);
  Timer1.attachInterrupt(TimerInterruptHandler);

  // Attach an interrupt handler to each SONAR's echo pin.
  attachInterrupt(kPinEcho0, Echo0InterruptHandler, CHANGE);
  attachInterrupt(kPinEcho1, Echo1InterruptHandler, CHANGE);
  attachInterrupt(kPinEcho2, Echo2InterruptHandler, CHANGE);
  attachInterrupt(kPinEcho3, Echo3InterruptHandler, CHANGE);
}

void TSonar::TimerInterruptHandler() {
  // The states of the state machine.
  typedef enum {
    COUNTDOWN,   // Count down timer ticks.
    PULSE_HIGH,  // Begin transmitting the SONAR pulse.
    PULSE_LOW  // Stop transmitting the SONAR pulse and begin listening for the
               // echo.
  } TTimerState;

  // The current state of the state machine.
  static volatile TTimerState state = COUNTDOWN;

  // Used to count timer pulses to drive the timing of each SONAR sensor.
  static volatile long countdown = kTimerCountsPerSamplingPeriod;

  if (--countdown == 0) {
    // Time to start the next SONAR sensor in the list of sensors.
    state = PULSE_HIGH;
    countdown = kTimerCountsPerSamplingPeriod;
  }

  switch (state) {
    case COUNTDOWN:
      // Continue counting so that the sensors are spaced
      // apart in time so they don't interfere with each other.
      break;

    case PULSE_HIGH:
      // Time to send out the ping for the next SONAR in the list.
      if ((g_next_sensor_index_ % 4) == 0) {
        digitalWrite(kPinTrigger0, HIGH);
      } else if ((g_next_sensor_index_ % 4) == 1) {
        digitalWrite(kPinTrigger1, HIGH);
      } else if ((g_next_sensor_index_ % 4) == 2) {
        digitalWrite(kPinTrigger2, HIGH);
      } else {
        digitalWrite(kPinTrigger3, HIGH);
      }

      state = PULSE_LOW;
      break;

    case PULSE_LOW:
      // Time to stop the ping output and begin listening for the
      // echo for the next SONAR in the list.
      if ((g_next_sensor_index_ % 4) == 0) {
        digitalWrite(kPinTrigger0, LOW);
      } else if ((g_next_sensor_index_ % 4) == 1) {
        digitalWrite(kPinTrigger1, LOW);
      } else if ((g_next_sensor_index_ % 4) == 2) {
        digitalWrite(kPinTrigger2, LOW);
      } else {
        digitalWrite(kPinTrigger3, LOW);
      }

      g_next_sensor_index_++;  // Control the next sensor in the list the next
                               // time around.
      state = COUNTDOWN;       // Begin counting down again.
      break;
  }
}

TSonar::TSonar() : TModule(TModule::kSonar) {
  for (size_t i = 0; i < kNumberSonars; i++) {
    // Initialize the array of readings used to computer the average distance
    // per sensor.
    for (size_t j = 0; j < kNumberReadingsToAverage; j++) {
      g_values_mm_history_[i][j] = 0;
    }
  }
}

TSonar& TSonar::singleton() {
  // A singleton pattern is used to support the TModule design pattern.
  if (!g_singleton_) {
    g_singleton_ = new TSonar();
  }

  return *g_singleton_;
}

uint8_t TSonar::g_next_sensor_index_ =
    0;  // Which sensor in the list is being controlled now.

TSonar* TSonar::g_singleton_ =
    nullptr;  // Used to ensure that only one instance of this class is ever
              // constructed.

int TSonar::g_values_mm_[TSonar::kNumberSonars] = {
    -1, -1, -1, -1};  // The last reading per sensor.

int TSonar::g_values_mm_history_
    [TSonar::kNumberSonars]
    [TSonar::kNumberReadingsToAverage];  // List of readings per sensur used to
                                         // computer an average per sensor.

float TSonar::g_average_value_m_[TSonar::kNumberSonars] = {
    0, 0, 0, 0};  // The computed average distance per sensor.

const float TSonar::g_time_to_mm_scaler_ =
    (10.0 / 2.0) / 29.1;  // For converting time to distance.

2022 11 02 Puck Status

The big result since the last status report is that more, very important safety systems have been implemented to protect the robot and its surroundings, and the robot now has more data about objects near to it as it moves so it can avoid running into those obstacles. And the new power and data control panel is finished, which makes it easier for me to disassemble the robot.

The last week and a half has seen a lot of work on Puck. Two of the time of flight sensors were replaced as the originals were either not sending distance measurements or the measurements were wildly wrong. While that layer of the physical robot case was opened to replace the sensors, I also color-coded all the cables for the time of flight and SONAR sensors, making it easier to trace the cables from the custom micro processor board (a.k.a the Teensy Monitor Board) to the physical sensors. The new power and data control panel was finished and all the new wiring is in place.

I also rewired the signal cable from the relay block to the emergency stop of the motor controller. This would allow the Teensy Monitor to immediately stop power going to the motors if some emergency situation were discovered.

A lot of work was done in the software of the Teensy Monitor. More diagnostic messages were added, which showed that there are still reliability issues with communication between the Teensy Monitor and the sensors, which is something I’ll have to solve eventually. I may just need to add better cable shielding.

The motor temperature sensors now are published to a topic, and the temperature values are now averaged as the sensors tend to be rather noisy. Code was added to detect an over-temperature situation for the motors. When the robot is driven for a long time, the temperature inside the motors can build up to the point that there is a risk of melting the wires inside.

Importantly, code to detect a runaway motor situation is now implemented. This happens especially when the encoder connections to the motor controller come loose. Remember that the robot vibrates a lot when it moves, and the connectors on the RoboClaw motor controller are not designed to stay well connected in the face of vibration.

The redundant motor current sensors were found to be useless, so the code that talked with those sensors is disabled for now. I’ll need to find some better sensors in the future. Meanwhile the motor current sensors built into the RoboClaw are working well enough. And code was added to detect an over-current situation, which can happen when the robot runs up against, say, a wall and the motor wheels stop turning or turn very slowly. Left in this situation, the wires in the motor would quickly overheat and literally melt.

With the three new fault detection code routines, the robot now get shut off via the emergency-stop signal to the RoboClaw if the motors get too hot, if the motors take off at high speed unexpectedly, or if the motors start drawing a lot of current, beyond normal operation. This is a three way protection to save the expensive motors and prevent the robot from running into something and just keep pushing forward.

With the now fully working SONAR and time of flight sensors, and the data being published at 25 to 30 frames per second, I finally figured out how to hook those sensors into the robot motion planning software. Now, if the robot is moving along and comes across an obstacle that only one of the proximity sensors can see (the LIDAR sees objects only at a specific height), the robot can make a plan to move around the object.

Outstanding work remains to discover why some of the sensor values are occasionally rejected by the planning software. It is likely that is because the data is too old, meaning that the data was more than a couple of milliseconds old by the time the planning software saw it. There may be a simple fix, but it needs to be investigated.

Adding SONAR-like sensors to Nav2

It is probably typical that the Navigation-2 (Nav2) setup under ROS 2 is just using LIDAR for generating costmaps. It is also possible to include SONAR-like sensors in your cost maps, that is sensors that publish a Range type of message. The process is easy.

Edit your YAML parameter configuration file for Nav2 as follows. In the “local_costmap” section, you will find a line that lists the plugins to be used to generate the local costmap. Add a “range_layer” to the list. Following the plugins line, you should see, e.g., sections describing the obstacle_layer and inflation_layer — that is, sections corresponding to the names in the plugins list. You will need to add a new section describing the range_layer.

Note, the names obstacle_layer, inflation_layer, range_layer, etc in the plugins list are arbitrary names, they don’t actually make the layer become, say, and obstacle layer. It’s what you supply as sub-parameters that make the section do what it does.

local_costmap:
  local_costmap:
    ros__parameters:
      plugins: ["obstacle_layer", "inflation_layer", "range_layer"]
      range_layer:
        plugin: "nav2_costmap_2d::RangeSensorLayer"
        topics:
          [
            "/sonar0Sensor",
            "/sonar1Sensor",
            "/sonar2Sensor",
            "/sonar3Sensor", 
            "/tof0Sensor",
            "/tof1Sensor",
            "/tof2Sensor",
            "/tof3Sensor",
            "/tof4Sensor",
            "/tof5Sensor",
            "/tof6Sensor",
            "/tof7Sensor",
          ]
      inflation_layer:
        .......rest of local_costmap YAML.....

Just like the section describing, say, the inflation_layer, you will add a new range_layer section by repeating the name given in the plugins list, followed by a colon. The first entry should describe the class name that will handle the data type. In the case of Range type values, the line should say:

plugin: "nav2_costmap_2d::RangeSensorLayer"

Then add a topics entry consisting of the word topics followed by a colon and then a single topic name in quotes, or a list of topic names formed by a left square bracket, then each quoted topic name in the list separated by commas and a closing right square bracket.

That’s it. If you bring up the nav stack, you should now see that when you wave you hand in front of your SONAR sensor, a new obstacle shows up in the local costmap.

There are a couple of caveats to consider. The published topic containing the Range message needs to have a valid frame_id in the header. And that frame_id needs to be described in the URDF for the robot, which, of course, needs to be published so that the transform system in ROS knows how to relate the Range value to the coordinate frames of the robot.

And, the time stamp in the header needs to be accurate and timely. This especially could be a problem if you are, say, publishing the data from an Arduino micro controller which may not have a notion of the time used by the rest of the ROS components. If you are publishing using microROS, there are ways to synchronize your Arduino with the main ROS system.

There are a few additional parameters besides plugin and topics that you can provide for the range_layer. They are described here.