I2C problems – progress report

I’ve been tinkering this morning with the hardware interrupt handling to try to track down how the I2C was being kicked to read data more than 500 times a second, when I’d reduced the sampling rate down to 500Hz.

So far, I’ve:

  • updated my custom GPIO code for yet higher performance on the hardware interrupt from the MPU-9250 by adding the EPOLLONESHOT which means epoll() only listens for the next rising edge event, and then disables the fd until it’s MODed – I’m assuming this should be faster that ADDing and DELeting each time and ensure we catch the next rising edge rather than any historic ones.
  • change the hardware interrupt to open-gate and added a pull-up on the GPIO input pin used to detect rising edge of the interrupt pulse
  • disabled the device tree and dropped back to the previous kernel device management
  • added the I2C baudrate=400000 back to /etc/modprobe.d/i2c_bcm2708.conf
  • added the I2C combined=1 to /etc/modprobe.d/i2c_bcm2708.conf but that caused garbage results and ultimately a kernel crash so I’ve backed that out.

I’m still getting I2C read errors, but at least the interrupt is now working in line with the baudrate: a reduced baudrate results in a reduced loop speed in the code as it’s taking longer to deliver data from the MPU-9250.

However, what I’d expect is that the interrupt would be in line with the sample rate, yet at 333Hz sample rate set for the MPU-9250, I’ve just got 799 sampling loops per second.  There’s something very odd about the interaction of the I2C baudrate, the data sample rate, and the data ready interrupt in the MPU-9250.

More digging anon.  Might be time to deploy the oscilloscope.

Diagnosing I2C problems (work in progress)

The symptoms of the I2C problems I’m seeing are python exceptions from the smbus library; when caught, as well as incrementing the ‘misses’ count, I have code that re-reads the sensors.  The trouble is, the I2C exceptions I’m catching are actually symptomatic of a bigger problem: duff data from reads which don’t trigger exceptions.  Just one set of duff data results in long lasting errors in angles, acceleration, gravity measurement etc etc etc.  Put simply, you can forget any chance of stability in a flight.

I’d solved the problem with Phoebe using the hardware interrupt to trigger the I2C read.  And it worked like a dream, and Chloë and Zoë inherited that solution.

But the problem came back with the birth of HoG (using the latest Raspbian distribution) and the swap to the MPU-9250.  So I’ve been trying things today to work out what’s gone wrong.

First step was to solder together a couple of pads on the MPU-9250 breakout to connect the pull-up resistor – this should be unnecessary as the Raspberry Pi I2C bus already has a pull-up and adding another could be detrimental.  Anyway, there was no change in behaviour – still I2C missing.

Next step was to track down whether it’s the sensor, the code, the A+ or the kernel causing the problem.  Zoë still lives in a half-alive state so I could swap SD cards around as see what happened:

  • HoG’s SD card @ kernel 3.18.7+ sees I2C read misses with MPU-9250 (HoG’s hardware)
  • Zoë’s SD card @ kernel 3.12.28+ sees I2C read misses with MPU-9250 (HoG’s hardware)
  • HoG’s @ kernel 3.18.7+ sees zero I2C read misses with MPU-6050 (Zoë’s hardware)
  • Zoë’s SD card @ kernel 3.12.28+ sees zero I2C read misses with MPU-6050 (Zoë’s hardware)

That rules out the kernel version, the A+ hardware, and my software.  It suggests a problem with the MPU-9250 breakout board, the PCB it’s connected to, the wiring or a long term I2C driver problem that only shows it’s ugly face with the MPU-9250.

My best guess is the I2C barometer on the same breakout; it uses the same bus, but as yet, I’ve not configured it in any way.  Perhaps as a result, it’s being noisy as a result, annoying all the other passengers on the bus as a result?


Update: on a whim, I reduced the data rate from 1kHz to 500Hz to allow more time for the sensor data to be read.  What I saw was the data rate go up from just under 700Hz to over 700Hz.  That suggests the data ready interrupt isn’t working; data is only being made ready at 500Hz, so where are those other pulses coming from?

I had a looking at my custom GPIO code, and a slightly contraversial change I made; I backed this out and tried again.  This time, no edge were detected at all!

Something in that area is very dodgy.  More tomorrow, no doubt.

News update

Sorry it’s been quiet here; thought I’d better update you what’s going on.  I’ve been doing test flights as the weather allows trying to track down a problem.  The symptoms have been inconsistency between flights, where a few are fantastic, but most end with a prop digging into the ground, and an alloy arm bent at the wrist.

For quite a long time since adding the hardware interrupt to announce to the code that new data was ready from the sensors, I’ve always received data I could trust from the MPU-6050.  With the build of HoG and the move to the MPU-9250, that’s no longer the case – I’m getting i2c read exceptions, yet the code in that area is untouched.

This might be related to the MPU-9250 registers, or it might be related to the latest distribution of Raspbian I installed on HoG.  Not clear yet.

I’ve also started looking at ‘Kitty’ – code using RaspiCam and picamera to identify the position of a laser pointer dot on the ground, so that the direction / distance of that dot could be turned into flight plan targets so that HoG would follow it.  Not a difficult piece of fun to add to HoG but while she’s not behaving, testing is restricted.

So it’s going to continue to be quiet here for a while until I’ve got I2C / GPIO hardware interupts back to the standard they were.

TTFN.

BYOAQ-BAT VI: Performance

If you haven’t already, please read the previous BYOAQ-BAT articles first – just search for the BYOAQ-BAT tag.


This is another short article about getting the code to run as fast as possible. I’ve taken numerous steps to ensure the code itself is fast including:

  • i2c 1 x 14 byte read rather than 14 x 1 byte reads
  • time.time() called once per cycle at the point the next sample is read – there’s an irony that calling time.time() takes a lot of time!
  • calculations of values used multiple times is done once – see the various matrices for an example
  • apply calibration / scaling once per motion not once per sample
  • separating data collection and motion processing into separate threads (although to be honest this is buggered by the GIL)
  • data logging to RAM disk and copied to SD card on flight completion

but without the best from the OS, they are pointless.

We’ve already overclocked the CPU and given as much memory as possible to the CPU.  Two last steps.

Get GPIO.tgz from GitHub is you don’t have it already.  Do the following:

tar xvf GPIO.tgz
sudo apt-install remove python-rpi.gpio
cd GPIO
sudo python setup.py install

That installs the GPIO replacement which is fast enough to catch the 50μS hardware interrupts from the sensors to say there is new data ready.

Then to read the data as fast as possible, you’ll need to edit /boot/config.txt adding:

dtparam=i2c_arm_baudrate=400000

Together I’m able to get about 700 loops per second each reading 14 bytes of data (a data rate of 78.4kbps) and process them.  The sensors is updating this data at 1kHz or 112kbps which is why the baudrate needs to be increased from its default of 100kbps.

The need for speed!

My code has been running at about 450 loops per second, and it seemed that whatever I tried to speed it up was having little effect.  The data from the MPU6050 was being updated 1000 times per seconds, so surely I could do better than 450?

Eventually, I started to suspect my customized GPIO python library was the cause – it waits for a hardware interrupt signalling fresh data is available – it calls epoll_wait() twice – could this explain why?  Is it only catching every other interrupt and hence reducing the speed to a maximum of 500 loops per second? It seemed plausible so I changed the code, and sure enough, processing speed has gone up to 760 loops per second.  The missing 240 loops are due to python motion processing, so now I can fine tune these and expect to get even better results.

Why does this matter?  By nearly doubling the amount of data received in a fixed period, I can get better averaging over that period, which means I can increase dlpf to a higher frequency, and so reduce the risk of filtering out good data.

I’ve updated the code on GitHub – you’ll need to remove the current GPIO code first before unpacking the GPIO.tgz and running the install thus.

sudo apt-get remove python-rpi.gpio
tar xvf GPIO.tgz
cd GPIO
sudo python setup.py install

Next step was to see what refinements I can make to the python code to speed the sensor data processing further. I moved the calibration and units checking from the main loop to the motion processing, and that upped the speed to 812 loops per seconds.

Now all I need to do is test it live!

GitHub update

I’ve just pushed a couple of things up to the GitHub repository:

  • the LibreOffice presentation I gave at yesterday’s CamJam
  • the RPi.GPIO python library with enhanced hardware interrupt performance.

To install the RPi.GPIO changes type

cd ~
tar xvf GPIO.tgz
cd GPIO
sudo python setup.py INSTALL

Then to use the improved performance, instead of calling GPIO.wait_for_edge(), call

  • GPIO.edge_detect_init(pin, edge) once at startup
  • GPIO.edge_detect_wait(pin) whenever you want to wait for a GPIO pin event
  • GPIO.edge_detect_term(pin)

GPIO.wait_for_edge() still exists as a wrapper for these 3 functions.  pin and edge are identical parameters as for the standard GPIO.wait_for_edge() call.

Propeller size and GPIO.wait_for_edge()

So I tried both of these today.

Dropping back to 10 x 3.3 T-motor propellers had no obvious effect so I’ll go back to the 11 x 3.7’s I’ve been using recently.

GPIO.wait_for_edge() is looking promising. In the official code, the Python call combines initialization, waiting, and cleanup into a single call to catch the hardware data-ready interrupt on the GPIO pin. This was 50% slower than my Python hard-loop reading the data ready register instead.

But by splitting this into three, edge_detect_init(), edge_detect_wait(), and edge_detect_term() with a wrapper to maintain back-compatible support for wait_for_edge(), my code now only calls edge_detect_wait() in the main processing loop – the other two are only called at start-up / shutdown. This means various bits of epoll() fd setup now only happens once rather than each processing loop.

The overall processing speed has only increased by perhaps a percent or two (expected), but critically, this represents getting marginally prompter indications of a new batch of sensor data availability. And hopefully therefore, reduce the risk significantly of dodgy sensor reads which mess up the velocity integration.

In my rewrite of the GPIO event handling code, I have broken some of the other event detection function I don’t use, but once I’ve reinstated that, I’ll see if Croston will accept these changes into the mainline GPIO Python library.

Hurry up, Phoebe

Flights are on hold at the moment until my daughter’s chicken pox clears and she’s allowed back to nursery, so I decided to have a tinker with the code’s performance.

Phoebe’s main processing loop had degraded since the last time I let her run at full speed from about 180 loops / sec (5.55ms per loop) to about 150 (6.66ms / loop).  At the same time, the sensor read was taking 60% of that 6.66ms or about 4ms per sensor read / processing.  The sensor processing is not heavy so I had to assume i2c was holding things up.

The default i2c clock frequency / data rate on the RPi is 100kbps, but the MPU6050 can operate up to 400kbps.  To do so, I created a new file i2c_bcm2708.conf in /etc/modprobe.d/ and added the line:

options i2c_bcm2708 baudrate=400000

At the same time, I increased the MPU6050 sample rate divider from 3 (250Hz data register update rate) to 1 (500Hz data register update rate).

Prior to the change, as I said, a sensor read took 4ms.  Afterwards, the code looped at 211.55 loops per second (comared to 150 previously), and only 43% of that time was spent on sensor reading leads to a 2ms sensor read – twice as fast.

Having been poking around alot in the sensor read code, I also spotted another area that could be improved: although there’s a data ready interrupt, I don’t use it; instead I loop polling the data ready register, only reading data once it transitions from low to high.  I wondered whether I could use the interrupt to save the hard-loop polling (and hence CPU) and at the same time fractionally increase the speed?

You may remember I use the RPIO library rather than the GPIO library as it offers hardware PWM to drive the ESCs.  RPIO claims to be 100% GPIO compatible, but that turns out not to be the case anymore.  The function I needed was a blocking call to waiting until there is a rising edge on the MPU6050 interrupt pin.  The GPIO API provides GPIO.wait_for_edge() which matches my needs.

So that meant using RPIO for PWM and GPIO for GPIO.  Luckily a little bit of hackery made that easy:

import RPIO.PWM as PWM
import RPi.GPIO as RPIO

No further changes to the code were needed to make the GPIO code replace the RPIO code.  A fiddly bit of config change to modify when MPU6050 interrupts are generated and cleared, and it all just worked.  Sadly, the loop speed dropped to 90Hz from the 211 previously seen which was dissapointing.  I’ve reverted to the previous high speed polling of the register as a result.  Still it was an interesting trial with interesting results.  I’ve left the interrupt code in place if I fancy another play later to track down the cause of the reduced performace – the obvious contender is the GPIO.wait_for_edge performance compared to the high speed polling.

P.S. All these tests were done with the ESCs not powered; it remains to be seen what effect this super-speedy sampling rate renders.  If nothing else, it means I can insert a lot more code (such as an RC) and retain the performance I’d had prior to these changes.

P.P.S. Just realized I’m using the complementary filter the wrong way.  It’s used for merging angles from the integrated gyro and the accelerometer passed through Euler angle conversion.  The integrated gyro is accuarate short term but drifts long term; the accelerometer is better long term but is noisy in the sort term.

Early in the project I’d heard that the gyro drifts, which is why it can’t be used long term.  I took that as read.  Now I believe that’s not correct otherwise the complementary filter still wouldn’t work, and it’s the integrated gyro that drifts (which makes complete sense to me).  That means that this performance improvement can provide more frequent gyro data, integrating more accurately, meaning it can be trusted for longer.  Hence tau can be increased to use the integrated gyro for longer whereas I’d been reducing it to include the accelerometer earlier as the noise reduced).  That also probably means I can probably increase the dlpf to let more good stuff through.

Get the bath tub ready…

I can feel a Eureka moment coming soon!

I can see at least two more problems to be fixed, but I can also see that unless my drone breaks the laws of physics, the number of problems really has to be nearing 0.

So the latest problem to be solved is how to avoid the junk / garbage data read from the MPU6050 sensors shown in the previous post.  That’s an answer for another day, as I need to understand and fix it first.  The problem is due to bad I2C interface reads, meaning the sensors appear to return values of -1, leading to the spikes you can see on the previous post.

Today’s fixed problem (I think) is how to ensure you only read data from the sensors when you can get your grubby mits on it safely?

  • the MPU6050 has two sets of sensor registers – ones that it fills in, and ones that are read by software
  • The software registers are updated periodically from the internal registers based upon a timer, and when that happens, the data ready interrupt gets triggered
  • The sample rate for the readable register updates depends on both the DLPF (1kHz if enabled, 8kHz if not used), and the configured sample rate divisor allowing user configuration of how often the sensor samples are updated.

So to read a full set of sensors, they need to be read in less time than the next sample gets pushed to the register.  This means I need to change my readSensorRaw code as follows

  • read the interrupt status register to clear any outstanding interrupts
  • spin waiting for a fresh interrupt
  • start a timer
  • read the sensors
  • stop the timer

And finally reconfigure the sample rate divider (in code) so that the elapsed time measured above is less than the sampling period.

In the quick tests I did, the reading of the sensors took ~6ms, so with the DLPF on, the internal sample rate is 1kHz, so I’ve set the sample divider to 10 so that the external registers change every 10ms which should be more than enough time to read the sensors.