Doesn’t bode well

I’ve been tinkering with the existing PCB layout (shown below) to see if power supply changes fix the Garmin LiDAR Lite (GLL) I²C interference.

Hermione's closeup

Hermione’s closeup

I’ve basically removed the black cuboid 1.5A 5V switching power regulator using the space instead to add a 680uF electrolytic capacitor between 5V and ground near the GLL as specified in the docs.  I then powered her up directly via the PCB, rather than indirectly via the RPi micro USB 5V output GPIO pin.  I used a lab power supply which can handle up to 5A.

Hermione showed identical I²C errors with the GLL plugged in, even if not used.  The PSU showed only 0.45mA at boot, 0.5mA with Hermione’s code running passively, and about 0.6A while she was ‘flying’.  This strongly suggests the new PCB won’t solve the problem either.  I am running out of fingers to cross 🙁

The option of using the GLL PWM output is still open, but I’m not keen to swap over to using the pigpio library this requires as it uses a daemon process in the background, unlike RPi.GPIO and RPIO which both directly access the /dev/gpio devices.  I wonder how hard it would be to write my own PWM C library with python wrapper to epoll the GLL PWM pin?  It would require a new PCB revision as would pigpio so there’s a PITA there too.  For some reason, probably because this all just works on Zoe, I still want to keep banging my head to solve the I²C problems.

Back on the wagon

I’ve flown my Mavic probably for 20 minutes over the course of 5 short flights simply to get familiar with the controls while dodging the rain showers of the last couple of days.  I’m back inside again trying to track down why Hermione has started throwing her I²C wobbly again.

Motion processing is working well, keeping processing close to the minimum 100Hz regardless of other sensor inputs – here 156 samples were processed in 1.724s.

Processing rate

Processing rate

Garmin’s height is running stably at the intended 20Hz and it’s well withing the accuracy possible for distances less than 1m

Garmin LiDAR v3

Garmin LiDAR v3

Here’s the problem though: the IMU is fine for 862 samples averaged into the 155 motion processing blocks, showing just gravity as Hermione sits on the ground, but suddenly the IMU values spike for no reason for the 156 sample average.  Note that this happens only when the Garmin is plugged in.  There are in fact two spikes: the first is shown, the second causes an I/O exception and the diagnostics are dumped:

IMU stats

IMU stats

I’ve tried power supplies up to 3.4A, both battery and mains powered; I’ve resoldered various critical PCB joins; I’ve added the 680uF capacitor as the Garmin spec suggests despite Zoe being fine without it, and I’ve used a newly flashed SD card, all to no avail.

I have two things left to try:

  • currently the Garmin is read every motion processing loop, despite being updated at 20Hz; the spec says there’s an interrupt, but as yet, I’ve not got it to work.  Must try harder!
  • Failing that, I’ll have to replace the MPU-9250 with another, and see if the current one is faulty.

Beyond these two, I’m out for ideas.

Hermione’s first flights

Hermione’s been out and about using the full X8; her first flight shot her into the air at hyper-speed due to a FIFO-overflow that the code didn’t catch.  With the code fixed and a replacement motor installed, I tried again.  After a couple of reasonable flights proving the X8 PIDs needed no tuning, they then started to fail for a couple of reasons:

  • FIFO overflows
  • I2C I/O errors

Both types were caught and killed the flights, but why were they happening?

  • The FIFO errors I think are due simply because I’m now pushing the limits of my A+’ performance; adding an exception to ensure the FIFO was processed as a priority when the FIFO fills over 50% has resolved that.  However…
  • The I2C I/O errors seem (after lots of testing) to be related to the Garmin LiDAR-Lite.  If it’s plugged in, regardless of whether I’m sampling it or not, I get the I2C I/O errors.  If it’s unplugged, all works perfectly.

I have various things to try out:

  1. It could be due to the extra power-draw when it’s plugged in.  Easy to test as I’ve just bought a huge 4S2P 6600mA LiPo:

    GensAce 4S2P 6600mAh LiPo

    GensAce 4S2P 6600mAh LiPo

  2. It could be confirmation of a rumour I’d heard up that the LiDAR-Lite V2 wasn’t good at sharing the I2C bus with anyone else – I hope the V3 doesn’t suffer from the same.
  3. It could be due to noise on the bus caused by the 20cm long cables.  I picked up some stuff from my PX4FLOW investigations about wiring requirements for long cables.
  4. Mentioned in the PX4FLOW docs, and also in the Garmin LiDAR Lite V3 instructions is the need for a decoupling capacitor between Vcc and ground.

If / when I can get the Garmin working together with the IMU, then it’s all systems go for sustained zero drift flights.

Say ‘Hi!’ to Hermione*

Hermione will be the mother of all that’s gone; she’s a bit heartless at the moment – her HoG will be an RPi A3 when it appears on the market.

Hermione flat-out (with banana for scale)

Hermione flat-out (with banana for scale)

Hermione flat-out (with banana for scale)

Hermione tip-toes (with banana for scale)

The main reason behind building yet another quadcopter is the frame:

  • loads more space for extra sensors
  • folding arms so it’s easily transportable
  • beautiful build – lots of CF and CNC machining
  • lots of extension platforms adding lots of options of where to put all the pieces – installed here are a power distribution board (in the plate sandwich), a plate for her HoG, and a GPS plate sticking out to the left.

The one thing that’s missing are legs – I’ve had the ones supplied before, and they crack and break with the down-falls my testing produces.  There are stronger ones I’ll be getting.  I’ve bought from quadframe.com previously for Chloe so I know the high quality of their frames; the previous frame I’d bought was premature, and sadly I ended up selling it.

But this time, I have a purpose for all the extra space: Hermione will be deployed with GPS, LiDAR x 2, PX4FLOW, a RPi Camera and anything else I can get onto her.  Also the frame supports an X8 layout as well as standard quad format; X8 is where each arm has 2 motors and props: the top motor prop is set up as normal; the lower one is upside down, and the prop is installed on the motor upside down.  The topside and underside props spin in opposite directions.  This format provides more power for heavy lifts and protection against motor / prop damage or failure.  Each prop has its own ESC and separate PWM feed.

Doing the above requires the A3 for its extra cores to do a lot more sensor processing and a new PCB to support the new GPIO pin requirements.

GPS, LiDAR and LEDDAR require UART connections; X8 requires another 4 GPIO pins for the 4 ESCs PWM feed.  One of the sensors – probably GPS – can use the USB port, assuming the A3 has built-in WiFi like the B3.  PXFLOW uses I2C.

My thinking is that LEDDAR and PXFLOW provide term fused inputs to the distance PIDs; GPS provides targets for the flight plan; compass is the yaw input with the the flight plan providing direction of travel so that a camera pointing forwards can always track her progress; Scance Sweep provides object detection overriding the GPS flight path short term to avoid objects.

Together, this means I could set a start and end position for a flight using GPS (either just start and finish or with intermediate check points), and then just set her loose autonomously to track against those points, whether this is simply flying from A to B or getting from the entrance to the centre of a maze, logging “where I’ve been” to ensure she always prioritises new paths through the maze.

Here’s the current PCB eagle layout for Phoebe and Chloe:

A+ PCB

A+ PCB

  • the right hand side 3 vertical PCB tracks are for the LiPo to 5V regulator – I’m planning on Hermione having a BEC on her PDB (power distribution board), so that opens up space for the rear X8 PWM pins.
  • the left hand side 4 vertical PCB track are I2C extensions in place for the URF, but the space is needed for the the front X8 PWM pins so the I2C extension is moved to the bottom edge and will be used for the PIX4FLOW.

Here’s the first draft of the revised PCB:

Hermione PCB beta

Hermione PCB beta

PIN usage:

  • pins 4, 6, 8, 10 and 12 are used for LEDDAR as now but extended outwards
  • the GPIO pins for PWM are completely reworked for X8 format as is the MPU9250 interrupt pin; I’ll probably end up adding X8 before I add Scanse Sweep simply to fill the gap between now and its delivery
  • I2C is extended on the lowerside for the PX4FLOW
  • That leaves a selection of pins and space on the topside for Scanse Sweep – I need to get wiring specs for this; and alternative is via USB using a UART to USB converter – I have one of these already for use configurating LEDDAR by my Windows PC.
  • GPS will also be via USB, assuming the WiFi is now build into the A3 board or I might use a mini USB hub like on of these that I used already for other projects.

Regarding WiFi, to extend the range, I’m going to replace the (hopefully) on board antenna with a HiRose U.FL connector, as per here.  That will then allow me to connect to a U.FL to RP-SMA cable, and place a higher gain antenna on the board.

I’ve still  left a few breadboard pins at the base as they may turn out to be useful for LEDs, buzzers etc.

As you can see, there’s a lot that needs to be done, but it breaks down into lots of separate blocks to keep me busy until the A3 and Scanse Sweep arrive.  The first step is simple: move Chloe’s HoG over to Hermione’s frame, and get her stable.


*Penelope (as in “Pitstop”) was a close runner up, but with my existing HoGWARTs WAP installation notes, Hermione (as in “Granger”) won the ballot outright.

Snow White

Just sharing a couple of photos.

The first is my new prototype Pi with a discrete level shifter used to drive the URF I2C at 5V, while using 3.3V over the RPi I2C interface.  The type of MOSFET is not critical as long as its Vgs is much less than 3.3V – many use the BSS138 or 2N7000 – I just used a couple from my stash which fit breadboards better.  The next step is yet another PCB for Phoebe*:

Discrete I2C level shifter

Discrete I2C level shifter

The second is Zoe who I’m in the process of upgrading to the latest Raspian Jessie Lite – this is a start-from-scratch ‘update’ so I’ll blog the step next.

Zoe litening

Zoe litening

Zoe’s upgrade has so far gone pretty smoothly except for one minor glitch with updating to the new networking interface model and not having a GUI.  The one thing that’s made it a lot easier is that I’ve not had to dismantle her at all.  Also, in the past I’ve used a 10-port powered USB hub for WiFi, keyboard and mouse, but here I have a 2-port USB hub attached to Zoe’s OTG USB connector.  In all, that’s made the upgrade process much less painful than I was expecting.


*Nope, this URF won’t work with Phoebe – even with the level shifter, when I updated the I2C baudrate to 400kbps, the simple test program blocked, just like Phoebe’s own URF does on 3.3V. I also ran Zoe at 100kbps by mistake during her litening transformation, and that proved 100kbps is not fast enough to empty the IMU FIFO as fast as it is being filled.

 

Progress report

Although the blog’s been quiet, I am still tinkering with both Zoe and Phoebe.

Zoe’s drift is better with 3 changes:

  • I’ve changed my 0g offset calibration so that it’s now done standing on her feet, and an average of 4 readings are taken with her rotated 90° each time and an average used
  • I’ve changed the double sided foam sticking her HoG to the frame – it’s all a bit of a balancing act between having her held down firmly, and yet also providing buffering against vertical prop vibrations
  • I’ve change the code which controls how many samples are per motion processing – there was a minor bug in the timing I knew of, but until now had not thought it worth the effort fixing it.

Here’s the result.

I don’t think it’s going to be possible to get things much better – my guess is the 0g offsets are the dominant contributor to the reduced drift, and they vary. Here’s the output from three runs – the first with her resting on her props, and the other two on her feet; the entries with the Z axis set to 0 are the averages of the preceding 4 calibration test entries. 8000 is roughly 1g to give you an idea of the size of these offsets.

0g calibration results

0g calibration results

The progress with Phoebe is less positive; with the SRF02 rangefinder installed, the code blocks during initializion; i2cdetect can see the sensor so I presume it’s because the sensor is only rated to 100kbps I2C baudrate.  There is a similar URF from a different manufacturer which does support 400kbps, but only if the I2C master (the Raspberry Pi) supports clock stretching; sadly the RPi I2C driver does not, although that may be fixed soon apparently.  However, until that happens, height control using the URF is blocked.

Without the sensor, she’s flying, but occasionally seems to decide to drift; it’s not a steady drift, it suddenly kicks in as though she gets a dodgy reading from the sensors.  I then have to kill the flight before she gains too much height.  It’s not clear to me yet what, if anything I can do about this.

The benefits of having a Brazilian…

comment on the blog: courtesy of Gustavo yesterday, I’m now able to empty the FIFO a batch at a time with a single I2C read (12 bytes) rather than 12 reads of 1 byte.  That’s made my code a lot faster, buying even more time to check other sensors etc.

It’s all to do with i2c.readList (i2c.smbus.read_i2c_block_data): I’d used readList when I was accessing the data registers direcly; reading 14 bytes from register 59 actually gave me a single byte each from registers 59 to 72 – the full set of sensor data I needed.  As a result, I’d ruled out using readList at register 116 (FIFO) getting me 12 bytes per read from just that single register; but that’s what Gustavo does and sure enough it worked for me too.

Sadly a quick flight this morning still showed the same negative G problems – have a look at the az values – these should read roughly 16384 at hover; these aren’t all the samples, just the ones showing negative G during the 8s flight:

[WARNING] (MainThread) __init__ 1397, Zoe is flying.
[WARNING] (MainThread) __init__ 312, SRD:, 1
[WARNING] (MainThread) __init__ 411, IMU core temp: 19.888756
[WARNING] (MainThread) fly 1522, fly = False, flight plan = , calibrate_0g = 0, hover_target = 150, shoot_video = False, vvp_gain = 400.000000, vvi_gain = 200.000000, vvd_gain= 0.000000, hvp_gain = 1.500000, hvi_gain = 0.100000, hvd_gain = 0.000000, prp_gain = 110.000000, pri_gain = 11.000000, prd_gain = 0.000000, rrp_gain = 90.000000, rri_gain = 9.000000, rrd_gain = 0.000000, yrp_gain = 80.000000, yri_gain = 8.000000, yrd_gain = 0.000000, test_case = 1, rtf_period = 1.500000, tau = 5.000000, diagnostics = False
[WARNING] (MainThread) load0gCalibration 551, 0g Offsets:, 0.000000, 0.000000, 0.000000
[WARNING] (MainThread) fly 1522, fly = True, flight plan = fp.csv, calibrate_0g = 0, hover_target = 380, shoot_video = False, vvp_gain = 400.000000, vvi_gain = 200.000000, vvd_gain= 0.000000, hvp_gain = 1.500000, hvi_gain = 0.100000, hvd_gain = 0.000000, prp_gain = 110.000000, pri_gain = 11.000000, prd_gain = 0.000000, rrp_gain = 90.000000, rri_gain = 9.000000, rrd_gain = 0.000000, yrp_gain = 80.000000, yri_gain = 8.000000, yrd_gain = 0.000000, test_case = 0, rtf_period = 1.500000, tau = 5.000000, diagnostics = False
[WARNING] (MainThread) load0gCalibration 551, 0g Offsets:, 0.000000, 0.000000, 0.000000
[WARNING] (MainThread) fly 1661, pitch -1.718589, roll -2.401715
[WARNING] (MainThread) fly 1662, egx 0.000000, egy 0.000000, egz 0.976289
[WARNING] (MainThread) fly 1739, 0 data errors; 0 i2c errors; 0 2g hits
[CRITICAL] (MainThread) readFIFO 469, ax: -20304; ay: 8588; az: -556; gx: 1501; gy: 1042; gz: 1181!
[CRITICAL] (MainThread) readFIFO 469, ax: 6156; ay: -784; az: -552; gx: -379; gy: 454; gz: -75!
[CRITICAL] (MainThread) readFIFO 469, ax: 1580; ay: -2772; az: -564; gx: -2126; gy: 304; gz: -339!
[CRITICAL] (MainThread) readFIFO 469, ax: -8784; ay: -1812; az: -608; gx: -2906; gy: 239; gz: -864!
[CRITICAL] (MainThread) readFIFO 469, ax: 5016; ay: 1420; az: -636; gx: -1649; gy: 2227; gz: 1277!
[CRITICAL] (MainThread) readFIFO 469, ax: 9652; ay: -1816; az: -732; gx: -1665; gy: 1730; gz: 751!
[CRITICAL] (MainThread) readFIFO 469, ax: -84; ay: 2328; az: -2392; gx: -2358; gy: 1965; gz: 926!
[CRITICAL] (MainThread) readFIFO 469, ax: -5884; ay: -1184; az: -108; gx: -75; gy: 1438; gz: 351!
[CRITICAL] (MainThread) readFIFO 469, ax: -2180; ay: 7628; az: -2316; gx: 493; gy: 2743; gz: 384!
[CRITICAL] (MainThread) readFIFO 469, ax: -152; ay: 5336; az: -2208; gx: 407; gy: -860; gz: -703!
[CRITICAL] (MainThread) readFIFO 469, ax: 21212; ay: -2948; az: -392; gx: -175; gy: -1606; gz: -538!
[CRITICAL] (MainThread) readFIFO 469, ax: -7704; ay: -5088; az: -1184; gx: -899; gy: 2470; gz: 10!
[CRITICAL] (MainThread) readFIFO 469, ax: 4904; ay: -2688; az: -676; gx: -1409; gy: 2067; gz: 649!
[CRITICAL] (MainThread) readFIFO 469, ax: 11732; ay: 3232; az: -976; gx: -1023; gy: 1170; gz: 1019!
[CRITICAL] (MainThread) readFIFO 469, ax: 4920; ay: 2740; az: -3212; gx: -552; gy: 958; gz: 667!
[CRITICAL] (MainThread) readFIFO 469, ax: -10364; ay: 5020; az: -2744; gx: -553; gy: 839; gz: 310!
[CRITICAL] (MainThread) readFIFO 469, ax: 4104; ay: -1516; az: -2756; gx: -693; gy: 46; gz: 654!
[CRITICAL] (MainThread) readFIFO 469, ax: -10048; ay: 5772; az: -20; gx: -686; gy: 283; gz: 1207!
[CRITICAL] (MainThread) readFIFO 469, ax: -6844; ay: 5884; az: -1868; gx: -424; gy: 974; gz: 703!
[CRITICAL] (MainThread) readFIFO 469, ax: 6428; ay: 1472; az: -368; gx: -394; gy: 496; gz: 582!
[CRITICAL] (MainThread) readFIFO 469, ax: -3064; ay: 9312; az: -1852; gx: -575; gy: 2588; gz: 801!
[CRITICAL] (MainThread) readFIFO 469, ax: -840; ay: 6392; az: -1944; gx: -366; gy: 1737; gz: 149!
[CRITICAL] (MainThread) readFIFO 469, ax: -2468; ay: 10792; az: -1580; gx: -737; gy: -727; gz: -65!
[CRITICAL] (MainThread) readFIFO 469, ax: -4448; ay: 2652; az: -3624; gx: -974; gy: 639; gz: -111!
[CRITICAL] (MainThread) readFIFO 469, ax: 4604; ay: 7728; az: -4000; gx: -560; gy: -61; gz: -529!
[CRITICAL] (MainThread) readFIFO 469, ax: -23048; ay: 9136; az: -4992; gx: 1366; gy: -524; gz: -5!
[CRITICAL] (MainThread) readFIFO 469, ax: -3860; ay: -1384; az: -488; gx: 1241; gy: -1463; gz: 293!
[CRITICAL] (MainThread) readFIFO 469, ax: -12488; ay: 6820; az: -516; gx: 2382; gy: -3340; gz: -664!
[CRITICAL] (MainThread) readFIFO 469, ax: -2024; ay: -7380; az: -40; gx: -1119; gy: -831; gz: 76!
[CRITICAL] (MainThread) readFIFO 469, ax: -3676; ay: 1472; az: -1756; gx: -687; gy: 6099; gz: 2686!
[WARNING] (MainThread) fly 2068, IMU core temp: 18.121548
[WARNING] (MainThread) fly 2069, motion_loops 934
[WARNING] (MainThread) fly 2070, sampling_loops 9522
[WARNING] (MainThread) fly 2072, 30 data errors; 0 i2c errors; 0 2g hits

That’s 30 errors detected out of 9522 samples, so not terrible, but not perfect.  I was flying her at her new high speed sampling rate of 1kHz and with the accelerometer low pass filter set to 0 (460Hz).  Next step is to turn off the protective code that skips these negative G values and have another go with alpf 2 (92Hz) to see what happens. Certainly the other errors of -32768 in the gyro have now gone, so it’s worth a try.

HoG & CoG

What have I been up to?

  1. Phoebe’s HoG PCB is now out for manufacturing – hopefully this will fix the I2C problems and allow me to progress with adding the magnetometer (orientation), URF (height) and camera (motion) sensors into the mix.  The new PCBs are due to arrive on or before 29th.
  2. Zoe’s undergone a minor rebuild of her frame to lift her CoG nearer to prop height to reduce the swinging in an otherwise stable flight – her main battery now resides on the top plate, and her power bank underneath.  I could improve this further by getting rid of the power bank completely, but that will require a new PCB to incorporate the regulator.

Other than that, I’m flying Zoe indoors daily tuning her PIDs ready for her upcoming performances.  She still drifts, and because she’s now using the IMU FIFO code, I’m back to considering sensor drift due to temperature.  She performs best in the freezing cold with the large difference between cold ambient and her IMU operating temperature.  However I won’t be investigating this further within the timeframe of the performances.

Yet another “One last chance”.

I simply couldn’t give up on Phoebe so easily: after posting, I spotted all the components needed to build a clean Zoe PCB which I did.  I attached it to Phoebe, and everything worked.  So the I2C problems are the fault of the breadboard-like PCB Phoebe uses.  So once more, Phoebe isn’t dead yet, and the new PCB layout is already in progress for the lovely peeps at Ragworm to produce in the next week or so.

Fed up of Phoebe

I’m giving up on Phoebe and adding new sensors using the IMU FIFO code.  The final nail in the coffin was I spotted I2C errors while I was trying to diagnose the squid problems.  I2C errors rule out using the IMU FIFO as the code then gets out of sync with what data it’s getting from the FIFO.  That means Phoebe has to switch back to using the data ready interrupt code, which in turn means she can’t periodically do something else like read other sensors because she can’t afford to miss the next interrupt and hence the next batch of data.

I’ve never been able to find the source of these I2C errors but I guess it’s the PCB, and I’m not going to get a revised version designed any time soon based on speculation alone.

Zoe on the other hand is still working perfectly – no I2C errors with identical code perhaps because she’s the daughter of Johnny Ball, a science presenter from my era, who also has cartoon appearances on the Raspberry Pi site.  But more likely is that she’s a Zero not an A+ and running a different PCB design as result (both mine and A+/Zero).  Unfortunately Zoe would need a new PCB to add the URF for vertical motion tracking, and she could never have a camera for the horizontal motion tracking – PiZero’s don’t have the camera socket.  So Zoe’s is essentially finished other than fine tuning.

So this project is grinding to a terminal halt for the moment until either an A2 appears on the market allowing me to split the DRI code from the motion processing and connect them with an OS FIFO, or I get a new PCB done for an A+ where there’s no guarantee this will fix anything.