Diagnoses and Cures

Hermione’s “reach for the stars” was due to I²C errors; I suspected powewr brown-outs.  Her regulator for the LiPo provided only 1.5A, so I tried her passively with mains PSU of 5V at 1A and 2.5A – the error was the same – shifted outputs from the IMU FIFO without any FIFO overflow.  That suggested a interaction with the I²C with the Garmin instead.  I rebuilt the cable with two UTPs (unshielded twisted pairs): SCL with Vss / GND and SDA with Vdd / 5V as per the PX4FLOW spec for long I²C wiring.  I was stunned – it just worked, regardless of whether the 1A or 2.5A power supply was used, I no longer got any I²C corruption.  Next step clearly is to test her live outdoors and check she no longer reaches for the stars.

I also had the bottle to let Zoe loose in the play room.  She still hardly got off the ground on the first flight, so it’s not temperature drift.  However, her second run was perfect, which reminded me that her first run was always cr@p for some reason.  Here’s the stats for both.  The stats are logging both accelerometer and Garmin / Camera distances.  There’s such a tight correlation between the very difference sensors that I’m very tempted to turn the fusion on.  Just a tad more bottle needed.  The key one for each is the bottom left: how high was she according to the two sensor sources.

Flight one

Flight one

Flight 2

Flight 2

I thinkthat’s my courage bottle empty for the day.  When it’s charged up tomorrow, I’ll take the sisters outside to test the above next steps.

CF props and near zero G

So I’ve finally worked out why Zoe doesn’t work with her carbon props.  2 factors:

  1. I’ve been using a hard take-off surface outdoors rather than the wet, soggy lawn.  This means that while the blades are winding up to hover speed, Zoe’s legs are imperceptibly bouncing up and down on the hard surface.  An upwards force from the blades is smooth, but as a leg hits the hard surface, the hard deceleration is picked up by the accelerometer as near zero or negative G.
  2. I’d still got the near zero G protective code in place, not recognising the near 0 G does happen when impacting with hard surfaces .

Hence the vertical velocity integration was ommitting the low or negative G forces, meaning the vertical velocity was showing much higher rates than expected, so the code wound down the prop speeds to slow down the perceived ascent.  Hence she never got off the ground.

Moving her to the saturated lawn and removing the protective code filtering out near 0G readings meant she finally took off.  She was very unstable but she took off.

So more physical buffering is required – I’ll be attaching sponge balls onto her feet to soften the hard impacts.  At the same time I have some softer (I think) foam pads coming too for attaching Zoe’s HoG to the frame.  Finally, I need to get her off the ground as quickly as possible rather than faffing around bringing her props up to speed in a nice controlled manner.  Oh, and I need to get her off the ground ASAP.

Buffing, the noise slayer!

OK, pushing my luck this time with the title of the post, but I spent too much time either side of the millenium watching Buffy The Vampire Slayer, and I have the full DVD set, so I wanted to involve Sarah Michelle Geller in here somehow.

Anyway, I mentioned previously I’ve balanced my props, but I’ve done it again since to much higher tolerances with a Dremel coarse buffing bit (geddit?) and a new maglev balancer – the old one had minor damage and the results were variable.

Anyhow, with that I did see a couple of internal flights with nigh on zero drift with the accelerometer low pass filter set to 2 (92Hz).  But I also still saw occasional drift in the front-left direction leading to a minor scratch on a table leg, and a damaged blade when it hit the metal log ‘basket’ in front of the fire.  Having ruled out the props, it’s time to check the motors; if there is still real noise, then the motors are at fault.  Roll the sequence…

Motor and Prop noise

Motor and Prop noise

This is a test case 1 flight (sudo python ./qc.py -h 220 –tc 1): spin each blade separately, and log the accelerometer readings for each; a sequence of 5s each for front left, front right, back left and finally back right blades..

Based upon the orange and blue X and Y accelerometer readings, back right’s the noisiest, followed by front left, back left and finally front right which is interesting / odd as it’s front right with the damaged prop.

I don’t think there’s much to deduce from this single run other than the new diagnostics I’ve added to test case one are worthwhile.

Diagnostics from indoors flights

I collected these diagnostics from a couple of indoor flights this morning.  The flight is 7.s long:

  • 1.5s to get the props spinning to hover speed on the ground
  • 2s climb at 30cm/s
  • 2s hover
  • 2s descent at -30cm/s.

This was using the complementary filter with tau set to 5s but the GERMS stats were collected to get a better idea what’s going on.

GERMS stats

GERMS stats

This shows two things – there’s a lot of noise in the raw data blue line, and the red trend line cleary shows an oscillation which is hiding the peaks and troughs of net acceleration / deceleration at 1.5, 3.5 and 5.5 seconds.

This set show raw accelerometer values, and the measure of gravity that comes after the raw data has been passed through the butterworth filter.  Three things of interest here:

Acceleration and Gravity

Acceleration and Gravity

  1. The grey line of raw acceleration is still noisy, but does show the peaks and troughs better at 1.5, 3.5 and 5.5s
  2. The green butterworth filter line is doing a lovely job of extracting gravity from the noisy accelerometer data
  3. There’s oscillation in both the X and Y accelerometer readings, best seen in the blue and orange trend lines.

The flights were perfect to look at – only these diagnostics show these subtle problems.

The next steps then are to sort increase the P gain and decrease the I gain to stop the oscillations.  With those gone, it’ll hopefully allow the GERMS stats to show only the deviations from real gravity, and thereby filter it out from the angle calculation as the complementary filter does now.

One point in passing, I have to drop the hover speed PWM value from the long standing 1500ms to 1400ms PWM; The new 4S batteries are clearly showing that 3S is not enough.

Why is Phoebe so manic?

Given that Phoebe and Chloe are so similar, why is their behaviour so different?  Chloe is like a Mum: calm, mellow and thoughtful; Phoebe is like her 3 year old daughter, bouncing around, screaming with delight right to the point she hurts herself and starts crying*.

I’ve been thinking about what could be through cause of this behaviour; the HoG’s are identical, which means hardware.

Motors:

  • Chloe’s has T-motor MN3501-16 12N14P (12 coils, 14 magnets)
  • Phoebe’s has T-motors MT2216-11 12N14P (12 coils, 14 magnets)

Because they have the same coil / magnet configuration, if they have the same ESC with the same PWM pulse width feeding it, they will rotate at the same speed, so this isn’t it.

Props

  • Chloe’s has T-motor 13 x 4.4 CF props – the larger of the two sizes recommended for use with 11.1 batteries and the motors (above) she has
  • Phoebe’s has T-motor 11 x 3.7 CF props – the middle of the 3 sets of props recommended for use with 11.1V batteries, and the motors (above) she has

So assuming they use identical PWM and ESCs and the motors above with identical coil / magnets, how much power to their props generate?  Dunno, but given Chloe weighs more than Phoebe, and has longer arms than Phoebe, there’s some logic that suggests she needs proportionally larger bigger props, which she does.  So this doesn’t seem a likely cause.

ESCs

  • Chloe’s has T-motor 30A opto ESCs
  • Phoebe has DJI 30A opto ESCs

The translation of the PWM pulse to the motors coil switching is very specific to each ESC – the details can only be found from the code running on the microcontroller inside the ESC so I have little to really prove it’s the ESCs fault except DJI have replaced the ESCs I use with new ones – perhaps related to Phoebe’s nutty behaviour – at least that’s plausible.

And last, but not least comes Noise

  • Chloe’s HoG is attached to the rest of her frame with very soft silicone grommets; there is no hard physically connect between her HoG and the motor / prop noise transmitted through the rest of the frame.
  • Phoebe on the other hand is directly connected to the rest of the frame and feels everything.

I already have silicone grommets on order (and they should have arrived today), so I shall be fitting those as soon as they arrive.

I may well also buy new ESCs for Phoebe that match Chloe, but I’ll need to check my bank account first as they’ll be £125 for 4!

Why bother?  Because Phoebe is running through props at probably 10 times the rate Chloe does, and so even a change in ESCs will pay for itself within a month!

P.S. While I was search for what 12N14P meant, I found this article about how to wind the coils of a 12N14P motor – worth a read if only for background knowledge.


*Through experience within earshot right know, the comparison I’ve made between Phoebe and Chloe, and my wife and daughter is absolutely accurate!

Phoebe vs. Chloe

So Chloe is working nigh on perfectly – each flight is as good or better than the recent video.

With the same code other than personal 0g tuning and PID gains, Phoebe is still a danger to herself and all around her.

The only real difference in hardware are the silicone grommets separating Chloe’s HoG from Chloe’s main frame noise.

I’d really like to get Phoebe up to Chloe’s standard, as she’s small and therefore easier to transport. Trouble is those grommet seem to be rare as hens teeth. The first source were out of stock, a US seller sent 3 instead of the 10 I ordered, but I do now have a UK supplier so I’ll have another go on Phoebe.

Prototyping Zoë

Zoë is the child of Phoebe and Chloë, taking the best of both and adding a little bit of magic to the mix.  She’s only in a partial build state currently – there’s no point in completing the build until I’m extremely confident in Phoebe’s flights.

At first glance, the finished Zoë will pretty much like Chloë:

Chloë

Chloë

The difference is in the details – and specifically the base plate(s).  Phoebe and Chloë only have a single base; Zoë has two:

Silicone Sandwich

Silicone Sandwich

The lower (base) plate handles power distribution from the LiPo on top out to the ESCs on the arms.  The new intermediate plate lies just a few millimeters above, and carries just the Raspberry Pi flight controller.  This intermediate plate ‘hovers’ above the base plate on a set of silicone grommets.  These are extremely soft, hopefully providing extreme isolation from the power frame’s vibrations, thus allowing the sensors to pick up only real acceleration.  Other than the ESC cables and the micro-USB cable, there is no hard connection between the intermediate plate and the rest of Zoë.

The only slight concern I have is the placement of Zoë’s battery bank.  It can’t sit on the intermediate plate next to the Raspberry Pi as with Phoebe; its too heavy and will squish the dampers on that side.

Power bank placement

Power bank placement

The picture shows it under the bottom plate, but as the arms don’t have hands (see Chloë’s photo at the top), the battery bank would be what hits the floor on landing, and experience tells me it won’t survive many landings that way unless I’m careful!  On the plus side, the way I’ve oriented the bank, the USB B sockets is closest to the base plate given some protection from the USB B plug wrenching the socket off the PSB inside the bank.  I think it’s worth trying as any other solution seems very ugly in comparison.

Last but not least, here’s the top side view; currently, that’s Chloë’s flight controller; Zoë will have her own as there are changes I’d like to make to Phoebe’s controller too.  The nuts and washers attached to the silicone are where Zoë’s arms will connect; they have a rock-solid connection to the base plate right through the centre of the silicone grommets maintaining structural solidity, while the intermediate plate (top in the picture) chills out in her vibration free environment.

Top-side

Top-side

Vibration damping

So if I can’t get the sampling up to 1kHz to average out vibration, perhaps I can do more to suppress the vibration physically before it gets to the sensors?  Currently just 3mm foam tape sticking the Raspberry Pi to the frame is all there is.  I’ve tinkered with additional vibration suppression in the past but gave up as there were much bigger problems to deal with.

And then just the other day, I stumbled on this site and a cunning plan hatched.

The idea is that the physical strength of the frame lies between the top plate and the top of each arm with 4 bolts per arm.  But the Raspberry Pi and sensor Beret sit on the lower plate with 2 bolts per arm.  If I could put vibration damping between the lower plate and the lower side of each arm, vibrations from the motors could be damped before reaching the Beret / sensors.

Vibration Damping

Vibration Damping

I already have the upper solution using rubber dampers from my previous attempt at vibration damping, but I prefer the lower solution using the silicone gel grommets.

Just one problem: both rely on M3 bolts, but the DJI Flamewheel F450 frame as used by Phoebe uses M2.5 bolts.  ‘Luckily’, the T-motor extension arms as used by Chloe use M3 bolts.

The only downside is that to save cash, I need to splice Phoebe and Chloe together – Chloë’s hardware mind melds with Phoebe’s software and electronics to make a hybrid for which I don’t have a suitable name – any ideas?

P.S. Zoë = Phoebe + Chloë; // my best choice so far.

Releasing the GIL in Python ‘C’ libraries

I found there was a way to release the GIL inside Python ‘C’ libraries:

    Py_BEGIN_ALLOW_THREADS
        // do something synchronous here like epoll_wait();
    Py_END_ALLOW_THREADS

My variant of (and the mainline) GPIO library does this when it blocks – specifically when it’s waiting for the hardware interrupt pulse with GPIO.edge_detect_wait() (mine) and GPIO.wait_for_edge() (standard version).  I wondered whether actually releasing the GIL at this point was bad as once released the Python interpreter then processes another 100 bytes of bitcode before releasing it again; I postulated this could possibly cause missed sensor reads.

So I updated my library code, and ran the thread+signal variant of the code – once more 800Hz, just like the serialized version.  Oh well, it was worth a try.

For now then I’ve definitely run out of ideas about threading and have moved onto vibration damping; the point of 1kHz sensor reads is integrating out noise / vibration – if the vibration is physically damped, then the sensor read frequency becomes a lot less important. .  More on that tomorrow, probably.