Soul destroying

Yet another slow period here on the blog, sorry.

Stuff is rattling around my head:

  1. should I buy the latest Dyson d8 handheld vacuum cleaner or stick with my current eol DC44 that’s slowly dying on me?
  2. should I buy an Apple iPhone SE or stick with my 4s that’s also slowly dying on me?
  3. should I sell my soul to pay for one or both of the above?
  4. should I give up on Phoebe and Zoe or keep banging my head against a brick wall with them and their consistent horizontal drift?

I’m guessing you’re probably only interested in the 4th item in the list above?  I’m working on the laser dot tracker to get height information and to take Phoebe for a walk on her laser leash.  Trouble is that because of the horizontal drift, there isn’t much time for the camera to latch onto the two fixed dots which track height, and (more importantly in this context) the third from the leash, especially as indoors, I’m only getting a camera shot or two per second.  Effectively the laser solution won’t work without a reducing the horizontal drift further without further sensors.

The 0g offsets of the X and Y accelerometers when horizontal seem to be the best contender so I’ve been putting a little more effort in that area.  A typical set of offsets from Zoe might read:

-138.571429 98.380962, -8313.238095

There are raw averaged accelerometer readings in the X, Y and Z directions.  In ms-2, they read

-0.166 0.118, -9.955

So Zoe when stationary and upside down thinks she’ll drift ≅ 3m backwards, 2.125m to the left over her standard 6s flight.  Which means in a real flight, she should be compensating for this by drifting 3m forwards and 2.125m to the right; yet from every recent flight shown in videos, she consistently and only drifts left by a meter or two.  So I’ve got something very wrong about 0g offsets or some area of horizontal flight control I can’t guess.

And this is frustrating enough that selling my soul to Apple or Dyson might actually be more satisfying!


28 thoughts on “Soul destroying

  1. starting a new thread again haha: didn’t you have a beer fridge to regulate temperature before? how stable could you get the temperature, i.e. what range of temp and how precise could you go down to? eg. 5C interval from 0-30C?

    • if you do the math, a few % in accel gain difference may not seem like a big deal instantaneously, but if you accumulate over time of flight, the drift is pretty significant, on the tune of 10cm per second if you were measuring earth’s gravity field (this is after correcting for offsets, but not gains).

      • Disagree strongly. As long as offsets are 0, then errors in accelerometer gain lead to fixed errors in velocity and distance travelled – they are all proportional to each other. A gain error doesn’t lead to a continuous drift, just an error in units of speed.

    • The beer fridge is fixed temperature though for full calibration I used the weather conditions 2 winters ago so from -5oC in the garage to 20+ indoors

  2. hey Hove, out of curiosity, I looked through ýour code briefly (under the theme of calibration), and in, L575, you say :
    for line in offs_file:
    ax_offset, ay_offset, az_offset = line.split()
    and then at L582, you go on to do:
    #AB! Override for safety
    self.ax_offset = 0.0
    self.ay_offset = 0.0
    self.az_offset = 0.0

    It seems to me that you are reading from a pre-configured calibration file and then deliberately writing zero back to the accel offsets. Was this intentional so that the user uses her own values?

    Also, how are you calculating the ax_offset, ay_offset and az_offset? It seems that you are averaging a couple of accel readings and treating the readings themselves as offsets? And any particular reason why you got rid of ax_gain (and ay, az)?

    • That’s an idea I tried a while back to see if it was the cause of horizontal drift. Imagine the quad sitting completely horizontal. Then the X and Y accelerometers should read 0. In the real world, they don’t and the error is not trivial. So the values are measured, stored in a file (0goffsets), and each flight, they are read in and subtracted from X and Y accelerometer values so they do read 0 when horizontal. These offsets are mentioned in the MPU6050 / 9250 spec, so I thought it worth having a play.

      It didn’t have the level of effect I was expecting, so depending on my mood, I comment out the code. The 0goffsets also seem to drift so it was an idea that was never going to work

        • Is there a reason why you removed the ax_gain? You used to have that in the code and most calibration procedure render it necessary. How did you previously calculate it?

          • I got rid of ax-, ay- and az gain calibration for a few reasons. The biggest is that there is no point – they were only out by only a % or two, and all that meant is it flew that few percent too slow / fast. The offsets seemed more important as they mean the sensors could read zero when in fact there is acceleration – I’m still not convinced I don’t need it.

            Gain and offset calibration required care – I have a perspex cube bigger than the quad. I’d stick the quad in it and stick it on a horizontal surface, and then flip the cube to sit on all 6 sides; that gives offset and gain for X, Y, and Z. Trouble is, commercial quads don’t do this, so it can’t be right. Finally, the offsets and gains drift over time and temperature, so however careful the calibration was done, it would be wrong by the time I took it to fly.

            • now what if you calibrated it for each axis (both + and – for each axis) at one temperature, then did it for multiple temperature and interpolated them? arducopter definitely required it for new quads. haven’t followed the latest solo or dji, but my guess is that they did all that at factory over multi-temperature and stick a good temp sensor in there

              • I used to do this in the past, but decided it couldn’t be necessary because it’s not done for commercial quads. Interesting to know arducopter do it

  3. Hi Alex, my brain’s caught up with you and the Porsche Cayman S doing 150 down the German Autobahn. All a fair point, but…isn’t the PWM update frequency a random variation around a fixed point where the fixed point is the desired PWM update frequency, and the random fluctuation is kernel scheduling of that PWM update? Surely then that gives me random drift which should average out as net zero? Whereas what I’m seeing is a constant port-side drift where velocity depends on hardware factors (props and noise primarily – see the latest video of indoor flight), but direction is consistently to port side? I’m not denying RPIO PWM could well be my weakest link, but I think I have a bigger problem biassing the system to hunt down first.

    • fair enough on hunting down the big problem first. one thing to note though, even though the variation can be assumed to be random, saying that you would have a net zero bias is a strong assumption, i.e. you’re assuming zero-mean variation. I could just as well assume that your mean is biased, i.e. the kernel tends to schedule the PWM such that your right motors spin faster or get updated quicker, more so than the left sets, such that you see a net port-side bias…

      also, while we’re on the BMW-Porsche RPM autobahn fanfair, you mentioned that you have never calibrated PWM before (much like the RC guys do: turn on controller, throttle max, turn on motor, throttle min, you should hear motors go double beep, then long beep when it’s done). I think your orevious comment was that because you’re not using RC, there’s no need to calibrate. Is this still true or have you implemented a calibration routine? TReason I ask is that the ESCs may come out of the box with different throttle/PWM max/min settings. so, while one motor may see PMW 1500 as ESC out duty cycle 50%, the other ESC may see it as 40%, so this may be another thing to consider. If you don’t want to do it with an RC controller, you can do change up the routine above with raspi-PWMOut max and min . The point is just to let the ESC know what to expect as max and min PWM from the controller and map its motor output duty cycle accordingly.I’ve done this with my quad and seen small imporvements.

      as you said, these are speculations to the weakest link, so you’ll have to decide which part of the Beemer to tune 🙂 PWM could be a small issue, but it is the output half of your system, and I am once again, obviously biased haha

      • Hi Alex,

        I’ll get back to you on most stuff tomorrow, but I will follow up now on ESC calibration: the only way I could see that it was worth doing is when using a remote control, where analogue joysticks are converted to digital signals passed to the Quadcopter and used to control it. I assumed the synchronisation was so the digital ESC knew the range of the analogue RC joysticks; because there was no analogue side in my system, I assume each ESC would leave the factory with the same ‘default’ digital settings and that was fine by me.

        • the part that would apply to your quad even without analog RC joystick is the “… digital signals passed to the Quadcopter and used to control it.” This digital signal would correspond to the pwm_update in your python code. The ESC would not care if that digital signal comes from an RC ADC or your raspi, and it only needs to know what is min and max really 🙂

  4. No problem on the bluntness. I’m just trying to figure this out as much as you are and I get the frustration. 🙂 when you say the pwm is hardware pwm, that is true for one channel only right since raspi only has one hardware pwm which is then split into four by means of the rpio software library. So the OS still could play into the timing.

    Likewise , have you considered ditching the veloctiy pids so you aren’t integrating noise inhenrent in the measurements?

    • Ooh, that’s interesting about how RPIO PWM works – I never bother to look – I didn’t realize it was split by software across the pins – doesn’t really change anything though – it’s sensor readings that need to be accurate so the PID feedback loops work.

      Ditching the velocity PIDs would be the perfect solution for getting rid of the drift but sadly it also throws away anything like autonomous control – it becomes a standard QC flight controller designed to include a human being with an RC in the main feedback loop. I always wanted this to be autonomous as my hand / eye coordintation is rubbish, so me and a remote control would be dangerous in a quadcopter feedback loop!

      • So the sensor readings need to be accurate, and as noise free as possible, or compensated by another sensor. But that only takes care of the input side. Now if pwm has some delay, between a set of four pwms (system lag), or if the pwm between each ESC is delayed, say you expect them to fire at 0, 5, 10, 15 , but they really go at 1, 3,11,14 that could throw off the attitude slightly for each loop. So even if you have the best sensor, the controller may still think your pwm is OK and it’s compensating for motion dynamics, but in reality its seeing the pwm out of sync. Reason why I think the OS may come into play is it intently controls when a loop runs, and if it delays, intentionally or otherwise, the pwm output because of some background task you can’t control or see, that may cause issues. Just a hunch, which may warrant comparing it with hardware pwm on all four channels being controlled (and also produced) by one low level microc while the raspi gives high level commands.

        On velocity pid, the autonomous case does require more inputs, so yes I suppose more sensors to improve velocity estimate makes sense.

        • Isn’t that missing the whole point of the feedback loop? The code is blind what it’s controlling beyond the final PID outputs. It has no idea of wind, weight distribution, loose bolts, damaged props, imperfect PWM, strange ESC firmware, dodgy axles on motors etc etc etc. All it knows is it’s put out some PID values, and the sensors pick up what’s happened as a result, compared it with what should have happened, and updated the PID output as a result. It doesn’t even matter if there are time delays – the integral of the PIDs will get that as just another factor of the feedback loop.

          All that matters is accurate sensors and code without bugs 😉

          • The integral term attempts to get rid of constant offsets from your desired state, eg what you’re seeing now actually– a horizontal drift. Thing is it needs time to counter that drift and that doesn’t take into account that the drift may not be constant and is changing so the integral term is always playing catch up…

            The feedback loop takes care of a lot of unknown shenanigans, IE mass change, prop unbalanced etc. , and your assumption that integral term may compensate for the pwm output inaccuracy could be true, if only the inaccuracy were consistent , which may not be the case with OS delays (CPU loads vary) . now if your feedback loop attempts to correct left drift at time t, pwm is inaccurately timed, and at time (t+1) feedback loop thinks it’s outputs are not good enough and ups the output again, but pwm may not be delayed this time so you overcompensate and fly way to the right this time, then pwm turns down ouput at time (t+2)….. You could say the feedback loop keeps trying to correct for the pwm inconsistent delays, but what’s true is also that your controller has no authority over the the output. It thinks the input is sensor, output is pwm and then feedback, but in reality, you’re adding in an unknown amount of delay / noise with potential pwm delays. This may not matter for if you don’t care about the drift, but every delay matters if you’re trying to hover over a spot autonomously. Having inaccurate pwm is like having shitty sensors, it’s half of the problem right there, IE the output part. It’s almost like you step on the accelerator of the Car the same amount and sometimes it’s revs up to 100 rpm, sometimes 200, sometimes 500, all for the same accelerator position..

            • Hmm, I’m not quite convinced here; if the sensors were perfect, and there was no noise etc so sensor data integrates perfectly to velocity, then regardless of any PWM oddities, if the velocity PID target is 0 in all 3 axes, then the sensors will always detect when the system is not stationary, and stop the drift. Of course while the motion is slowing down and stopping then there is drift.

              Trouble is, you’ve hit the limit of my brain to take in and process all the details of the conversation together – I could easily miss a detail in what you’re saying. I need a beer or two to resolve this confusion!

              • let’s take your example and say zero drift, perfect sensor: the only issue is pwm irregularity. yes sensor picks that up and tries to correct it, but the problem is that it is trying to control a constantly changing system , i.e. the potential for pwm delays… so now, your control system is not dealing with a deterministic command-to-output subsystem (i.e. what it commands may not be what is output to the escs themselves, because of the delays).

                take the car gas pedal analogy: your goal is to be at 70kmh on a windless day with a perfect german autobahn. you put your foot on the gas pedal to a certain position, rpm goes to a certain point, say 3000 rpm and you can maintain that speed 70kmh as long as you keep your foot there (and winds and surface roughness are all the same, which it is). now the problem with this pwm delay is that even though you put your foot at the same position, the engine rpm goes all over the place (due to delays): 3100, 3150, 2800, 2950… is it close to 3000? sure, but it isn’t exactly 3000,so your speed is fluctuating around
                70kmh and not exactly on it because you may try to chase the needle at 70kmh by changing foot position and what makes it worse is you have no prediction of how the rpm will respond to your foot position. so same goes for pwm delays, which is analogous to the engine rpm fluctuations (since the pwm delays correspond directly to how the pwm output is timed, or how fast the motor can respond to speed change commands). problem may be what you command (software pwm out / foot position) may not be what the motor esc sees (pwm signal seen by esc / rpm of engine), so you’re constantly chasing a moving system / target.

                that’s my hunch of what may be a weak link in your system, just a guess haha. enjoy yo beer 🙂

  5. my vote is on 4, and I have my obvious bias 🙂 how much hori. drift do you get, and how long are your average flights? running at 1000Hz loop for the software?

    i have seen a similar issue too with horizontal drift, but my code is pretty old, so my hunch is that this is a fundamental issue with the sensor and possibly PWM output. MEMs silicon IMUs drift like crazy, nothing new. if you did a test and left the current sensor sitting on a desk and just record data for half an hour, you’d get my drift 🙂 so the next issue is, why not pair it with a complimentary sensor, like say an ultrasound sensor that give you sub-cm accuracy probably.

    another note: ar drone 2 has something similar to your image solution, which is having their bottom camera and front facing camera do position lock, but one thing they have going for them too is their ultrasound sensor, which is a standard sensor in drones today. so something to consider? my guess is that they need the ultrasound to tell them the “measured” height, so they can compare their calibrated height measurement relative to a pixel ((i.e. actual distance / pixel) to the ultrasound “measured” height, which then gives them insight into how much they are drifting (because the video resolution is fixed, i.e. width pixles X height pixels).

    on the pwm issue, this is my hunch, but when things don’t work, question everything. you set out to prove to people that it is possible to output pwm with a non-microcontroller, ie raspi and you have successfully done so, I’d argue, i.e. takeoff, hover and land a drone for 15-30seconds without any major crash. but if you extend the flight duration to 10-15 minute, that’s a different ballgame. if you let it hover for that long, it would be in the next room and then some… my hunch is that the raspi may be outputting pwm (or really the rpio library), but 1. it’s software pwm vs hardware pwm 2. the library author himself strongly suggests against using it for real-time control and 3. you have linux OS that is doing multiple things while trying to output pwm, whereas hardware pwm (a dedicated chip if you will), or even just a microcontroller has a limited task scope. so really, you do not have a fine-grained control over the pwm that is output, eg. they do not simultaneously update with 4 parallel channels and they do may not output consistently at a fixed rate (say 200Hz on the dot because the OS could be doing something else when you want to output pwm, hence introducing a lag to the output and the system response)

    I honestly think that with the sensor drift and pwm not being instantaneous / real-time, the combined effects could lead to this horizontal drift (and other issues). these are literally the input and output of the system, which have extremely low tolerance for errors / ambiguity. add on the need for potentially more sensors (ultrasound?) and we may see the small effects of each imperfect system become a major issue.

    my bet is to investigate hardware pwm, or a dedicated micocontroller to output pwm (maybe even read sensor data) and investigate ultrasound sensor. if after you look at them and they do not solve yout drift issue, then you can conclusively say that microC/ hardware pwm and additional sensors are not needed.

    • PWM accuracy doesn’t matter for a few reasons:

      1. The sensors and PIDs provide a feedback loop to correct any under- / over-shoot in the PWM

      2. The raspberry Pi has hardware PWM buit in so there’s no need for an outside board. I’ve check the output of the RPi PWM on an oscilloscope and it’s accurate. Because it’s hardware PWM, it’s not interrupted by the OS.

      3. The PWM to motor phase as defined by the ESCs is not standard so it’s pointless worrying about the “right” PWM – there is no such thing – it depends on the firmware the ESC is running.

      I’m running 1kHz sampling and motion processing batches of 10 from the IMU FIFO roughly every 100Hz. Clock is taking from the IMU sampling, not RPi to guarantee accuracy. 10 samples in motion processing = 10ms elapsed.

      I have ultrasound installed for height but the code doesn’t use it because the 3 dot LED solution gives height and drift horizontal and vertical) without needing ultrasound for calibration. The drift I’m most worried about is horizontal which a downward facing URF won’t help, hence the LED option I’m considering.

      Normal quads with RC don’t have the same PID layout as my autonomous one – I’m never going to get long stable flights without drift unless I have other sensors – there’s too much integration in my code for velocity which normal RC control flight controllers don’t need.

      I think the problem lies all with the MEMs IMU drift and that’s why I’m looking at more sensors to add stability.

      • P.S. Sorry if this all comes across as a bit rude, but having spent 3 years on this project, I’ve tried an awful lot and learned an awful lot, tweaked and refined an awful lot, and so occasionally I can be a bit blunt when others are helpfully suggesting solutions I’ve already covered many time over.

  6. Are you using the data provided by the motion estimation block of the H264 encoder in any of your image processing?

    • Thanks for the reminder Jeremy – I haven’t got round to trying that yet either. Can’t remember why it stalled I think I’m simply not willing to move on until I understand why horizontal drift is as bad it is.

      I’ll have a go with the h.264 macro block motion code tomorrow to at least collect some data and see whether it looks right.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.