The death of kitty++

The h.264 macro-block data that the GPU spits out can’t be used for an accurate, long term motion tracking sensor complementing the integrated accelerometer for short term accuracy and high resolution. Period 🙁

It works perfectly detecting parts of a video frame that have moved since the previous video frame, and therefore optimize compression of the new frame by only including the changes to the previous frame.  That’s what it’s designed to do and it does it well.

But what’s required for my horizontal motion tracking is an accurate measure of how much and in which direction every pixel has moved from one frame to the next.  And it almost does a perfect job of this, except for two reasons which completely rule out its usage for this cause:

  • periodically, the macro-blocks are reset for the frame – that means there is no measure of the velocity of pixels compared to the previous frame.
  • each macro-block motion vector is accompanies with a sum of absolute differences (SAD) – a numerical value for the level of confidence in the vectors accuracy.  Testing yesterday revealed that these values are big and fluctuate smoothly across a large range of values for a pair of frames without obvious high and low confidence values.  And that means it’s not possible to combine the vectors and the SAD values to get an accurate vector for the whole frame.

So it’s back to the laser dot tracking as the only possible solution until the Kickstarter  LiDAR I backed is launched.


6 thoughts on “The death of kitty++

    • Wow, the Phantom 4 is absolutely overloaded with sensors for drift prevention and object avoidance. They must have some sideways facing sensors to stop it drifting into everyone watching the flight from a few meters away? I can’t see how down facing cameras (1 or 2) could do that. It’s this horizontal drift I need to kill.

      I just had a quick look at OpenCV and it lost me within a couple of sentences. I’ll try to have a proper look but I have a feeling it’s beyond me!

  1. At least you gave it a shot! 🙂

    I was thinking about the SAD values and wondering just how they could be utilized. If you assume that the only movement in the scene is due to movement of the copter then theoretically any “reasonably confident” macro block should describe the copter motion. If you can find 3 or 4 “reasonably confident” macro blocks that all seem to agree on the same kind of motion then that might be a suitable solution. You would need to interpret motion (translation, rotation, etc.) but hopefully the other sensors would serve to reinforce said interpretation. It would get more complicated when you consider the possibility of other objects moving in the scene but I feel like the extraneous information could be filtered out to avoid skewing the results.

    Just a thought.

    • Hi Jeremy, I could (and might out of interest) spend more time looking at the vectors and SAD values for every macro-block in a single frame to see if the values are consistent enough to just ignore SAD values, but that still leaves the problem that occasional, the macro-block processor resets and you get 0, 0, 0 for each macro-block in the frame – the link with the previous frame is broken and so accurate velocity tracking is lost. And it’s this that’s required to be accurate long term to compensate for accelerometer integration errors.

      • Hi,
        If I remember correctly you can configure the period for the keyframes, including a value of zero. This will prevent the camera from outputting keyframes that reset the motion vectors.
        Another similar option is to use the cyclic refresh that resets each macroblock individually, not the whole frame at once.

        I really wanted to pursue this as well as it seems like a very nice way to compensate for drift. Hope you find a way to get better data out of it 🙂

        • That’s interesting about disabling keyframes completely as that would allow some long term ‘measurement’. I’ll have a look once I find some motivation! Both Phoebe and Zoe are misbehaving at the moment with drift – Phoebe vertically and Zoe horizontally, and understanding why is getting frustrating as it feels like I’d been here so many times!

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.