Continued from sublimehq/package_control_channel#9005 (comment)
Hm, indeed. Since you seem to re-render all phantoms on every update, when doing it async will make the entire document jump around. I wonder if you could leave any unchanged phantoms alone.... It's been a while since I went deep into phantoms, but I don't think I ever had that issue with ColorHints. I do notice your PhantomSet doesn't have a key, that might help?
Continued from sublimehq/package_control_channel#9005 (comment)