Tom:
Hi team,
I’ve been plating with injecting and recovering 1-year orbital period planets. I started using a dumb median filter but got stuck with detections always occurring after earth points through faulty filtering at data gaps.
I then tried using Gal’s detrender and am finding it causes massive spikes after every earth point. What am I doing wrong? I’ve attached a plot showing the a dumb median filter in red and Gal’s one in blue. I tried playing with the window size with little success.
By the way, the data files contain a column called SAP_QUALITY. The values in this column mean things, ie. there is a numerical value for ‘earth-point just happened’. I’ve started trying just throwing away the data for two days after every earth-point.
Cheers,
Tom
Bekki:
We're working on various approaches to avoid detections at data gaps. For example, what I'm doing now is:
- Identify change points based on time gaps and/or big flux jumps
- Detrend each segment between change points separately using running median filter with 3-day width
- Delete data 3 days before and after each gap
- Use wavelets to interpolate between gaps
And I think others are working on different things. But basically for now most of us are sacrificing data near gaps.
Ruth is working with Gal's median filter too and I think she's implementing something to try to address this issue.
Tom:
Thanks for the update Bekki. I too am finding that sacrificing data near gaps is a necessary evil.
Why do you remove data before the gaps? These data shouldn’t be affected by the nasty ‘thermal ramps’, the spacecraft doesn’t know its about to do an Earth-point.
(I guess we should probably be having this conversation on Gibhub, oops)
Hope all is well there with you guys! It’s rainy here in SF!! in June!!
Tom:
Hi team,
I’ve been plating with injecting and recovering 1-year orbital period planets. I started using a dumb median filter but got stuck with detections always occurring after earth points through faulty filtering at data gaps.
I then tried using Gal’s detrender and am finding it causes massive spikes after every earth point. What am I doing wrong? I’ve attached a plot showing the a dumb median filter in red and Gal’s one in blue. I tried playing with the window size with little success.
By the way, the data files contain a column called SAP_QUALITY. The values in this column mean things, ie. there is a numerical value for ‘earth-point just happened’. I’ve started trying just throwing away the data for two days after every earth-point.
Cheers,
Tom
Bekki:
We're working on various approaches to avoid detections at data gaps. For example, what I'm doing now is:
And I think others are working on different things. But basically for now most of us are sacrificing data near gaps.
Ruth is working with Gal's median filter too and I think she's implementing something to try to address this issue.
Tom:
Thanks for the update Bekki. I too am finding that sacrificing data near gaps is a necessary evil.
Why do you remove data before the gaps? These data shouldn’t be affected by the nasty ‘thermal ramps’, the spacecraft doesn’t know its about to do an Earth-point.
(I guess we should probably be having this conversation on Gibhub, oops)
Hope all is well there with you guys! It’s rainy here in SF!! in June!!