-
Notifications
You must be signed in to change notification settings - Fork 85
Handle case where no samples in a rangline are valid #327
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -33,7 +33,8 @@ | |
| import numpy as np | ||
| import isce3 | ||
| from isce3.core import DateTime, TimeDelta, LUT2d, Attitude, Orbit | ||
| from isce3.focus import make_los_luts, fill_gaps, make_cal_luts, Notch | ||
| from isce3.focus import (make_los_luts, fill_gaps, make_cal_luts, Notch, | ||
| find_bad_rangline_slices) | ||
| from isce3.geometry import los2doppler | ||
| from isce3.io.gdal import Raster, GDT_CFloat32 | ||
| from isce3.product import (RadarGridParameters, | ||
|
|
@@ -1750,6 +1751,16 @@ def get_caltone_algorithm(cfg, fc, fs, n, is_dithered): | |
|
|
||
| return algorithm, wavelets | ||
|
|
||
|
|
||
| def log_bad_pulses(swaths, max_slices=10): | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I assume the purpose of the |
||
| n_bad_pulses, bad_slices = find_bad_rangline_slices(swaths) | ||
| log.info(f"Number of pulses with no valid samples = {n_bad_pulses}") | ||
| if n_bad_pulses > 0: | ||
| log.warning(f"Bad pulses appear in {len(bad_slices)} unique blocks") | ||
| for s in bad_slices[:max_slices]: | ||
| log.warning(f"Bad ranglines in pulse {s}") | ||
|
|
||
|
|
||
| def focus(runconfig, runconfig_path=""): | ||
| # Strip off two leading namespaces. | ||
| cfg = runconfig.runconfig.groups | ||
|
|
@@ -2050,6 +2061,7 @@ def temp(suffix): | |
| swaths = raw.getSubSwaths(channel_in.freq_id, tx=pol[0]) | ||
| swaths = swaths[:, pulse_begin:pulse_end, :] | ||
| log.info(f"Number of sub-swaths = {swaths.shape[0]}") | ||
| log_bad_pulses(swaths) | ||
|
|
||
| rawfd = temp(f"_{frequency}{pol}_raw.c8") | ||
| log.info(f"Decoding raw data to memory map {rawfd.name}.") | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even though in reality and during warmup the entire pulse is not invalid but first 2/3rd and then 1/3rd is missing, looks like we have decided to mark all invalid in L0B. Anyways that is an upstream discussion which we already had. So that is ok to give up on the warm up part in RSLC.
Out of curiosity what happens if the entire L0B has a mask that marks all pulses invalid? Does focus detect and complain, or with this minor change we are letting that go through?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess that's true about the first 1/3rd of the swath. We'd want to test that EAP does the right thing in that case.
If all pulses are marked completely invalid then yes processing would continue and the result would be an image of all zeros. Similarly, a block of 5 seconds of missing data would no longer be a fatal error. That's why I added the extra logging so there's at least some visibility into those sorts of edge cases.
The L0B .met file contains information about missing rangelines, so PCM could implement a rule to avoid RLSC processing stuff with missing data.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I agree that it is the responsibility of job manager (PCM) to check if we want a threshold for not submitting job. I think it is actually a good thing to let data be processed even if there is 5 sec or whatever gap in the data. Again somebody else's job (e.g, QA) to accept the partial SLC or not!