Remove future night redshifts when running purge night #2430
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This solves issue #2388. desi_purge_night as it exists in the main branch removes all of the directories and files associated with the specified night, but doesn't remove future cumulative redshifts that used the data being deleted. This update now does that by leveraging the existing
desi_purge_tilenight
code, which will remove future redshifts for a specified set of tiles.In implementing I've also refactored the scripts so that the code is under
/py/desispec/scripts/purge_night.py
. I've also updated it to usedesiutil.log
rather thanprint
statements, and to share common code withdesi_purge_tilenight
.Note: lots of tests are failing because of a
fitsio
issue that appears to be unrelated to this PR.Tests
Tests can be found here:
/global/cfs/cdirs/desi/users/kremin/PRs/purge_night_futurezs
I set up 3 tests that all used the same fake prod. Each prod have 11 tiles (80000-80010) and 8 nights (20211010, 20211031, 20221010, 20221031, 20231010, 20231031, 20241010, and 20241031). The exposure tables and production tables indicate that 10-11 of these tiles are observed on each of the 8 nights. I then ran 3 tests to verify that the
desi_purge_night
did the right thing. I first ran it on an early night to make sure it removed all of the future redshifts (test 1). I then did the same for a night in the middle and verified it removed future redshifts but not past redshifts (test 2). Finally, I randesi_purge_night
on a later night to remove some redshifts, then ran it on an earlier night to remove more redshifts. All tests passed and did what I expected them to do.Test 1:
Ran
desi_purge_night -n 20211010 --not-dry-run
And verified that all files from the night were removed along with redshifts for 20211010, 20211031, 20221010, 20221031, 20231010, 20231031, 20241010, and 20241031.
Test 2:
Ran
desi_purge_night -n 20221031 --not-dry-run
Verified that it removed all files from the given night. Also verified that it removed redshifts for 20221031, 20231010, 20231031, 20241010, and 20241031. Finally, I verified that it did not remove redshifts for 20211010, 20211031, and 20221010.
Test 3:
Ran
desi_purge_night -n 20241010 --not-dry-run
Verified that it removed all files from the given night. Also verified that it removed redshifts for 20241010 and 20241031. Then verified that the code did not remove redshifts for 20211010, 20211031, 20221010, 20221031, 20231010, and 20231031.
After this, I ran
desi_purge_night -n 20221031 --not-dry-run
which ran without any issues. I verified that it removed all files from the given night. Also verified that it removed redshifts for 20221031, 20231010, and 20231031 (2024 nights were already gone). Finally, I verified that redshifts were not removed for 20211010, 20211031, and 20221010.