Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove future night redshifts when running purge night #2430

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

akremin
Copy link
Member

@akremin akremin commented Jan 10, 2025

Summary

This solves issue #2388. desi_purge_night as it exists in the main branch removes all of the directories and files associated with the specified night, but doesn't remove future cumulative redshifts that used the data being deleted. This update now does that by leveraging the existing desi_purge_tilenight code, which will remove future redshifts for a specified set of tiles.

In implementing I've also refactored the scripts so that the code is under /py/desispec/scripts/purge_night.py. I've also updated it to use desiutil.log rather than print statements, and to share common code with desi_purge_tilenight.

Note: lots of tests are failing because of a fitsio issue that appears to be unrelated to this PR.

Tests

Tests can be found here: /global/cfs/cdirs/desi/users/kremin/PRs/purge_night_futurezs
I set up 3 tests that all used the same fake prod. Each prod have 11 tiles (80000-80010) and 8 nights (20211010, 20211031, 20221010, 20221031, 20231010, 20231031, 20241010, and 20241031). The exposure tables and production tables indicate that 10-11 of these tiles are observed on each of the 8 nights. I then ran 3 tests to verify that the desi_purge_night did the right thing. I first ran it on an early night to make sure it removed all of the future redshifts (test 1). I then did the same for a night in the middle and verified it removed future redshifts but not past redshifts (test 2). Finally, I ran desi_purge_night on a later night to remove some redshifts, then ran it on an earlier night to remove more redshifts. All tests passed and did what I expected them to do.

Test 1:

Ran desi_purge_night -n 20211010 --not-dry-run
And verified that all files from the night were removed along with redshifts for 20211010, 20211031, 20221010, 20221031, 20231010, 20231031, 20241010, and 20241031.

Test 2:

Ran desi_purge_night -n 20221031 --not-dry-run
Verified that it removed all files from the given night. Also verified that it removed redshifts for 20221031, 20231010, 20231031, 20241010, and 20241031. Finally, I verified that it did not remove redshifts for 20211010, 20211031, and 20221010.

Test 3:

Ran desi_purge_night -n 20241010 --not-dry-run
Verified that it removed all files from the given night. Also verified that it removed redshifts for 20241010 and 20241031. Then verified that the code did not remove redshifts for 20211010, 20211031, 20221010, 20221031, 20231010, and 20231031.

After this, I ran desi_purge_night -n 20221031 --not-dry-run
which ran without any issues. I verified that it removed all files from the given night. Also verified that it removed redshifts for 20221031, 20231010, and 20231031 (2024 nights were already gone). Finally, I verified that redshifts were not removed for 20211010, 20211031, and 20221010.

@akremin akremin requested a review from sbailey January 10, 2025 23:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant