Internal wavelength calibration errors

[segasai]

After tackling the sky corrections, I wanted to take a look at the internal wavelength corrections done by trace_shifts.
Specifically so far I just reran the traceshifts on the representative sample of exposures with different conditions (that i used for previous patch)
For each run I saved the dwave corrections in different wavelength bins for each fiber that are computed by compute_dy_using_boxcar_extraction.

I then saved the wavelength offsets for each wavelength bin and each fiber
Here's an good example of b9 with expid=126067.

Each row of panel show different wavelength bin. The right panels show the ylim of -3,3 the left panels show the zoom in on -.2, .2

One problem that i see is shown here for example. The uncertainties assigned to shifts seem kind'a wrong most of the time
because they are clearly too small.

There are some cases where the uncertainties seem right like here

I do see that the polynomial model that fits these offsets has some way of dealing with outliers, but I don't quite know if it behaves well if all the errors are vastly underestimated. Also I havent' checked if there are cases where the offsets for the whole frame are completely wrong (as they all are just uniformly scattered between -3 and 3 )

Just in case the images for all the frames I fitted are available here /desi/users/koposov/wavelength_fix/outpsf/

[segasai]

Now switching to a more concrete question.

I am not sure I understand the reason for setting an ivar to be equal to ivar*reference flux here.

desispec/py/desispec/trace_shifts.py

Line 363 in ab265e5

dwave,err = compute_dy_from_spectral_cross_correlation(flux[fiber,ok],wave[ok],reference_flux[ok],ivar=ivar[fiber,ok]*reference_flux[ok],hw=3., calibrate=True)

This may be harmless but I still don't understand the rationale.

This was introduced in this patch
8457e26

[julienguy]

I must have added this to enhance the weight of sky lines, but I agree it's questionable and it can be revisited. Formally we should not add this term if the model spectrum we are comparing to is representative of the data but this is not the case here: the flux of the objects can be very different from the reference flux.
Changes to the weights could improve the measurements but have to be tested in various conditions (inc. the backup survey!).

[segasai]

Thanks @julienguy
I have previously thought that the internal wavelength calibration is harmless for backup because one can still use the continuum solar spectra for this cross-correlation, but I haven't realised that the internal calibration step is done with spectra that have objects in them as well.

Altogether, I think the blue corrections even in the dark time are highly problematic (other than in two wavelength bins with bright emission lines.

Also my concern here (if I understand the code correctly) is what will happen if we are observing a large number of stars with not very bright sky, this correction will kind'a try to bring them all to same velocity.

So I think this internal correction need at least to use the same mask for sky as we use for external. I.e. ignore all other pixels other than those that are near known bright lines (and maybe use the same highpass filter).

The only thing that is lacking now is a clear plot how these corrections affect the final data-products, so I can diagnose if things are actually better if I change anything.

(my current intuition looking at the measured offsets is that if I disable this internal wavelength calibration, the accuracy of velocities will increase for a large fraction of exposures)
(and I agree that any change needs to be tested in a range of conditions -- and I'm still planning to use my set of exposures in /global/cfs/cdirs/desi/users/koposov/wavelength_fix/refit_exp_list.csv which covers all the cases pretty well as far as I can see.

[segasai]

I believe this issue is now fixed with #2386.