Add SpinQuant pass

[jambayk]

Describe your changes

Add a new pass do weight rotation using SpinQuant.

• Similar to QuaRot, this pass also only performs offline weight rotation.
• The concept is similar to QuaRot but the rotation weights are trained on a calibration dataset to improve activation quantization quality.
• Only per_token and per_tensor activation quantization are supported. Groupwise quatization is not supported yet since we don't expect it to be used in subsequent QDQ models.
• Common training utils have been abstracted from the lora passes.

Checklist before requesting a review

• Add unit tests for this change.
• Make sure all tests can pass.
• Update documents if necessary.
• Lint and apply fixes to your code by running lintrunner -a
• Is this a user-facing change? If yes, give a description of this change to be included in the release notes.
• Is this PR including examples changes? If yes, please remember to update example documentation in a follow-up PR.

(Optional) Issue link

[xiaoyu-work]

Is stiefel always true here?

[jambayk]

yes, it is required to be True to do the cayley sgd for orthagonal matrices. without it, it behaves the same as normal sgd. original implementation here https://github.com/facebookresearch/SpinQuant/blob/44dbc26056ee9e319dd8ce24bfbf7203785f5c77/optimize_rotation.py#L109

[xiaoyu-work]

Curious why the stiefel default is False, and it also implements the case when stiefel is False?

[jambayk]

I am not sure about the reason but yes, it implements the False case too.

I got it from https://github.com/facebookresearch/SpinQuant/blob/main/train_utils/optimizer.py#L57 which was in based on the torch sgd implementation https://github.com/pytorch/pytorch/blob/main/torch/optim/sgd.py#L26.
Maybe they wanted to have both modes be available in the same optimizer for completeness.

[xiaoyu-work]

IMO, if we don't (and won't) support the other case, we should remove those dead codes.

[jambayk]

sounds good! I will remove the False case and simplify the optimizer.

[jambayk]

I tried doing this, but the options are more tied than I expected. There are if/else conditions that involve stiefel=true/else + related parameters. I cannot verify the correctness of a modified optimizer so decided to keep it as is. Even the original implementation of spinquant copied the optimizer directly from the source code of the paper that introduced the algorithm.