[Question]: How was edit sim for code tasks calculated?

[cornzz]

@pzs19

In the longbench evaluation fuzzywuzzy.ratio was used to calculate edit similarity (Levenshtein distance) for the predicted code lines (see here).

However I noticed that this function returns different results depending on whether python-Levenshtein, an optional dependency of fuzzywuzzy, is installed or not. Therefore I wanted to know if you used python-Levenshtein or not.

On another note, in both cases this library does not actually return the correct Levenshtein distance: THUDM/LongBench#96