AbsTaskSummarization using STSEvaluator instead of SummarizationEvalutor?

[alt-glitch]

Hey!

While going through your implementation to add FinMTEB to MTEB, I found that the summarization abstract class uses STSEvaluator instead of the SummarizationEvaluator

FinMTEB/finance_mteb/abstasks/AbsTaskSummarization.py

Lines 40 to 48 in 272ac35

	evaluator = STSEvaluator(
	data_split["text"],
	data_split["summary"],
	normalized_scores,
	task_name=self.metadata.name,
	**kwargs,
	)
	scores = evaluator(model)

This seemed like a bug so I thought I would let you know.

If this is intended, I'd like to better understand why so!

Thank you!

[yixuantt]

Hi, it's not a bug. We do not have a human_summaries/relevance score for the summarization task. Instead, we compare the Spearman correlation based on the semantic similarity between the summary and the text. The semantic meaning of a summary must be highly correlated to the text.

To simplify, we directly use the STSEvaluator here, which serves the same function.

[alt-glitch]

Understood. Thank you for the explanation!