Prevent leading whitespace in markdown code blocks from being stripped

[peytondmurray]

This PR prevents leading whitespace from being stripped from markdown code blocks before they are parsed for highlighting. For certain languages, leading whitespace is syntactically significant. Fixes jupyter/nbviewer#1021.

I tried to go as far back as I could through the git history to figure out why this was added, but it appears in the earliest tag, 4.0.0. If there is a reason to strip leading whitespaces, it's not explicitly tested for in the test suite. If you have any context around this choice, I'd be interested in hearing about it.

I also added a test to check that whitespaces aren't stripped from code blocks. I've also added a RUF001 rule to ignore uses of ambiguous unicode characters in tests/exporters/test_html.py, because of the "ɩ" which is apparently a valid APL function.

[krassowski]

Does it also fix #2156?

[krassowski]

I think this change makes sense. Should we add an option to allow to opt-in into the old behaviour so that users who may have relied on it are not blocked from upgrading?

[peytondmurray]

Does it also fix #2156?

I don't think so - this PR only addresses code blocks inside markdown. The issue above ☝️ is about being able to split statements across multiple code cells - for example, putting a single (empty) if, with, or while etc in one cell, and having indented code in subsequent cells further down be part of the scope of the first cell:

Maybe this is a discussion for that issue, but I'm not sure that makes sense to do here - at the moment, each cell needs to be a syntactically complete snippet of python code, which isn't true in the use case provided.

About your other suggestion: I like it, I'll add the option to pass arbitrary lexer options to the pygments lexers now!

[peytondmurray]

I've added a test for the lexer options. The user can revert to the old behavior by setting

HTMLExporter(lexer_options={"stripall": True})

The option should be documented by traitlets, so I haven't included anything extra about reverting back to the old behavior. But if some additional documentation would be helpful please let me know and I can add an explicit section about this to the docs.