fix: IllegalArgumentException in Colorizer event processing.

[andrewL-avlq]

The Colorizer should not process stale tokens if they go over the end of the line in the current document. This prevents IllegalArgumentException occurring when the Tokenizer thread runs slow or is delayed relative to the textChanged notification in the reconciler.

[andrewL-avlq]

I think this should fix issues like #408.

[sebthom]

Thanks for your PR. However I am not sure if this is the right fix or if we should better suppress the IllegalArgumentException as well as the BadLocationExceptions in the catch block and only log them when tracing is enabled. This new check does not fully prevent the occurrence of race conditions in the following lines. Also I am not sure if continuing to colorize subsequent lines makes sense if the tokens of one line are stale.

[andrewL-avlq]

Sure, this PR is a starting point for discussion if nothing else.

The edits that the Tokenizer is processing might not include changes to the following lines, so exiting the colorizer early didn't seem quite right. On the other hand, when the Tokenizer thread has updated the tokens, the colorizer will run again, so I am fine with throwing if you want.

Would you rather catch the IllegalArgumentException, or detect the condition early (such as having getTokenLength detect that it is about to return a negative value) and throw a custom exception (maybe even BadLocationException)?

Given that the Tokenizer thread runs asynchronously, I was not trying to prevent any racing, but rather to have the colorizer cope with what goes wrong if the model is out of sync with the document (as there already appear to be conditions to handle other cases of this). Also for this reason, I agree that the logging of the BadLocationExceptions should be guarded and only logged when tracing is enabled.

[sebthom]

I think I would just add another } catch (final IllegalArgumentException | BadLocationException ex) { error = ex; TMUIPlugin.logTrace(ex); } block with some explanation before

tm4e/org.eclipse.tm4e.ui/src/main/java/org/eclipse/tm4e/ui/text/Colorizer.java

Lines 169 to 172 in 83e998c

	} catch (final Exception ex) {
	error = ex;
	TMUIPlugin.logError(ex);
	} finally {

[sebthom]

Thanks!

[andrewL-avlq]

No problem.
What is the timescale for seeing this in a release?

[sebthom]

There is no fixed plan. I can create a release when the license check works again. I already opened a support ticket.

[andrewL-avlq]

That would be great thanks.

[sebthom]

@andrewL-avlq A new release 0.14.1 is available