Coloring character literals

[dvanhorn]

DrRacket mis-colors some character literals, likely due to a mismatch between the documentation on character literals and the implementation of the reader. See racket/racket#3506 and racket/racket#3507 for examples of character literals that fall outside the spec given in the document, but are considered valid.

So #\U000000AA should be colored as a character literal, but is currently colored as a literal followed by an identifier AA. #\71 is colored as a character literal but should be red since it is not valid.

[rfindler]

The relevant code is here and here's a drracket-free version of the code:

#lang racket/base
(require syntax-color/racket-lexer)
(define (get-tokens str)
  (define sp (open-input-string str))
  (let loop ()
    (define-values (maybe-eof token paren start end)
      (racket-lexer sp))
    (cond
      [(eof-object? maybe-eof) '()]
      [else (cons (list token start end) (loop))])))

(get-tokens "#\\U000000AA")
(get-tokens "#\\71")

[rfindler]

I'm going to try to transfer this to racket/syntax-color (from racket/drracket)