For your lexer you would like the text of
q'[Here's Johnny]' and
'Here''s Johnny' to be matched. Assuming that you want the match to include all characters comprising the string token, including quotes, brackets, etc, this regular expression should work:
The two relevant pieces are:
q'\[.*?\](?=')'is the basis for q-quoted strings, and the rest of the appropriate brackets have their own statements, and
'(?:[^']|'')*', matches single-quoted strings.
You can see matching examples here.