I want to write an Editor in my Eclipse Plugin that can handle the language of my tool. The language is based on C/C++ with additional keywords and annotations. I was successfull in writing my own Language class that extends GPPLanguage and make use of the according CDT extension point.
The main problem I have is that my language has keywords inside comments.
In this language something like this is often used:
/** @ctPrint
*
* This is a real comment, describing this block.
*
* @author Sadik // This looks like a comment, but it's part of the syntax. The compiler will treat this in a special way.
* @tag CT-001 // The @tag is part of syntax, the real comment starts with //
* @result TRUE // Again, @result is part of the syntax.
*
* This is a final description.
*/
So some parts inside the comments are not treated as comments by my compiler (extending g++). That's why I want to display those parts inside eclipse with highlighting.
As far as I understand from this answer is that the first one to become active during the processing of the raw source code is the Lexer. This is part of the documentation of CDT Lexer:
In short this class converts line endings (to '\n') and trigraphs
(to their corresponding character), removes line-splices, comments and whitespace other than newline. Returns preprocessor tokens.
So the Lexer throws parts of my source code away and doesn't tokenize it. Since the Lexer is declared final, it's not designed to extend it and reuse it's functionality.
If I have a parser that extends CDT's GNUCPPParser, that parser will fail to 'see' my comments, because there are no tokens for them. How can I have tokens for that part too?
While comment tokens are not handed to the parser in the usual way (e.g. they will not appear as a result of consume()), they are not discarded altogether: they are retained by the preprocessor, and made available as IASTComment nodes via IASTTranslationUnit.getComments(). You could use that to query the comments as a post-parsing step, parse their contents, and then highlight the source ranges corresponding to your special tokens.