Custom IntelliSense system










0















For a personal project I am looking forward to building a custom IntelliSense system, the project in question is a simple code editor.



What i had some problems with is the IntelliSense aspect of the editor, and in specifics, the highlighting of the code. How does the system know which words are keywords?



A solution is adding them manually, but wouldnt that mean that for third party libraries, it wont work since the program doesnt have the functions from that library or API?



So in short, how is the IntelliSense code highlighting logic constructed, and is there an archive of some type on internet where i can get a language's keywords? Keep in mind that i want to build this code editor as a multi-language one (supports many languages).










share|improve this question






















  • All the editors I've ever used seem to handle syntax highlighting and code intellisense in completely different modules. Are you interested only in the former?

    – Álvaro González
    Nov 12 '18 at 12:46











  • Well what's the difference actually? Yes i'm mainly interested in the syntax highlighting at the moment, but later on i will need to make also autocomplete since thats part of the IntelliSense

    – Christian Panov
    Nov 12 '18 at 12:56











  • I don't really know why (I'm just a user). Perhaps it's for reusability and performance reasons. Syntax highlighting is relatively simple task that can be done with a simple tokenizer or even a regexp engine. Static analysis is an entirely different beast. If you're anyway interested in the latter you may want to check Language Server Protocol by Microsoft.

    – Álvaro González
    Nov 12 '18 at 13:49
















0















For a personal project I am looking forward to building a custom IntelliSense system, the project in question is a simple code editor.



What i had some problems with is the IntelliSense aspect of the editor, and in specifics, the highlighting of the code. How does the system know which words are keywords?



A solution is adding them manually, but wouldnt that mean that for third party libraries, it wont work since the program doesnt have the functions from that library or API?



So in short, how is the IntelliSense code highlighting logic constructed, and is there an archive of some type on internet where i can get a language's keywords? Keep in mind that i want to build this code editor as a multi-language one (supports many languages).










share|improve this question






















  • All the editors I've ever used seem to handle syntax highlighting and code intellisense in completely different modules. Are you interested only in the former?

    – Álvaro González
    Nov 12 '18 at 12:46











  • Well what's the difference actually? Yes i'm mainly interested in the syntax highlighting at the moment, but later on i will need to make also autocomplete since thats part of the IntelliSense

    – Christian Panov
    Nov 12 '18 at 12:56











  • I don't really know why (I'm just a user). Perhaps it's for reusability and performance reasons. Syntax highlighting is relatively simple task that can be done with a simple tokenizer or even a regexp engine. Static analysis is an entirely different beast. If you're anyway interested in the latter you may want to check Language Server Protocol by Microsoft.

    – Álvaro González
    Nov 12 '18 at 13:49














0












0








0








For a personal project I am looking forward to building a custom IntelliSense system, the project in question is a simple code editor.



What i had some problems with is the IntelliSense aspect of the editor, and in specifics, the highlighting of the code. How does the system know which words are keywords?



A solution is adding them manually, but wouldnt that mean that for third party libraries, it wont work since the program doesnt have the functions from that library or API?



So in short, how is the IntelliSense code highlighting logic constructed, and is there an archive of some type on internet where i can get a language's keywords? Keep in mind that i want to build this code editor as a multi-language one (supports many languages).










share|improve this question














For a personal project I am looking forward to building a custom IntelliSense system, the project in question is a simple code editor.



What i had some problems with is the IntelliSense aspect of the editor, and in specifics, the highlighting of the code. How does the system know which words are keywords?



A solution is adding them manually, but wouldnt that mean that for third party libraries, it wont work since the program doesnt have the functions from that library or API?



So in short, how is the IntelliSense code highlighting logic constructed, and is there an archive of some type on internet where i can get a language's keywords? Keep in mind that i want to build this code editor as a multi-language one (supports many languages).







intellisense






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked Nov 12 '18 at 12:17









Christian PanovChristian Panov

437




437












  • All the editors I've ever used seem to handle syntax highlighting and code intellisense in completely different modules. Are you interested only in the former?

    – Álvaro González
    Nov 12 '18 at 12:46











  • Well what's the difference actually? Yes i'm mainly interested in the syntax highlighting at the moment, but later on i will need to make also autocomplete since thats part of the IntelliSense

    – Christian Panov
    Nov 12 '18 at 12:56











  • I don't really know why (I'm just a user). Perhaps it's for reusability and performance reasons. Syntax highlighting is relatively simple task that can be done with a simple tokenizer or even a regexp engine. Static analysis is an entirely different beast. If you're anyway interested in the latter you may want to check Language Server Protocol by Microsoft.

    – Álvaro González
    Nov 12 '18 at 13:49


















  • All the editors I've ever used seem to handle syntax highlighting and code intellisense in completely different modules. Are you interested only in the former?

    – Álvaro González
    Nov 12 '18 at 12:46











  • Well what's the difference actually? Yes i'm mainly interested in the syntax highlighting at the moment, but later on i will need to make also autocomplete since thats part of the IntelliSense

    – Christian Panov
    Nov 12 '18 at 12:56











  • I don't really know why (I'm just a user). Perhaps it's for reusability and performance reasons. Syntax highlighting is relatively simple task that can be done with a simple tokenizer or even a regexp engine. Static analysis is an entirely different beast. If you're anyway interested in the latter you may want to check Language Server Protocol by Microsoft.

    – Álvaro González
    Nov 12 '18 at 13:49

















All the editors I've ever used seem to handle syntax highlighting and code intellisense in completely different modules. Are you interested only in the former?

– Álvaro González
Nov 12 '18 at 12:46





All the editors I've ever used seem to handle syntax highlighting and code intellisense in completely different modules. Are you interested only in the former?

– Álvaro González
Nov 12 '18 at 12:46













Well what's the difference actually? Yes i'm mainly interested in the syntax highlighting at the moment, but later on i will need to make also autocomplete since thats part of the IntelliSense

– Christian Panov
Nov 12 '18 at 12:56





Well what's the difference actually? Yes i'm mainly interested in the syntax highlighting at the moment, but later on i will need to make also autocomplete since thats part of the IntelliSense

– Christian Panov
Nov 12 '18 at 12:56













I don't really know why (I'm just a user). Perhaps it's for reusability and performance reasons. Syntax highlighting is relatively simple task that can be done with a simple tokenizer or even a regexp engine. Static analysis is an entirely different beast. If you're anyway interested in the latter you may want to check Language Server Protocol by Microsoft.

– Álvaro González
Nov 12 '18 at 13:49






I don't really know why (I'm just a user). Perhaps it's for reusability and performance reasons. Syntax highlighting is relatively simple task that can be done with a simple tokenizer or even a regexp engine. Static analysis is an entirely different beast. If you're anyway interested in the latter you may want to check Language Server Protocol by Microsoft.

– Álvaro González
Nov 12 '18 at 13:49













1 Answer
1






active

oldest

votes


















0














Building decent support for even one non-trivial language is a lot of work. Consider using a library that will do it for you. Most of these will not be free, but are probably the way to go, unless the point of this exercise is you learning compiler theory.



An example of such a library is Microsoft's C# compiler (Roslyn). It's available as a free open source library that you can use in other .NET applications. You give it some code as input (as a string), and it tells you everything there is to know about that piece of code: where symbols and keywords are, what each symbol is, what to offer for autocomplete at which point, where compile-time errors in the code are, etc...



If you want to learn how to build something like this yourself, I suggest getting a compiler theory book and setting aside a few months. While it's not rocket science, it's not trivial either.



Once you have a basic grasp of lexers and parsers, then you'll probably want to use a parser/lexer generator like Antlr to do the tedious work of building a lexer and parser. After that, you still have a lot of work to process the syntax tree that the parser gives you. If you're building autocomplete (intellisense) keep in mind that while the user is typing, the code that they're typing is probably not valid, which might throw the parser off and result in bad autocomplete suggestions. As I said, it's a lot of work to do this right.



That said, if syntax highlighting is all you need, you can probably use an editor that already has it. Depending on the technology you're using, you might be able to make use Scintilla or AvalonEdit. You won't get intellisense (autocomplete) or error highlighting or any other features that depend on understanding the meaning of the code (semantic analysis), though, but you might get half-decent syntax highlighting.






share|improve this answer

























  • Thank you for the detailed answer, things got much clearer now. Only a couple more questions. The editor i am developing is in C# WPF, so AvalonEdit seems like the perfect choice. If you are familiar with it, any thoughts on it about performance, does it implement this compiler theory etc.?

    – Christian Panov
    Nov 12 '18 at 12:58











  • Yep, AvalonEdit is a really good choice if you're in .NET. It has support for syntax highlighting for ~20 languages. The syntax highlighting is based on regular expressions, so it's rather basic. It does not understand anything about the user's code, so it will not give you autocomplete, error squiggles, symbol tool-tips etc., you'd have to have a separate library/ies to parse the user's code so you can offer meaningful autocomplete, error highlighting etc...

    – anakic
    Nov 12 '18 at 13:23












  • Okay, thank you very much. But what would a complex syntax highlighter mean if the regex method is rather basic? I imagine it's still a very good solution, especially if you expand on it, but still out of curiosity.

    – Christian Panov
    Nov 12 '18 at 13:31











  • A complex syntax highlighter will never be coded in Regex. it would use a compiler frontend : Lexer and Parser to determine symbols etc. At this point you can then expand and add Type tooltips, squiggles etc

    – Theophile Dano
    Nov 12 '18 at 14:09










Your Answer






StackExchange.ifUsing("editor", function ()
StackExchange.using("externalEditor", function ()
StackExchange.using("snippets", function ()
StackExchange.snippets.init();
);
);
, "code-snippets");

StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "1"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53262028%2fcustom-intellisense-system%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









0














Building decent support for even one non-trivial language is a lot of work. Consider using a library that will do it for you. Most of these will not be free, but are probably the way to go, unless the point of this exercise is you learning compiler theory.



An example of such a library is Microsoft's C# compiler (Roslyn). It's available as a free open source library that you can use in other .NET applications. You give it some code as input (as a string), and it tells you everything there is to know about that piece of code: where symbols and keywords are, what each symbol is, what to offer for autocomplete at which point, where compile-time errors in the code are, etc...



If you want to learn how to build something like this yourself, I suggest getting a compiler theory book and setting aside a few months. While it's not rocket science, it's not trivial either.



Once you have a basic grasp of lexers and parsers, then you'll probably want to use a parser/lexer generator like Antlr to do the tedious work of building a lexer and parser. After that, you still have a lot of work to process the syntax tree that the parser gives you. If you're building autocomplete (intellisense) keep in mind that while the user is typing, the code that they're typing is probably not valid, which might throw the parser off and result in bad autocomplete suggestions. As I said, it's a lot of work to do this right.



That said, if syntax highlighting is all you need, you can probably use an editor that already has it. Depending on the technology you're using, you might be able to make use Scintilla or AvalonEdit. You won't get intellisense (autocomplete) or error highlighting or any other features that depend on understanding the meaning of the code (semantic analysis), though, but you might get half-decent syntax highlighting.






share|improve this answer

























  • Thank you for the detailed answer, things got much clearer now. Only a couple more questions. The editor i am developing is in C# WPF, so AvalonEdit seems like the perfect choice. If you are familiar with it, any thoughts on it about performance, does it implement this compiler theory etc.?

    – Christian Panov
    Nov 12 '18 at 12:58











  • Yep, AvalonEdit is a really good choice if you're in .NET. It has support for syntax highlighting for ~20 languages. The syntax highlighting is based on regular expressions, so it's rather basic. It does not understand anything about the user's code, so it will not give you autocomplete, error squiggles, symbol tool-tips etc., you'd have to have a separate library/ies to parse the user's code so you can offer meaningful autocomplete, error highlighting etc...

    – anakic
    Nov 12 '18 at 13:23












  • Okay, thank you very much. But what would a complex syntax highlighter mean if the regex method is rather basic? I imagine it's still a very good solution, especially if you expand on it, but still out of curiosity.

    – Christian Panov
    Nov 12 '18 at 13:31











  • A complex syntax highlighter will never be coded in Regex. it would use a compiler frontend : Lexer and Parser to determine symbols etc. At this point you can then expand and add Type tooltips, squiggles etc

    – Theophile Dano
    Nov 12 '18 at 14:09















0














Building decent support for even one non-trivial language is a lot of work. Consider using a library that will do it for you. Most of these will not be free, but are probably the way to go, unless the point of this exercise is you learning compiler theory.



An example of such a library is Microsoft's C# compiler (Roslyn). It's available as a free open source library that you can use in other .NET applications. You give it some code as input (as a string), and it tells you everything there is to know about that piece of code: where symbols and keywords are, what each symbol is, what to offer for autocomplete at which point, where compile-time errors in the code are, etc...



If you want to learn how to build something like this yourself, I suggest getting a compiler theory book and setting aside a few months. While it's not rocket science, it's not trivial either.



Once you have a basic grasp of lexers and parsers, then you'll probably want to use a parser/lexer generator like Antlr to do the tedious work of building a lexer and parser. After that, you still have a lot of work to process the syntax tree that the parser gives you. If you're building autocomplete (intellisense) keep in mind that while the user is typing, the code that they're typing is probably not valid, which might throw the parser off and result in bad autocomplete suggestions. As I said, it's a lot of work to do this right.



That said, if syntax highlighting is all you need, you can probably use an editor that already has it. Depending on the technology you're using, you might be able to make use Scintilla or AvalonEdit. You won't get intellisense (autocomplete) or error highlighting or any other features that depend on understanding the meaning of the code (semantic analysis), though, but you might get half-decent syntax highlighting.






share|improve this answer

























  • Thank you for the detailed answer, things got much clearer now. Only a couple more questions. The editor i am developing is in C# WPF, so AvalonEdit seems like the perfect choice. If you are familiar with it, any thoughts on it about performance, does it implement this compiler theory etc.?

    – Christian Panov
    Nov 12 '18 at 12:58











  • Yep, AvalonEdit is a really good choice if you're in .NET. It has support for syntax highlighting for ~20 languages. The syntax highlighting is based on regular expressions, so it's rather basic. It does not understand anything about the user's code, so it will not give you autocomplete, error squiggles, symbol tool-tips etc., you'd have to have a separate library/ies to parse the user's code so you can offer meaningful autocomplete, error highlighting etc...

    – anakic
    Nov 12 '18 at 13:23












  • Okay, thank you very much. But what would a complex syntax highlighter mean if the regex method is rather basic? I imagine it's still a very good solution, especially if you expand on it, but still out of curiosity.

    – Christian Panov
    Nov 12 '18 at 13:31











  • A complex syntax highlighter will never be coded in Regex. it would use a compiler frontend : Lexer and Parser to determine symbols etc. At this point you can then expand and add Type tooltips, squiggles etc

    – Theophile Dano
    Nov 12 '18 at 14:09













0












0








0







Building decent support for even one non-trivial language is a lot of work. Consider using a library that will do it for you. Most of these will not be free, but are probably the way to go, unless the point of this exercise is you learning compiler theory.



An example of such a library is Microsoft's C# compiler (Roslyn). It's available as a free open source library that you can use in other .NET applications. You give it some code as input (as a string), and it tells you everything there is to know about that piece of code: where symbols and keywords are, what each symbol is, what to offer for autocomplete at which point, where compile-time errors in the code are, etc...



If you want to learn how to build something like this yourself, I suggest getting a compiler theory book and setting aside a few months. While it's not rocket science, it's not trivial either.



Once you have a basic grasp of lexers and parsers, then you'll probably want to use a parser/lexer generator like Antlr to do the tedious work of building a lexer and parser. After that, you still have a lot of work to process the syntax tree that the parser gives you. If you're building autocomplete (intellisense) keep in mind that while the user is typing, the code that they're typing is probably not valid, which might throw the parser off and result in bad autocomplete suggestions. As I said, it's a lot of work to do this right.



That said, if syntax highlighting is all you need, you can probably use an editor that already has it. Depending on the technology you're using, you might be able to make use Scintilla or AvalonEdit. You won't get intellisense (autocomplete) or error highlighting or any other features that depend on understanding the meaning of the code (semantic analysis), though, but you might get half-decent syntax highlighting.






share|improve this answer















Building decent support for even one non-trivial language is a lot of work. Consider using a library that will do it for you. Most of these will not be free, but are probably the way to go, unless the point of this exercise is you learning compiler theory.



An example of such a library is Microsoft's C# compiler (Roslyn). It's available as a free open source library that you can use in other .NET applications. You give it some code as input (as a string), and it tells you everything there is to know about that piece of code: where symbols and keywords are, what each symbol is, what to offer for autocomplete at which point, where compile-time errors in the code are, etc...



If you want to learn how to build something like this yourself, I suggest getting a compiler theory book and setting aside a few months. While it's not rocket science, it's not trivial either.



Once you have a basic grasp of lexers and parsers, then you'll probably want to use a parser/lexer generator like Antlr to do the tedious work of building a lexer and parser. After that, you still have a lot of work to process the syntax tree that the parser gives you. If you're building autocomplete (intellisense) keep in mind that while the user is typing, the code that they're typing is probably not valid, which might throw the parser off and result in bad autocomplete suggestions. As I said, it's a lot of work to do this right.



That said, if syntax highlighting is all you need, you can probably use an editor that already has it. Depending on the technology you're using, you might be able to make use Scintilla or AvalonEdit. You won't get intellisense (autocomplete) or error highlighting or any other features that depend on understanding the meaning of the code (semantic analysis), though, but you might get half-decent syntax highlighting.







share|improve this answer














share|improve this answer



share|improve this answer








edited Nov 12 '18 at 12:44

























answered Nov 12 '18 at 12:38









anakicanakic

1,26411924




1,26411924












  • Thank you for the detailed answer, things got much clearer now. Only a couple more questions. The editor i am developing is in C# WPF, so AvalonEdit seems like the perfect choice. If you are familiar with it, any thoughts on it about performance, does it implement this compiler theory etc.?

    – Christian Panov
    Nov 12 '18 at 12:58











  • Yep, AvalonEdit is a really good choice if you're in .NET. It has support for syntax highlighting for ~20 languages. The syntax highlighting is based on regular expressions, so it's rather basic. It does not understand anything about the user's code, so it will not give you autocomplete, error squiggles, symbol tool-tips etc., you'd have to have a separate library/ies to parse the user's code so you can offer meaningful autocomplete, error highlighting etc...

    – anakic
    Nov 12 '18 at 13:23












  • Okay, thank you very much. But what would a complex syntax highlighter mean if the regex method is rather basic? I imagine it's still a very good solution, especially if you expand on it, but still out of curiosity.

    – Christian Panov
    Nov 12 '18 at 13:31











  • A complex syntax highlighter will never be coded in Regex. it would use a compiler frontend : Lexer and Parser to determine symbols etc. At this point you can then expand and add Type tooltips, squiggles etc

    – Theophile Dano
    Nov 12 '18 at 14:09

















  • Thank you for the detailed answer, things got much clearer now. Only a couple more questions. The editor i am developing is in C# WPF, so AvalonEdit seems like the perfect choice. If you are familiar with it, any thoughts on it about performance, does it implement this compiler theory etc.?

    – Christian Panov
    Nov 12 '18 at 12:58











  • Yep, AvalonEdit is a really good choice if you're in .NET. It has support for syntax highlighting for ~20 languages. The syntax highlighting is based on regular expressions, so it's rather basic. It does not understand anything about the user's code, so it will not give you autocomplete, error squiggles, symbol tool-tips etc., you'd have to have a separate library/ies to parse the user's code so you can offer meaningful autocomplete, error highlighting etc...

    – anakic
    Nov 12 '18 at 13:23












  • Okay, thank you very much. But what would a complex syntax highlighter mean if the regex method is rather basic? I imagine it's still a very good solution, especially if you expand on it, but still out of curiosity.

    – Christian Panov
    Nov 12 '18 at 13:31











  • A complex syntax highlighter will never be coded in Regex. it would use a compiler frontend : Lexer and Parser to determine symbols etc. At this point you can then expand and add Type tooltips, squiggles etc

    – Theophile Dano
    Nov 12 '18 at 14:09
















Thank you for the detailed answer, things got much clearer now. Only a couple more questions. The editor i am developing is in C# WPF, so AvalonEdit seems like the perfect choice. If you are familiar with it, any thoughts on it about performance, does it implement this compiler theory etc.?

– Christian Panov
Nov 12 '18 at 12:58





Thank you for the detailed answer, things got much clearer now. Only a couple more questions. The editor i am developing is in C# WPF, so AvalonEdit seems like the perfect choice. If you are familiar with it, any thoughts on it about performance, does it implement this compiler theory etc.?

– Christian Panov
Nov 12 '18 at 12:58













Yep, AvalonEdit is a really good choice if you're in .NET. It has support for syntax highlighting for ~20 languages. The syntax highlighting is based on regular expressions, so it's rather basic. It does not understand anything about the user's code, so it will not give you autocomplete, error squiggles, symbol tool-tips etc., you'd have to have a separate library/ies to parse the user's code so you can offer meaningful autocomplete, error highlighting etc...

– anakic
Nov 12 '18 at 13:23






Yep, AvalonEdit is a really good choice if you're in .NET. It has support for syntax highlighting for ~20 languages. The syntax highlighting is based on regular expressions, so it's rather basic. It does not understand anything about the user's code, so it will not give you autocomplete, error squiggles, symbol tool-tips etc., you'd have to have a separate library/ies to parse the user's code so you can offer meaningful autocomplete, error highlighting etc...

– anakic
Nov 12 '18 at 13:23














Okay, thank you very much. But what would a complex syntax highlighter mean if the regex method is rather basic? I imagine it's still a very good solution, especially if you expand on it, but still out of curiosity.

– Christian Panov
Nov 12 '18 at 13:31





Okay, thank you very much. But what would a complex syntax highlighter mean if the regex method is rather basic? I imagine it's still a very good solution, especially if you expand on it, but still out of curiosity.

– Christian Panov
Nov 12 '18 at 13:31













A complex syntax highlighter will never be coded in Regex. it would use a compiler frontend : Lexer and Parser to determine symbols etc. At this point you can then expand and add Type tooltips, squiggles etc

– Theophile Dano
Nov 12 '18 at 14:09





A complex syntax highlighter will never be coded in Regex. it would use a compiler frontend : Lexer and Parser to determine symbols etc. At this point you can then expand and add Type tooltips, squiggles etc

– Theophile Dano
Nov 12 '18 at 14:09

















draft saved

draft discarded
















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53262028%2fcustom-intellisense-system%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Use pre created SQLite database for Android project in kotlin

Darth Vader #20

Ondo