Searching in extension store turned out nothing, but I found something in their master branch repo:
https://github.com/Microsoft/vscode/tree/master/extensions/groovy
But not quite sure if this is the extension that I want, if it is, do I just extract this into my .vscode/extensions/ folder?
The extension you linked is built-in, meaning the Change Language Mode command should already list Groovy for you:
It provides syntax highlighting and snippets, see the "contributes" section of the package.json.
However, I'm guessing you want something that provides more than just highlighting and snippets (i.e. completion). Interestingly enough, there does appear to be a language server implementation for Groovy (found on langserver.org), but I couldn't find a matching VSCode extension. Documentation seems a bit lacking, but after searching the repo's issues for a bit, it seems like there's no VSCode integration yet:
It's not integrated with VS code yet. Currently we are using it with just Monaco and some glue code which is not open sourced.
In theory, VSCode integration shouldn't be too hard if the language server spec was followed. There's some documentation / examples on language server development here.