Tokenizer
A guide to Fig's tokenizer.
User Input | Tokenized Output |
---|---|
Standardgit push origin main | ["git", "push", "origin", "main"] |
Quotes | |
Double Quotesgit commit -m "brendan's commit message" | ["git", "commit", "-m", "brendan's commit message"] |
Single Quotesgit commit -m 'my "commit" msg' | ['git', 'commit', '-m', 'my "commit" msg'] |
Escape Charactersecho hello\ world 1234 | ["echo", "hello world", "1234"] |
Options with Arguments | |
Long options--message="hello" and --message "hello" | ["--message", "hello"] |
Short options-m "hello" and -m="hello" | ["-m", "hello"] |
Short options with arguments not separated by = or spacegrep -C50 See exceptions to this below | ["grep", "-C" "-50"] |
Chained short options | |
Normal chained optionsps -aux | ["ps", "-a", "-u", "-x"] |
Chained options with argument at endgrep -abC555 → See exceptions to this below | ["grep", "-a", "-b", "-C", "555"] |
Settings You can add settings to Fig's completion spec to affect the tokenizer | |
Non posix compliant options posixNoncompliantFlags ps -aux | ["ps", "-aux"] |
Where Fig's Tokenizer breaks:
- Chained short options where the second option or after is not a lowercase or uppercase letter or the number
1
(1
is surprisingly a common flagls
)- e.g.
ls -ga@
will output["ls", "-g", "-a", "@"]
instead of["ls", "-g", "-a", "-@"]
- e.g.
ls -g%aG
will output["ls", "-g", "%aG"]
instead of["ls", "-g", "%", "a", "G"]
- BUT
ls -ga1
will work as expected because Fig consider 1 to be an optionls -% -@
will work as the@
and%
symbols are the first option, not the secondgit push -4 -6
will work as4
and6
are the first option in each token, not the second
- e.g.
- Short options that take argument that starts with a lowercase letter, uppercase letter, or 1
- e.g.
grep -C1
will output["grep", "-C", "-1"]
instead of["grep", "-C", "1"]
- e.g.
grep -C100
will output["grep", "-C", "-1", "00"]
instead of["grep", "-C", "100"]
- e.g.
git commit -mhello
will output["git", "commit", "-m", "h", "e", "l", "l", "o"]
instead of["git", "commit", "-m", "hello"]
- BUT
grep -C51
will work as the 5 will trigger the rest of the token to be considered an argument
- e.g.