Tokenizer
A guide to Fig's tokenizer.
| User Input | Tokenized Output |
|---|---|
Standardgit push origin main | ["git", "push", "origin", "main"] |
| Quotes | |
Double Quotesgit commit -m "brendan's commit message" | ["git", "commit", "-m", "brendan's commit message"] |
Single Quotesgit commit -m 'my "commit" msg' | ['git', 'commit', '-m', 'my "commit" msg'] |
Escape Charactersecho hello\ world 1234 | ["echo", "hello world", "1234"] |
| Options with Arguments | |
Long options--message="hello" and --message "hello" | ["--message", "hello"] |
Short options-m "hello" and -m="hello" | ["-m", "hello"] |
Short options with arguments not separated by = or spacegrep -C50 See exceptions to this below | ["grep", "-C" "-50"] |
| Chained short options | |
Normal chained optionsps -aux | ["ps", "-a", "-u", "-x"] |
Chained options with argument at endgrep -abC555 → See exceptions to this below | ["grep", "-a", "-b", "-C", "555"] |
| Settings You can add settings to Fig's completion spec to affect the tokenizer | |
Non posix compliant options posixNoncompliantFlagsps -aux | ["ps", "-aux"] |
Where Fig's Tokenizer breaks:
- Chained short options where the second option or after is not a lowercase or uppercase letter or the number
1(1is surprisingly a common flagls)- e.g.
ls -ga@will output["ls", "-g", "-a", "@"]instead of["ls", "-g", "-a", "-@"] - e.g.
ls -g%aGwill output["ls", "-g", "%aG"]instead of["ls", "-g", "%", "a", "G"] - BUT
ls -ga1will work as expected because Fig consider 1 to be an optionls -% -@will work as the@and%symbols are the first option, not the secondgit push -4 -6will work as4and6are the first option in each token, not the second
- e.g.
- Short options that take argument that starts with a lowercase letter, uppercase letter, or 1
- e.g.
grep -C1will output["grep", "-C", "-1"]instead of["grep", "-C", "1"] - e.g.
grep -C100will output["grep", "-C", "-1", "00"]instead of["grep", "-C", "100"] - e.g.
git commit -mhellowill output["git", "commit", "-m", "h", "e", "l", "l", "o"]instead of["git", "commit", "-m", "hello"] - BUT
grep -C51will work as the 5 will trigger the rest of the token to be considered an argument
- e.g.