pull down to refresh
Tokenizer makes tokens from text. This says: convert your stuff to this first, then feed to a tokenizer. The examples are "converted" to their XML thingy. Hence, make it a feature of the tokenizer and don't bother people with conversions. I'm sure GPT could have told them this too.
reply
Why would that even be better? Isn’t the whole point to cut down on token costs? That’s an ignorant question! ~lol