In many cases, AutoTokenizer will return the result of token_type_ids:
Explanation of token_type_ids:
For two sentence pairs, the previous sentence is marked as 0, and the next sentence is marked as 1.
In many cases, AutoTokenizer will return the result of token_type_ids:
Explanation of token_type_ids:
For two sentence pairs, the previous sentence is marked as 0, and the next sentence is marked as 1.