Tokenizer 정리

참고 링크 : HuggingFace https://huggingface.co/docs/transformers/tokenizer_summary SentencePiece Tokenizer Demystified , towardsdatascience https://towardsdatascience.com/sentencepiece-tokenizer-demystified-d0a3aac19b15 HuggingFace Tokenizers https://huggingface.co/docs/tokenizers/index https://huggingface.co/docs/tokenizers/pipeline Unicode Normalization https://towardsdatascience.com/what-on-earth-is-unicode-normalization-56c005c55ad0 관련 링크 : Unicode equivalence , Wikipedia https://en.wikipedia....