Question

what is tokonizatin In atomata
​

Answer 1

Answer:

Tokenization is the act of breaking up a sequence of strings into pieces such as words, keywords, phrases, symbols and other elements called tokens. ... In the process of tokenization, some characters like punctuation marks are discarded. The tokens become the input for another process like parsing and text mining.

Explanation:

Thanks dear hope it's helpful for you