Home »
MCQs »
Generative AI MCQs
'Tokenization' in Natural Language Processing helps ____?
25. 'Tokenization' in Natural Language Processing helps ____?
- In encoding the data
- In creating token for transfer over network
- Breaking down text into smaller units for processing
- None of the above
Answer
The correct answer is: C) Breaking down text into smaller units for processing
Explanation
'Tokenization' in Natural Language Processing helps in breaking down text into smaller units like words, phrases. It helps generative models to understand and process text efficiently.