[ML] refactoring internal tokenization logic for NLP (#83835)

This simplifies the internal logic used to pass tokenization results around while streamlining building the request sent to the model.

This helps lay some of the ground work for windowing as collapsing request building && token results will be required (as a single sequence could result in a batch request).

Additionally, many of the intellij warnings are addressed and code is modernized (i.e. taking advantage of records)