How to handle duplicate data entries

[jennyzhang-petuum]

Is your feature request related to a problem? Please describe.
When we choose to perform tokenize for multiple processors in the same pipeline, there would be the same entry type generated for the result which is ft.onto.base_ontology.Token. This would result in duplicate data. For example, the processor later in the pipeline would get access to these mixed entries of data for counting or prediction and get the confused result.

Describe the solution you'd like
Multiple solutions could help with solving this problem
The current solution in consideration is to create subclass ontology to diferentiate between these entries. For example, to create ontology entries of Token subclass separately for multiple processors that perform tokenize.

Describe alternatives you've considered
Create another field entry record in the datapack file for separation but it's a bit bulky for the datapack file.

[hunterhector]

The same problem happens for all entry types, for example, duplicated Sentence. So we can use the same idea to handle all of them.