• Dr Cog
    link
    fedilink
    arrow-up
    2
    ·
    1 year ago

    Tokens are usually never multiple words. Think of them like “information units”. If you have a plural word “hats”, there are two tokens: the “hat” one and the "s’ that adds more information about it being plural. Combinations of words only really occur for proper names.