Quote:

In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities.

Related:

  • 𝕊𝕚𝕤𝕪𝕡𝕙𝕖𝕒𝕟@programming.devOPMEnglish
    2·
    1 year ago

    Has anyone else tried these models? I find them very impressive. Here is a completion I got from the 1M one (prompt in bold):

    Once upon a time, there was a little girl called Anne. She was three years old and loved to play outside. One day, Anne was playing in the garden when she saw a big, shiny object. She wanted to pick it up, but it was too high up.

    This is surprisingly coherent coming from a model with only 1 million parameters (GPT-3.5 has 175 billion). Unfortunately, I couldn’t generate more text after this (“No text was generated”). I’m not really familiar with Hugging Face or how these models work but it would be interesting to experiment with it more.