we explore how it makes inference 10x faster plus we offer the best curated list of the freshest ML news and papers
  • Finadil@lemmy.worldEnglish
    1·
    9 months ago

    That with a fp16 model? Don’t be scared to try even a 4 bit quantization, you’d be surprised at how little is lost and how much quicker it is.