I’m looking at starting a service that involves hosting a lot of LLM models, which are often going to be 16GB+ (compressed). I did a bit of searching for cloud storage providers with cheap egress, and the cheapest I could find is $0.01 per GB, which would still be $0.16+ per download.

How do sites like Huggingface or CivitAI do it? Lots of VC funding?

  • Akisamb@programming.dev
    link
    fedilink
    English
    arrow-up
    9
    ·
    1 year ago

    R2 by cloudfare does not charge egress costs. It’s 0.015$/GB/month for storage. Read operations are 0.36$ per million.

    I do have a hard time believing that they will remain this cheap though, but who knows.

  • Oliver Lowe@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    7
    ·
    1 year ago

    If the files are not going to be changing much, then what is typically done is to use a CDN service (e.g. Cloudflare, Akamai, Fastly). The idea is you have an “origin” which could be any old server which serves your files over HTTP (even a VPS running nginx). The CDN is configured to proxy requests to the origin, building up a cache of the files it serves. The CDN can serve files from cache on their own (very large) infrastructure. See also What is a CDN?

    • Oliver Lowe@lemmy.sdf.org
      link
      fedilink
      English
      arrow-up
      5
      ·
      edit-2
      1 year ago

      So I got curious and wondered how HuggingFace hosts their files. It’s AWS CloudFront:

      $ curl -LI https://huggingface.co/facebook/musicgen-large/resolve/main/state_dict.bin
      
      HTTP/2 302 
      content-type: text/plain; charset=utf-8
      content-length: 1198
      location: https://cdn-lfs.huggingface.co/repos/81/...
      date: Fri, 23 Jun 2023 07:09:39 GMT
      x-powered-by: huggingface-moon
      x-request-id: Root=1-64954533-1eb79eed4ea500203f6435cb
      access-control-allow-origin: https://huggingface.co
      vary: Origin, Accept
      access-control-expose-headers: X-Repo-Commit,X-Request-Id,X-Error-Code,X-Error-Message,ETag,Link,Accept-Ranges,Content-Range
      x-repo-commit: c19300a6b2b62d29b345ae9eb7b163278e65238a
      accept-ranges: bytes
      x-linked-size: 6514108999
      x-linked-etag: "1f0cf17b5e65c5dd8ba71767371c377f174e4ce1db44bc4d6657825769f26ffd"
      x-cache: Miss from cloudfront
      via: 1.1 65c7d0c3355767ac8658c2122c8280b6.cloudfront.net (CloudFront)
      x-amz-cf-pop: SYD1-C1
      x-amz-cf-id: XAU8X4yneUeudylCi_9MeAYZmISCr8OHiBcgjAGcBQT-edrBF6wGCA==
      
      HTTP/2 200 
      content-type: application/octet-stream
      content-length: 6514108999
      date: Fri, 23 Jun 2023 07:06:20 GMT
      last-modified: Thu, 08 Jun 2023 19:05:02 GMT
      etag: "44ef1b51c0cc2200e29fed5cddbf8e27-408"
      x-amz-storage-class: INTELLIGENT_TIERING
      x-amz-server-side-encryption: AES256
      x-amz-version-id: upVN9_QvGmQZMDWfVECqbytHgWjHTz4t
      content-disposition: attachment; filename*=UTF-8''state_dict.bin; filename="state_dict.bin";
      accept-ranges: bytes
      server: AmazonS3
      x-cache: Hit from cloudfront
      via: 1.1 dff3fc94ddb54b32b708edf2668b23d2.cloudfront.net (CloudFront)
      x-amz-cf-pop: SIN52-P1
      x-amz-cf-id: JWTqKUDp8bVPD7Tt4DBySj8zbgT8G60sCA5BMC1cYF4vK9nx45f1Iw==
      age: 200
      vary: Origin
      
    • minorninth@lemmy.world
      link
      fedilink
      English
      arrow-up
      0
      ·
      1 year ago

      That’s true, but I just checked a few CDNs and you won’t find one for less than $.01/GB. The lowest I found was $.03/GB.

        • omegastickOP
          link
          fedilink
          English
          arrow-up
          3
          ·
          1 year ago

          I was considering this. The hosting provider we use for model training runs doesn’t charge for ingress/egress. Their storage costs would eat us alive though haha. OVH looks much more promising.

  • jeff 👨‍💻@programming.dev
    link
    fedilink
    English
    arrow-up
    6
    ·
    edit-2
    1 year ago

    Lots of VC funding?

    Probably. They might have gotten additional discounts off of the advertized price by talking with sales and committing to the service for a year or other ways.

    • omegastickOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      1 year ago

      Interesting, I’ll have to have a look at doing something like that. If I remember correctly, the CivitAI devs are active on Discord, so maybe I could just ask them directly.

      • jeff 👨‍💻@programming.dev
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 year ago

        There’s no harm in messaging the sales team directly. Whatever deals CivitAI got might not work for you, they might not even be legally allowed to mention specifics. But you can shop around and see what different providers will offer you.

  • minorninth@lemmy.world
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    Would BitTorrent be an option?

    There are also ad-supported download sites.

    You could offer users a choice: BitTorrent, ads, or $1 for an instant download.

    • omegastickOP
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      We’d be offering a service training models for users, so I don’t see bittorrent working. Each file would be unique to the user and probably only downloaded once. We might be able to give free downloads of a LoRA (kind of a very compressed “diff” of the training run), along with tools to download and merge it with the base model from Huggingface, then charge for a download of the full model maybe.

  • liori@lemm.ee
    link
    fedilink
    English
    arrow-up
    2
    ·
    edit-2
    1 year ago

    Storj does it at 7 USD/TB. And there are providers that technically provide unlimited bandwidth, like Hetzner’s dedicated servers; they still have some abuse limits, but even working within the limits should make it much cheaper. This means custom engineering though.

  • pe1uca@lemmy.pe1uca.dev
    link
    fedilink
    English
    arrow-up
    1
    ·
    1 year ago

    Is this for VPS instance bandwidth?
    Does S3 also counts towards the bandwidth? Or is it independent?