cross-posted from: https://lemmy.world/post/15706364

Transparent compression layer on Linux?

My use-case: streaming video to a Linux mount and want compression of said video files on the fly.

Rclone has an experimental remote for compression but this stuff is important to me so that’s no good. I know rsync can do it but will it work for video files, and how I get rsync to warch the virtual mount-point and automatically compress and move over each individual file to rclone for upload to the Cloud? This is mostly to save on upload bandwidth and storage costs.

Thanks!

  • exu@feditown.comEnglish
    13·
    6 months ago

    You could use BTRFS, ZFS or BcacheFS to do compression on the filensystem level, but it’s not gonna compress video files or other already compressed media.

  • bizdelnick
    131·
    6 months ago

    Isn’t that video stream already compressed? Or you want to convert it using another codec/bitrate?

  • 𝒍𝒆𝒎𝒂𝒏𝒏@lemmy.dbzer0.com
    7·
    6 months ago

    I do something similar for my property’s exterior cameras, which are streamed to my VPS in ‘real-time’.

    You will need to re-encode the footage - videos are already pretty well compressed, so traditional compression methods like 7z (lzma), gz, zip etc being layered on top can’t compress them further.

    For your solution, I’d probably run a find every minute w/ cron to look for these files in a staging/watch folder, move them to another folder so they aren’t picked up on the next run, then re-encode with ffmpeg. Do note that when you re-encode footage, you always lose quality, even if you’re on a high quality preset.

    I have a feeling that the Handbrake project can do this with a watch folder though, so might be worth looking into that. After a quick search this looks easier to setup than my solution:

    https://github.com/HandBrake/HandBrake
    https://github.com/shannah/handbrake-watcher

    • MigratingtoLemmy@lemmy.worldOP
      1·
      6 months ago

      Thank you, this is exactly my use-case, along with some live streaming.

      However,

      1. How are you running a find command on a warch folder with continuously streaming video? I believe the IPC process has to end (system has to finish writing to a file) for post-processing (reencoding) to take place? That’s why I wanted to do this transparently, since I have no local storage to speak of to maintain a watch folder/cache.
      2. Thanks for the reencoding solution, I’ll take a look at handbrake