I just lost 44 GB of data that I downloaded in the past two hours. My PC started to get laggy and it got stuck so I just shut it down believing that all will be well since the data has already been written to disk…

  • kaitco@lemmy.world
    link
    fedilink
    English
    arrow-up
    51
    arrow-down
    2
    ·
    1 year ago

    That’s not how PCs work. Once you’ve downloaded the data, it’s on your hard drive, not the RAM. This sounds very much like you’ve experienced hard drive failure.

    • AProfessional@lemmy.world
      link
      fedilink
      English
      arrow-up
      19
      arrow-down
      1
      ·
      1 year ago

      This is incorrect too. The OS buffers writes to drives for performance, a portion is kept in memory and flushed to disk when possible. A sudden power loss can easily result in a partial write.

      That said a drive failure is also possible.

      The filesystems journal will be the source of truth.

      • kaitco@lemmy.world
        link
        fedilink
        English
        arrow-up
        19
        ·
        1 year ago

        44 GB over two hours, though? If it were the last couple GBs in the last maybe 10 ten minutes, but unless the OP is running a PC from 2002, the data should have already moved from RAM to disk in that time.

        But yeah, the filesystem journal will explain what happened there.

      • Yendor@reddthat.com
        link
        fedilink
        English
        arrow-up
        9
        ·
        1 year ago

        The buffer is flushed every 6 seconds. OP was pulling down 44GB/7200sec = 6MB/s. OP would have only lost 36MB to disk caching.

        If it was a single 44GB file and OP turned off less than 6 seconds after it finished, then it could have been caching. But that’s very unlikely.

    • Swordman5@lemmy.world
      link
      fedilink
      English
      arrow-up
      17
      ·
      edit-2
      1 year ago

      Well, depending on the drive and if this 40gb of loss was one file and how quickly he shut down after saving the data, it actually is possible. Some drives will use write cache to speed up the perception of the write speeds of a drive where new data is very quickly written to a faster cache, then is transferred a little more slowly to permanent storage. But this write cache isn’t always power loss protected. If you do a normal shut down, the computer waits until any data on the write cache gets transferred to permanent storage before it fully shuts down. If you just nix the power though, that data could be gone, and if it was part of a larger file, it would corrupt that file.

      Edit: Here’s a source to back up my info. Though, it looks like it may actually be an os feature instead of a drive feature. https://www.iolosystem.com/resources/disk-write-caching.html

    • Duckman@lemmy.world
      link
      fedilink
      English
      arrow-up
      4
      ·
      1 year ago

      Mem cache is definitely still a thing. Non-volatile storage has gotten faster in recent years but it’s still not as fast as RAM. Depending how his system is configured, data loss is definitely possible.

      That being said, unless it was one big file and he lost critical data that made the file readable (say an MBR on a disk image) there’s no way he should have lost 44GB.

    • sourcepie@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      1
      ·
      edit-2
      1 year ago

      It actually might be, because now a magic file appeared on my hard drive (a file that was once deleted) that I can’t delete. When I try to delete it, even with admin rights, it says No such file or directory

  • drspod
    link
    fedilink
    English
    arrow-up
    12
    arrow-down
    1
    ·
    1 year ago

    What filesystem are you using? Filesystems should be crash consistent these days.

    • AProfessional@lemmy.world
      link
      fedilink
      English
      arrow-up
      5
      ·
      1 year ago

      A modern filesystem cannot make magic happen, if the data was in the process of being written it is still in memory and lost.

      Only copy-on-write filesystems are safe here as they don’t overwrite old data. NTFS is not safe.

      • drspod
        link
        fedilink
        English
        arrow-up
        6
        arrow-down
        1
        ·
        1 year ago

        If 44GB had not been flushed to disk yet then that implies that OP has >44GB of RAM in use as page cache. SSD write caches are only around 2GB while HDD write caches are not more than about 512MB. I don’t know how much RAM OP has but this seems unlikely to be the case.

        I would bet on one of the following:

        • the transfer was still in progress when the system was halted
        • the transfer was not a network download but actually a copy from another locally mounted partition and the OS was still transferring data
        • the filesystem driver did write that 44GB to inodes but didn’t link the inodes into the filesystem yet, in which case the filesystem’s fsck tool should be able to recover data blocks that were already written
        • OP is using a filesystem which does not use either journaling or COW (eg. FAT32) in which case the data which was written but not linked into the filesystem is lost as orphaned inodes.

        3 & 4 being the reason I was asking what filesystem OP is using.

        Only copy-on-write filesystems are safe here as they don’t overwrite old data.

        COW filesystems make no difference in this case since this was not data overwriting an existing file. They still have the same behavior with write caching in the OS and on the drive and still require a tool (eg. btrfs-check) to recover unlinked changes in the event of a crash.

        NTFS is not safe.

        NTFS is a journaling filesystem and is crash-consistent. That doesn’t mean it can do “magic” if the data hasn’t hit the disk, but it does mean that it won’t corrupt the filesystem or lose data which was already written in the case of a crash.

    • wmassingham@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      ·
      1 year ago

      Wait it out, or fix whatever is making it “laggy”, like a program using all the CPU time.

      And OP can probably recover that data with standard recovery tools, if they haven’t continued using the PC. Or just redownload it, since recovery would take longer than two hours.

      • sourcepie@lemmy.worldOP
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 year ago

        The 44 GB of data that got lost consisted of 191 files each around ~220 MiB in one single folder. I used testdisk to inspect my drive and I couldn’t find the data…

  • djsaskdja@reddthat.com
    link
    fedilink
    English
    arrow-up
    2
    ·
    1 year ago

    I mean it’s definitely not recommended, but sometimes you have no choice. As others have mentioned, your situation sounds unique.

  • Thetimefarm@lemm.ee
    link
    fedilink
    English
    arrow-up
    2
    arrow-down
    1
    ·
    1 year ago

    Do you have a RAID array of some sort? As other pointed out it’s odd for so much data to be lost with just a hard reboot.

    • sourcepie@lemmy.worldOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      edit-2
      1 year ago

      I have no clue what a RAID is to be honest. I was using a VM with 6 GB RAM to do some work in another operating system and passed through my physical hard drive (Host to Guest) to write the data on when suddenly my VM got stuck.

    • EpeeGnome@lemmy.fmhy.ml
      link
      fedilink
      English
      arrow-up
      1
      ·
      1 year ago

      Almost all of the data is certainly *somewhere *on the hard drive. It sounds like it was a write to the file system records that lost the record of where they are.