Thread: Archiving the massive raw media

Page 5 of 5 FirstFirst ... 345
Results 41 to 49 of 49
  1. #41  
    Senior Member Peter J. DeCrescenzo's Avatar
    Join Date
    Apr 2012
    Location
    Portland, Oregon USA
    Posts
    2,000
    Since "RAID" is being mentioned in this thread, here's my obligatory caution:

    A RAID is not a backup, unless it's one of at least two storage systems containing identical data.

    A RAID is relatively big, fast storage, and may include certain redundancy features for reliability, but by itself it's not a backup. "RAID=backup" is an oft-repeated misconception. So, to repeat: A RAID by itself is not a backup, unless it's one of at least two storage systems containing identical data.

    A backup is a complete, up-to-date, identical copy of your data, on physical media or a remote server, that is separate from your online storage. Backups can be on a variety of media, such as individual hard disk drives, HDDs in a RAID, LTO tape, BluRay or DVD data disks, etc.

    Ideally you should have more than one set of backups, with at least one located off-site to protect from disasters such as fire, theft, earthquake, flood, etc.

    Whatever data storage technique you're using, one thing is certain: It will fail at some point. If you only have 1 backup of your data, at the moment the original storage system fails, you no longer have a backup: You only have 1 copy of the data on one storage system. There's a very real possibility that whatever physical cause, human error, software or hardware glitch that caused one copy of the file(s) to go missing will cause another copy to meet the same fate soon thereafter. That's why you should always have at least 2 backup copies (in addition to the original data), and keep at least one backup copy off-site.

    Cheers.
    Reply With Quote  
     

  2. #42  
    Senior Member Adam Jeal's Avatar
    Join Date
    Apr 2012
    Location
    South-West Scotland, UK
    Posts
    218
    Quote Originally Posted by Jason M. View Post
    Backups and archiving are two different things, though they are obviously related.

    Hard drives are good for simple backup and data redundancy, but not good for archival purposes. They are not shelf-stable, and given the way CDs and DVDs degrade, I'm pretty wary of the longevity of optical formats (though I've heard Blu-ray is more stable).

    Tape is a better option, but as with any digital archival technology, data migration after a certain period of time (usually 7-10 years) is key.

    Even a 250 GB Blu-ray is only going to carry half an hour of RAW footage from this camera, which will make archiving any significant project quite tedious.

    In terms of longevity and easy readability (both incredibly important for archival purposes), the best archival format for motion pictures is still black and white separation masters made on film, hands down. Prohibitively expensive for any independent production, though.
    +1 to everything you've said! You cannot overstate the importance of long term archiving. Unless you want to lose your material.

    Adam
    Reply With Quote  
     

  3. #43  
    Senior Member Adam Jeal's Avatar
    Join Date
    Apr 2012
    Location
    South-West Scotland, UK
    Posts
    218
    Quote Originally Posted by Peter J. DeCrescenzo View Post
    Since "RAID" is being mentioned in this thread, here's my obligatory caution:

    A RAID is not a backup, unless it's one of at least two storage systems containing identical data.

    A RAID is relatively big, fast storage, and may include certain redundancy features for reliability, but by itself it's not a backup. "RAID=backup" is an oft-repeated misconception. So, to repeat: A RAID by itself is not a backup, unless it's one of at least two storage systems containing identical data.

    A backup is a complete, up-to-date, identical copy of your data, on physical media or a remote server, that is separate from your online storage. Backups can be on a variety of media, such as individual hard disk drives, HDDs in a RAID, LTO tape, BluRay or DVD data disks, etc.

    Ideally you should have more than one set of backups, with at least one located off-site to protect from disasters such as fire, theft, earthquake, flood, etc.

    Whatever data storage technique you're using, one thing is certain: It will fail at some point. If you only have 1 backup of your data, at the moment the original storage system fails, you no longer have a backup: You only have 1 copy of the data on one storage system. There's a very real possibility that whatever physical cause, human error, software or hardware glitch that caused one copy of the file(s) to go missing will cause another copy to meet the same fate soon thereafter. That's why you should always have at least 2 backup copies (in addition to the original data), and keep at least one backup copy off-site.

    Cheers.
    Listen to this man! RAID's should never be considered as backup.

    Adam
    Reply With Quote  
     

  4. #44  
    Did any of you see the announcement for Amazon Glacier? It's like S3 but it's designed for long term backup (with infrequent access). Basic features:

    - $0.01 per GB/Month. Or $120 per TB/year.
    - Can access 5% of your data per month for free, accessing more will cost you
    - Can seed or retrieve by sending a hard drive if you need to for a fee
    - Fully redundant

    At a glance it seems like a pretty good solution for an emergency backup. Much less hassle than tape. Thoughts?
    Reply With Quote  
     

  5. #45  
    I know this thread is ancient, but I'm really curious to hear how thoughts have changed about the cloud vs LTO-5 vs HDDs. What's your currently backup strategy?
    Reply With Quote  
     

  6. #46  
    Senior Member
    Join Date
    May 2012
    Posts
    1,132
    M-disc BDR 100GB discs can be written with most current gen Blu-ray writers and can be read by any Blu-ray drive.
    True permanent archival medium under normal storage conditions. Cost is about $150/TB for media. No special software requirements beyond normal BDR data disc writing apps. For large libraries some sort of management software would be useful.
    A combination of Slimraw compression and these is readily accessible to anyone at low cost.
    https://en.m.wikipedia.org/wiki/M-DISC#Overview
    Google is beta testing a combination cloud storage + M-disc archiving service.
    http://www.mdisc.com
    Reply With Quote  
     

  7. #47  
    Senior Member
    Join Date
    Sep 2013
    Posts
    631
    Quote Originally Posted by razz16mm View Post
    M-disc BDR 100GB discs can be written with most current gen Blu-ray writers and can be read by any Blu-ray drive.
    True permanent archival medium under normal storage conditions. Cost is about $150/TB for media. No special software requirements beyond normal BDR data disc writing apps. For large libraries some sort of management software would be useful.
    A combination of Slimraw compression and these is readily accessible to anyone at low cost.
    https://en.m.wikipedia.org/wiki/M-DISC#Overview
    Google is beta testing a combination cloud storage + M-disc archiving service.
    http://www.mdisc.com
    I've looked at this option but my concern is longterm support of Blu-ray in general. Will they continue to make Blu-ray devices? Seems like the computer manufacturers are supporting optical discs less and less. I think they would like to see a complete virtualization.

    To me, we are still looking at the same archive hierarchy:
    - film
    - tape (LTO)
    - optical
    - hard drives

    What we need is the holy grail of archive:
    - Non proprietary
    - Nearly indestructible
    - affordable
    - large capacity

    I don't think we've found that holy grail yet. I wish Kodak was able to be the one that finds it though. There would be something cool about that.
    Reply With Quote  
     

  8. #48  
    Senior Member
    Join Date
    May 2012
    Posts
    1,132
    Quote Originally Posted by stevesherrick View Post
    I've looked at this option but my concern is longterm support of Blu-ray in general. Will they continue to make Blu-ray devices? Seems like the computer manufacturers are supporting optical discs less and less. I think they would like to see a complete virtualization.

    To me, we are still looking at the same archive hierarchy:
    - film
    - tape (LTO)
    - optical
    - hard drives

    What we need is the holy grail of archive:
    - Non proprietary
    - Nearly indestructible
    - affordable
    - large capacity

    I don't think we've found that holy grail yet. I wish Kodak was able to be the one that finds it though. There would be something cool about that.
    I agree with your hierarchy, but think that no particular digital technology or format is guaranteed to exist long term as things progress. Blu-ray is going forward into the next 4k generation, so should be around for some years to come. It is less popular as a distribution medium in a cloud streaming age, but still very useful for long term general purpose data storage in a secure physical format. Unlike dye based optical discs, M-discs have the capability to outlive the format without data loss. Low capacity is something of an issue too.
    Physically M-disc is a superior long term storage option to any magnetic media, which are subject to bit rot and degradation over relatively short time frames. Tape and hard drives need to be transferred to fresh media every so often.
    M-disc is readily accessible to anyone with minimal hardware investment at modest media cost.
    Reply With Quote  
     

  9. #49  
    Senior Member rick.lang's Avatar
    Join Date
    Apr 2012
    Location
    Victoria BC Canada
    Posts
    3,956
    Archival storage at $150/TB seems expensive. I haven't checked LTO prices lately but isn't the Media something no like $15 per tape and that sores more than a TB? The LRO tape recorder may be $1,5000, but by the time you sabpve 11 TB, the drive has paid for itself. Besides the apparent longevity of LTO tape, it still seems the least expensive option for indies although Frank's method of a simple relatively inexpensive HDD is also attractive.

    If you are a major Studio producing a dozen features a year and you want to keep your footag for posterity, modern celluloid may also be an option. Until there's a flood or fire that ruins your canisters.

    I do agree that if the media is too complex, it will eventually no longer have Support. All digital media will expire at some point and thirty years from now who is going to copy your precious footage from the protocol about to lose Support to the newest and greatest protocol that may lose support in another 20 years. Real long term retention is a "bag of hurt." For most indies, before the time the current protocols expire, interest in the footage likely will have already expired. Unless you're franchised and shooting Fast and Furious XXVII or Aliens: Utopía or 28,000 Days Later or ...
    Last edited by rick.lang; 05-20-2017 at 10:39 AM.
    Reply With Quote  
     

Bookmarks
Bookmarks
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •