Thread: Fastest Way To Offload Footage

Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 28
  1. #11  
    Quote Originally Posted by Joe Grasso View Post
    I'm just using Windows explorer.
    That's probably the issue right there. As an OSX user I can't speak to the performance penalty of using Windows Explorer--or even writing thousands of small DNG files on NTFS volumes--but I will say that you're much better served by using an application designed for this kind of work rather than the file browser in the OS. Take a look at ShotPut Pro or SlimRAW and see if there are any performance gains.
    Reply With Quote  
     

  2. #12  
    Junior Member
    Join Date
    Mar 2016
    Posts
    25
    Will do. Thanks for the suggestion.
    Reply With Quote  
     

  3. #13  
    Junior Member
    Join Date
    Mar 2017
    Posts
    5
    Quote Originally Posted by Joe Grasso View Post
    Now that you guys are talking about offloading, I was wondering what kind of transfer speeds you guys normally get when offloading 4k raw files. I often shoot in raw 4:1 with my Ursa Mini 4k, and offload with a Lexar CR1 via USB 3.0 to either an SSD in my desktop, my WD Raid 0 HDD external, or a standard 7200 rpm HDD internal. My transfer speeds are always around 30 MB/s regardless of the storage type I'm using. I've tried all of the different USB 3.0 ports in both my desktop and laptop and the speed didn't change. Also, I'm using Lexar 3400X 128 GB Cfast cards. I know the individual raw files should take longer to transfer than the Prores files, but it's taking on average about 10X longer, which sucks when I'm trying to transfer in the field. I truly appreciate any thoughts and considerations you guys might have.
    I don't mean to harp on about Hedge (which won't help you since you're on Windows) but they've published a blog post that sheds some light on your issue: https://medium.hedgeformac.com/backu...117#.nxeptmcpm
    Reply With Quote  
     

  4. #14  
    Senior Member
    Join Date
    Mar 2016
    Posts
    474
    Quote Originally Posted by Alex.Mitchell View Post
    The kind of files shouldn't make a difference in a large sequential transfer. To really diagnose though, we need more information. What software are you using to copy the files? Are you positive that your ports aren't defaulting back to USB2 because the cables are either malfunctioning or not inserted all the way? etc.

    Actually it does, especially if you aren't using a multi-threaded transfer system (like using the OS built in copy) so DNG would take long to copy than say prores because it's a bunch of files, it has to find each one process then start the copy and so on for each file. It hads up to a lot in large volumes.

    Ideally you want to use a mulit-threaded transfer system that also does MD5 comparisons.
    Reply With Quote  
     

  5. #15  
    Senior Member
    Join Date
    Oct 2013
    Posts
    260
    Quote Originally Posted by Jason Finnigan View Post
    Actually it does, especially if you aren't using a multi-threaded transfer system (like using the OS built in copy) so DNG would take long to copy than say prores because it's a bunch of files, it has to find each one process then start the copy and so on for each file. It hads up to a lot in large volumes.

    Ideally you want to use a mulit-threaded transfer system that also does MD5 comparisons.
    My understanding is the same, it takes a lot of time to transfer DNGs because of the large amount of independent files.

    What kind of software would be able to bypass that? Would SSDs help? I would be interested in hearing how pro DITs work.
    Reply With Quote  
     

  6. #16  
    Senior Member
    Join Date
    Mar 2016
    Posts
    474
    Quote Originally Posted by townio View Post
    My understanding is the same, it takes a lot of time to transfer DNGs because of the large amount of independent files.

    What kind of software would be able to bypass that? Would SSDs help? I would be interested in hearing how pro DITs work.
    Something like ShoutPut Pro will do multithreaded http://www.imagineproducts.com/index...products_id=33

    Now on the free end non-Video orinted file transfer tools can do this too, like Tera Copy, Rich Copy (now defunct)
    Last edited by Jason Finnigan; 03-21-2017 at 01:54 PM.
    Reply With Quote  
     

  7. #17  
    Quote Originally Posted by Jason Finnigan View Post
    Actually it does, especially if you aren't using a multi-threaded transfer system (like using the OS built in copy) so DNG would take long to copy than say prores because it's a bunch of files, it has to find each one process then start the copy and so on for each file. It hads up to a lot in large volumes.

    Ideally you want to use a mulit-threaded transfer system that also does MD5 comparisons.
    You're right; you're definitely not going to get peak performance from copying thousands of small individual files compared to fewer, larger files. It shouldn't be dipping to 30MB/s though. I'm regularly hitting 230-250MB/s when I offload DNG footage, and a portable HDD should at least be hitting 80-100MB/s.

    Also, in every case where you have a choice it's better to use XXHash-64 instead of MD5.
    Reply With Quote  
     

  8. #18  
    Senior Member
    Join Date
    Mar 2016
    Posts
    474
    Quote Originally Posted by Alex.Mitchell View Post
    Also, in every case where you have a choice it's better to use XXHash-64 instead of MD5.
    Yes XXHash is much faster, it was never designed for encryption only comparison unlike MD5 Hashing.
    Reply With Quote  
     

  9. #19  
    Quote Originally Posted by jhnkng View Post
    I use the Lexar Professional Thunderbolt hub because I can download multiple cards at once without slowing down, and I use Hedge (http://hedgeformac.com) to download my cards. The only problem with the Lexar hub is that it requires power, but each card reader can be removed and plugged straight into the computer via USB3 if you're battery only.
    I have been pulling my hair out with the workload of my ursa. Yesterday I shoot a dusk shot, half on my drone and osmo raw (which is a data problem on its own!) and the rest with my ursa. Shooting into the sun I have found raw quite a bit better than process 1080p 4444 (I only need 1080p) - so I have been having to shoot raw- and stuck with dealing with 4.6k 60p. So I shot 2 x 64gb and 2 x 128gb cards (along with the osmo) and I was up till 2am- where I left the last osmo ssd going while I went to sleep. The editor needed transcoded footage, so I was up at 6am to transcode (3 hours) so I could rush the footage over. I didnt have time to backup.
    I am finding this workload too hard for the speeds things need to be done.

    Can I download many cards at the same time with this software (if I buy a few more cfast readers)? then go to sleep while it churns away?

    I have also thought about getting a couple of older cheap 13inck macbook pros to be dedicated downloading machines.

    paul
    Paul Ross Jones


    -------------------------
    www.paulrossjones.com
    Reply With Quote  
     

  10. #20  
    Junior Member
    Join Date
    Mar 2016
    Posts
    25
    Quote Originally Posted by Jason Finnigan View Post
    Something like ShoutPut Pro will do multithreaded http://www.imagineproducts.com/index...products_id=33

    Now on the free end non-Video orinted file transfer tools can do this too, like Tera Copy, Rich Copy (now defunct)
    Great, thanks for the suggestions. Have you used Tera Copy? Is it stable/reliable? I guess what I'm asking is, is it worth it to spend the $100 on the Shoutput vs using the free Tera Copy if I just need to speed up my DNG transfers when I'm in the field?
    Reply With Quote  
     

Similar Threads

  1. Fastest way to off load CinemaDNG Raw file from a Cfast card.
    By amsh89es335 in forum General Discussion
    Replies: 1
    Last Post: 01-10-2017, 10:54 AM
  2. Replies: 9
    Last Post: 09-25-2015, 02:22 AM
  3. Replies: 10
    Last Post: 08-01-2013, 09:17 PM
  4. Simple Question- Fastest way to transcode 4TB of Raw footage?
    By SamuelJacobPauling in forum Workflow
    Replies: 8
    Last Post: 05-26-2013, 11:37 PM
  5. Playback of raw DNG after offload
    By Martin in forum General Discussion
    Replies: 5
    Last Post: 09-10-2012, 10:48 AM
Bookmarks
Bookmarks
Posting Permissions
  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •