Transferring huge data from my PC to NAS

[icube]

I have just bought a new NAS and I would like to move 100GB+ of data to the NAS's storage.. what is the fastest way to do that?

 

[jtjt00]

Use TeraCopy

Ensure you have the following:
1. Gigabit network port on PC side
2. Gigabit router/switch
3. Cat. 6 LAN cable

 

[icube]

Use TeraCopy

Ensure you have the following:
1. Gigabit network port on PC side
2. Gigabit router/switch
3. Cat. 6 LAN cable

Hmm ok... but any guide out there to do this? I mean how should the connection be like?
Where can i get the Cat6 LAN cable? Is there any other way other than using this? I don't want to just buy a Cat6 LAN cable just for this purpose...

I am using Synology NAS btw...

Thanks!

 

[iCuteCube]

Hmm ok... but any guide out there to do this? I mean how should the connection be like?
Where can i get the Cat6 LAN cable? Is there any other way other than using this? I don't want to just buy a Cat6 LAN cable just for this purpose...

I am using Synology NAS btw...

Thanks!

Hi brother!

You can get Cat6 lan cable at challenger (easier for you), if you want cheaper and waste some time travel to SLS, it's more cost effective there, but considering time spent, confirm not worth it.

Cable is the fastest way, i not sure what are you talking about, maybe there is a faster way, which is buying a fiber connector

 

[szeli]

using windows? robocopy.
it has logs n resume on errors.
after e job finish jus check e logs to see if any file got missed out.

 

[Solkarnar]

100GB is not huge.... And if it's just a one off affair, just use normal cable cat 5e can liao la. Just start copy and go shower/ watch the news come back finish already. If you have 10TB, THEN thats huge.

 

[davidktw]

I have just bought a new NAS and I would like to move 100GB+ of data to the NAS's storage.. what is the fastest way to do that?

Assuming your data is found at /source, and assume your NAS has SSH running and the target directory is /destination

The code below will be fast.

tar -cf - /source | ssh username@nas.hostname 'cd /destination; tar -xf -'

This approach will be saturate your network bandwidth in situations where there are large number of files to be copied.

Update: Just saw you are using Synology, then this approach will definitely work for your case. A typical harddisk can easily saturate the a single 1Gbps network for sequential data, it shouldn't take you more than 30mins to transfer 100GB of data via SSH using CAT5e cable, check your NIC that you are on negotiated on 1Gbps connection.

 

[yukita]

@davinktw
wah this one good info, although like very chim.

@TS
but 100GB is nothing "huge", considering transferring at relatively low speed of 50MB/s it takes 35 minutes (rounding)

even when i transferring my 7 TB (7000GB) personally i dont consider mine huge lol

 

[davidktw]

@davinktw
wah this one good info, although like very chim.

It's 2nd nature to most *nix developers. Do feel free to learn more about it if you are interested.

"tar" is a tape archiver utility for tape backup, but modern days we use it more for consolidation of directories of files into one single file for transport or storage. You can apply numerous features to the archive file like splitting it into chunks, applying compression. It helps to preserve unix symbolic links, permissions of files and so forth. A must have tool across all *nix environments.

"ssh" is your friendly secure shell neighbour always available for any client platforms to establish secure connect to the servers and invoke any program on the server-side, not just limited to shell program. Besides just secure channel, it can perform tunnelling, socks proxy, piping of commands remotely and even STDIN data in this case.

Basically the idea is archive your source directory of files, output to the STDOUT of the terminal, pipe the tar STDOUT stream into the SSH's STDIN stream. SSH will transmit this stream over the network to the server side(NAS in this case). At the server side, invoke the archiver utility again, pipe the STDOUT of the SSH server end to the STDIN of the remote archiver utility, where the extraction will extract the archived stream back into it's form just the same as the source directories of files.

That's the beauty of unix files and streams abstraction which I often say is the heaven for developers. With this skill set alone, you can mix and match numerous orthogonal functionalities utilities that can accomplish a set of complicated tasks. Each tool excel in what it does best.

I could have used secure copy known as (scp), but it will be very slow when copying a large number of small files. The reason is each file will need to be open, read serialized send across. Using tar, tar will consolidate all small files into one single stream, instead of SSH doing it which has too much I/O switching between the network and the disk. When using 2 utilities, these 2 utilities are concurrently working using 2 processes, which can be spread across multicores or sockets. Here the tasks are not CPU bounded, but having 2 different processes helps versus 1 single process doing all the work.

Without such approach, large number of small files will likely not saturate the network to have an efficient network transfer. Compression could be used for smaller transfer, but will incur higher computation which is not recommended on a slow Synology processor. If you know your files are highly compressible, you can opt for a fast gzip in the command as such

tar -czf - | ... | tar -xzf -

Normally I wouldn't bother to compress since the average ratio is roughly halved. Using a fast cipher, such as ARC4 for SSH also helps, but marginal, provided you are not concern with security, but requires more configuration on server and client end. So hope this information will be helpful to the interested parties