Data transfer

Depending on the connection protocol you are using, there are multiple ways to transfer between the mass storage and another storage device (e.g., your computer). The recommended method vary based on the type of data your are willing to transfer.

Use rsync if:

  • you want to transfer large (or small files).

  • you are connected to the mass storage over SSH.

Use SAMBA if:

  • you want to transfer small files.

  • you are connected to the mass storage over SAMBA.

Transferring files over SSH

Note

Make sure to set up your SSH hosts before you follow this method.

This technique requires uses rsync and is the recommended way to transfer large (and small) files. To transfer data using SSH, follow this procedure:

  1. Open a new terminal window.

  2. Locate the source and the destination file/directory. If you are trying to transfer a file from your computer to the mass storage, the path to the source file/directory is the path on your computer, while the destination path is the path on the mass storage and should be prefixed with store.giga:. Do the opposite for the other way round.

  3. Run the synchronisation command:

    rsync \
   --progress --partial --hard-links --links --recursive --times --verbose \
   <source> <destination>

    or, in short

    rsync -PHlrtv <source> <destination>

  4. Run a sanity check by repeating the same command with an added --checksum (-c) flag:

    rsync -PHlrtcv <source> <destination>

    This will check the integrity of the transferred files using a MD5 checksum.

Hint

  • If you want to transfer a folder and its content, you must omit the / at the end of the source path.

  • If you want to transfer the content of a folder, but not the folder itself, you must include the / at the end of the source path.

  • The rsync command accepts regular expressions to specify the files to transfer.

If you are not sure of the paths to use, please contact the GIGA bioinformatics team.

Warning

If you are not in a screen or tmux session and close the terminal or loose connection to the mass storage, the transfer will be aborted. If this happens, repeat the rsync command to resume the transfer.

Transferring files over SAMBA

Attention

Using this method for files larger than a few GB in size may result in long transfer time, and potential system freeze. It is recommended to use rsync to transfer large files.

To transfer files using SAMBA, you have to:

  1. Connect to the mass storage with SAMBA.

  2. Drag and drop the files or folders you want to transfer using your file explorer.

Other resources