How can I copy files to lilac?

 
1
0
-1
    CommentAdd your comment...

    1 answer

    1.  
      1
      0
      -1

      To copy files from other clusters, first ssh -A into the other cluster to forward your keys.

      ssh -A $USERNAME@$CLUSTER

      We recommend rsync -va to copy files and directories.

      Then copy the data to lilac:

      rsync -av --progress $SOURCEPATH lilac:$DESTPATH


      Note that the hal cluster is outside the MSKCC network, and does not have access to lilac. To transfer data first:

      ssh -A $USERNAME@lilac.mskcc.org

      Then pull files from hal:

      rsync -av --progress hal:$SOURCEPATH $DESTPATH


      Make sure you know how much you will copy to lilac, and you have enough space on lilac to avoid hitting your hard quota. lilac uses data replication to for safety, so a file containing 1G of data consumes 2G of quota on lilac.

      You can see the size of files and directories with du, which will show 2G for 1G of file data due to replication. To see file size without replication overhead use du --apparent-size instead:

      du -sh --apparent-size $DIRECTORY

      hal also uses data replication, so a 1G file on hal will consume 2G of quota on both hal and lilac. Unreplicated hal directories include grlab, TCGA shared data, and /cbio/archives.

      Depending on the size and number of files to copy, you may run multiple rsync commands simultaneously to copy different directories.

      The HPC private network is faster than the MSKCC campus network, so using short names (lilac, saba, luna, selene, etc.) will often make transfers faster than using fully qualified domain names such as luna.mskcc.org. This does not apply to hal, though.





      See also: How can I ssh into lilac compute nodes?

        CommentAdd your comment...