Copy While Organizing Files Over Existing SSH Connections

Photo by Sai Kiran Anagani on Unsplash

Often, I find myself copying files via ssh to a remote machine, and then shelling into that machine to organize them (or vice versa). A good example of this is how I organize my photos: I organize them into directories named YYYY/YYYY-MM-DD/ based on the exif data of the file. For example:

2020
├── 2020-11-01
│   ├── DSC_0536.NEF
│   ├── DSC_0537.NEF
│   ├── DSC_0538.NEF
├── 2020-11-07
│   ├── DSC_0001.NEF
│   ├── DSC_0002.NEF
│   ├── DSC_0003.NEF
...

This presents a slight problem when copying over ssh. I either need to:

• Organize the files on my sd card before copying. I try to avoid moving a lot of data on the sd card to prolong it’s life.
• Login twice via ssh (once to copy, and then again via an interactive shell to organize). Even with ssh keys, this can be annoying if you use any 2FA method like yubikey or duo.

After some research, I found a more elegant solution: determine the image destination and then copy them through an existing open ssh connection. Below, I detail how to achieve this.

Determining the Photo’s Destination

First, I need to know in which directory the image should be copied to. exiftool can extract the date and time of when the picture was taken:

exiftool -T -createdate -d  "%Y-%m-%d" image_file

That outputs the destination folder in YYYY-MM-DD format.

I’ll save that in a variable, and also extract the year:

DATE=$(exiftool -T -createdate -d  "%Y-%m-%d" image_file)
YEAR=$(date --date="$DATE" "+%Y")

Using A Single SSH Connection

Next, I open an ssh connection using the ControlMaster mode. When you connect to a remote host with this mode, future commands use the existing connection until the first connection is closed.

ssh -o ControlPath=~/.ssh/control-%h-%p-%r-o ControlMaster=yes user@host

There are two ways to use it. The first is to simply set the options when you form the first connection:

ssh -o ControlPath=~/.ssh/control-%h-%p-%r-o ControlMaster=yes user@host

and then future connections can be made with:

ssh -o ControlPath=~/.ssh/control-%h-%p-%r user@host

The second way is more elegant IMO. Edit your ~/.ssh/config file to set the control path for each host:

Host host
    User user
    HostName (IP/DNS)
    ControlPath ~/.ssh/control-%h-%p-%r

and then making the first connection is rather easy to type:

ssh -M host

Future commands can be made with simply:

ssh host

In both cases, you close the first connection with:

ssh -O exit

Putting It All Together

Now, the logic to simultaneously copy and organize the files over ssh is pretty straightforward:

First, open an ssh connection using the ControlMaster mode:

ssh -M host

Next, for each image, we’ll:

• Determine it’s destination folder
• Ensure the destination exists using open ssh connection
• Copy the image file using existing ssh connection

for image_file in "${images[@]}"; do
    
    # Determine it's destination folder
    DATE=$(exiftool -T -createdate -d  "%Y-%m-%d" image_file)
    YEAR=$(date --date="$DATE" "+%Y")

    # Ensure the destination exists using open ssh connection
    ssh host "mkdir -p ./Pictures/$YEAR/$DATE"

    # Copy the image
    scp image_file host:"./Pictures/$YEAR/$DATE"
done

And finally close the ssh connection:

ssh -O exit

Conclusions

The ssh Control* features make it nice and easy to connect to a remote host and organize files while they are copied over. As noted by Geoffrey Thomas , it also speeds up subsequent file transfers since you don’t need to re-auth each time.

I’ve packaged the logic discussed above into a bash script on GitHub. Let me know if you use it!

Originally published at https://lou.dev.