I have a strange problem with scp.
When sending a large file (mov file originally recorded on an iphone) from one computer on the network to another (same network), the file transfer progress seems to be fine and the file builds, then when it is finished, the file appears truncated/overwritten by a smaller file with the same name.
I am using a laptop running Ubuntu 24.04.2 to ssh on to a server running Raspbian OS to copy the file to an RPi running Batocera 41. The versions of ssh running on the client and server are different but from what I've found they handshake ok and it shouldn't be having an effect.
So the set up is:
Media server:
Linux media-server 6.12.25+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.12.25-1+rpt1 (2025-04-30) aarch64;- Ssh version:
OpenSSH_9.2p1 Debian-2+deb12u6, OpenSSL 3.0.16 11 Feb 2025
- Ssh version:
Media client:
Batocera 41 Linux 6.6.56- Ssh version:
Dropbear v2024.85
- Ssh version:
Note, authentication at the moment is with strong passwords. We set up with certs but they stopped working. We'll debug that separately.
The copy is initiated on the server with the command:
scp -Cr /home/ToLocalize/* root@batocera.local:/media/LocalStore/transfer >~/log/scp.'date +"%Y-%m-%d_%H:%M:%S"'.log 2>&1
Now let's say there's a file called VideoConference.mov in the server ToLocalize directory that is 1.5GB in size. Using the command above, I can see a file called VideoConference.mov on the client that builds to 1.5GB.
When the command ends, the size of VideoConference.mov is 3.5MB. It seems to be the start of the MOV file but I'm not 100% sure.
I have tried removing the -C from the scp command in case it is a compression issue which doesn't change the behavior.
The log file was empty. I added -v to the command (still without -C) and the log file contains this:
Executing: program /usr/bin/ssh host batocera.local, user root, command sftp
OpenSSH_9.2p1 Debian-2+deb12u6, OpenSSL 3.0.16 11 Feb 2025
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug1: Connecting to batocera.local [192.168.1.8] port 22.
debug1: Connection established.
debug1: identity file /home/mediaserver/.ssh/id_rsa type -1
debug1: identity file /home/mediaserver/.ssh/id_rsa-cert type -1
debug1: identity file /home/mediaserver/.ssh/id_ecdsa type -1
debug1: identity file /home/mediaserver/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/mediaserver/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/mediaserver/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/mediaserver/.ssh/id_ed25519 type -1
debug1: identity file /home/mediaserver/.ssh/id_ed25519-cert type -1
debug1: identity file /home/mediaserver/.ssh/id_ed25519_sk type -1
debug1: identity file /home/mediaserver/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/mediaserver/.ssh/id_xmss type -1
debug1: identity file /home/mediaserver/.ssh/id_xmss-cert type -1
debug1: identity file /home/mediaserver/.ssh/id_dsa type -1
debug1: identity file /home/mediaserver/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u6
debug1: Remote protocol version 2.0, remote software version dropbear_2024.85
debug1: compat_banner: no match: dropbear_2024.85
debug1: Authenticating to batocera.local:22 as 'root'
debug1: load_hostkeys: fopen /home/mediaserver/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
debug1: load_hostkeys: fopen /home/mediaserver/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'batocera.local' is known and matches the ED25519 host key.
debug1: Found key in /home/mediaserver/.ssh/known_hosts:1
debug1: ssh_packet_send2_wrapped: resetting send seqnr 3
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 3
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/mediaserver/.ssh/id_rsa
debug1: Will attempt key: /home/mediaserver/.ssh/id_ecdsa
debug1: Will attempt key: /home/mediaserver/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/mediaserver/.ssh/id_ed25519
debug1: Will attempt key: /home/mediaserver/.ssh/id_ed25519_sk
debug1: Will attempt key: /home/mediaserver/.ssh/id_xmss
debug1: Will attempt key: /home/mediaserver/.ssh/id_dsa
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-256,ssh-rsa>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/mediaserver/.ssh/id_rsa
debug1: Trying private key: /home/mediaserver/.ssh/id_ecdsa
debug1: Trying private key: /home/mediaserver/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/mediaserver/.ssh/id_ed25519
debug1: Trying private key: /home/mediaserver/.ssh/id_ed25519_sk
debug1: Trying private key: /home/mediaserver/.ssh/id_xmss
debug1: Trying private key: /home/mediaserver/.ssh/id_dsa
debug1: Next authentication method: password
Authenticated to batocera.local ([192.168.1.8]:22) using "password".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: Sending environment.
debug1: channel 0: setting env LC_ADDRESS = "en_US.UTF-8"
debug1: channel 0: setting env LC_NAME = "en_US.UTF-8"
debug1: channel 0: setting env LC_MONETARY = "en_US.UTF-8"
debug1: channel 0: setting env LANG = "en_US.UTF-8"
debug1: channel 0: setting env LC_PAPER = "en_US.UTF-8"
debug1: channel 0: setting env LC_IDENTIFICATION = "en_US.UTF-8"
debug1: channel 0: setting env LC_TELEPHONE = "en_US.UTF-8"
debug1: channel 0: setting env LC_MEASUREMENT = "en_US.UTF-8"
debug1: channel 0: setting env LC_TIME = "en_US.UTF-8"
debug1: channel 0: setting env LC_NUMERIC = "en_US.UTF-8"
debug1: Sending subsystem: sftp
debug1: SSH2_MSG_KEXINIT received
debug1: SSH2_MSG_KEXINIT sent
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: rekeying in progress
debug1: rekeying in progress
debug1: rekeying in progress
debug1: rekeying in progress
debug1: rekeying in progress
debug1: rekeying in progress
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
debug1: ssh_packet_send2_wrapped: resetting send seqnr 40509
debug1: ssh_set_newkeys: rekeying out, input 1600580 bytes 179596 blocks, output 1070880804 bytes 133839643 blocks
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: ssh_packet_read_poll2: resetting read seqnr 40778
debug1: SSH2_MSG_NEWKEYS received
debug1: ssh_set_newkeys: rekeying in, input 1600600 bytes 179598 blocks, output 1070880804 bytes 0 blocks
debug1: rekey in after 134217728 blocks
scp: debug1: truncating at 25165824
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
Transferred: sent 1509395880, received 2258432 bytes, in 86.0 seconds
Bytes per second: sent 17554380.1, received 26265.7
debug1: Exit status 0
With the logfile lines (at the end) seeming interesting:
scp: debug1: truncating at 25165824
and
Transferred: sent 1509395880, received 2258432 bytes, in 86.0 seconds
The low received number seems to be contradicted by the fact the destination file builds to 1.5GB before snapping to 3.5MB when the command exits.
Additional points: I have tried looking here: Using scp to copy files from remote to home machine and here: https://help.ubuntu.com/community/SSH/TransferFiles but neither seem to address this issue.
Anyone have any idea what is going on and how to resolve?
I'd like to avoid going back to USB flash drives to move large files
