How to transfer files between two different projects

How-to Guides
#1

Transfer files between two different projects with SSH or SCP

Goal

The goal is to transfer files between two separate projects.

Assumptions

You will need two projects set up on Platform.sh:

  • One project (undntpvafhdn4 in this guide) that acts as the data source.
  • One project (xksjd6v6od7iq) that is the data sink (where we want to copy data to).

Problems

Communications between projects relies on SSH, so managing the keys needs to be automated.

Steps

1. Preparing the data sink

On the data sink, add .ssh as a writeable mount (this is to allow SSH to write into known_hosts). Your .platform.app.yaml has to contain this block:

mounts:
    "/.ssh":
      source: local
      source_path: .ssh

In the build hook, install the Platform.sh CLI. This is used to dynamically retrieve the data source’s SSH connection string. If the remote will not change, you can also hardcode the path and avoid this step.

hooks:
    build: |
      echo "### INSTALLING Platform.sh CLI ... ###"
      curl -sS https://platform.sh/cli/installer | php

2. Transferring files

Connect to the data sink project using the -A option for SSH (this allows SSH key forwarding and connections to the data source project from the data sink project):

> ssh -A "$(platform ssh -p xksjd6v6od7iq --pipe)"

 ___ _      _    __                    _
| _ \ |__ _| |_ / _|___ _ _ _ __    __| |_
|  _/ / _` |  _|  _/ _ \ '_| '  \ _(_-< ' \
|_| |_\__,_|\__|_| \___/_| |_|_|_(_)__/_||_|


 Welcome to Platform.sh.

 This is environment master-7rqtwti
 of project xksjd6v6od7iq.

web@pff7myd47w7zhnm43cetvkeujy:~$

and run the rsync command:

web@pff7myd47w7zhnm43cetvkeujy:~$ rsync -avrz "$(platform ssh -p undntpvafhdn4 -e master --pipe)":/app/web/*.txt /tmp/
Warning: Permanently added 'ssh.eu-3.platform.sh,63.34.120.202' (RSA) to the list of known hosts.
receiving incremental file list
1.txt
2.txt

sent 62 bytes  received 147 bytes  418.00 bytes/sec
total size is 4  speedup is 0.02
web@pff7myd47w7zhnm43cetvkeujy:~$

3. Automating it with a SSH key

If you do not want to rely on the SSH -A switch to enable key forwarding, you can add an SSH key to have the connection set up automatically.

On your local machine, generate a new key with no passphrase:

> ssh-keygen -f id_rsa_transfer
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa_transfer.
Your public key has been saved in id_rsa_transfer.pub.
The key fingerprint is:
SHA256:<randomstring>
The key's randomart image is:
<key_image>

Add the generated public key into .ssh/authorized_keys in the data source project and commit the change:

> cat id_rsa_transfer.pub > <datasourceprojectpath>/.ssh/authorized_keys
> cd <datasourceprojectpath>
> git commit -m "Add SSH key"
> git push

Add the key in the data sink project:

> cat id_rsa_transfer > <datasinkprojectpath>/.ssh/id_rsa
> cat id_rsa_transfer.pub > <datasinkprojectpath>/.ssh/id_rsa.pub
> cd <datasinkprojectpath>

Edit the build hook to set the correct permissions on the keys:

hooks:
    build: |
      <... other commands ...>
      chmod 0600 .ssh/id_rsa
	  chmod 0600 .ssh/id_rsa.pub

Commit the changes:

> git commit -m "Add SSH key"
> git push

On the data source project, add a separate Platform.sh user in the Console and add the newly generated public key in the new account. Provide the least privileges possible (Viewer), for security reasons.

Now, connect in the data sink project without the -A switch:

> ssh xksjd6v6od7iq-master-7rqtwti--app@ssh.eu-3.platform.sh

 ___ _      _    __                    _
| _ \ |__ _| |_ / _|___ _ _ _ __    __| |_
|  _/ / _` |  _|  _/ _ \ '_| '  \ _(_-< ' \
|_| |_\__,_|\__|_| \___/_| |_|_|_(_)__/_||_|


 Welcome to Platform.sh.

 This is environment master-7rqtwti
 of project xksjd6v6od7iq.

web@pff7myd47w7zhnm43cetvkeujy:~$

and rsync still works:

web@pff7myd47w7zhnm43cetvkeujy:~$ rsync -avrz "$(platform ssh -p undntpvafhdn4 -e master --pipe)":/app/web/*.txt /tmp/
Warning: Permanently added 'ssh.eu-3.platform.sh,63.35.24.107' (RSA) to the list of known hosts.
receiving incremental file list
1.txt
2.txt

sent 62 bytes  received 147 bytes  418.00 bytes/sec
total size is 4  speedup is 0.02

Conclusion

The two projects were set up to transfer files in an automated way.

2 Likes
#2
#3