Commit 8e3857c1 authored by Dillenn Terumalai's avatar Dillenn Terumalai
Browse files

Update README.md

parent 6c3a3b50
...@@ -8,6 +8,7 @@ This is the GitLab repo of the official transfer-tool for SPSP. ...@@ -8,6 +8,7 @@ This is the GitLab repo of the official transfer-tool for SPSP.
- [Set up a shared drive between users of the same SPSP group](#set-up-a-shared-drive-between-users-of-the-same-SPSP-group) - [Set up a shared drive between users of the same SPSP group](#set-up-a-shared-drive-between-users-of-the-same-SPSP-group)
- [Upload the SSH public key](#upload-the-SSH-public-key) - [Upload the SSH public key](#upload-the-SSH-public-key)
- [Installation](#installation) - [Installation](#installation)
- [Sign the public key](#sign-the-public-key)
- [Use the transfer-tool](#use-the-transfer-tool) - [Use the transfer-tool](#use-the-transfer-tool)
- [Use the automatic mode](#use-the-automatic-mode) - [Use the automatic mode](#use-the-automatic-mode)
- [Authors](#authors) - [Authors](#authors)
...@@ -20,9 +21,10 @@ These instructions will allow you to use locally the tool to transfer your data. ...@@ -20,9 +21,10 @@ These instructions will allow you to use locally the tool to transfer your data.
If you want to be able to use the tool smoothly, make sure that you have: If you want to be able to use the tool smoothly, make sure that you have:
- OS: macOS or Linux - This script can only run on those operative systems - OS: macOS or Linux - This script can only run on those operative systems, it might be possible to run it on Windows 10 with bash installed but it has not been tested
- SSH: public key uploaded - This script is assuming that you have generated and transferred your public SSH key to the SPSP SFTP Server. If not, please read the [Installation](#installation) chapter - SSH: public key uploaded - This script is assuming that you have generated and transferred your public SSH key to the SPSP SFTP Server. If not, please read the [Installation](#installation) chapter
- GPG: gpg avaiable - This script uses [GnuPG](https://gnupg.org/) to encrypt the data , make sure that it is installed and you can run this command (`which gpg`) - GPG: gpg available - This script uses [GnuPG](https://gnupg.org/) to encrypt the data , make sure that it is installed and you can run this command (`which gpg`) and that you already have a private key set ([more info](#sign-the-public-key)
- COMMANDS: commands available - The transfer-tool relies on multiple commands such as: `sha256sum`, `tar`, `sftp`, `nc` or `gpg`, to perform different operations needed. Make sure that all of them are available (`which sha256sum` for example)
- (Optional) CRON: automatic mode - If you want to activate the automatic mode of the transfer-tool make sure that you can setup a CRON task - (Optional) CRON: automatic mode - If you want to activate the automatic mode of the transfer-tool make sure that you can setup a CRON task
Due to the secure environment where SPSP is hosted, data cannot be directly uploaded via the SPSP online platform. Instead SPSP users should use a dedicated drive within their institution to submit data to SPSP. Due to the secure environment where SPSP is hosted, data cannot be directly uploaded via the SPSP online platform. Instead SPSP users should use a dedicated drive within their institution to submit data to SPSP.
...@@ -96,15 +98,48 @@ Let's start by setting up the transfer-tool. To do so, type: ...@@ -96,15 +98,48 @@ Let's start by setting up the transfer-tool. To do so, type:
This will make sure that some commands are available, that the script is executable and will also import the public key to your own list of keys. This will make sure that some commands are available, that the script is executable and will also import the public key to your own list of keys.
### Sign the public key
At one point, you will be asked to sign the SPSP provided public key. This public key is used for [asymmetric encryption](https://en.wikipedia.org/wiki/Public-key_cryptography) of your data.
Before the transfer can be allowed to use it, you need check the fingerprint of the public key and manually sign the key (see below). If the public used for encryption is not exactly the one provided with the tranfer-tool, that means that we won't be able to decrypt the data correctly.
/!\ **TO SIGN THE PUBLIC KEY YOU NEED YOUR OWN PRIVATE KEY, TO DO SO PLEASE CHECK THIS [LINK](https://www.gnupg.org/gph/en/manual.html#AEN26) OR THIS [LINK](https://help.github.com/en/github/authenticating-to-github/generating-a-new-gpg-key)** /!\
```bash
pub rsa2048/48B70E724BAFE0A3
created: 2019-12-16 expires: never usage: SC
trust: full validity: unknown
Primary key fingerprint: ABC9 FC14 AAC9 52E7 767F D14A 48B7 0E72 4BAF E0A3
SPSP SFTP <spsp-support@sib.swiss>
How carefully have you verified the key you are about to sign actually belongs
to the person named above? If you don't know what to answer, enter "0".
(0) I will not answer. (default)
(1) I have not checked at all.
(2) I have done casual checking.
(3) I have done very careful checking.
Your selection? (enter '?' for more information):
```
Make sure that the "Primary key fingerprint" corresponds to:
**ABC9 FC14 AAC9 52E7 767F D14A 48B7 0E72 4BAF E0A3**
If it doesn't, please [contact us](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Wrong%20Public%20Key) and send us the public key (.pub file in the directory).
Then, type **3** to validate your choice
Next, you need to configure the .env file to use the correct ID. Open the file with a text editor and change the line below: Next, you need to configure the .env file to use the correct ID. Open the file with a text editor and change the line below:
``` ```
ID=TST_0001 #CHANGE THIS LINE BY USING YOUR OWN ID PROVIDED BY THE SPSP BOARD ID=TST_0001 #CHANGE THIS LINE BY USING YOUR OWN ID PROVIDED BY THE SPSP BOARD
HOST=spsp-sftp.vital-it.ch HOST=spsp-sftp.vital-it.ch #DO NOT CHANGE THIS LINE
SFTP_URL=${ID}@${HOST}:/data SFTP_URL=${ID}@${HOST}:/data #DO NOT CHANGE THIS LINE
``` ```
If everything went well, congratulations, you are ready to use the transfer-tool. If not, please check the output or contact the [support](mailto:spsp-support@sib.swiss). If everything went well, congratulations, you are ready to use the transfer-tool. If not, please check the output or contact the [support](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Support).
## Use the transfer-tool ## Use the transfer-tool
...@@ -113,7 +148,7 @@ The following commands are available: ...@@ -113,7 +148,7 @@ The following commands are available:
- `./spsp compress <folder>` - compress a folder to tar.gz archive - `./spsp compress <folder>` - compress a folder to tar.gz archive
- `./spsp encrypt <file>` - encrypts a file using gpg command and SPSP public key (which needs to be in your own GPG keys list) - `./spsp encrypt <file>` - encrypts a file using gpg command and SPSP public key (which needs to be in your own GPG keys list)
- `./spsp transfer <file>` - transfers a file through sftp to SPSP server (your SSH key needs to be validated by SPSP to use this command) - `./spsp transfer <file>` - transfers a file through sftp to SPSP server (your SSH key needs to be validated by SPSP to use this command)
- `./spsp auto`- automatically run the transfer-tool (this needs to be combined with a CRON task, see below for more information) - `./spsp auto`- automatically run the transfer-tool (this needs to be combined with a CRON task, see below for more information), add `--no-archive` or `no-archive` to keep the sent files
- `./spsp help` - displays the help - `./spsp help` - displays the help
For more information, don't hesite to type: For more information, don't hesite to type:
...@@ -126,31 +161,34 @@ For more information, don't hesite to type: ...@@ -126,31 +161,34 @@ For more information, don't hesite to type:
To use the automatic mode that will automatically compress, encrypt and transfer your data, you need to set up a [CRON](https://en.wikipedia.org/wiki/Cron) task. To use the automatic mode that will automatically compress, encrypt and transfer your data, you need to set up a [CRON](https://en.wikipedia.org/wiki/Cron) task.
We recommand the following settings: We recommend the following settings:
``` ```
0 5 * * * /path/to/spsp/spsp auto >> /path/to/spsp.log 0 5 * * * /path/to/spsp/spsp auto >> /path/to/spsp.log
``` ```
This will run the spsp script at 5 AM every day of the week using the automatic mode and save the output inside a `spsp.log` file. In order, this what happens: This will launch the transfer-tool at 5 AM every day of the week using the automatic mode and save the output inside a file called `spsp.log` (this will be the main log file).
In order, this is what happens:
1) Check that the `outbox`, `sent` and `logs` folders exist. 1) Check that the `.outbox`, `sent`, `viruses`, `bacteria` and `.logs` folders exist.
2) Create a log file using the current date 2) Create a log file using the current date inside `.logs` directory
3) Check if the connection to SPSP works 3) Check if the connection to SPSP works
4) Scan the directory for any folder (ignoring `outbox`, `sent` and `logs`); if one is found, check that it contains `.fastq` or `.fastq.gz` and `.xlsx` files 4) Scan the two `viruses` and `bacteria` directories for any folder; if one is found, check that it contains `.fastq` or `.fastq.gz` and `.xlsx` files at least
5) Compress the folder to tar.gz and move it to `outbox`, then delete the initial folder 5) Compress the folder to tar.gz and move it to `.outbox` directory, then delete the initial folder
6) Then for every file inside `outbox`, sign the file using SHA-256 6) Then for every file inside `outbox`, sign the file using SHA-256
7) Encrypt the file using the SPSP public key and delete the initial unencrypted compressed file 7) Encrypt the file using the SPSP public key and delete the initial unencrypted compressed file
8) Transfer `.sha256` (signature) and `.gpg` (encrypted tar.gz) files and then move them to the `sent` folder 8) Transfer `.sha256` (signature) and `.gpg` (encrypted tar.gz) files to the corresponding subdirectory (`viruses` or `bacteria`) on the remote server
9) (Optional)If you used the automatic mode with the `--no-archive` option, the sent files will be moved to the `sent` folder, if not they will be erased
If an error occurs during the process, the script will output the error in the log file and will automatically stop to avoid any more errors. If an error occurs during the process, the script will output the error in the log file inside the `.logs` directory and will automatically stop to avoid any more errors.
Keep in mind that in the CRON task, we are returning the output of the automatic mode of the script inside a file called `spsp.log`. This should be your starting point to check if any error occured. Then, you can check the log file inside the `logs` folder for more information. Keep in mind that in the CRON task, we are returning the output of the automatic mode of the script inside a file called `spsp.log`. This should be your starting point to check if any error occured. Then, you can check the log file inside the `.logs` folder for more information.
Also, be sure that when you copy the `fastq` or `fastq.gz` files inside the directory, the process should be completed before 5 AM, or the script will send an incomplete folder. Also, be sure that when you copy the `fastq` or `fastq.gz` files inside the directory, the process should be completed before 5 AM (based on the recommended settings), or the script will send incomplete files.
Finally, as files may be quite large (several Gb per file), it is up to each institution to decide if all the archives should be kept inside the `sent` folder (default behavior). Finally, as files may be quite large (several Gb per file), it is up to each institution to decide if all the archives should be kept inside the `sent` folder (default behavior) or not (use the `--no-archive` option).
## Authors ## Authors
* **Dillenn TERUMALAI** - *Initial work* - [dillenn.terumalai@sib.swiss](mailto:dillenn.teruamlai@sib.swiss) * **Dillenn TERUMALAI** - *Initial work* - [dillenn.terumalai@sib.swiss](mailto:dillenn.terumalai@sib.swiss)
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment