Commit 42430a97 authored by Dillenn Terumalai's avatar Dillenn Terumalai
Browse files

Completed the

parent a515c369
......@@ -5,6 +5,7 @@ This is the GitLab repo of the official transfer-tool for SPSP.
## Table of Contents
- [Getting Started](#getting-started)
- [Set up a shared drive between users of the same SPSP group](#set-up-a-shared-drive-between-users-of-the-same-SPSP-group)
- [Upload the SSH public key](#upload-the-SSH-public-key)
- [Installation](#installation)
- [Use the transfer-tool](#use-the-transfer-tool)
......@@ -24,6 +25,16 @@ If you want to be able to use the tool smoothly, make sure that you have:
- GPG: gpg avaiable - This script uses GPG to encrypt the data, make sure that you can run this command (`which gpg`)
- (Optional) CRON: automatic mode - If you want to activate the automatic mode of the transfer-tool make sure that you can setup a CRON task
Due to the secure environment where SPSP is hosted, data cannot be directly uploaded via the SPSP online platform. Instead SPSP users should use a dedicated drive within their institution to submit data to SPSP.
Note: The dedicated drive is to be setup by each institution, with the support of SIB ([see below](#set-up-a-shared-drive-between-users-of-the-same-SPSP-group)). Upon registration of your group to SPSP, SIB will ask you to liaise with your IT department to set this up. Data cannot be submitted to SPSP before this drive has been set up.
## Set up a shared drive between users of the same SPSP group
SPSP users must belong to a SPSP group. All the data submitted by a user of a group is visible to all the users of this group. Thus, if multiple SPSP groups are registered to SPSP in your institution, please make sure to set up separate shared drives for each SPSP group.
The shared drive should be hosted on a Linux server, and require authentication using e.g. your institution LDAP. As explained below, data transferred to SPSP is not signed by the user but by the SPSP group. Hence, in order to be able to trace back the origin of potential malware submissions, it is essential that access to the shared drive be controlled at the user level.
## Upload the SSH public key
Before using the script, you need to make sure that you create an SSH key pair for user authentication.
......@@ -31,7 +42,7 @@ Before using the script, you need to make sure that you create an SSH key pair f
Start by generating a key pair, make sure to replace `user` by your specific ID provided by the board of SPSP. Open a terminal and type:
ssh-keygen -t rsa -b 4096 -C #PLEASE REPLACE USER WITH YOUR OWN ID
ssh-keygen -t rsa -b 4096 -C #PLEASE REPLACE user WITH YOUR OWN ID
You will be asked to `Enter file in which to save the key (/Users/user/.ssh/id_rsa):`, leave it by default by typing the return key.
......@@ -46,7 +57,7 @@ For the next step, you will need to upload your key. Start by copying your key.
Then click [here]( to send your key. Once the key has been validated, you will notified by mail.
Then click [here]([SPSP-SFTP]Request%20Authorization) to send your key. Once the key has been validated, you will notified by mail.
## Installation
......@@ -85,6 +96,14 @@ Let's start by setting up the transfer-tool. To do so, type:
This will make sure that some commands are available, that the script is executable and will also import the public key to your own list of keys.
Next, you need to configure the .env file to use the correct ID. Open the file with a text editor and change the line below:
If everything went well, congratulations, you are ready to use the transfer-tool. If not, please check the output or contact the [support](
## Use the transfer-tool
......@@ -97,8 +116,41 @@ The following commands are available:
- `./spsp auto`- automatically run the transfer-tool (this needs to be combined with a CRON task, see below for more information)
- `./spsp help` - displays the help
For more information, don't hesite to type:
./spsp help
## Use the automatic mode
To use the automatic mode that will automatically compress, encrypt and transfer your data, you need to set up a [CRON]( task.
We recommand the following settings:
0 5 * * * /path/to/spsp/spsp auto >> /path/to/spsp.log
This will run the spsp script at 5 AM every day of the week using the automatic mode and save the output inside a `spsp.log` file. In order, this what happens:
1) Check that the `outbox`, `sent` and `logs` folders exist.
2) Create a log file using the current date
3) Check if the connection to SPSP works
4) Scan the directory for any folder (ignoring `outbox`, `sent` and `logs`), if one is found, check that it contains `.fastq` or `.fastq.gz` and `.xlsx` files
5) Compress the folder to tar.gz and move it to `outbox`, then delete the initial folder
6) Then for every file inside `outbox`, sign the file using SHA-256
7) Encrypt the file using SPSP public key and delete initial unencrypted compressed file
8) Transfer `.sha256` (signature) and `.gpg` (encrypted tar.gz) files and then move them to `sent` folder
If an error occurs during the process, the script will output the error in the log file and will automatically stop to avoid any more errors.
Keep in mind, that in the CRON task we are returning the output of the automatic mode of the script inside a file called `spsp.log`. This should be your starting point to check if any error occured. Then, you can check the log file inside the `logs` folder for more information.
Also, be sure that when you copy the `fastq` or `fastq.gz` files inside the directory, the process should be completed before 5 AM, or the script will send incomplete folder.
Finally, as files may be quite large (several Gb per file). It is up to you to decide if all the archives should stay inside the `sent` folder (default behavior).
## Authors
* **Dillenn TERUMALAI** - *Initial work* - [](
......@@ -228,8 +228,8 @@ if [ $# -gt 0 ]; then
compressFolder $DIRECTORY &> /dev/null
echo "SPSP: Sucessfully compressed!"
echo "SPSP: Removing the source and moving the tar.gz to outbox folder."
mv $DIRECTORY.tar.gz ./outbox/.
echo "[`date +"%T"`] local.ERROR: Missing files (*.fastq/.fastq.gz and/or *.xlsx) inside ${DIRECTORY}" >> $LOGFILE
echo "SPSP: Error detected please check $LOGFILE"
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment