Commit 08e89ee7 authored by Dillenn Terumalai's avatar Dillenn Terumalai
Browse files

Update README.md

parent 02084697
Pipeline #1246 passed with stage
in 3 minutes and 18 seconds
# SPSP Transfer-tool # SPSP Transfer Tool
This is the GitLab repo of the official transfer-tool for SPSP. This is the GitLab repo of the official Transfer Tool(TT) for SPSP.
## Table of Contents ## Table of Contents
- [How does the Transfer Tool (TT) work?](#how-does-the-transfer-tool-tt-work) - [How does the Transfer Tool (TT) work?](#how-does-the-transfer-tool-tt-work)
- [Getting Started](#getting-started) - [Getting Started](#getting-started)
- [Set up a shared drive between users of the same SPSP group](#set-up-a-shared-drive-between-users-of-the-same-SPSP-group) - [Set up a shared drive between users of the same SPSP group](#set-up-a-shared-drive-between-users-of-the-same-spsp-group)
- [Upload the SSH public key](#upload-the-SSH-public-key) - [Upload the SSH public key](#upload-the-SSH-public-key)
- [Installation](#installation) - [Installation](#installation)
- [Configure the .env file](#configure-the-env-file) - [Configure the .env file](#configure-the-env-file)
- [Verify the public key](#verify-the-public-key) - [Verify the public key](#verify-the-public-key)
- [Use the transfer-tool](#use-the-transfer-tool) - [Use the Transfer Tool](#use-the-transfer-tool)
- [Use the automatic mode](#use-the-automatic-mode) - [How to prepare FASTQ files with the metadata file](#how-to-prepare-fastq-files-with-the-metadata-file)
- [How to transfer files easily](#how-to-transfer-files-easily)
- [Use the automatic mode in combination with a CRON task](#use-the-automatic-mode-in-combination-with-a-cron-task)
- [Debugging](#debugging)
- [Authors](#authors) - [Authors](#authors)
## How does the Transfer Tool (TT) work? ## How does the Transfer Tool (TT) work?
...@@ -45,8 +48,8 @@ If you want to be able to use the tool smoothly, make sure that you have: ...@@ -45,8 +48,8 @@ If you want to be able to use the tool smoothly, make sure that you have:
- OS: macOS or Linux - This script can only run on those operative systems, it might be possible to run it on Windows 10 with bash installed but it has not been tested - OS: macOS or Linux - This script can only run on those operative systems, it might be possible to run it on Windows 10 with bash installed but it has not been tested
- SSH: public key uploaded - This script is assuming that you have generated and transferred your public SSH key to the SPSP SFTP Server. If not, please read the [Installation](#installation) chapter - SSH: public key uploaded - This script is assuming that you have generated and transferred your public SSH key to the SPSP SFTP Server. If not, please read the [Installation](#installation) chapter
- GPG: gpg available - This script uses [GnuPG](https://gnupg.org/) to encrypt the data , make sure that it is installed and you can run this command (`which gpg`) - GPG: gpg available - This script uses [GnuPG](https://gnupg.org/) to encrypt the data , make sure that it is installed and you can run this command (`which gpg`)
- COMMANDS: commands available - The transfer-tool relies on multiple commands such as: `sha256sum` or `shasum`, `tar`, `sftp`, `nc` and `gpg`, to perform different operations needed. Make sure that all of them are available (`which sha256sum` for example) - COMMANDS: commands available - The Transfer Tool relies on multiple commands such as: `sha256sum` or `shasum`, `tar`, `sftp`, `nc` and `gpg`, to perform different operations needed. Make sure that all of them are available (`which sha256sum` for example)
- (Optional) CRON: automatic mode - If you want to activate the automatic mode of the transfer-tool make sure that you can setup a CRON task - (Optional) CRON: automatic mode - If you want to activate the automatic mode of the Transfer Tool make sure that you can setup a CRON task
Due to the secure environment where SPSP is hosted, data cannot be directly uploaded via the SPSP online platform. Instead SPSP users should use a dedicated drive within their institution to submit data to SPSP. Due to the secure environment where SPSP is hosted, data cannot be directly uploaded via the SPSP online platform. Instead SPSP users should use a dedicated drive within their institution to submit data to SPSP.
...@@ -65,7 +68,7 @@ Before using the script, you need to make sure that you create an SSH key pair f ...@@ -65,7 +68,7 @@ Before using the script, you need to make sure that you create an SSH key pair f
Start by generating a key pair, make sure to replace `user` by your specific ID provided by the board of SPSP. Open a terminal and type: Start by generating a key pair, make sure to replace `user` by your specific ID provided by the board of SPSP. Open a terminal and type:
```bash ```bash
ssh-keygen -o -a 64 -t ed25519 -f ~/.ssh/id_ed25519 -C "user@spsp.sib.swiss" #PLEASE REPLACE user WITH YOUR OWN LAB/INSTITUTION ID GIVE BY SPSP SUPPORT ssh-keygen -o -a 64 -t ed25519 -f ~/.ssh/id_ed25519 -C "user@spsp.sib.swiss" #PLEASE REPLACE user WITH YOUR OWN LAB/INSTITUTION ID GIVEN BY SPSP SUPPORT
``` ```
You will be asked to `Enter file in which to save the key (/Users/user/.ssh/id_ed25519 or /home/user/.ssh/id_ed25519):`, leave it by default by typing the return key. You will be asked to `Enter file in which to save the key (/Users/user/.ssh/id_ed25519 or /home/user/.ssh/id_ed25519):`, leave it by default by typing the return key.
...@@ -92,9 +95,9 @@ Then click [here](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Request%20Aut ...@@ -92,9 +95,9 @@ Then click [here](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Request%20Aut
### Getting started ### Getting started
A step by step series of commands that tell you how to setup properly the transfer tool. A step by step series of commands that tell you how to setup and use properly the Transfer Tool.
Start by download the transfer-tool on your local machine: Start by downloading the last version of Transfer Tool on your local machine:
[Download](https://gitlab.sib.swiss/SPSP/transfer-tool/-/releases) [Download](https://gitlab.sib.swiss/SPSP/transfer-tool/-/releases)
...@@ -105,32 +108,31 @@ cd ~/path/to/transfer-tool ...@@ -105,32 +108,31 @@ cd ~/path/to/transfer-tool
ls -la ls -la
``` ```
Your terminal should output 4 folders (logs,sent,viruses,bacteria), 2 files (README.md, spsp), 1 hidden folder (.outbox) and 2 hidden files (.env, .pub). Here is a short description of each folder and file: Your terminal should output 4 folders (logs,sent,viruses,bacteria), 2 files (README.md, spsp), 1 hidden folder (.outbox) and 1 hidden file (.pub). Here is a short description of each folder and file:
- **viruses** - main repository where you should move your folder which contains your **viruses** fastq files and metadata file that you want to send - **viruses** - main repository where you should copy your folder which contains your **viruses** fastq files and metadata file that you want to send
- **bacteria** - main repository where you should move your folder which contains your **bacteria** fastq files and metadata file that you want to send - **bacteria** - main repository where you should copy your folder which contains your **bacteria** fastq files and metadata file that you want to send
- **sent** - contains encrypted files with their SHA256 hash that have been properly sent - **sent** - contains encrypted files with their SHA256 hash that have been properly sent
- **logs** - contains all the log files when you use the auto mode (log files record only errors) - **logs** - contains all the log files when you use the auto mode (log files record only errors)
- ***.outbox*** - contains files to be sent to the SPSP server through sftp - ***.outbox*** - contains files to be sent to the SPSP server through sftp
- README.md - user guide - README.md - user guide
- spsp - script containing all the commands to run, type `./spsp help` to display the commands - spsp - script containing all the commands to run, type `./spsp help` to display the commands
- *.env* - setting file to be configured by the user
- *.pub* - public key of SPSP for encryption - *.pub* - public key of SPSP for encryption
Let's start by setting up the transfer-tool. To do so, type: Let's start by setting up the Transfer Tool. To do so, type:
```bash ```bash
sh spsp init sh spsp init
``` ```
This will make sure that the needed commands are available, that the script is executable, that your .env file is properly set and it will also import the public key to your own list of keys. This will make sure that the needed commands are available, that the script is executable, that your .env file is properly set and it will also import the public key to your own list of keys. Refer to the terminal output in case of any error.
### Configure the .env file ### Configure the .env file
/!\ **THIS STEP IS EXTREMELY IMPORTANT, WITHOUT THE CORRECT SETUP, THE TRANSFER WILL FAIL** /!\ /!\ **THIS STEP IS EXTREMELY IMPORTANT, WITHOUT THE CORRECT SETUP, THE TRANSFER WILL FAIL** /!\
You need to have a properly configured .env file to connect to the SFTP server of SPSP. Normally you should have prompted to fill some informations while using the command `sh spsp init`. But **if it is not the case**, you can manually create and file. Create the .env file by using the following commands: You need to have a properly configured .env file to connect to the SFTP server of SPSP. Normally, in the previous step, you should have been prompted to fill some informations while using the command `sh spsp init`. But **if it is not the case**, you can manually create the needed file. Create an .env file by using the following commands:
```bash ```bash
echo 'ID=LAB_ID' > .env #REPLACE LAB_ID BY YOUR OWN ID PROVIDED BY THE SPSP BOARD echo 'ID=LAB_ID' > .env #REPLACE LAB_ID BY YOUR OWN ID PROVIDED BY THE SPSP BOARD
...@@ -148,9 +150,9 @@ If it doesn't, please [contact us](mailto:spsp-support@sib.swiss?subject=[SPSP-S ...@@ -148,9 +150,9 @@ If it doesn't, please [contact us](mailto:spsp-support@sib.swiss?subject=[SPSP-S
### Conclusion ### Conclusion
If everything went well, congratulations, you are ready to use the transfer-tool. If not, please check the output or contact the [support](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Support). If everything went well, congratulations, you are ready to use the Transfer Tool. If not, please check the terminal output or contact the [support](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Support).
## Use the transfer-tool ## Use the Transfer Tool
The following commands are available: The following commands are available:
...@@ -161,15 +163,51 @@ The following commands are available: ...@@ -161,15 +163,51 @@ The following commands are available:
- `./spsp auto`- automatically run the transfer-tool (this needs to be combined with a CRON task, see below for more information), add `--no-archive` or `-NA` to not keep the sent files - `./spsp auto`- automatically run the transfer-tool (this needs to be combined with a CRON task, see below for more information), add `--no-archive` or `-NA` to not keep the sent files
- `./spsp help` - displays the help - `./spsp help` - displays the help
For more information, don't hesite to type: ## How to prepare FASTQ files with the metadata file
```bash This step assumes that you already followed the guide on [spsp.ch](https://spsp.ch/) and will only tell you how you should orgnaize your files for the Transfer Tool to work properly
./spsp help
``` - Start by **identifying** if your sequences are from **viruses** or **bacteria** (if mixed, you need to separate them)
- Make sure that the sequences described in your metadata file **match** the FASTQ files
- **Create a subfolder** as the date of the day (for example: 26-06-12) **inside bacteria/viruses directory** depending on their type
- **COPY** your FASTQ files and the metadata file inside the freshly created folder
**IT IS VERY IMPORTANT TO ALWAYS PUT YOUR FASTQ FILES AND METADATA FILE INSIDE A SUBFOLDER IN THE BACTERIA/VIRUSES DIRECTORY, OR THE TT WILL IGNORE THE FILES**
## How to transfer files easily
If you want to quickly and easily send a batch of FASTQ files with their metadata, just follow those instructions:
## Use the automatic mode - Follow the instruction on [How to prepare FASTQ files with the metadata file](#how-to-prepare-fastq-files-with-the-metadata-file)
- Launch the pipeline by typing `./spsp auto` which will trigger the **automatic** mode
- Once the transfer is over, you should find the sent files (encrypted archive and hash file) in the `sent` folder
To use the automatic mode that will automatically compress, hash, encrypt, and transfer your data, you need to set up a [CRON](https://en.wikipedia.org/wiki/Cron) task. Before the transfer, your directory should look like this:
- /bacteria
- /26-06-20
- sequence1.fastq
- sequence2.fastq
- sequence3.fastq
- metadata-file.xlsx
- /viruses
- /sent
- /logs
- spsp
- README.md
After the transfer, it shoud look like this:
- /bacteria
- /viruses
- /sent
- 26-06-20.tar.gz.gpg
- 26-06-20.tar.gz.sha256
- /logs
- spsp
- README.md
## Use the automatic mode in combination with a CRON task
If you want to use the automatic mode on daily basis, you need to set up a [CRON](https://en.wikipedia.org/wiki/Cron) task.
We recommend the following settings: We recommend the following settings:
...@@ -191,13 +229,17 @@ In order, this is what happens: ...@@ -191,13 +229,17 @@ In order, this is what happens:
8) Transfers `*.sha256` (hash) and `*.gpg` (encrypted tar.gz) files to the corresponding subdirectory (`viruses` or `bacteria`) on the remote server 8) Transfers `*.sha256` (hash) and `*.gpg` (encrypted tar.gz) files to the corresponding subdirectory (`viruses` or `bacteria`) on the remote server
9) (Optional) If you used the automatic mode with the `--no-archive` option, the sent files will not be moved to the `sent` folder and **will be erased** 9) (Optional) If you used the automatic mode with the `--no-archive` option, the sent files will not be moved to the `sent` folder and **will be erased**
If an error occurs during the process, the script will output the error in the log file inside the `.logs` directory and will automatically stop to avoid any more errors. If any error occurs during the process, the script will output the error in the log file inside the `.logs` directory and will automatically stop to avoid any more errors.
Keep in mind that in the CRON task, we are returning the output of the automatic mode of the script inside a file called `spsp.log`. This should be your starting point to check if any error occured. Then, you can check the log file inside the `.logs` folder for more information. Keep in mind that in the CRON task, we are returning the output of the automatic mode of the script inside a file called `spsp.log`. This should be your starting point to check if any error occured. Then, you can check the log file inside the `.logs` folder for more information.
Also, be sure that when you copy the `fastq` or `fastq.gz` files inside the directory, the copy process should be completed before 5 AM (based on the recommended settings), or the script will send incomplete files. Also, be sure that when you copy the `fastq` or `fastq.gz` files inside the directory, the copy process should be completed before 5 AM (based on the recommended settings), or the script will send incomplete files.
Finally, as files may be quite large (several Gb per file), it is up to each institution to decide if all the archives should be kept inside the `sent` folder (default behavior) or not (use the `--no-archive` option). Finally, as files may be quite large (several GB per file), it is up to each institution to decide if all the archives should be kept inside the `sent` folder (default behavior) or not (use the `--no-archive` option).
## Debugging
The Transfer Tool will automatically exit on any error. If you need to debug it, make sure to always check the `logs` folder which contains all the logs. You can quickly identify the log of the day by looking at the name. If you open the log file with a text editor, you will see a short description of what went wrong, allowing you to understand what needs to be fixed. If you don't understand the error message or you don't know what to do, don't hesitate to [contact us](mailto:spsp-support@sib.swiss).
## Authors ## Authors
......
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment