Commit f6e144c7 authored by Dillenn Terumalai's avatar Dillenn Terumalai 🏠

Version 1.1.0

parent 574d2fab
*
!logs
!build.sh
!.outbox
!sent
!.gitlab-ci.yml
......
......@@ -6,20 +6,43 @@ image: centos:latest
before_script:
- dnf install -y nc
- dnf install -y openssh-clients
- dnf install -y zip
- dnf install -y bzip2
test_init:
stage: test
script:
- echo "Testing init function"
- sh spsp init --without-env
# after_script:
# - echo "After script section"
# - echo "For example you might do some cleanup here"
test1:
test_compress:
stage: test
script:
- echo "Do a test here"
- echo "For example run a test suite"
- echo "Testing compress function"
- echo "Hello World!" > test.txt
- sh spsp compress test.txt --without-env
- rm test.txt*
test2:
test_sign:
stage: test
script:
- echo "Do another parallel test here"
- echo "For example run a lint test"
- echo "Testing sign function"
- echo "Hello World!" > test.txt
- sh spsp sign test.txt --without-env
test_help:
stage: test
script:
- echo "Testing help function"
- sh spsp help --without-env
archive:
script: sh build.sh
artifacts:
paths:
- builds/transfer-tool*.zip
- builds/transfer-tool*.tar
- builds/transfer-tool*.tar.gz
- builds/transfer-tool*.tar.bz2
......@@ -8,7 +8,8 @@ This is the GitLab repo of the official transfer-tool for SPSP.
- [Set up a shared drive between users of the same SPSP group](#set-up-a-shared-drive-between-users-of-the-same-SPSP-group)
- [Upload the SSH public key](#upload-the-SSH-public-key)
- [Installation](#installation)
- [Sign the public key](#sign-the-public-key)
- [Configure the .env file](#configure-the-.env-file)
- [Verify the public key](#verify-the-public-key)
- [Use the transfer-tool](#use-the-transfer-tool)
- [Use the automatic mode](#use-the-automatic-mode)
- [Authors](#authors)
......@@ -23,7 +24,7 @@ If you want to be able to use the tool smoothly, make sure that you have:
- OS: macOS or Linux - This script can only run on those operative systems, it might be possible to run it on Windows 10 with bash installed but it has not been tested
- SSH: public key uploaded - This script is assuming that you have generated and transferred your public SSH key to the SPSP SFTP Server. If not, please read the [Installation](#installation) chapter
- GPG: gpg available - This script uses [GnuPG](https://gnupg.org/) to encrypt the data , make sure that it is installed and you can run this command (`which gpg`) and that you already have a private key set ([more info](#sign-the-public-key)
- GPG: gpg available - This script uses [GnuPG](https://gnupg.org/) to encrypt the data , make sure that it is installed and you can run this command (`which gpg`)
- COMMANDS: commands available - The transfer-tool relies on multiple commands such as: `sha256sum` or `shasum`, `tar`, `sftp`, `nc` and `gpg`, to perform different operations needed. Make sure that all of them are available (`which sha256sum` for example)
- (Optional) CRON: automatic mode - If you want to activate the automatic mode of the transfer-tool make sure that you can setup a CRON task
......@@ -44,22 +45,28 @@ Before using the script, you need to make sure that you create an SSH key pair f
Start by generating a key pair, make sure to replace `user` by your specific ID provided by the board of SPSP. Open a terminal and type:
```bash
ssh-keygen -o -a 64 -t ed25519 -f ~/.ssh/id_ed25519 -C "user@spsp.sib.swiss" #PLEASE REPLACE user WITH YOUR OWN ID
ssh-keygen -o -a 64 -t ed25519 -f ~/.ssh/id_ed25519 -C "user@spsp.sib.swiss" #PLEASE REPLACE user WITH YOUR OWN LAB/INSTITUTION ID GIVE BY SPSP SUPPORT
```
You will be asked to `Enter file in which to save the key (/Users/user/.ssh/id_ed25519):`, leave it by default by typing the return key.
You will be asked to `Enter file in which to save the key (/Users/user/.ssh/id_ed25519 or /home/user/.ssh/id_ed25519):`, leave it by default by typing the return key.
Then you will be asked to `Enter passphrase (empty for no passphrase):`, leave it empty and type two times the return key to confirm your choice.
Then you will be asked to `Enter passphrase (empty for no passphrase):`, you can leave it empty or type your own passphrase.
You will then be prompted that your SSH public key has been saved to `/Users/user/.ssh/id_ed25519.pub`. This is your public key that needs to be authorized on the SPSP SFTP Server.
You will then be prompted that your SSH public key has been saved to `/Users/user/.ssh/id_ed25519.pub` or `or /home/user/.ssh/id_ed25519.pub`. This is your public key that needs to be authorized on the SPSP SFTP Server.
For the next step, you will need to upload your key. Start by copying your key. Type the following to display the public key:
For macOS:
```bash
cat /Users/user/.ssh/id_ed25519.pub #PLEASE REPLACE user WITH YOUR LOCAL ACCOUNT
```
Then click [here](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Request%20Authorization) to send your key. Once the key has been validated, you will notified by mail.
For linux:
```bash
cat /home/user/.ssh/id_ed25519.pub #PLEASE REPLACE user WITH YOUR LOCAL ACCOUNT
```
Then click [here](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Request%20Authorization) to send your key. Once the key has been validated, you will be notified by mail.
## Installation
......@@ -67,62 +74,59 @@ Then click [here](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Request%20Aut
A step by step series of commands that tell you how to setup properly the transfer tool.
Start by cloning the transfer-tool on your local machine:
Start by download the transfer-tool on your local machine:
```bash
git clone https://gitlab.isb-sib.ch/SPSP/transfer-tool.git
```
[Download](https://gitlab.sib.swiss/SPSP/transfer-tool/-/releases)
Then get inside the directory:
Extract the downloaded archive where you want and access it with your terminal:
```bash
cd transfer-tool
cd ~/path/to/transfer-tool
ls -la
```
Your terminal should output 3 folders (logs,outbox,sent) and 2 files (README.md, spsp) and 2 hidden files (.env, .pub). Here is a short description of each folder and file:
Your terminal should output 4 folders (logs,sent,viruses,bacteria), 2 files (README.md, spsp), 1 hidden folder (.outbox) and 2 hidden files (.env, .pub). Here is a short description of each folder and file:
- **viruses** - main repository where you should move your folder which contains your **viruses** fastq files and metadata file that you want to send
- **bacteria** - main repository where you should move your folder which contains your **bacteria** fastq files and metadata file that you want to send
- **sent** - contains encrypted files with their SHA256 hash that have been properly sent
- **.logs** - contains all the log files when you use the auto mode (log files record only errors)
- **.outbox** - contains files to be sent to the SPSP server through sftp
- *README.md* - user guide
- *spsp* - script containing all the commands to run, type `./spsp help` to display the commands
- **logs** - contains all the log files when you use the auto mode (log files record only errors)
- ***.outbox*** - contains files to be sent to the SPSP server through sftp
- README.md - user guide
- spsp - script containing all the commands to run, type `./spsp help` to display the commands
- *.env* - setting file to be configured by the user
- *.pub* - public key of SPSP for encryption
- *.gitlab* - some markdown files for GitLab templating
(.git and .gitignore are juste GIT related files)
Let's start by setting up the transfer-tool. To do so, type:
```bash
./spsp init
sh spsp init
```
This will make sure that some commands are available, that the script is executable and will also import the public key to your own list of keys.
This will make sure that the needed commands are available, that the script is executable, that your .env file is properly set and it will also import the public key to your own list of keys.
### Verify the public key
### Configure the .env file
At one point, the terminal should output the fingerprint of the imported key. Please make sure that the fingerprint corresponds to:
/!\ **THIS STEP IS EXTREMELY IMPORTANT, WITHOUT THE CORRECT SETUP, THE TRANSFER WILL FAIL** /!\
**ABC9 FC14 AAC9 52E7 767F D14A 48B7 0E72 4BAF E0A3**
If it doesn't, please [contact us](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Wrong%20Public%20Key) and send us the public key (.pub file in the directory).
You need to have a properly configured .env file to connect to the SFTP server of SPSP. Normally you should have prompted to fill some informations while using the command `sh spsp init`. But **if it is not the case**, you can manually create and file. Create the .env file by using the following commands:
### Configure the .env file
```bash
echo 'ID=LAB_ID' > .env #REPLACE LAB_ID BY YOUR OWN ID PROVIDED BY THE SPSP BOARD
echo 'HOST=spsp-sftp.vital-it.ch' >> .env #DO NOT CHANGE THIS LINE
echo 'SFTP_URL=${ID}@${HOST}:/data' >> .env #DO NOT CHANGE THIS LINE
```
/!\ **THIS STEP IS EXTREMELY IMPORTANT, WITHOUT THE CORRECT SETUP, THE TRANSFER WILL FAIL** /!\
### Verify the public key
At one point, the terminal should output the fingerprint (in green) of the imported key. Please make sure that the fingerprint corresponds to:
Next, you need to configure the .env file to use the correct ID. Open the file with a text editor and change the line below:
**ABC9 FC14 AAC9 52E7 767F D14A 48B7 0E72 4BAF E0A3**
```
ID=TST_0001 #CHANGE THIS LINE BY USING YOUR OWN ID PROVIDED BY THE SPSP BOARD
HOST=spsp-sftp.vital-it.ch #DO NOT CHANGE THIS LINE
SFTP_URL=${ID}@${HOST}:/data #DO NOT CHANGE THIS LINE
```
If it doesn't, please [contact us](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Wrong%20Public%20Key) and send us the public key (.pub file in the directory).
### Conclusion
If everything went well, congratulations, you are ready to use the transfer-tool. If not, please check the output or contact the [support](mailto:spsp-support@sib.swiss?subject=[SPSP-SFTP]Support).
......@@ -163,14 +167,14 @@ In order, this is what happens:
5) Compress the folder to tar.gz and move it to `.outbox` directory, then delete the initial folder
6) Then for every file inside `outbox`, sign the file using SHA-256
7) Encrypt the file using the SPSP public key and delete the initial unencrypted compressed file
8) Transfer `.sha256` (signature) and `.gpg` (encrypted tar.gz) files to the corresponding subdirectory (`viruses` or `bacteria`) on the remote server
9) (Optional)If you used the automatic mode with the `--no-archive` option, the sent files will be moved to the `sent` folder, if not they will be erased
8) Transfer `*.sha256` (signature) and `*.gpg` (encrypted tar.gz) files to the corresponding subdirectory (`viruses` or `bacteria`) on the remote server
9) (Optional) If you used the automatic mode with the `--no-archive` option, the sent files will not be moved to the `sent` folder and **will be erased**
If an error occurs during the process, the script will output the error in the log file inside the `.logs` directory and will automatically stop to avoid any more errors.
Keep in mind that in the CRON task, we are returning the output of the automatic mode of the script inside a file called `spsp.log`. This should be your starting point to check if any error occured. Then, you can check the log file inside the `.logs` folder for more information.
Also, be sure that when you copy the `fastq` or `fastq.gz` files inside the directory, the process should be completed before 5 AM (based on the recommended settings), or the script will send incomplete files.
Also, be sure that when you copy the `fastq` or `fastq.gz` files inside the directory, the copy process should be completed before 5 AM (based on the recommended settings), or the script will send incomplete files.
Finally, as files may be quite large (several Gb per file), it is up to each institution to decide if all the archives should be kept inside the `sent` folder (default behavior) or not (use the `--no-archive` option).
......
#!/usr/bin/env bash
# abort on errors
set -e
# set colors for output
ESC_SEQ="\033["
COL_CYAN=${ESC_SEQ}"0;36m"
COL_RED=${ESC_SEQ}"0;31m"
COL_GREY=${ESC_SEQ}"1;30m"
BACK_CYAN=${ESC_SEQ}"0;30;46m"
BACK_RED=${ESC_SEQ}"0;30;41m"
BACK_GREEN=${ESC_SEQ}"0;30;42m"
COL_RESET=${ESC_SEQ}"39;49;00m"
if [ ! -f spsp ]; then
TIME=`date +"%T"`
printf "${BACK_RED} ERROR ${COL_RESET} ${COL_RED}Stopped with 1 error${COL_RESET} ${COL_GREY} ${TIME} ${COL_RESET}\n"
printf "${BACK_RED} error ${COL_RESET} File not found: spsp script has not been found, please check your current directory '$(pwd)'."
exit 2
else
version=$(awk -F'"' '/^VERSION=/ {print $2}' spsp )
fi
if [ ! -f README.md ]; then
TIME=`date +"%T"`
printf "${BACK_RED} ERROR ${COL_RESET} ${COL_RED}Stopped with 1 error${COL_RESET} ${COL_GREY} ${TIME} ${COL_RESET}\n"
printf "${BACK_RED} error ${COL_RESET} File not found: README.md has not been found, please check your current directory '$(pwd)'."
exit 2
fi
if [ ! -f .pub ]; then
TIME=`date +"%T"`
printf "${BACK_RED} ERROR ${COL_RESET} ${COL_RED}Stopped with 1 error${COL_RESET} ${COL_GREY} ${TIME} ${COL_RESET}\n"
printf "${BACK_RED} error ${COL_RESET} File not found: .pub (PGP SPSP public key) has not been found, please check your current directory '$(pwd)'."
exit 2
fi
OUTBOX=".outbox"
if [ ! -d "$OUTBOX" ]; then
mkdir $OUTBOX
fi
SENT=sent
if [ ! -d "$SENT" ]; then
mkdir $SENT
fi
LOGS=logs
if [ ! -d "$LOGS" ]; then
mkdir $LOGS
fi
VIRUSES=viruses
if [ ! -d "$VIRUSES" ]; then
mkdir $VIRUSES
fi
BACTERIA=bacteria
if [ ! -d "$BACTERIA" ]; then
mkdir $BACTERIA
fi
#Creating builds folder if necessary or clean it
if [ ! -d builds ]; then
mkdir builds
else
rm -rf ./builds/*
fi
echo "- Archiving version ${version} of the transfer-tool..."
#Creating ZIP archive and test it
zip ./builds/transfer-toolv${version}.zip * .* -qq -x .git/ .gitignore .gitlab/ .gitlab-ci.yml .env ../ build.sh logs/** bacteria/** viruses/** builds/ .outbox/** sent/** .DS_Store
printf "${BACK_GREEN} DONE ${COL_RESET} Build archived. The ${COL_CYAN}zip${COL_RESET} archive has been created.\n"
#Creating TAR archive
tar cf ./builds/transfer-toolv${version}.tar --exclude='.DS_Store' --exclude='.git' --exclude='.gitlab' --exclude='.gitignore' --exclude='.gitlab-ci.yml' --exclude='.env' --exclude='build.sh' --exclude "logs/*.*" --exclude='bacteria/*.*' --exclude='viruses/*.*' --exclude='.outbox/*.*' --exclude='sent/*.*' --exclude='builds' .
printf "${BACK_GREEN} DONE ${COL_RESET} Build archived. The ${COL_CYAN}tar${COL_RESET} archive has been created.\n"
#Creating TAR.GZ archive
tar cfz ./builds/transfer-toolv${version}.tar.gz --exclude='.DS_Store' --exclude='.git' --exclude='.gitlab' --exclude='.gitignore' --exclude='.gitlab-ci.yml' --exclude='.env' --exclude='build.sh' --exclude "logs/*.*" --exclude='bacteria/*.*' --exclude='viruses/*.*' --exclude='.outbox/*.*' --exclude='sent/*.*' --exclude='builds' .
printf "${BACK_GREEN} DONE ${COL_RESET} Build archived. The ${COL_CYAN}tar.gz${COL_RESET} archive has been created.\n"
#Creating TAR.BZ2 archive
tar cfj ./builds/transfer-toolv${version}.tar.bz2 --exclude='.DS_Store' --exclude='.git' --exclude='.gitlab' --exclude='.gitignore' --exclude='.gitlab-ci.yml' --exclude='.env' --exclude='build.sh' --exclude "logs/*.*" --exclude='bacteria/*.*' --exclude='viruses/*.*' --exclude='.outbox/*.*' --exclude='sent/*.*' --exclude='builds' .
printf "${BACK_GREEN} DONE ${COL_RESET} Build archived. The ${COL_CYAN}tar.bz2${COL_RESET} archive has been created.\n"
printf "${BACK_GREEN} DONE ${COL_RESET} All build archived. The ${COL_CYAN}zip, tar, tar.gz and tar.gz2${COL_RESET} archives has been created.\n"
printf "${BACK_CYAN} INFO ${COL_RESET} Check out official releases at ${COL_CYAN}https://gitlab.sib.swiss/SPSP/transfer-tool/-/releases${COL_RESET}\n\n\n"
exit 0
\ No newline at end of file
......@@ -73,6 +73,7 @@ showHelp() {
options="${COL_LYELLOW}Options:\n${COL_RESET}"
options="$options --help, -h Shows Help (this screen)\n"
options="$options --version, -V, version Show Version\n"
options="$options --without-env, -W Use environment variables from the system and the .env file\n"
examples="${COL_LYELLOW}Examples:\n${COL_RESET}"
examples="$examples ${COL_CYAN}$ ./script compress myfolder${COL_RESET}\n"
......@@ -142,12 +143,14 @@ cleanup() {
SENT="sent"
VIRUSES="viruses"
BACTERIA="bacteria"
BUILDS="builds"
rm -rf ./$OUTBOX/*
rm -rf ./$LOGS/*
rm -rf ./$SENT/*
rm -rf ./$VIRUSES/*
rm -rf ./$BACTERIA/*
rm -rf ./$BUILDS/*
}
initMode() {
......@@ -417,13 +420,18 @@ generateFakeFiles() {
echo "rm fake*" | sftp ${SFTP_URL}/viruses || exit 1
echo "rm fake*" | sftp ${SFTP_URL}/bacteria || exit 1
echo "Files has been properly removed!"
message="Removing the ${COL_LGREEN}local fake files${COL_RESET} from SFTP..."
printf "$message\n"
cleanup
echo "Files has been properly removed!"
exit 0
}
# If we pass any arguments...
if [ $# -gt 0 ]; then
if [[ "$@" == *--without-env* ]] || [[ "$@" == *-without-env* ]]; then
if [[ "$@" == *--without-env* ]] || [[ "$@" == *-W* ]]; then
echo "Ignoring the .env file"
echo "Using environment variables"
else
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment