Commit 34ba5308 authored by Dillenn Terumalai's avatar Dillenn Terumalai
Browse files

Version 1.2.0

parent 6b19a280
......@@ -23,12 +23,20 @@ test_compress:
- sh spsp compress test.txt --without-env
- rm test.txt*
test_sign:
test_hash:
stage: test
script:
- echo "Testing sign function"
- echo "Testing hash function"
- echo "Hello World!" > test.txt
- sh spsp sign test.txt --without-env
- sh spsp hash test.txt --without-env
test_encrypt:
stage: test
script:
- echo "Testing encrypt function"
- echo "Hello World!" > test.txt
- gpg --import --fingerprint .pub
- sh spsp encrypt test.txt --without-env
test_help:
stage: test
......@@ -36,7 +44,7 @@ test_help:
- echo "Testing help function"
- sh spsp help --without-env
archive:
production:
script: sh build.sh
artifacts:
paths:
......@@ -44,5 +52,7 @@ archive:
- builds/transfer-tool*.tar
- builds/transfer-tool*.tar.gz
- builds/transfer-tool*.tar.bz2
only:
- master
.gitlab/diagram.png

478 KB | W: | H:

.gitlab/diagram.png

234 KB | W: | H:

.gitlab/diagram.png
.gitlab/diagram.png
.gitlab/diagram.png
.gitlab/diagram.png
  • 2-up
  • Swipe
  • Onion skin
......@@ -17,19 +17,20 @@ This is the GitLab repo of the official transfer-tool for SPSP.
## How does the Transfer Tool (TT) work?
To understand how the TT works, it helps to look at a diagram. In the example below, Labo1 wants to send DNA sequences of bacteria/viruses in a protected manner. The SPSP server has a public key and a private key, which are two mathematically related encryption keys. The public key can be shared with anyone, but only the SPSP server has the private key.
To understand how the TT works, it helps to look at a diagram. In the example on the right, Labo1 wants to send DNA sequences of bacteria/viruses in a protected manner. To do so, the data exchange will rely on asymetric/public key encryption. The SPSP server has a public key and a private key, which are two mathematically related encryption keys. The public key can be shared with any laboratory, but only the SPSP server has the private key.
First, Lab1 uses the TT to compress the sequences files (*.fastq) and metadata file (*.xlsx) into a tar.gz archive.
Then, the TT signs the previously generated archive using SHA-256 algorithm which generates a hash specific to the archive, meaning that if the content changes even slightly, the hash will be completely different.
Then, the TT generates a unique hash of the previously generated archive using SHA-256 algorithm, meaning that if the content changes even slightly, the hash will be completely different.
After that, the TT uses SPSP’s public key to encrypt the archive, turning it into something called ciphertext – scrambled, seemingly random characters.
Finally, the encrypted archive and the hash are uploaded using SFTP protocol which runs over the SSH protocol (which provides communication security and strong encryption).
On the server side, once the archive is properly uploaded, the server will try to decrypt the encrypted archive using its own private key. Then, it will compare the uploaded hash to the hash generated by the server from the decrypted archive to ensure that the content was never changed during the transfer. Finally, if everything goes well, the metadata file is parsed and loaded inside the database. As the server cannot guess for which project of the laboratory the uploaded sequences belong to, an assignment task will be generated.
On the server side, once the archive is properly uploaded, the server will decrypt the encrypted archive using its own private key. Then, it will compare the uploaded hash to the hash generated by the server from the decrypted archive to ensure that the content was never changed during the transfer. Finally, if everything goes well, the metadata file is parsed and loaded inside the database. As the server cannot guess for which project of the laboratory the uploaded sequences belong to, an assignment task will be generated.
Advantages of the Transfer Tool and SFTP services:
**It keeps the data safe from hackers.** Asymetric/public key encryption means only the laboratory sender and the SPSP server have access to the unencrypted data. While in transit, the data remains completely encrypted.
**It uses a secured tunnel.** SFTP protocol protects the integrity of the data using encryption and cryptographic hash functions, and authenticates both the server and the laboratory.
**It checks that the data is left untouched.** By generating a hash, the TT makes sure that the data was never modified during the whole process.
Advantages of the transfer-tool and SFTP services:
- **It keeps the data safe from hackers.** End-to-end encryption means fewer parties have access to the unencrypted data.
- **It uses a secured tunnel.** SFTP protocol protects the integrity of the data using encryption and cryptographic hash functions, and authenticates both the server and the user.
- **It checks that the data is left untouched.** By generating a hash, the TT makes sure that the data was never modified during the whole process.
![Transfer Tool Diagram](.gitlab/diagram.png)
......@@ -55,7 +56,7 @@ Note: The dedicated drive is to be setup by each institution, with the support o
SPSP users must belong to a SPSP group. All the data submitted by a user of a group is visible to all the users of this group. Thus, if multiple SPSP groups are registered to SPSP in your institution, please make sure to set up separate shared drives for each SPSP group.
The shared drive should be hosted on a Linux server, and require authentication using e.g. your institution LDAP. As explained below, data transferred to SPSP is not signed by the user but by the SPSP group. Hence, in order to be able to trace back the origin of potential malware submissions, it is essential that access to the shared drive be controlled at the user level.
The shared drive should be hosted on a Linux server, and require authentication using e.g. your institution LDAP. As explained below, data transferred to SPSP is not done by the user but by the SPSP group. Hence, in order to be able to trace back the origin of potential malware submissions, it is essential that access to the shared drive be controlled at the user level.
## Upload the SSH public key
......@@ -155,6 +156,7 @@ The following commands are available:
- `./spsp compress <folder>` - compress a folder to tar.gz archive
- `./spsp encrypt <file>` - encrypts a file using gpg command and SPSP public key (which needs to be in your own GPG keys list)
- `./spsp hash <file>` - generates the hash of a file using SHA-256 algorithm
- `./spsp transfer <file>` - transfers a file through sftp to SPSP server (your SSH key needs to be validated by SPSP to use this command)
- `./spsp auto`- automatically run the transfer-tool (this needs to be combined with a CRON task, see below for more information), add `--no-archive` or `-NA` to not keep the sent files
- `./spsp help` - displays the help
......@@ -167,7 +169,7 @@ For more information, don't hesite to type:
## Use the automatic mode
To use the automatic mode that will automatically compress, sign, encrypt and transfer your data, you need to set up a [CRON](https://en.wikipedia.org/wiki/Cron) task.
To use the automatic mode that will automatically compress, hash, encrypt, and transfer your data, you need to set up a [CRON](https://en.wikipedia.org/wiki/Cron) task.
We recommend the following settings:
......@@ -175,18 +177,18 @@ We recommend the following settings:
0 5 * * * /path/to/spsp/spsp auto >> /path/to/spsp.log
```
This will launch the transfer-tool at 5 AM every day of the week using the automatic mode and save the output inside a file called `spsp.log` (this will be the main log file).
This will launch the Transfer Tool at 5 AM every day of the week using the automatic mode and save the output inside a file called `spsp.log` (this will be the main log file).
In order, this is what happens:
1) Check that the `.outbox`, `sent`, `viruses`, `bacteria` and `.logs` folders exist.
2) Create a log file using the current date inside `.logs` directory
3) Check if the connection to SPSP works
4) Scan the two `viruses` and `bacteria` directories for any folder; if one is found, check that it contains `.fastq` or `.fastq.gz` and `.xlsx` files at least
5) Compress the folder to tar.gz and move it to `.outbox` directory, then delete the initial folder
6) Then for every file inside `outbox`, sign the file using SHA-256
7) Encrypt the file using the SPSP public key and delete the initial unencrypted compressed file
8) Transfer `*.sha256` (signature) and `*.gpg` (encrypted tar.gz) files to the corresponding subdirectory (`viruses` or `bacteria`) on the remote server
1) Checks that the `.outbox`, `sent`, `viruses`, `bacteria` and `.logs` folders exist.
2) Creates a log file using the current date inside `.logs` directory
3) Checks if the connection to SPSP works
4) Scans the two `viruses` and `bacteria` directories for any folder; if one is found, checks that it contains `.fastq` or `.fastq.gz` and `.xlsx` files at least
5) Compresses the folder to tar.gz and move it to `.outbox` directory, then delete the initial folder
6) Then for every file inside `outbox`, generates the hash of the file using SHA-256
7) Encrypts the file using the SPSP public key and delete the initial unencrypted compressed file
8) Transfers `*.sha256` (hash) and `*.gpg` (encrypted tar.gz) files to the corresponding subdirectory (`viruses` or `bacteria`) on the remote server
9) (Optional) If you used the automatic mode with the `--no-archive` option, the sent files will not be moved to the `sent` folder and **will be erased**
If an error occurs during the process, the script will output the error in the log file inside the `.logs` directory and will automatically stop to avoid any more errors.
......
......@@ -17,8 +17,10 @@ COL_MAGENTA=${ESC_SEQ}"0;35m"
# Define the UID of the recipient for GPG encryption
RECIPIENT="spsp-support@sib.swiss"
# Check for the machine compatibility
# Exit immediately if a simple command exits with a nonzero exit value
set -e
# Check for the machine compatibility
UNAMEOUT="$(uname -s)"
case "${UNAMEOUT}" in
Linux*) MACHINE=linux;;
......@@ -63,9 +65,9 @@ showHelp() {
commands="${COL_LYELLOW}Commands:\n${COL_RESET}"
commands="$commands compress <folder> Compress the ${COL_LGREEN}folder${COL_RESET} or the ${COL_LGREEN}file${COL_RESET} to a ${COL_LYELLOW}tar.gz${COL_RESET} file\n"
commands="$commands encrypt <file> Encrypt a ${COL_LGREEN}file${COL_RESET} using the SPSP public key\n"
commands="$commands sign <file> Sign a file using ${COL_LGREEN}SHA256${COL_RESET}\n"
commands="$commands hash <file> Generates a hash using ${COL_LGREEN}SHA256${COL_RESET}\n"
commands="$commands transfer <file> Transfer the encrypted file using ${COL_LGREEN}SFTP${COL_RESET} to the SPSP server\n"
commands="$commands auto Automatically run the commands to ${COL_CYAN}Compress${COL_RESET}, ${COL_GREEN}Encrypt${COL_RESET}, ${COL_LYELLOW}Sign${COL_RESET} & ${COL_MAGENTA}Transfer${COL_RESET}, add --no-archive option to delete the files after the transfer\n"
commands="$commands auto Automatically run the commands to ${COL_CYAN}Compress${COL_RESET}, ${COL_GREEN}Encrypt${COL_RESET}, ${COL_LYELLOW}Digest${COL_RESET} & ${COL_MAGENTA}Transfer${COL_RESET}, add --no-archive option to delete the files after the transfer\n"
commands="$commands help Shows help screen\n"
commands="$commands test Dry run with fake generated files\n"
commands="$commands cleanup Clean the following directories: logs, sent, viruses, bacteria and .outbox\n"
......@@ -77,9 +79,9 @@ showHelp() {
examples="${COL_LYELLOW}Examples:\n${COL_RESET}"
examples="$examples ${COL_CYAN}$ ./script compress myfolder${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script encrypt myfile.tar.gz${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script sign myencryptedfile${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script transfer myencryptedfile${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script encrypt myfile${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script hash myfile${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script transfer myfile${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script auto${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script test${COL_RESET}\n"
examples="$examples ${COL_CYAN}$ ./script cleanup${COL_RESET}\n"
......@@ -113,7 +115,7 @@ transferFile() {
}
signFile() {
hashFile() {
message="Hashing the file ${COL_LGREEN}$1${COL_RESET} using SHA-256..."
printf "$message\n"
$SHACMD $1 > $1.sha256
......@@ -190,14 +192,11 @@ initMode() {
chmod +x spsp
echo "SPSP: Importing SPSP public key for encryption protocol..."
$GPGCMD --import --fingerprint .pub
echo "SPSP: Please consult: https://gitlab.sib.swiss/SPSP/transfer-tool, to make sure that the fingerprint below matches"
printf "Fingerprint: ${COL_LGREEN}"
$GPGCMD --with-colons --import-options show-only --import --fingerprint < .pub | awk -F: '$1 == "fpr" {print $10;}' | head -n 1
$GPGCMD --with-colons --import-options show-only --import --fingerprint .pub | awk -F: '$1 == "fpr" {print $10;}' | head -n 1
printf "${COL_RESET}"
# echo "SPSP: Verifying key... Please consult https://gitlab.sib.swiss/SPSP/transfer-tool#sign-the-public-key,"
# echo "to check the fingerprint of the key (below) before validating anything. Once checked, please type '3'"
# echo "and then 'y' to sign the key with your own key."
# $GPGCMD --sign-key --ask-cert-level spsp-support@sib.swiss
echo "SPSP: Complete!"
echo "SPSP: You can now use SPSP transfer-tool"
......@@ -312,9 +311,9 @@ autoMode() {
echo "SPSP: Encrypting $FILE using SPSP public key..."
encryptFile $FILE
echo "SPSP: $FILE is now encrypted"
echo "SPSP: Signing $FILE using SHA-256 algorithm..."
signFile "$FILE"
echo "SPSP: $FILE is now signed!"
echo "SPSP: Digesting $FILE using SHA-256 algorithm..."
hashFile "$FILE"
echo "SPSP: $FILE is now hashed!"
if [ -f "$FILE.sha256" ] && [ -f "$FILE.gpg" ]; then
echo "SPSP: Starting the transfer"
transferFile $FILE.gpg /$VIRUSES
......@@ -335,7 +334,7 @@ autoMode() {
fi
else
cd ../..
echo "[`date +"%T"`] local.ERROR: Missing the encrypted version or the signature for $FILE" >> $LOGFILE
echo "[`date +"%T"`] local.ERROR: Missing the encrypted version or the hash for $FILE" >> $LOGFILE
echo "SPSP: Error detected please check $LOGFILE"
echo "########################################"
exit 2
......@@ -352,9 +351,9 @@ autoMode() {
echo "SPSP: Encrypting $FILE using SPSP public key..."
encryptFile $FILE
echo "SPSP: $FILE is now encrypted"
echo "SPSP: Signing $FILE using SHA-256 algorithm..."
signFile "$FILE"
echo "SPSP: $FILE is now signed!"
echo "SPSP: Digesting $FILE using SHA-256 algorithm..."
hashFile "$FILE"
echo "SPSP: $FILE is now hashed!"
if [ -f "$FILE.sha256" ] && [ -f "$FILE.gpg" ]; then
echo "SPSP: Starting the transfer"
transferFile $FILE.gpg /$BACTERIA
......@@ -375,7 +374,7 @@ autoMode() {
fi
else
cd ../..
echo "[`date +"%T"`] local.ERROR: Missing the encrypted version or the signature for $FILE" >> $LOGFILE
echo "[`date +"%T"`] local.ERROR: Missing the encrypted version or the hash for $FILE" >> $LOGFILE
echo "SPSP: Error detected please check $LOGFILE"
echo "########################################"
exit 2
......@@ -498,10 +497,10 @@ if [ $# -gt 0 ]; then
transferFile $@
exit 0
# Sign a afile given as an input
elif [ "$1" == "sign" ]; then
# Digest a file given as an input
elif [ "$1" == "hash" ]; then
shift 1
signFile $@
hashFile $@
exit 0
# Encrypt a folder/file given as an input
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment