-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
1 changed file
with
54 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,40 +1,56 @@ | ||
# [Oneway-Filesync](https://github.com/danlapid/oneway-filesync/) | ||
|
||
Sync files between different hosts over a one way network link such as a Data Diode | ||
|
||
[](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ABuild) | ||
[](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ATest) | ||
[](https://coveralls.io/github/danlapid/oneway-filesync?branch=main) | ||
[](https://opensource.org/licenses/MIT) | ||
|
||
## Usage | ||
|
||
### Download binaries from [Releases](https://github.com/danlapid/oneway-filesync/releases) | ||
|
||
Create a config.toml file for both the sender and the receiver similarly to the example in the repo | ||
|
||
On the source machine: | ||
|
||
``` | ||
./sender | ||
./watcher | ||
``` | ||
|
||
On the target machine: | ||
|
||
``` | ||
./receiver | ||
``` | ||
|
||
To send a specific file through that is not in the watched folder: | ||
|
||
``` ./sendfiles <file/dir path> ``` | ||
|
||
## Data flow | ||
|
||
### Sender side: | ||
|
||
QueueReader (From DB) -> FileReader -> FecEncoder -> BandwidthLimiter -> UdpSender | ||
|
||
### -> Data Diode -> | ||
|
||
### Receiver side: | ||
|
||
UdpReceiver -> ShareAssember -> FecDecoder -> FileWriter -> FileCloser (Updates receiver DB) | ||
|
||
## Config | ||
|
||
- ReceiverIP : The IP the receiver will listen on and the sender will send to | ||
- ReceiverPort : The port the receiver will listen on and the sender will send to | ||
- BandwidthLimit : in Bytes/Second the sender will limit itself to this amount, suggested to be a little under link speed, if you get "buffers are filling up" error code then you might need more compute power on the receiver | ||
- ChunkSize : Data length sent in each udp datagram should be 42 bytes smaller than link MTU (14 Ethernet, 20 IP, 8 UDP) for compute efficiency it is suggested to increase the link MTU and then increase this value as well | ||
- EncryptedOutput : If true the files will be encrypted in a zip file with password `filesync` before being sent and saved to the receiver as the encrypted zip | ||
- ChunkFecRequired : Reed Solomon FEC parameter, the amount of shares that must arrive for the chunk to be reconstructed | ||
- ChunkFecTotal : Reed Solomon FEC parameter, the total amount of shares that will be sent, it is suggested that this will be a multiple of ChunkFecRequired | ||
- OutDir : Directory to write output files to on the receiver, the original directory structure will be perserved and appended to this path | ||
- WatchDir : Directory the watcher will detect file changes on and send every changed files from | ||
|
||
go run oneway-filesync/cmd/receiver \ | ||
go run oneway-filesync/cmd/sender | ||
|
||
|
||
|
||
DB Reader: | ||
output: paths | ||
FileReader | ||
input: paths | ||
output: chunks | ||
-> | ||
FEC | ||
input: chunks | ||
output: chunks | ||
-> | ||
Rate Limiter | ||
input: chunks | ||
output: chunks | ||
-> | ||
Socket Sender | ||
input: chunks | ||
|
||
|
||
Socker reader | ||
output: chunks | ||
-> | ||
DeFEC | ||
input: chunks | ||
output: chunks | ||
-> | ||
File Writer | ||
input: chunks: | ||
output: Finished files | ||
|
||
|
||
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT * FROM files" | ||
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "DELETE FROM files" | ||
TODO: different db for sent and received |