Skip to content

Commit

Permalink
Update README.md
Browse files Browse the repository at this point in the history
  • Loading branch information
danlapid authored Sep 1, 2022
1 parent c9276a0 commit c070e86
Showing 1 changed file with 54 additions and 38 deletions.
92 changes: 54 additions & 38 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,40 +1,56 @@
# [Oneway-Filesync](https://github.com/danlapid/oneway-filesync/)

Sync files between different hosts over a one way network link such as a Data Diode

[![Build Status](https://github.com/danlapid/oneway-filesync/actions/workflows/build.yml/badge.svg)](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ABuild)
[![Test Status](https://github.com/danlapid/oneway-filesync/actions/workflows/test.yml/badge.svg)](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ATest)
[![Coverage Status](https://coveralls.io/repos/github/danlapid/oneway-filesync/badge.svg?branch=main)](https://coveralls.io/github/danlapid/oneway-filesync?branch=main)
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)

## Usage

### Download binaries from [Releases](https://github.com/danlapid/oneway-filesync/releases)

Create a config.toml file for both the sender and the receiver similarly to the example in the repo

On the source machine:

```
./sender
./watcher
```

On the target machine:

```
./receiver
```

To send a specific file through that is not in the watched folder:

``` ./sendfiles <file/dir path> ```

## Data flow

### Sender side:

QueueReader (From DB) -> FileReader -> FecEncoder -> BandwidthLimiter -> UdpSender

### -> Data Diode ->

### Receiver side:

UdpReceiver -> ShareAssember -> FecDecoder -> FileWriter -> FileCloser (Updates receiver DB)

## Config

- ReceiverIP : The IP the receiver will listen on and the sender will send to
- ReceiverPort : The port the receiver will listen on and the sender will send to
- BandwidthLimit : in Bytes/Second the sender will limit itself to this amount, suggested to be a little under link speed, if you get "buffers are filling up" error code then you might need more compute power on the receiver
- ChunkSize : Data length sent in each udp datagram should be 42 bytes smaller than link MTU (14 Ethernet, 20 IP, 8 UDP) for compute efficiency it is suggested to increase the link MTU and then increase this value as well
- EncryptedOutput : If true the files will be encrypted in a zip file with password `filesync` before being sent and saved to the receiver as the encrypted zip
- ChunkFecRequired : Reed Solomon FEC parameter, the amount of shares that must arrive for the chunk to be reconstructed
- ChunkFecTotal : Reed Solomon FEC parameter, the total amount of shares that will be sent, it is suggested that this will be a multiple of ChunkFecRequired
- OutDir : Directory to write output files to on the receiver, the original directory structure will be perserved and appended to this path
- WatchDir : Directory the watcher will detect file changes on and send every changed files from

go run oneway-filesync/cmd/receiver \
go run oneway-filesync/cmd/sender



DB Reader:
output: paths
FileReader
input: paths
output: chunks
->
FEC
input: chunks
output: chunks
->
Rate Limiter
input: chunks
output: chunks
->
Socket Sender
input: chunks


Socker reader
output: chunks
->
DeFEC
input: chunks
output: chunks
->
File Writer
input: chunks:
output: Finished files


psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT * FROM files"
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "DELETE FROM files"
TODO: different db for sent and received

0 comments on commit c070e86

Please sign in to comment.