rfc doc

makeomatic · Jul 10, 2019 · f866653 · f866653
1 parent c57e797
commit f866653
Show file tree

Hide file tree

Showing 3 changed files with 120 additions and 66 deletions.
diff --git a/csrc/doc/rfc/00_design.md b/csrc/doc/rfc/00_design.md
@@ -0,0 +1,117 @@
+# Proposal
+As described in Redis Module API modules have more benefints than general LUA scripts. In current environment, lua script locking db for a long period. The main reason of this module: provide non db-locking access for time-consuming operations.
+
+# Design
+* Projects consists from one redis module `filtered_sort` and node.js wrapper `redis-filtered-sort`.
+
+* By default, module going to have 2 build options: Docker based build for redis:5.0-alpine and local system `gcc`
+
+* Exported functions for dev purpose prefixed with "c" prefix
+
+* Current module behaviour copied from existing scripts for compat with lua.
+
+* in Redis Cluster environment: all keys and sets MUST use {prefix} notation, to be stored on same node. Otherwise module won't work correctly.
+
+
+## Expected API and behaviour
+Module will export into Redis command namespace these commands:
+#### cfsort 
+Sorts, filters and paginates provided SET `idSetKey` using data from `metaKeyPattern`.`hashKey` using `sortOrder`,`filter` and stores processed data for `expire` seconds
+
+* **Method usage**: `cfsort idSetKey metaKeyPattern hashKey sortOrder filter(jsonFormattedString) curTime:unixtime [offset:int] [limit:int] [expire:int - ttl in seconds] [keyOnly]`;
+    - **idSetKey**: Redis SET containing id values
+    - **metaKeyPattern**: Pattern to find HASHes of record, "*" is replaced by id from `idSetKey`, eg: _*-metakey_ produce  _id-231-metakey_
+    - **hashKey**: field name from HASHkey formed from `metakeyPattern`
+    - **filter**: Json string with filters
+* **General behaviour**: During work process function going to store postprocessed data into 2 temporary sets(sorted and filterd) in format `{idSetKey}:{order}:{metaKey}:{hasKey}:{filterString}` and reuse them while `ttl` not past. 
+
+    Creates index of this sets in ZSET `tempKeysSet`.
+    > _Definitely I don't understand why this needed, but we have cacheBuster_ in lua. #SeeLater
+
+    - If `metaKeyPattern` and `hashKey` not provided, function going to use `idSetKey` values for sorting, otherwise fetch data from `metaKeyPattern.gsub("*", idSetMemberValue).providedHashKey` and use them for sort.
+
+    - Same as prev behaviour going to be used in filtering process.
+
+    - If more than one filter provided: `filterString` will contain only "#" filter;
+
+
+### cfsortBust :
+Checking caches for specified `idSetKey` that was created in result of `cfsort` method work. Deletes outdated tempKeys from ZSET `tempKeysSet`
+
+**Method usage**: `cfsort idSetKey curTime:unixtime [expire:int]`
+
+### cfsortAggregate:
+Groups elements by `aggregateParam` from set `idSet` using meta available under `metaKeyPattern` and reports count of each type of criteria.
+
+Currently supports only `sum`;
+
+* **Method usage**: `cfsortAggregate idSetKey metaKeyPattern aggregateParam`
+    - **idSetKey**: Redis SET containing id values
+    - **metaKeyPattern**: Pattern to find HASHes of record, "*" is replaced by id from `idSetKey`, eg: _*-metakey_ produce  _id-231-metakey_
+    - **aggregateParam** : JsonString containing aggregate criteria.
+        ```Javascript
+            {
+                "fieldName": "sum"
+            }
+        ```
+
+
+## Filter Format `filter` parameter:
+Accepting `json` formated object:
+```Javascript
+    {
+        "fieldToFiler": {
+            //Filter params...
+        }
+        "fieldToFilter": "String to find"
+    }
+```
+
+- **Available criteria**:
+    - `lte`,`gte` - (int) value comparison
+    - `eq`,`ne` - equal or not
+    - `any` - Object containing criteria list to Exclude
+    - `some` - Object containing criteria list to Include
+    - `match` - String contains 
+    - `exists` - if value exists
+
+- **MultiField(#multi)**: Reserved field name for assigning criterias to multiple fields from `meta` record(currently supports only _String contains check_)
+    ```Javascript
+        {
+            "#multi": {
+                "fields": [ 
+                    //Field list 
+                    ...
+                ],
+                "match": "criteria"
+            }
+        }
+    ```
+
+
+**Example**
+```Javascript
+    {
+        "metaFieldName": "contained string",
+        "metaFieldName2": { //Filter params object
+            "gte": 12,
+            "lte": 13,
+        },
+        "metaFieldAnyParams": {
+            "any": {
+                "0": {
+                    "gte": 12,
+                    "lte": 13,
+                },
+                "1": {
+                    "gte": 28,
+                    "lte": 30,
+                }
+            }
+        },
+
+
+    }
+    ```
+
+
diff --git a/csrc/doc/rfc/01_bench.md b/csrc/doc/rfc/01_bench.md
@@ -0,0 +1,2 @@
+# Benchmarking
+...TODO
diff --git a/csrc/redis-filtered-sort/README.md b/csrc/redis-filtered-sort/README.md
@@ -1,67 +1,2 @@
 # Filtered Sort C lang ver
-
-    [ ] Сделать реализацию
-        [ ] Сортировка (+-)
-        [ ] Фильтрация
-    [ ] Ввести thread pool, увести все операции в отдельные трэды
-    [ ] Провести сравнительные тесты
-    [ ] Привести все в порядок
-
-###
-    `cfsort id-set-key meta-key-pattern hash-key sort order json-filter curTime [offset] [limit] [expire] [keyOnly]`
-
-    `cfsortBust id-set-key curTime [expire]`
-
-    #!todo
-
-### Build system
-    * Сборка модуля docker'ом в отдельно контейнера под alpine linux.
-    * Сборка custom redis контейнера с включенным модулем.
-
-### Dependencies
-    `jansson` - для lua cjson
-    *использовать как можно меньше зависимостей.*
-### 
-
-#### Из хорошего
-Просмотрев архитектуру RedisModule и так же RediSearch RedisMl RedisAI Можно сказать что результат возможен.
-
-Если использовать `low-level` API = так называемый прямой доступ к ключам,
-отдельный трэд и блокировку клиента, можно добиться нужного результата.
-
-# Из плохого
-
-Категорически избегать блокирующих операций. Как на чтение так и на запись.
-
-Встает вопрос в работе с кластером, но лучше проверить.
-
-
-Update:
-
-!!!! Категорически не будет работать в кластере, как и оригинальный скрипт.
-На данный момент lua скрипт падает с ошибкой " ReplyError: CROSSSLOT Keys in request don't hash to the same slot"
-!!!!!!!!!!!
-
-
-`
-5332a9007251add48219e2866d8e2b1c2943a915 127.0.0.1:7006@17006 slave 7819ac6cf3703f1e942a7f38cfa6a390e1a14cee 0 1562695256598 7 connected
-
-d85d87b27c61066ac7c1506fec03c34b0e1422ac 127.0.0.1:7004@17004 slave 088ad3854d8953d6756b295cf74603ec9630cbf2 0 1562695255788 5 connected
-
-ab68df2c82dccea54d9c9a5129dd96d64e48ec8b 127.0.0.1:6379@16379 myself,master - 0 1562695256000 1 connected 0-5460
-
-e74a870ff4eac1507d6dfc0e8805d705f80b075c 127.0.0.1:7003@17003 slave ab68df2c82dccea54d9c9a5129dd96d64e48ec8b 0 1562695256806 4 connected
-
-7819ac6cf3703f1e942a7f38cfa6a390e1a14cee 127.0.0.1:7002@17002 master - 0 1562695257000 3 connected 10923-16383
-
-088ad3854d8953d6756b295cf74603ec9630cbf2 127.0.0.1:7001@17001 master - 0 1562695257831 2 connected 5461-10922
-
-c7944ef74f4f3272c8b38fa4e98751e0481e5b26 127.0.0.1:7005@17005 slave ab68df2c82dccea54d9c9a5129dd96d64e48ec8b 0 1562695257316 6 connected`
-
-`"no-cluster"**: The command should not register in Redis Cluster
-since is not designed to work with it because, for
-example, is unable to report the position of the
-keys, programmatically creates key names, or any
-other reason. `
-
-
+Full docs soon