feat: implement Sliding Window Bloom Filter (#715)

This feature enhances the existing Bloom Filter capabilities by allowing time-based item tracking, making it suitable for use cases requiring temporary membership checks. Signed-off-by: Ernesto Alejandro Santana Hidalgo <[email protected]> Signed-off-by: Rueian <[email protected]>
valkey-io · Jan 13, 2025 · 97ea601 · 97ea601
1 parent e3957fc
commit 97ea601
Show file tree

Hide file tree

Showing 6 changed files with 1,190 additions and 8 deletions.
diff --git a/valkeylimiter/README.md b/valkeylimiter/README.md
@@ -1,13 +1,13 @@
 
 # valkeylimiter
 
-This module provides an interface for token bucket rate limiting with precise control over limits and time windows. Inspired by GitHub's approach to scaling their API with a sharded, replicated rate limiter in Redis ([github.blog](https://github.blog/engineering/infrastructure/how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/)).
+This module provides an interface for token bucket rate limiting with precise control over limits and time windows. Inspired by GitHub's approach to scaling their API with a sharded, replicated rate limiter in Valkey ([github.blog](https://github.blog/engineering/infrastructure/how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/)).
 
 ## Features
 
 - **Token Bucket Algorithm**: Implements a token bucket algorithm to control the number of actions (e.g., API requests) a user can perform within a specified time window.
 - **Customizable Limits**: Allows configuration of request limits and time windows to suit various application requirements.
-- **Distributed Rate Limiting**: Leverages Redis to maintain rate limit counters, ensuring consistency across distributed environments.
+- **Distributed Rate Limiting**: Leverages Valkey to maintain rate limit counters, ensuring consistency across distributed environments.
 - **Reset Information**: Provides `ResetAtMs` timestamps to inform clients when they can retry requests.
 
 ## Installation
@@ -86,8 +86,8 @@ func main() {
 
 Creates a new rate limiter with the specified options:
 
-- `ClientOption`: Options to connect to Redis.
-- `KeyPrefix`: Prefix for Redis keys used by this limiter.
+- `ClientOption`: Options to connect to Valkey.
+- `KeyPrefix`: Prefix for Valkey keys used by this limiter.
 - `Limit`: Maximum number of allowed requests per window.
 - `Window`: Time window duration for rate limiting. Must be greater than 1 millisecond.
 
@@ -134,8 +134,8 @@ result, err := limiter.AllowN(ctx, "user_identifier", 3)
 
 ## Implementation Details
 
-The `valkeylimiter` module employs Lua scripts executed within Redis to ensure atomic operations for checking and updating rate limits. This approach minimizes race conditions and maintains consistency across distributed systems.
+The `valkeylimiter` module employs Lua scripts executed within Valkey to ensure atomic operations for checking and updating rate limits. This approach minimizes race conditions and maintains consistency across distributed systems.
 
-By utilizing Redis's expiration capabilities, the module automatically resets rate limits after the specified time window, ensuring efficient memory usage and accurate rate limiting behavior.
+By utilizing Valkey's expiration capabilities, the module automatically resets rate limits after the specified time window, ensuring efficient memory usage and accurate rate limiting behavior.
 
-For more information on the design and implementation of Redis-based rate limiters, refer to GitHub's detailed account of scaling their API with a sharded, replicated rate limiter in Redis ([github.blog](https://github.blog/engineering/infrastructure/how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/)).
+For more information on the design and implementation of Valkey-based rate limiters, refer to GitHub's detailed account of scaling their API with a sharded, replicated rate limiter in Valkey ([github.blog](https://github.blog/engineering/infrastructure/how-we-scaled-github-api-sharded-replicated-rate-limiter-redis/)).
diff --git a/valkeylimiter/limiter.go b/valkeylimiter/limiter.go
@@ -12,7 +12,7 @@ import (
 
 var (
 	ErrInvalidTokens   = errors.New("number of tokens must be non-negative")
-	ErrInvalidResponse = errors.New("invalid response from Redis")
+	ErrInvalidResponse = errors.New("invalid response from Valkey")
 )
 
 type Result struct {

diff --git a/valkeyprob/README.md b/valkeyprob/README.md
@@ -134,3 +134,62 @@ func main() {
     fmt.Println(count) // 1
 }
 ```
+
+### Sliding Window Bloom Filter
+
+It is a variation of the standard Bloom filter that adds a sliding window mechanism. 
+Useful for use cases where you need to keep track of items for a certain amount of time.
+
+Example:
+
+```go
+package main
+
+import (
+	"context"
+	"fmt"
+	"time"
+
+	"github.com/valkey-io/valkey-go"
+	"github.com/valkey-io/valkey-go/valkeyprob"
+)
+
+func main() {
+	client, err := valkey.NewClient(valkey.ClientOption{
+		InitAddress: []string{"localhost:6379"},
+	})
+	if err != nil {
+		panic(err)
+	}
+
+	sbf, err := NewSlidingBloomFilter(client, "sliding_bloom_filter", 1000, 0.01, time.Minute)
+
+	err = sbf.Add(context.Background(), "hello")
+	if err != nil {
+		panic(err)
+	}
+
+	err = sbf.Add(context.Background(), "world")
+	if err != nil {
+		panic(err)
+	}
+
+	exists, err := sbf.Exists(context.Background(), "hello")
+	if err != nil {
+		panic(err)
+	}
+	fmt.Println(exists) // true
+
+	exists, err = sbf.Exists(context.Background(), "world")
+	if err != nil {
+		panic(err)
+	}
+	fmt.Println(exists) // true
+
+	count, err := sbf.Count(context.Background())
+	if err != nil {
+		panic(err)
+	}
+	fmt.Println(count) // 2
+}
+```