How to batch lots of HGET from different keys with caching?

[0x079]

I'm trying to fetch a lot of values (>10,000 records) from different hashsets HSET but this is sequential. Although rueidis might be able to batch individual requests with requests coming from other goroutines, they will probably result in a lot of roundtrips and end up taking a really long time?

	for i := 0; i < 100; i++ {
		key := fmt.Sprintf("k%v", i)
		client.DoCache(ctx, client.B().Hget().Key(key).Field("1").Cache(), time.Minute)
	}

What's the best way to improve this? Looks like there is MGET but is there a version for hashsets?

If I write it this way, and only some of the entries are cached, will the client fetch all of them (including the ones that are already cahced) as a MULTI transaction or will it only go fetch the ones that are not available locally? What's the best way to do this more efficiently? Thank you!

	cmds := make([]rueidis.CacheableTTL, 0, 100)
	for i := 0; i < 100; i++ {
		key := fmt.Sprintf("k%v", i)
		cmds = append(cmds, rueidis.CT(client.B().Hget().Key(key).Field("1").Cache(), time.Minute))
	}
	for _, resp := range client.DoMultiCache(ctx, cmds...) {
		if err := resp.Error(); err != nil {
			panic(err)
		}
	}

I guess maybe I'm confused about the difference between c.B().Multi() and c.DoMulti(...), what's the difference? Multi() is for the "MULTI" command transaction, DoMulti() is simply for batching commands, but they might not be executed as a single transaction?

Another question is, how many concurrent connections will the client open? Is it something we can control? I couldn't find documentations on it.

[rueian]

You are right. Your first example will result in a lot of round trips.

And your second example is the best you can do. It will incur only one round trip and send only those cache missed commands.

c.DoMulti is simply a batched version of c.Do that sends multiple commands in one round trip. So does c.DoMultiCache with consulting local cache first.

c.B().Multi() is just the redis MULTI command, since c.B() is a command builder. To use redis transaction, you can freely construct related commands to suite your needs, for example:

client.Dedicated(func(c rueidis.DedicatedClient) error {
    // watch keys first
    c.Do(ctx, c.B().Watch().Key("k1", "k2").Build())
    // perform read here
    c.Do(ctx, c.B().Mget().Key("k1", "k2").Build())
    // perform write with MULTI EXEC
    c.DoMulti(
        ctx,
        c.B().Multi().Build(),
        c.B().Set().Key("k1").Value("1").Build(),
        c.B().Set().Key("k2").Value("2").Build(),
        c.B().Exec().Build(),
    )
    return nil
})

[0x079]

Thank you! That's really helpful!

Would you mind explaining how the client decides how many connections to open? Is it something we can configure? And what are some best practices there?

[rueian]

You can try using client.Dedicated() or client.Dedicate() to acquire dedicated connections from the pool to benchmark whether using multiple connections has improvements.

Also please note that redis is basically single-threaded.