Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[NFR]: Use mGet method when fetching multiple keys from Redis #16689

Closed
RayHughes opened this issue Jan 27, 2025 · 6 comments · Fixed by #16706
Closed

[NFR]: Use mGet method when fetching multiple keys from Redis #16689

RayHughes opened this issue Jan 27, 2025 · 6 comments · Fixed by #16706
Assignees
Labels
5.0 The issues we want to solve in the 5.0 release new feature request Planned Feature or New Feature Request

Comments

@RayHughes
Copy link

RayHughes commented Jan 27, 2025

Is your feature request related to a problem? Please describe.
The current implementation of getMultiple on Phalcon's Redis Cache Adapter is using a loop to fetch keys individually. This makes multiple requests to the Redis server and decreases application performance by increasing latency to the Redis Server.

  • PHP

https://github.com/phalcon/phalcon/blob/86e136b75c022b6f86c394b0bd4a0d4a76cccab8/src/Cache/AbstractCache.php#L181

  • Zephir

protected function doGetMultiple(var keys, var defaultValue = null) -> array

Describe the solution you'd like
Remove the loop logic for getMultiple and use PhpRedis's mGet method.

https://github.com/phpredis/phpredis?tab=readme-ov-file#mget

Describe alternatives you've considered
N/A

Additional context
We store realtime pricing data in Redis with a sub 3 second TTL as the data is updated very frequently. To reduce complexity, the dataset is stored using distributed keys based on category / market. A simple request can fetch upwards of 200 individual items from Redis.

Using Phalcon implementation, reading is delayed by increased latency. Using PhpRedis mGet performance is comparable to a single fetch.

@RayHughes RayHughes added the new feature request Planned Feature or New Feature Request label Jan 27, 2025
@niden
Copy link
Member

niden commented Jan 28, 2025

@RayHughes I am not sure if this change will be faster.

I checked this out last night and yes mget brings back all the values for all keys that you have specified. Then all we have to do is array_combine them to get the key => value format back.

    protected function doGetMultiple(
        iterable $keys,
        mixed $default = null
    ): iterable {
        $this->fireManagerEvent('cache:beforeGetMultiple', $keys);

        $adapterClass = get_class($this->adapter);
        if ($adapterClass === Redis::class) {
            $results = $this->adapter->getAdapter()->mget($keys);
            $results = array_combine($keys, $results);
        } else {
            $results = [];
            foreach ($keys as $element) {
                $results[$element] = $this->get($element, $default);
            }
        }

        $this->fireManagerEvent('cache:afterGetMultiple', $keys);

        return $results;
    }

This has introduces problems though. For keys that do not exist, mget returns false (even when setting the serializer to NONE. As such we need to loop:

    protected function doGetMultiple(
        iterable $keys,
        mixed $default = null
    ): iterable {
        $this->fireManagerEvent('cache:beforeGetMultiple', $keys);

        $adapterClass = get_class($this->adapter);
        if ($adapterClass === Redis::class) {
            $results = $this->adapter->getAdapter()->mget($keys);
            $results = array_map(
                function ($element) use ($default) {
                    return false === $element ? $default : $element;
                },
                $results
            );
            $results = array_combine($keys, $results);
        } else {
            $results = [];
            foreach ($keys as $element) {
                $results[$element] = $this->get($element, $default);
            }
        }

        $this->fireManagerEvent('cache:afterGetMultiple', $keys);

        return $results;
    }

But now the results come back OK only if I set the serializer to NONE. If the serializer is anything else I get the serialize data back i.e.:

 Array (
-    '679904c64de0c' => 'test1'
+    '679904c64de0c' => 's:5:"test1";'
     'unknown' => 'default-unknown'
-    '679904c64de0d' => 'test2'
+    '679904c64de0d' => 's:5:"test2";'
 )

I guess we can incorporate the unserlialization in the same array_map loop that handles the default value. However, the underlying adapter classes do not expose the serializer so changes have to be made there also:

The resulting code becomes:

    protected function doGetMultiple(
        iterable $keys,
        mixed $default = null
    ): iterable {
        $this->fireManagerEvent('cache:beforeGetMultiple', $keys);

        $adapterClass = get_class($this->adapter);
        if ($adapterClass === Redis::class) {
            $results = $this->adapter->getAdapter()->mget($keys);
            $serializer = $this->adapter->getSerializer();
            $results = array_map(
                function ($element) use ($serializer, $default) {
                    $serializer->unserialize($element);
                    return false === $element
                        ? $default
                        : $serializer->getData()
                    ;
                },
                $results
            );
            $results = array_combine($keys, $results);
        } else {
            $results = [];
            foreach ($keys as $element) {
                $results[$element] = $this->get($element, $default);
            }
        }

        $this->fireManagerEvent('cache:afterGetMultiple', $keys);

        return $results;
    }

I really don't like putting IF statements in the code based on a specific class instead of delegating this to the respective class. If this is the way forward I would have to figure something out so that this is only for Redis and the Cache class stays the same.

Thoughts?

@niden
Copy link
Member

niden commented Jan 28, 2025

As a quick solution for now, you can use mget directly if you wish

$cache->getAdapter()->getAdapter()->mget($keys)

@RayHughes
Copy link
Author

I was expecting functionality to work similar to the phpredis method. Your response does make sense, especially since the vast majority of people are using redis with Phalcon are probably using a serializer and using it to store db cache etc.

We wound up using hashes for what we needed to accomplish anyway, but figured id open a NFR regardless.

@RayHughes
Copy link
Author

What about updating mGet to use pipelines instead?

So everything is in the same request to the driver. It should save the issue you described above.

@niden niden self-assigned this Feb 28, 2025
@niden niden added the 5.0 The issues we want to solve in the 5.0 release label Feb 28, 2025
@niden niden added this to Phalcon v5 Feb 28, 2025
@niden niden moved this to In Progress in Phalcon v5 Feb 28, 2025
@niden
Copy link
Member

niden commented Feb 28, 2025

What about updating mGet to use pipelines instead?

So everything is in the same request to the driver. It should save the issue you described above.

It will be the same really since we will need to have that if statement to know whether the Redis adapter is used or not.

I am going to change the storage classes to expose getMultiple() This way the cache adapter would just make the same call and we can adjust each adapter accordingly.

I am not sure which one is fastest, mget or a pipeline.

Also I want to create some sort of a storage adapter for Redis to take advantage of hashes. It is a bit tricky though because you can store a bunch of data in each hash - I will have to research that

@niden niden mentioned this issue Feb 28, 2025
5 tasks
@niden niden linked a pull request Feb 28, 2025 that will close this issue
5 tasks
@niden niden moved this from In Progress to Implemented in Phalcon v5 Feb 28, 2025
@niden
Copy link
Member

niden commented Feb 28, 2025

This has been fixed in #16706

@niden niden closed this as completed Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5.0 The issues we want to solve in the 5.0 release new feature request Planned Feature or New Feature Request
Projects
Status: Implemented
Development

Successfully merging a pull request may close this issue.

2 participants