[Labs-l] Redis now available in Tool Labs

[Yuvi Panda]

Hello Magnus!

On Sat, Jun 29, 2013 at 1:26 AM, Magnus Manske
<magnusmanske at googlemail.com> wrote:
> Hi Yuvi,
>
> * Is the storage in RAM only, or does it use a disk DB? When the redis is
> restarted, will all entires be gone?

We are running Redis with durability turned to max, so when it is
restarted, your entries will *not* be gone. However, total size of the
data that can stay on Redis is limited by memory, and right now we
have 1G set for it. We can obviously increase it if necessary.

> * Is there a time limit on how long data stays in there? Is it based on last
> request? Can I set one myself?

You can set limits for it yourself, otherwise it gets evicted when
memory limits are exceeded. See http://redis.io/commands for the
wealth of commands available.

> * Is there a regis PHP module installed on tools, or do we have to each
> install out own copy?

Not yet, but I'm fixing that now :) We use phpredis library in
production (https://github.com/nicolasff/phpredis) so will install
that. Will also install appropriate python libraries globally.

Thanks!

>
> Cheers,
> Magnus
>
>
> On Fri, Jun 28, 2013 at 5:42 PM, Yuvi Panda <yuvipanda at gmail.com> wrote:
>>
>> Hello!
>>
>> Redis is now available for tools to use on Tool Labs! Tool Labs
>> specific documentation available at
>> https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help#Redis
>>
>> For those who haven't used Redis before - it is a persistant key-value
>> store (like memcache) but with a lot more features. It can be easily
>> used to do publish/subscribe between processes, and also maintain
>> persistant queues. The Wikipedia Article[1] has more information.
>>
>> It is already being used by SuchABot (Gerrit <-> GitHub sync) for the
>> last few weeks, so should be stable for use by other tools too :)
>> Redis enables/makes easier a lot of things in Tool Labs, such as queue
>> processing. I'll write up docs about those later on if people are
>> interested.
>>
>> Note on security, copied from the docs:
>>
>> Redis has no access control mechanism, so other users can
>> accidentally/intentionally overwrite and access the keys you set. To
>> prevent this, it is highly recommended that you prefix all your keys
>> with an application specific, lengthy, randomly generated secret key.
>> We have also disabled the redis commands that let users 'list' keys.
>> Even if you are not worried about security, you should still prefix
>> your keys in order to prevent accidental key collisions with other
>> apps - it is highly probably that multiple tools would try to use the
>> same key (such as lastupdated, etc). To emphasize, PLEASE PREFIX YOUR
>> KEYS!.
>>
>> You can very simply generate a good enough prefix by running the
>> following command:
>>
>> openssl rand -base64 64
>>
>> Thanks to Petan, Andrew Bogott, Ryan Lane and Coren for helping get this
>> in!
>>
>> [1]: https://en.wikipedia.org/wiki/Redis
>>
>> --
>> Yuvi Panda T
>> http://yuvi.in/blog
>>
>> _______________________________________________
>> Labs-l mailing list
>> Labs-l at lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
>



--
Yuvi Panda T
http://yuvi.in/blog