Redis Collections

Redis Collections are a set of basic Python collections backed by Redis.

Redis is a great key-value storage. There is well-designed client for Python, but sometimes working with it seems to be too low-level. You just call methods with names of corresponding Redis commands. Such approach is great when dealing with cutting edge software tasks, but if you write just a simple app or command line script for your own, you might appreciate a bit different interface.

Aim of this small library is to provide such interface when dealing with collections. Redis has support for several types: strings, hashes, lists, sets, sorted sets. Why not to bring them to Python in a pythonic way?

>>> from redis_collections import Dict
>>> d = Dict()
>>> d['answer'] = 42
>>> d
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'answer': 42}>
>>> d.items()
[('answer', 42)]

In my Redis I can see a hash structure under key fe267c1dde5d4f648e7bac836a0168fe. Using dict-like notation, it’s value is following:

{'answer': 'I42\n.'}

The value is pickled, because Redis can store only strings. On Python side, you can do almost any stuff you are used to do with standard dict instances:

>>> d.update({'hasek': 39, 'jagr': 68})
>>> d
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'answer': 42, 'jagr': 68, 'hasek': 39}>
>>> del d['answer']
>>> d
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'jagr': 68, 'hasek': 39}>

Every such operation atomically changes data in Redis.

Installation

Current version is 0.1.7.

The Cheese Shop:

pip install redis-collections

In case you have an adventurous mind, give a try to the source:

pip install git+https://github.com/honzajavorek/redis-collections.git#egg=redis-collections

Persistence

When creating the Dict object, your collection gets a unique Redis key. If you keep this key, you can summon your collection any time in the future:

>>> d.key
fe267c1dde5d4f648e7bac836a0168fe
>>> Dict(key='fe267c1dde5d4f648e7bac836a0168fe')
<redis_collections.Dict at fe267c1dde5d4f648e7bac836a0168fe {'jagr': 68, 'hasek': 39}>

In case you wish to wipe all its data, use clear() method, which is available to all collections provided by this library:

>>> d.clear()
>>> d.items()
[]

If I look to my Redis, key fe267c1dde5d4f648e7bac836a0168fe completely disappeared.

Note

If you provide your own key string, collection will be successfully created. If there is no key corresponding in Redis, it will be created and initialized as an empty collection. This means you can set up your own way of assigning unique keys dependent on your other code. For example, by using IDs of records from your relational database you can have exactly one unique collection in Redis for every record from your SQL storage.

Redis connection

By default, collection uses a new Redis connection with its default values, which is highly inefficient, but needs no configuration. If you wish to use your own Redis instance, pass it in redis keyword argument:

>>> from redis import StrictRedis
>>> r = StrictRedis()
>>> d = Dict(redis=r)
>>> l = List(redis=r)  # using the same connection as Dict above

There are several operations between collections resulting into creation of new instances of Redis Collections. These new instances always use the same Redis connection as the original object:

>>> from redis import StrictRedis
>>> from redis_collections import List
>>> r = StrictRedis()
>>> l = List([1, 2], redis=r)
>>> l
<redis_collections.List at 196e407f8fc142728318a999ec821368 [1, 2]>
>>> l + [4, 5, 6]  # result is using the same connection
<redis_collections.List at 7790ef98639043c9abeacc80c2de0b93 [1, 2, 4, 5, 6]>

Pickling

If you don’t like the standard way of data serialization made by pickle, you can set your own. Use pickler keyword argument:

>>> import pickle
>>> l = List(pickler=pickle)  # this makes no sense

pickler can be anything having two methods (or functions): dumps() for conversion of data to string, and loads() for the opposite direction. You can write your own module or class with such interface, or you can use one of those which are already available:

>>> import json
>>> l = List(pickler=json)

New instances of collections coming from operations between them use the same pickler. It is propagated as well as Redis connection.

Philosophy

  • All operations are atomic.

    Warning

    If an operation has race conditions, it is a bug. Please, report it.

  • Redis Collections stick to API of the original data structures known from Python standard library. To have the same (expected) behaviour is considered to be more important than efficiency.

    Warning

    If a collection has the method you want to use, but it does not behave as the original built-in and it does not raise NotImplementedError, then it is a bug. Please, report it.

    If there is more efficient approach than the one complying with the model interface, new method exposing this approach should be introduced.

  • Cases where different than standard approach would lead to better efficiency are mentioned and highlighted in API documentation as notes. Known incompatibilities with the original API are marked as warnings.

  • Behavior of nested Redis Collections containing other Redis Collections is undefined. It is not recommended to create such structures. Use collection of keys instead.

API Documentation

Redis Collections are composed of only several classes. All items listed below are exposed as public API, so you can (and you should) import them directly from redis_collections package.

base

class redis_collections.base.RedisCollection(data=None, redis=None, key=None, pickler=None)[source]

Abstract class providing backend functionality for all the other Redis collections.

__init__(data=None, redis=None, key=None, pickler=None)[source]
Parameters:
  • data – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.
  • pickler

    Implementation of data serialization. Object with two methods is expected: dumps() for conversion of data to string and loads() for the opposite direction. Examples:

    import json, pickle
    Dict(pickler=json)
    Dict(pickler=pickle)  # default
    

    Of course, you can construct your own pickling object (it can be class, module, whatever). Default serialization implementation uses pickle.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

clear()[source]

Completely cleares the collection’s data.

copy(key=None)[source]

Return a copy of the collection.

Parameters:key (string) – Key of the new collection. Defaults to auto-generated.
key = None

Redis key of the collection.

pickler = None

Class or module implementing pickling. Standard pickle module is set as default.

redis = None

Redis client instance. StrictRedis object with default connection settings is used if not set by __init__().

dicts

Collections based on dict interface.

class redis_collections.dicts.Dict(*args, **kwargs)[source]

Mutable mapping collection aiming to have the same API as the standard mapping type, dict. See dict for further details. The Redis implementation is based on the hash type.

Warning

In comparing with original dict type, Dict does not implement methods viewitems(), viewkeys(), and viewvalues().

Note

Some operations, which are usually not used so often, can be more efficient than their “popular” equivalents. For example, get() should be preffered over the classic d[key] approach.

__contains__(key)[source]

Return True if Dict instance has a key key, else False.

__delitem__(key)[source]

Remove d[key] from dictionary. Raises a KeyError() if key is not in the map.

Note

Due to implementation on Redis side, this method of deleting items is not very efficient. If possible, use discard().

__getitem__(key)[source]

Return the item of dictionary with key key. Raises a KeyError if key is not in the map.

If a subclass of Dict defines a method __missing__(), if the key key is not present, the d[key] operation calls that method with the key key as argument. The d[key] operation then returns or raises whatever is returned or raised by the __missing__(key) call if the key is not present.

Note

Due to implementation on Redis side, this method of retrieving items is not very efficient. If possible, use get().

__init__(*args, **kwargs)[source]

Breakes the original dict API, because there is no support for keyword syntax. The only single way to create Dict object is to pass iterable or mapping as the first argument.

Parameters:
  • data (iterable or mapping) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.
  • pickler

    Implementation of data serialization. Object with two methods is expected: dumps() for conversion of data to string and loads() for the opposite direction. Examples:

    import json, pickle
    Dict(pickler=json)
    Dict(pickler=pickle)  # default
    

    Of course, you can construct your own pickling object (it can be class, module, whatever). Default serialization implementation uses pickle.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

Warning

As mentioned, Dict does not support following initialization syntax: d = Dict(a=1, b=2)

__iter__()[source]

Return an iterator over the keys of the dictionary.

__len__()[source]

Return the number of items in the dictionary.

__setitem__(key, value)[source]

Set d[key] to value.

discard(key)[source]

Remove d[key] from dictionary if it is present.

Note

Due to implementation on Redis side, this method of retrieving items is more efficient than classic approach over using the __delitem__() protocol.

classmethod fromkeys(seq, value=None, **kwargs)[source]

Create a new dictionary with keys from seq and values set to value.

Note

fromkeys() is a class method that returns a new dictionary. value defaults to None. It is possible to specify additional keyword arguments to be passed to __init__() of the new object.

get(key, default=None)[source]

Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.

Note

Due to implementation on Redis side, this method of retrieving items is more efficient than classic approach over using the __getitem__() protocol.

getmany(*keys)[source]

Return the value for keys. If particular key is not in the dictionary, return None.

items()[source]

Return a copy of the dictionary’s list of (key, value) pairs.

iter()[source]

Return an iterator over the keys of the dictionary. This is a shortcut for iterkeys().

iteritems()[source]

Return an iterator over the dictionary’s (key, value) pairs.

iterkeys()[source]

Return an iterator over the dictionary’s keys.

itervalues()[source]

Return an iterator over the dictionary’s values.

keys()[source]

Return a copy of the dictionary’s list of keys.

pop(key, default=<missing value>)[source]

If key is in the dictionary, remove it and return its value, else return default. If default is not given and key is not in the dictionary, a KeyError is raised.

popitem()[source]

Remove and return an arbitrary (key, value) pair from the dictionary.

popitem() is useful to destructively iterate over a dictionary, as often used in set algorithms. If the dictionary is empty, calling popitem() raises a KeyError.

setdefault(key, default=None)[source]

If key is in the dictionary, return its value. If not, insert key with a value of default and return default. default defaults to None.

update(other=None, **kwargs)[source]

Update the dictionary with the key/value pairs from other, overwriting existing keys. Return None.

update() accepts either another dictionary object or an iterable of key/value pairs (as tuples or other iterables of length two). If keyword arguments are specified, the dictionary is then updated with those key/value pairs: d.update(red=1, blue=2).

values()[source]

Return a copy of the dictionary’s list of values.

class redis_collections.dicts.Counter(*args, **kwargs)[source]

Mutable mapping collection aiming to have the same API as collections.Counter. See Counter for further details. The Redis implementation is based on the hash type.

Warning

Not available in Python 2.6.

Warning

In comparing with original collections.Counter type supports only integers. Counter also does not implement methods viewitems(), viewkeys(), and viewvalues().

Note

Unlike Dict, Counter has the same efficiency of c[key] and get() operations.

__getitem__(key)[source]

Return the item of dictionary with key key. Returns zero if key is not in the map.

Note

Unlike Dict, Counter has the same efficiency of c[key] and get() operations.

__init__(*args, **kwargs)[source]

Breakes the original Counter API, because there is no support for keyword syntax. The only single way to create Counter object is to pass iterable or mapping as the first argument. Iterable is expected to be a sequence of elements, not a sequence of (key, value) pairs.

There is no support for pickler setting, because all values are integers only.

Parameters:
  • data (iterable or mapping) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

Warning

As mentioned, Counter does not support following initialization syntax: c = Counter(a=1, b=2)

elements()[source]

Return an iterator over elements repeating each as many times as its count. Elements are returned in arbitrary order. If an element’s count is less than one, elements() will ignore it.

classmethod fromkeys(seq, value=None, **kwargs)[source]

This class method is not implemented for Counter objects.

inc(key, n=1)[source]

Value of key will be increased by n. n defaults to 1. If n is negative integer, value of key will be decreased. Value after the increment (decrement) operation is returned.

Note

Whole operation is ignored if n is zero.

Return type:integer
most_common(n=None)[source]

Return a list of the n most common elements and their counts from the most common to the least. If n is not specified, most_common() returns all elements in the counter. Elements with equal counts are ordered arbitrarily.

subtract(other)[source]

Elements are subtracted from an iterable or from another mapping (or counter). Like dict.update() but subtracts counts instead of replacing them.

update(other)[source]

Elements are counted from an iterable or added-in from another mapping (or counter). Like dict.update() but adds counts instead of replacing them. Also, the iterable is expected to be a sequence of elements, not a sequence of (key, value) pairs.

lists

Collections based on list interface.

class redis_collections.lists.List(*args, **kwargs)[source]

Mutable sequence collection aiming to have the same API as the standard sequence type, list. See list for further details. The Redis implementation is based on the list type.

Warning

In comparing with original list type, List does not implement methods sort() and reverse().

Note

Some operations, which are usually not used so often, can be more efficient than their “popular” equivalents. Some operations are shortened in their capabilities and can raise NotImplementedError for some inputs (e.g. most of the slicing functionality).

__add__(values)[source]

Returns concatenation of the list and given iterable. New List instance is returned.

__delitem__(index)[source]

Item of index is deleted.

Warning

Slicing is generally not supported. Only empty lists are accepted if the operation leads into trimming:

del l[2:]
del l[:2]
del l[:]
__getitem__(index)[source]

Returns item of sequence on index. Origin of indexes is 0. Accepts also slicing.

Note

Due to implementation on Redis side, l[index] is not very efficient operation. If possible, use get(). Slicing without steps is efficient. Steps are implemented only on Python side.

__init__(*args, **kwargs)[source]
Parameters:
  • data (iterable) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.
  • pickler

    Implementation of data serialization. Object with two methods is expected: dumps() for conversion of data to string and loads() for the opposite direction. Examples:

    import json, pickle
    Dict(pickler=json)
    Dict(pickler=pickle)  # default
    

    Of course, you can construct your own pickling object (it can be class, module, whatever). Default serialization implementation uses pickle.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

__iter__()[source]

Return an iterator over the sequence.

__len__()[source]

Length of the sequence.

__mul__(n)[source]

Returns n copies of the list, concatenated. New List instance is returned.

__reversed__()[source]

Returns iterator for the sequence in reversed order.

__setitem__(index, value)[source]

Item of index is replaced by value.

Warning

Slicing is generally not supported. Only empty lists are accepted if the operation leads into trimming:

l[2:] = []
l[:2] = []
l[:] = []
append(value)[source]

Insert value at end of list.

count(value)[source]

Returns number of occurences of value.

Note

Implemented only on Python side.

extend(values)[source]

values are appended at the end of the list. Any iterable is accepted.

get(index, default=None)[source]

Return the value for index if index is not out of range, else default. If default is not given, it defaults to None, so that this method never raises a IndexError.

Note

Due to implementation on Redis side, this method of retrieving items is more efficient than classic approach over using the __getitem__() protocol.

index(value, start=None, stop=None)[source]

Returns index of the first occurence of value.

If start or stop are provided, returns the smallest index such that s[index] == value and start <= index < stop.

insert(index, value)[source]

Insert value before index. Can only work with index == 0.

pop(index=-1)[source]

Item on index is removed and returned.

Warning

Only indexes 0 and -1 (default) are supported, otherwise NotImplementedError is raised.

remove(value)[source]

Remove the first occurence of value.

sets

Collections based on set interface.

class redis_collections.sets.Set(*args, **kwargs)[source]

Mutable set collection aiming to have the same API as the standard set type. See set for further details. The Redis implementation is based on the set type.

__and__(*args)[source]

Return a new set with elements common to the set and the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

The same behavior as at __sub__() applies.

__contains__(elem)[source]

Test for membership of elem in the set.

__iand__(*args)[source]

Update the set, keeping only elements found in it and the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

The same behavior as at __isub__() applies.

__init__(*args, **kwargs)[source]
Parameters:
  • data (iterable) – Initial data.
  • redis (redis.StrictRedis) – Redis client instance. If not provided, default Redis connection is used.
  • key (str) – Redis key of the collection. Collections with the same key point to the same data. If not provided, default random string is generated.
  • pickler

    Implementation of data serialization. Object with two methods is expected: dumps() for conversion of data to string and loads() for the opposite direction. Examples:

    import json, pickle
    Dict(pickler=json)
    Dict(pickler=pickle)  # default
    

    Of course, you can construct your own pickling object (it can be class, module, whatever). Default serialization implementation uses pickle.

Note

uuid.uuid4() is used for default key generation. If you are not satisfied with its collision probability, make your own implementation by subclassing and overriding internal method _create_key().

__ior__(*args)[source]

Update the set, adding elements from the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

The same behavior as at __isub__() applies.

__isub__(*args)[source]

Update the set, removing elements found in other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

If other is instance of Set, operation is performed completely in Redis. Otherwise it’s performed in Python and results are sent to Redis.

__iter__()[source]

Return an iterator over elements of the set.

__ixor__(*args)[source]

Update the set, keeping only elements found in either set, but not in both.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:None

Note

The same behavior as at __isub__() applies.

__len__()[source]

Return cardinality of the set.

__or__(*args)[source]

Return a new set with elements from the set and the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

The same behavior as at __sub__() applies.

__sub__(*args)[source]

Return a new set with elements in the set that are not in the other.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

If other is instance of Set, operation is performed completely in Redis. Otherwise it’s performed in Python and results are sent to Redis.

__xor__(*args)[source]

Update the set, keeping only elements found in either set, but not in both.

Parameters:other – Set object (instance of collections.Set ABC, so built-in sets and frozensets are also accepted), otherwise TypeError is raised.
Return type:type of the first operand

Note

The same behavior as at __sub__() applies.

add(elem)[source]

Add element elem to the set. Returns False if elem was already present in the set.

Return type:boolean

Warning

Original add() in set returns no value.

difference(*others, **kwargs)[source]

Return a new set with elements in the set that are not in the others.

Parameters:
  • others – Iterables, each one as a single positional argument.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

If all others are Set instances, operation is performed completely in Redis. If return_cls is provided, operation is still performed in Redis, but results are sent back to Python and returned with corresponding type. All other combinations are performed in Python and results are sent to Redis. See examples:

s1 = Set([1, 2])
s2 = Set([2, 3])
s3 = set([2, 3])  # built-in set

# Redis (whole operation)
s1.difference(s2, s2, s2)  # = Set

# Python (operation) → Redis (new key with Set)
s1.difference(s3)  # = Set

# Python (operation) → Redis (new key with Set)
s1.difference(s2, s3, s2)  # = Set

# Redis (operation) → Python (type conversion)
s1.difference(s2, return_cls=set)  # = set

# Redis (operation) → Python (type conversion)
s1.difference(s2, return_cls=list)  # = list

# Redis (operation) → Python → Redis (new key with List)
s1.difference(s2, return_cls=List)  # = List
difference_update(*others)[source]

Update the set, removing elements found in others.

Parameters:others – Iterables, each one as a single positional argument.
Return type:None

Note

If all others are Set instances, operation is performed completely in Redis. Otherwise it’s performed in Python and results are sent to Redis. See examples:

s1 = Set([1, 2])
s2 = Set([2, 3])
s3 = set([2, 3])  # built-in set

# Redis (whole operation)
s1.difference_update(s2, s2)  # = None

# Python (operation) → Redis (update)
s1.difference(s3)  # = None

# Python (operation) → Redis (update)
s1.difference(s2, s3, s2)  # = None
discard(elem)[source]

Remove element elem from the set if it is present.

intersection(*others, **kwargs)[source]

Return a new set with elements common to the set and all others.

Parameters:
  • others – Iterables, each one as a single positional argument.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

The same behavior as at difference() applies.

intersection_update(*others)[source]

Update the set, keeping only elements found in it and all others.

Parameters:others – Iterables, each one as a single positional argument.
Return type:None

Note

The same behavior as at difference_update() applies.

issubset(other)[source]

Test whether every element in the set is in other.

Parameters:other – Any kind of iterable.
Return type:boolean
issuperset(other)[source]

Test whether every element in other is in the set.

Parameters:other – Any kind of iterable.
Return type:boolean
pop()[source]

Remove and return an arbitrary element from the set. Raises KeyError if the set is empty.

random_sample(k=1)[source]

Return a k length list of unique elements chosen from the set. Elements are not removed. Similar to random.sample() function from standard library.

Parameters:k – Size of the sample, defaults to 1.
Return type:list

Note

Argument k is supported only for Redis of version 2.6 and higher.

remove(elem)[source]

Remove element elem from the set. Raises KeyError if elem is not contained in the set.

symmetric_difference(other, **kwargs)[source]

Return a new set with elements in either the set or other but not both.

Parameters:
  • others – Any kind of iterable.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

The same behavior as at difference() applies.

symmetric_difference_update(other)[source]

Update the set, keeping only elements found in either set, but not in both.

Parameters:others – Any kind of iterable.
Return type:None

Note

A bit different behavior takes place in comparing with the one described at difference_update(). Operation is always performed in Redis, regardless the types given. If others are instances of Set, the performance should be better as no transfer of data is necessary at all.

union(*others, **kwargs)[source]

Return a new set with elements from the set and all others.

Parameters:
  • others – Iterables, each one as a single positional argument.
  • return_cls – Keyword argument, type of result, defaults to the same type as collection (Set, if not inherited).
Return type:

Set or collection of type specified in return_cls argument

Note

The same behavior as at difference() applies.

update(*others)[source]

Update the set, adding elements from all others.

Parameters:others – Iterables, each one as a single positional argument.
Return type:None

Note

The same behavior as at difference_update() applies.

Author

My name is Jan (Honza) Javorek. See my GitHub profile for further details. I have a blog and Twitter account, but you wouldn’t probably understand a word as it is in Czech only.

License: ISC

© 2013 Jan Javorek <jan.javorek@gmail.com>

This work is licensed under ISC license.

Indices and tables

Table Of Contents

This Page