A Web Admin Console for Redis, Part Two
In my last post I put together a simple web server infrastructure that could issue scans to a remote Redis server and display the results, available for paging back and forth, to a web interface. It's useful for seeing which keys are available (more so than the command line interface), but could be a lot more useful. For starters, it's not displaying the value associated with the key. This is a little trickier than it sounds, since not only can the interface not assume that data is textual, there are different sorts of values that Redis entries can take: string, list, hash, set and sorted set (there are also bitmaps and hyperloglogs which I won't examine here). Redis's API does provide a way to check to see what sort of value a key corresponds to, but the only way to determine if the underlying value is binary or text is to load it and scan it.
Besides displaying values, it would also be useful to edit and even remove entries from the UI. I'll address these issues in this post.
In order to return non-string values back to the client, I'll have to first query each key for its type:
if page == len(activeQuery['pages']) - 1:
activeQuery['pages'].append({'cursor': cursor, 'position': position})
tpipe = redisConn.pipeline()
for key in pageKeys:
tpipe.type(key)
types = tpipe.execute()
Listing 1: get types of keys
When this completes, the types array will contain an entry for each of the keys that was retrieved. Now I can
modify the value retrieval loop to retrieve the right sort of value (note that I have to do this in two batches, since I need
to know the value type before I retrieve it):
vpipe = redisConn.pipeline()
i = 0
for key in pageKeys:
if types[i] == 'string':
vpipe.get(key)
elif types[i] == 'list':
vpipe.lrange(key, 0, 10)
elif types[i] == 'set':
vpipe.smembers(key)
elif types[i] == 'hash':
vpipe.hgetall(key)
elif types[i] == 'zset':
vpipe.zrange(key, 0, 10)
i += 1
values = vpipe.execute()
# converts sets to lists so they can be converted to JSON
values = [[e for e in x] if isinstance(x, set) else x for x in values]
self.send_text({'page': list(zip(pageKeys, values)), 'more': more})
Listing 2: Retrieve values based on key type
The strange-looking list comprehension toward the end converts sets to lists, since Python sets aren't
JSON serializable.
Notice that for lists and zsets, I'm just returning the first 10 values, and for sets and hashes, I return all available values. For all four types, I'd do better to have a way to "drill down" into the members. What I'll do instead of returning values (or representatives of values) is return the type, along with the value in the case of a string, and leave it up to the client to call back for more info if desired.
vpipe = redisConn.pipeline()
i = 0
for key in pageKeys:
if types[i] == 'string':
vpipe.get(key)
i += 1
values = vpipe.execute()
pageResponse = []
i = 0
j = 0
for key in pageKeys:
pageEntry = {'key': key, 'type': types[i]}
if (types[i] == 'string'):
pageEntry['value'] = values[j]
j += 1
i += 1
pageResponse.append(pageEntry)
self.send_text({'page': pageResponse, 'more': more})
Listing 3: return types instead of value summaries
This requires a slight change on the client, since I'm not returning an array of tuples any more:
for (var i = 0; i < result.page.length; i++) {
tblHtml += '<tr><td>' + result.page[i]['key'] + '</td>'
switch (result.page[i]['type']) {
case 'string':
tblHtml += '<td>' + result.page[i]['value'] + '</td>';
break;
case 'list':
tblHtml += '<td><a href="/list?key=' + result.page[i]['key'] +
'&page=0&size=10">list</a></td>';
break;
case 'set':
tblHtml += '<td><a href="/set?key=' + result.page[i]['key'] +
'&cursor=0&size=10">set</a></td>';
break;
case 'hash':
tblHtml += '<td><a href="/hash?key=' + result.page[i]['key'] +
'&cursor=0&size=10">hash</a></td>';
break;
case 'zset':
tblHtml += '<td><a href="/zset?key=' + result.page[i]['key'] +
'&cursor=0&size=10">zset</a></td>';
break;
}
tblHtml += '</tr>'
}
Listing 4: displaying keys and values (or links)
I've also introduced four new routes, though: list, set, hash & zset.
Of the four, list is the easiest to handle, since there's a well defined order, but I do
want to go ahead and support pagination. I can actually handle all of the UI on the server side here, since I don't have to
deal with partial pages like I did with my top level query: I always want to query everything, so my results will either be
an entire page of the size requested or a partial page indicating that I've reached the end (but I do need to double-check
for the case where the last page is exactly on a page boundary).
elif self.path.startswith('/list'):
params = self.parseQuery(self.path)
if self.ensure_required_params(params, ['key', 'size', 'page']):
key = params['key']
size = int(params['size'])
page = int(params['page'])
llen = redisConn.llen(key)
values = redisConn.lrange(key, page * size, ((page + 1) * size) - 1)
table = ['<tr><td>' + value + '</td></tr>' for value in values]
table += ('<tr><td>' +
('<a href="/list?key=' + key + '&page=' + str(page - 1) + '&size=' +
str(size) + '">prev</a>' if page > 0 else '') +
'</td><td>' +
('<a href="/list?key=' + key + '&page=' + str(page + 1) + '&size=' +
str(size) + '">next</a>' if ((page + 1) * size) < llen else '') +
'</td></tr>');
self.send_html('<html><body><table>' + ''.join(table) +
'</table></body></html>')
Listing 5: Paging through lists
Sets, sorted sets and hashes are slightly more complex (but not much more): you can't request a "range" of values for them
but instead you issue a type-specific scan command that works exactly like the top-level scan
command. In fact, this command can be used to filter out results inside types as well, but I won't take advantage of that
here — I'm assuming that the nested types I look at are relatively small. They're so similar that I can go ahead
and handle them in effectively the same code block:
elif self.path.startswith('/set') or self.path.startswith('/zset')
or self.path.startswith('/hash'):
params = self.parseQuery(self.path)
if self.ensure_required_params(params, ['key', 'cursor', 'size']):
key = params['key']
size = int(params['size'])
cursor = int(params['cursor'])
if self.path.startswith('/set'):
(cursor, values) = redisConn.sscan(key, cursor, '*', size)
table = ['<tr><td>' + value + '</td></tr>' for value in values]
elif self.path.startswith('/zset'):
(cursor, values) = redisConn.zscan(key, cursor, '*', size)
table = ['<tr><td>' + value[0] + '</td><td>' + str(value[1]) +
'</td></tr>' for value in values]
elif self.path.startswith('/hash'):
(cursor, values) = redisConn.hscan(key, cursor, '*', size)
table = ['<tr><td>' + key + '</td><td>' + values[key] +
'</td></tr>' for key in values.keys()]
table += ('<tr><td></td><td>' +
('<a href="/set?key=' + key + '&size=' + str(size) + '&cursor=' + str(cursor) +
'">next</a>' if cursor != 0 else '') + '</td></tr>')
self.send_html('<html><body><table>' + ''.join(table) +
'</table></body></html>')
Listing 6: Paging through sets, sorted sets and hashes
All of this works correctly with string content, but not with binary content. In fact, the inclusion of the
decode_responses=True parameter when the redisConn is created causes the server to fail if non-string
content is encountered: you'll get the error UnicodeDecodeError: 'utf-8' codec can't decode byte 0xac in position 0:
invalid start byte if you try. I can address this with some extra code by invoking decode manually on
each entry and catching and handling the decode exception if it occurs.
tpipe = redisConn.pipeline()
for key in pageKeys:
tpipe.type(key)
types = tpipe.execute()
types = [t.decode('UTF-8') for t in types]
vpipe = redisConn.pipeline()
i = 0
for key in pageKeys:
if types[i] == 'string':
vpipe.get(key)
i += 1
values = vpipe.execute()
pageResponse = []
i = 0
j = 0
for key in pageKeys:
pageEntry = {'key': key.decode('UTF-8'), 'type': types[i]}
if (types[i] == 'string'):
try:
pageEntry['value'] = values[j].decode('UTF-8')
except:
pageEntry['value'] = str(values[j])
j += 1
i += 1
pageResponse.append(pageEntry)
self.send_text({'page': pageResponse, 'more': more})
Listing 7: basic support for binary types
For some silly reason, you have to manually decode even the type response in spite of the fact
that it can only ever return strings.
Container types are a little bit easier again, since they can't contain other containers:
elif self.path.startswith('/list'):
params = self.parseQuery(self.path)
if self.ensure_required_params(params, ['key', 'size', 'page']):
key = params['key']
size = int(params['size'])
page = int(params['page'])
llen = redisConn.llen(key)
values = redisConn.lrange(key, page * size, ((page + 1) * size) - 1)
values = [value.decode('UTF-8') if value.isascii() else str(value)
for value in values]
table = ['<tr><td>' + value + '</td></tr>' for value in values]
table += ('<tr><td>' +
('<a href="/list?key=' + key + '&page=' + str(page - 1) + '&size=' +
str(size) + '">prev</a>' if page > 0 else '') +
'</td><td>' +
('<a href="/list?key=' + key + '&page=' + str(page + 1) + '&size=' +
str(size) + '">next</a>' if ((page + 1) * size) < llen else '') +
'</td></tr>');
self.send_html('<html><body><table>' + ''.join(table) +
'</table></body></html>')
elif self.path.startswith('/set') or self.path.startswith('/zset')
or self.path.startswith('/hash'):
params = self.parseQuery(self.path)
if self.ensure_required_params(params, ['key', 'cursor', 'size']):
key = params['key']
size = int(params['size'])
cursor = int(params['cursor'])
if self.path.startswith('/set'):
(cursor, values) = redisConn.sscan(key, cursor, '*', size)
values = [value.decode('UTF-8') if value.isascii() else str(value) for value in values]
table = ['<tr><td>' + value + '</td></tr>' for value in values]
elif self.path.startswith('/zset'):
(cursor, values) = redisConn.zscan(key, cursor, '*', size)
values = [(value[0].decode('UTF-8'), value[1])
if value[0].isascii() else str(value) for value in values]
table = ['<tr><td>' + value[0] + '</td><td>' +
str(value[1]) + '</td></tr>' for value in values]
elif self.path.startswith('/hash'):
(cursor, values) = redisConn.hscan(key, cursor, '*', size)
table = ['<tr><td>' + key.decode('UTF-8') + '</td><td>' +
values[key].decode('UTF-8') + '</td></tr>'
if key.isascii() else '' for key in values.keys()]
table += ('<tr><td></td><td>' +
('<a href="/set?key=' + key + '&size=' + str(size) + '&cursor=' +
str(cursor) + '">next</a>' if cursor != 0 else '') +
'</td></tr>')
self.send_html('<html><body><table>' + ''.join(table) +
'</table></body></html>')
Listing 8: binary container types
Spartan though it is, this admin console can now support most of the Redis keyspaces you're going to come across. However, once you've found the value you're interested in, you're likely to want to edit or remove it. Additional support for modifications will be the topic of my next post.
Add a comment:
Completely off-topic or spam comments will be removed at the discretion of the moderator.
You may preserve formatting (e.g. a code sample) by indenting with four spaces preceding the formatted line(s)
