<    April 2017    >
Su Mo Tu We Th Fr Sa  
                   1  
 2  3  4  5  6  7  8  
 9 10 11 12 13 14 15  
16 17 18 19 20 21 22  
23 24 25 26 27 28 29  
30
00:00 <felixjet> it is possible to configure redis using env vars'
00:10 enigma_raz joined
00:14 svm_invictvs joined
00:21 saurabhjha joined
00:31 indi_ joined
00:40 <wCPO> Anyone running Redis on kubernetes in some sort of HA mode? I have only found this example: https://github.com/kubernetes/kubernetes/tree/master/examples/storage/redis
00:43 soveran joined
01:05 saurabhjha joined
01:19 saurabhjha joined
01:32 saurabhjha joined
01:44 ogres joined
01:56 Guest96 joined
02:00 Guest96 joined
02:08 wCPO joined
02:09 edrocks joined
02:11 hkdsun joined
02:13 Guest96 joined
02:15 EyePulp joined
02:30 rxc joined
02:34 indi_ joined
02:36 chipotle joined
02:39 Guest96 joined
02:46 soveran joined
02:48 Guest96 joined
02:55 Guest96 joined
03:08 basiclaser joined
03:10 edrocks_ joined
03:12 enigma_raz joined
03:15 Guest96 joined
03:15 wCPO_irssi joined
03:18 EyePulp joined
03:18 edrocks joined
03:20 saurabhjha joined
03:47 soveran joined
04:09 saurabhjha joined
04:37 indi_ joined
04:39 saurabhjha joined
04:47 Guest96 joined
04:52 shinnya joined
04:58 saurabhjha joined
05:25 xiaomiao joined
05:32 Guest96 joined
05:49 soveran joined
05:50 Cohedrin joined
06:17 edvorg joined
06:27 saurabhjha joined
06:38 soveran joined
06:40 indi_ joined
06:46 <minus> felixjet: no, but using cmdline args
06:50 saurabhjha joined
07:17 soveran joined
07:20 saurabhjha joined
07:25 tarkus joined
07:30 Guest96 joined
07:38 Guest96 joined
07:42 timg__ joined
07:54 Guest96 joined
08:15 Guest96 joined
08:15 winem_ joined
08:17 tavish joined
08:19 saurabhjha joined
08:29 EyePulp joined
08:31 bannakaf_ joined
08:46 saurabhjha joined
08:48 timg__ joined
08:55 efphe joined
09:03 chipotle joined
09:04 fractalsea joined
09:08 chipotle joined
09:24 saurabhjha joined
10:22 saurabhjha joined
10:45 EyePulp joined
10:52 soveran joined
11:02 fakenerd joined
11:13 saurabhjha joined
11:18 fakenerd joined
11:34 saurabhjha joined
11:37 shesek joined
12:00 sfa joined
12:03 timg__ joined
12:04 saurabhjha joined
12:08 edvorg joined
12:21 Guest96 joined
12:25 Guest96 joined
12:26 H1ccup joined
12:29 <H1ccup> Hello folks! First time here, so apologies if this turns out to be a noobish question. In our company, we've been using Redis as a dumb cache that can only be queried with a particular set of keys. So far so good, but I would love to do a bit more than that and make the querrying more intelligent. This is pretty much the data-set. Every day, a set of activities per user gets added to redis. Now, I would like to query all activities
12:29 Guest96 joined
12:30 <H1ccup> Also, I would like to query atleast one more level, say, all the activities where the person did X. Now, I've considered using ZRange/ZAdd to play around with UTC timestamp as an index, but that takes care of one level. What about querrying another level in? What could be a good solution for this?
12:31 <H1ccup> Hello folks! First time here, so apologies if this turns out to be a noobish question. In our company, we've been using Redis as a dumb cache that can only be queried with a particular set of keys. So far so good, but I would love to do a bit more than that and make the querrying more intelligent. This is pretty much the data-set. Every day, a set of activities per user gets added to redis. Now, I would like to query all activities
12:39 saurabhjha joined
12:53 chipotle joined
12:54 soveran joined
12:54 mikecmpbll joined
12:57 <minus> H1ccup: your message got cut off, seems like the web client doesn't break it into multiple messages automatically
12:57 fractalsea joined
12:57 <minus> you can't really search in redis
13:00 <H1ccup> @minus what about using a solution like this? https://gist.github.com/observerss/7129377
13:00 <H1ccup> minus: what about using a solution like this? https://gist.github.com/observerss/7129377
13:00 <minus> got you the first time
13:01 <minus> a typo in the headline doesn't make a good picture :D
13:01 <H1ccup> haha, I initially thought that it was a wrapper around Redis. And then, well.
13:01 EyePulp joined
13:03 <minus> i'd probably go with a RDBMS instead of Redis if i did something like that
13:03 <H1ccup> I do have RDBMS (postgres) sitting underneath all this
13:04 <minus> is it too slow to handle the full load?
13:04 <H1ccup> but the idea is to serve millions and millions of row faster, where people will mostly query between range
13:04 <H1ccup> and at most kind of activity the person is doing
13:08 saurabhjha joined
13:09 <minus> https://gist.github.com/observerss/7129377 ← the lua script in this should be using ZREVRANGEBYSCORE btw
13:11 <H1ccup> yes, I was talking more about the idea. I think I would go with my own implementation. But wanted to know how scalable such a solution, as this might end up creating too many keys, and too much overlap of data
13:12 <minus> scalable? well, if you can separate it by user you can throw it at redis cluster and scale just fine
13:12 <H1ccup> hmm, makes sense.
13:13 <H1ccup> is there any need of writing advanced data-structures on top of redis? Say like what twitter did to make it work for our needs?
13:13 <minus> my main concern would be how to get the data into redis reliably and consistently
13:14 <H1ccup> This redis cluster would be just used for reads. Everyday evening, or late night, new data will be sorted and dumped into it and made available again.
13:15 <minus> reasonable.
13:16 <minus> how about just adding a few postgres slaves instead? i can't imagine it being slow when you've got the index on the right columns
13:18 <H1ccup> well, if a table is say around 100 GB
13:19 <H1ccup> and you want to query a bunch of things a particular user has done from it, knowing that it doesn't change realtime, won't it be extremely slow?
13:21 <minus> if you're querying by time and you have the time column indexed? blazing fast. filtering by time and specific tasks: just as fast (without some special index for it). faster in redis if you create the right index, but will need some extra memory
13:25 <H1ccup> surely, json data indexed by time on redis, and aggregated by time over redis would be much, much quicker than doing the same with postgres?
13:25 <H1ccup> just for the fact that redis is in-memory.
13:26 <minus> indeed
13:28 <H1ccup> my only concern is this
13:29 <H1ccup> how denormalize can you possibly make a data-set. My dumb maths and some intuition suggests that only till secondary indexing - dealing with two indexes. Tomorrow if we go deeper, and want to query another subset of the data, data stored on Redis becomes useless
13:30 <H1ccup> or rather, it becomes too much to handle
13:37 <minus> what do you mean by secondary indexing?
13:41 <H1ccup> I mean, querying a data using another index. So, say you can query a dataset using timestamp, or by a set of activities theuser does
13:46 saurabhjha joined
13:47 <minus> so you'd create indices for every thing you'd like to be able to query, containing ids of the data; for querying you fetch the ids from the indices, intersect them and fetch the respecitive stored data. doing that client-side will be rather slow because of network latency; doing it in lua has the potential of blocking redis if it's a lot of data.
13:52 <H1ccup> so how would you suggest to go about it?
13:55 <minus> 1) write a baseline benchmark for postgres 2) get data into redis 3) get indices for data queried in 1) into redis (zset for time-based indices, sets for fixed-word ones) 4) write client-sided query solution for redis + benchmark 5) translate to lua and benchmark that as well
13:56 <minus> benchmark for maybe 2-3 most common queries
13:57 edvorg joined
14:00 <H1ccup> hmm
14:01 <H1ccup> why do the lua part? won't a python wrapper work good here?
14:02 <minus> yeah, you can probably do it decently in python if you batch up the operations well
14:02 <minus> but lua is probably a bunch faster
14:05 <H1ccup> Cool
14:05 <H1ccup> Thanks a lot sir.
14:08 <minus> you're welcome, have fun building it. would be cool if you could show me some benchmark results when you get it done
14:09 timg__ joined
14:09 <H1ccup> Most definitely.
14:09 saurabhjha joined
14:33 forgotmynick joined
14:41 ahtik__ joined
14:49 saurabhjha joined
14:50 cyborg-one joined
14:56 soveran joined
14:57 atomi joined
15:08 edvorg joined
15:17 etehtsea joined
15:18 EyePulp joined
15:18 saurabhjha joined
15:25 saurabhjha joined
15:51 mortenie joined
15:58 soveran joined
16:04 shinnya joined
16:09 SkyRocknRoll joined
16:23 saurabhjha joined
16:25 daxelrod joined
16:28 daxelrod1 joined
16:53 saurabhjha joined
16:55 saurabhj_ joined
17:24 hkdsun joined
17:25 hkdsun joined
17:25 Cohedrin joined
17:29 tavish joined
17:33 MrXXIV joined
17:43 saurabhjha joined
17:46 iamchrisf joined
17:47 iamchrisf joined
17:54 ChrisJames02170 joined
17:55 saurabhjha joined
17:55 sfa joined
18:00 soveran joined
18:07 efphe_ joined
18:22 chipotle joined
18:44 edrocks joined
18:48 saurabhjha joined
18:53 saurabhjha joined
18:56 fractalsea joined
19:13 GreenJello joined
19:15 Guest96 joined
19:15 jud joined
19:15 jud joined
19:22 fractalsea joined
19:22 SkyRocknRoll joined
19:29 ppang joined
19:44 Cohedrin joined
19:59 ogres joined
20:02 soveran joined
20:52 ragecryx joined
21:05 edge226 joined
21:16 soveran joined
21:16 soveran joined
21:17 saurabhjha joined
21:20 Cohedrin joined
21:35 wCPO joined
21:35 wCPO joined
21:35 cyborg-one joined
22:26 chipotle joined
22:47 bandroidx joined
23:00 enigma_raz joined
23:09 forgotmynick joined
23:14 timg___ joined
23:17 fakenerd_ joined
23:23 Guest96 joined
23:41 soveran joined
23:43 enigma_raz joined
23:44 enigma_r_ joined
23:45 rpap joined
23:50 enigma_raz joined