<    March 2017    >
Su Mo Tu We Th Fr Sa  
          1  2  3  4  
 5  6  7  8  9 10 11  
12 13 14 15 16 17 18  
19 20 21 22 23 24 25  
26 27 28 29 30 31
00:22 FunnyLookinHat joined
00:28 gazarsgo left
00:42 FunnyLookinHat joined
00:57 cyborg-one joined
01:19 edvorg joined
01:23 hphuoc25 joined
01:39 edvorg joined
01:41 unbalancedparen joined
01:41 soveran joined
01:42 minimalism joined
01:44 baiwfg2 joined
01:45 <baiwfg2> hi. I'm new
02:02 daxelrod joined
02:10 FunnyLookinHat joined
02:10 FunnyLookinHat joined
02:13 FunnyLookinHat joined
02:14 roelmonnens joined
02:43 soveran joined
02:47 chipotle joined
02:58 fakenerd joined
03:08 EyePulp joined
03:16 cyborg-one joined
03:24 hphuoc25 joined
03:28 hphuoc25 joined
04:02 roelmonnens joined
04:06 RemiFedora joined
04:10 rchavik joined
04:36 fakenerd joined
04:44 soveran joined
04:55 fakenerd joined
05:03 ppang joined
05:06 bngle joined
05:23 hue joined
05:34 raspado joined
05:45 ppang_ joined
05:47 tavish joined
05:50 roelmonnens joined
05:54 tavish joined
05:55 Fabricio20 joined
06:02 soveran joined
06:09 minimalism joined
06:18 roelmonnens joined
06:19 roelmonnens joined
06:20 roelmonnens joined
06:24 svm_invictvs joined
06:27 edge226 joined
06:30 tarkus joined
06:32 hphuoc25 joined
06:41 [1]moteus joined
06:51 MonkOfCity joined
06:53 EyePulp joined
06:57 hphuoc25_ joined
07:01 [1]moteus joined
07:16 tavish joined
07:42 Dave_R joined
07:51 tavish joined
07:57 lukasdboer joined
08:00 tavish joined
08:02 hphuoc25 joined
08:03 lukasdboer joined
08:06 mungustas joined
08:07 <mungustas> hi, I want to search for keys by pattern "*queue*"
08:07 <mungustas> https://pastebin.com/raw/FL32TL9B
08:08 <mungustas> how come KEYS *queue* and --scan --pattern "*queue*" gives different results ?!
08:11 roelmonn_ joined
08:20 efphe joined
08:30 hphuoc25 joined
08:36 tavish joined
08:42 <mstaack> be careful using keys command in production
08:49 <badboy_> mungustas: the dataset probably changed in between those two calls
08:57 cyborg-one joined
09:00 <mungustas> yeah I know
09:01 <mungustas> I'm trying to avoid using KEYS
09:01 <mungustas> by using --scan --pattern
09:01 <mungustas> but was scratching my head why both commands give different results
09:01 <mungustas> I am pretty sure dataset hasn't changed in between those calls
09:05 <mungustas> https://pastebin.com/raw/sSbdPgJ1
09:05 <mungustas> current results
09:05 MonkOfCity joined
09:06 MonkOfCity joined
09:07 MonkOfCity joined
09:09 <mungustas> hm but maybe it does change
09:09 EyePulp joined
09:14 <badboy_> ;)
09:14 <badboy_> otherwise you would have found a serious bug and I doubt that
09:14 <badboy_> > reports:webform_opens:processing:queue:2
09:15 <badboy_> this looks like it is from some job queue, probably workers storing their progress in Redis
09:15 <badboy_> would make sense that those keys are gone when the workers finished
09:16 ABK joined
09:18 MonkOfCity joined
09:20 <ABK> we just updated redis to version 3.2.8 but can't get the sentinel client-reconfig-script to work anymore. Any ideas?
09:27 SkyRocknRoll joined
09:30 dcesario joined
09:34 maneta left
09:36 raspado joined
10:05 programmingcool joined
10:09 <badboy_> ABK: there was a similar bug report
10:09 <badboy_> ABK: what version did you have before?
10:11 <badboy_> https://github.com/antirez/redis/issues/3903
10:15 etehtsea joined
10:15 roelmonnens joined
10:20 roelmonn_ joined
10:37 hphuoc25 joined
10:39 Mr__Anderson joined
10:42 pila joined
10:53 bf_ joined
11:03 roelmonnens joined
11:06 rendar joined
11:07 hphuoc25 joined
11:16 fakenerd joined
11:25 EyePulp joined
11:26 moteus joined
11:28 mikecmpbll joined
11:28 <mikecmpbll> is the redis sentinel suitable for client failover? if so, do people ever use redis sentinel and something like haproxy, if so why?
11:29 moteus joined
11:30 r23r joined
11:30 <r23r> hi
11:31 HelgeO joined
11:36 <Inge-> o_O
11:41 <mstaack> hi
11:43 EyePulp joined
11:43 hphuoc25 joined
12:03 wlightning-fuel joined
12:04 HelgeO_ joined
12:23 <ABK> @badboy that's my colleague that wrote that questions. We upgraded from 2.0.x
12:23 <badboy_> ah :D
12:24 <badboy_> let me check my sentinel setup
12:24 <ABK> we can't find any docs other than the small inline ones in the configuration file.
12:25 <badboy_> yeah
12:25 <badboy_> well, it should work.
12:25 <badboy_> give me a second, I'll try to reproduce it
12:26 Guest96 joined
12:31 <badboy_> ABK: I can't reproduce it at all
12:32 <badboy_> 1. Check configuration of all Sentinels
12:32 <badboy_> 2. Check permissions of that script on all Sentinels (the script is on all Sentinel servers, right?)
12:32 <badboy_> 3. Check that it has an appropiate shebang in the file
12:33 <badboy_> 4. Check that the name nuvomaster is correct
12:33 bauruine joined
12:36 Electron^- joined
12:46 leito joined
12:47 HelgeO joined
12:48 chipotle joined
12:50 ante joined
12:50 bauruine joined
12:56 hphuoc25 joined
13:02 edrocks joined
13:06 soveran joined
13:16 wlightning-fuel joined
13:19 EyePulp joined
13:28 enigma_raz joined
13:33 soveran joined
13:33 soveran joined
13:35 wlightning-fuel joined
13:37 raspado joined
13:38 daxelrod joined
13:43 chipotle joined
13:48 hphuoc25 joined
13:53 daxelrod joined
13:57 daxelrod joined
13:57 pila joined
14:07 <ABK> @badboy 1) it's on every sentinel server and they are correct. 2 & 3) The script was getting executed before the update. 4) It's correct
14:08 <badboy_> 2&3) still check it again ;)
14:08 <ABK> @badboy Does your scripts get executed on failover?
14:08 <badboy_> let me check the actual logs
14:08 <ABK> @badboy We just did that now
14:08 <ABK> :)
14:10 hphuoc25 joined
14:12 <badboy_> funnily enough the docu is actual wrong it seems
14:12 <badboy_> # <state> is currently always "failover"
14:12 <badboy_> it's always "start"
14:14 byoungb joined
14:14 <ABK> @badboy We have a echo statement at the top of the script and it seams that they are never executed
14:14 <badboy_> Notification Mi 29. Mär 14:31:19 CEST 2017
14:14 <badboy_> +switch-master mymaster 6381 6380
14:15 <badboy_> ABK: bash scripts?
14:15 <byoungb> Does anyone know of a way too keep a slave that is just coming online from being used as a read slave from a redis sentinel client?
14:16 <ABK> @badboy So your scripts are getting executed? Yes, it's bash scripts
14:16 <badboy_> yes, they are executed
14:16 <byoungb> When I add a redis slave node to the cluster I get a bunch of "BusyLoadingError: Redis is loading the dataset in memory" from clients trying to do reads on the not fully replicated slave
14:16 <badboy_> ABK: again, do you have a shebang in those scripts?
14:16 <badboy_> byoungb: that is expected
14:17 <ABK> @badboy The perms used to be root:ubuntu, and that worked fine with 2.0.x; now we've set them to redis:redis, and they dont run. Yes, has a shebang, scripts havent been modified since the apt upgrade
14:17 <byoungb> anyway to mitigate this?
14:18 <byoungb> I will look at the python client and see if I can try another slave automatically
14:18 <ABK> @badboy And just to clarify, we set the perms to redis:redis after the problem manifested
14:19 <badboy_> byoungb: no. while the dataset is loaded, Redis will refuse to answer reads or writes
14:19 <badboy_> ABK: no more ideas then.
14:19 <badboy_> ABK: start tracing/debugging what's happening
14:20 <ABK> ok thx
14:22 <byoungb> are slaves that are loading the dataset marked as odown or sdown?
14:25 <byoungb> @badboy I understand that... just doesn't seem to be very HA if adding a server to the sentinel cluster causes issues with some of my client connections
14:26 <badboy_> byoungb: a slave loading a dataset will answer to PINGS with -LOADING and thus considered up
14:27 <badboy_> answering to requests without a fully loaded dataset would mean to answer inconsistently, which is not much better than answering with an error
14:27 <badboy_> (of course depending on your requirements)
14:27 <badboy_> and your client can always decide to just ask the master in that case
14:33 edrocks joined
14:37 <byoungb> okay well looks like I get to maybe contribute to the python redis client
14:37 <badboy_> :)
14:38 unbalancedparen joined
14:43 shinnya joined
14:44 bleb left
14:48 bandroidx joined
14:49 etehtsea joined
14:51 felixjet joined
15:00 felixjet joined
15:04 roelmonnens joined
15:11 hphuoc25 joined
15:14 LSN joined
15:14 <LSN> Anyone know what "-script-timeout" means in the redis-sentinel.log?
15:15 daxelrod joined
15:16 al-damiri joined
15:22 Electron^- joined
15:28 timg__ joined
15:41 <badboy_> LSN: raised if any scripts runs for more than 60 seconds
15:41 <badboy_> (and the script is then killed)
15:43 raspado joined
15:45 raspado_ joined
15:51 leito joined
15:56 efphe joined
15:57 <LSN> Thanks badboy. Our scripts are placed on Amazon's NFS (EFS), its not fast, guess that could be a problem.
15:59 timg___ joined
16:00 <byoungb> badboy_: the LOADING error is not sent at connection time.... is there anyway that could be enabled? I just went through redis-py and how everything is handled... and there is no good way for me to fail over to another slave or to the master when the LOADING error is thrown at the command level and not at the connection level.
16:00 soveran joined
16:01 <badboy_> no
16:03 <byoungb> any ideas? because redis-py sets up a connection pool with all of the slaves in a round robin fasion and then fails to the master.... but this all happens when creating the connection pool
16:04 <byoungb> so if a slave exists even if it is currently loading it is the only thing that reads will be sent to since it only adds the master if there are no slaves it could connect to
16:05 <byoungb> I guess I could add a ping during the connection block to make sure it does not respond with LOADING
16:05 <byoungb> does that sound acceptable?
16:06 svm_invictvs joined
16:10 <badboy_> I don't know the internals
16:11 <byoungb> yeah I guess I was just confirming that PINGing a slave right after connecting to it will return LOADING if it is loading in the dataset
16:11 <badboy_> it should
16:12 <byoungb> badboy_: cool thanks
16:22 soveran joined
16:23 Mr__Anderson joined
16:24 tavish joined
16:27 rchavik joined
16:32 saltsa joined
16:51 soveran joined
16:51 SkyRocknRoll joined
16:58 efphe joined
17:02 roelmonnens joined
17:02 bandroidx joined
17:11 svm_invictvs joined
17:17 tarkus joined
17:18 bandroidx joined
17:19 maxmatteo joined
17:31 roelmonnens joined
17:36 byoungb left
17:49 bandroidx joined
18:03 mikecmpbll joined
18:13 daxelrod joined
19:46 soveran joined
19:46 soveran joined
19:57 hashpuppy joined
20:09 timg__ joined
20:15 gheegh joined
20:17 chipotle joined
20:31 dahl joined
21:11 Mr__Anderson joined
21:25 fredsted joined
21:29 _Alaric_ joined
21:30 Azure joined
21:51 chipotle joined
21:53 atomi joined
21:54 raspado joined
22:11 atomi joined
22:14 atomi joined
22:31 chipotle_ joined
22:40 GreenJello joined
22:59 efphe joined
23:23 chipotle joined
23:54 sz0 joined
23:55 lexAngeles joined
23:56 chipotle joined