<     May 2017     >
Su Mo Tu We Th Fr Sa  
    1  2  3  4  5  6  
 7  8  9 10 11 12 13  
14 15 16 17 18 19 20  
21 22 _2_3 24 25 26 27  
28 29 30 31
00:23 bturker joined
01:05 djnym joined
01:24 bturker joined
01:41 djnym joined
02:05 djnym joined
02:14 Technocrat7 joined
02:24 bturker joined
03:25 bturker joined
04:26 bturker joined
05:05 djnym joined
05:36 bturker joined
06:36 bturker joined
07:46 bturker joined
08:07 bturker joined
09:06 djnym joined
10:23 <s2hc_johan> Hi, I've recently started a roling upgrade from 1.4.12 to 2.0.9, and looking at the first node that got 2.0.9 it's been repairing keys due to aae. I know there is some change to aae on disk in 2.0.9, but I'm qurious as to what reparid keys means
10:23 <s2hc_johan> is it just that there are missmatches in the aae hashes, or were there actuall differences in the key/value replicas?
11:22 djnym joined
11:44 andyt1 joined
12:06 <russelldb> s2hc_johan: I think it means that there are divergent values for some keys
12:08 <s2hc_johan> so the object themselves are corrupt in some way?
12:09 <s2hc_johan> cause the cluster have been up for several years and a few days ago we upgraded the first node and now we have several 100 000 keys getting repaired
12:11 <s2hc_johan> looking at riak-admin aae-status the last section "keys repaired" I've got 601344 as max on one of the rows, and similar numers on a few others
12:11 <s2hc_johan> I'm thinking if it could be something with the aae-change that was done to 2.0.9
12:15 <s2hc_johan> maybe it was stupid of me not to hold back the new aae thing, but I thought I needed to hold back only if I needed to roll back
12:25 <russelldb> s2hc_johan: I have a similar feeling (that something is up with AAE)
12:26 <russelldb> I colleague has been starting to dig into it, as we also see massive numbers of exchanges after a remove->add of a node, even after hand off has completed, all the keys are then AAE repaired…doesn't seem right
12:52 kota__ joined
12:54 Necro|senseless joined
13:28 <s2hc_johan> russelldb: yea that was my feeling as well, the traffice after taking a node up seems right, even some repairs would be to expect since we update somehere around 100k objects an hour, but going on with repairing a few keys for days on, feels funky
13:30 <s2hc_johan> we're seeing lots of this in the logs [info] <0.21175.19>@riak_kv_exchange_fsm:key_exchange:263 Repaired 213 keys during active anti-entropy exchange of {1101835218769001028176996768415010245287480393728,3} between {1101835218769001028176996768415010245287480393728,'riak@<node4>'} and {1113253200310648707225463056170606206378542366720,'riak@<node1>'}
13:35 Necromantic joined
14:44 codenamedmitri joined
15:34 metaphysician joined
16:21 codenamedmitri joined
16:33 Necromantic joined
17:34 <MitchellSalad> is strong consistency support still experimental in riak?
17:35 <codenamedmitri> I’m pretty sure that’s still the case. (experimental)
17:36 <MitchellSalad> hmm... okay
17:36 <MitchellSalad> does anyone have recommendations for doing distributed transactions?
17:37 <codenamedmitri> I think the first recommendation is - double and triple-check that you actually need em.
17:37 <codenamedmitri> but in general, you have a couple of options.
17:38 <MitchellSalad> I do actually need them, for account-creation type actions that must be serializable
17:38 <codenamedmitri> the general pattern that I’ve seen recommended is — go through the steps, and at the end, to “commit” the transaction, write the success object. And only consider the transaction to have succeeded if that object exists.
17:39 <MitchellSalad> oh, within riak you mean. I was asking about other off-the-shelf solutions, actually (but working within riak is an option I hadn't thought of)
17:39 <codenamedmitri> so like, step 1) write to the ‘users’ bucket, 2) write to ‘user-account’ bucket, etc, … step X) write a ‘user-account-success’ bucket object
17:39 <codenamedmitri> outside of riak,
17:40 <codenamedmitri> you basically have two options. Either use a distributed consensus algorithm like Paxos, or pick a single-threaded service to be the point of serialization
17:40 <codenamedmitri> so for example, in RiakCS, a single server is designated just for account creation.
17:41 <codenamedmitri> all other read/write operations are distributed & eventually consistent. but account creation (and bucket creation, I think) specifically is serialized because there’s just one actor
17:42 <codenamedmitri> Paxos and company are more robust and more available, but harder to run, operationally. Picking a single-threaded point of serialization is much simpler to run. but, then you need fallbacks & failovers in case the service goes down
17:42 <MitchellSalad> yeah, a consensus algorithm is required of course, but I was actually trying to ask a much higher-level question :)
17:43 <MitchellSalad> which is - any distributed SQL databases you (or whomever) can recommend looking into?
17:43 <codenamedmitri> oh?
17:43 <codenamedmitri> ah I see
17:43 <codenamedmitri> not sure about SQL. possibly the wrong room for it :)
17:43 <MitchellSalad> oh, haha
17:44 <codenamedmitri> (interestingly enough, the Datomic database kind of uses that second approach.
17:44 <MitchellSalad> SQL may have been the wrong term, I simply need linearizable transactions
17:44 <codenamedmitri> which is, reads are parallel and multi-actor, but *writes* are performed by a single-threaded service. That’s their approach to transactions)
17:44 <MitchellSalad> when I think transaction, I think SQL, so, terminology error on my part
17:45 <MitchellSalad> I wonder exactly how functional strong consistency is in riak
17:47 <MitchellSalad> is it totally broken? or just sufficiently new that they are wary of advertising the feature without a warning?
17:48 <codenamedmitri> it’s not totally broken. more like.. still needs some work, and management doesn’t want to focus on it & spend any more cycles/resources.
17:49 <MitchellSalad> ah
17:49 <MitchellSalad> do you work at basho, codenamedmitri?
17:49 <codenamedmitri> I do not.
18:13 Necromantic joined
18:26 kleptocroc- joined
18:36 bturker joined
18:37 djnym joined
18:57 jmeredith joined
19:00 djnym joined
19:03 riak051 joined
19:04 <riak051> exit
19:04 riak051 left
19:27 djnym joined
19:37 bturker joined
20:15 djnym joined
20:24 bturker joined
21:24 bturker joined
21:49 codenamedmitri joined
22:09 codenamedmitri joined
22:26 bturker joined
23:27 bturker joined
23:47 Tom- joined
23:54 bturker joined