21:00:42 #startmeeting swift 21:00:43 Meeting started Wed Jul 5 21:00:42 2017 UTC and is due to finish in 60 minutes. The chair is notmyname. Information about MeetBot at http://wiki.debian.org/MeetBot. 21:00:44 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 21:00:46 The meeting name has been set to 'swift' 21:00:48 who's here for the swift team meeting? 21:00:52 hey 21:00:52 o/ 21:00:52 o/ 21:00:53 o/ 21:01:17 o/ 21:01:20 weee 21:01:22 hello 21:01:48 hi 21:01:50 cschwede won't be able to make it this week 21:01:51 o/ 21:02:19 tdasilva: around? 21:02:30 hello 21:02:55 welcome everyone 21:03:15 I think this will be a relatively quick meeting this week 21:03:24 Link https://wiki.openstack.org/wiki/Meetings/Swift 21:03:44 #topic upcoming events 21:03:51 the PTG and the summit are coming up 21:03:59 #link https://etherpad.openstack.org/p/swift-ptg-queens 21:04:09 that's the etherpad to start collecting topics for the PTG 21:04:37 please register for the PTG and get your hotel soon so that you have a close place before they are all booked 21:05:01 is the conference hotel already full? 21:05:08 I hope not! 21:05:23 I just booked mine last week, so I'd guess there is still room 21:05:36 https://www.openstack.org/ptg has all the PTG info 21:06:12 I booked ok last week, except the Saturday before was full 21:06:26 the schedule for the PTG will be similar to the last one in atlanta: monday-tuesday will have a lot of "cross project" things to invest in. wednesday-friday will have rooms dedicated to the swift team 21:06:47 please, *please* encourage all ops people to attend the PTG 21:06:52 Just checking. There is room. 21:07:00 if you have to choose, choose the PTG over the sydney summit 21:07:00 the last one was good, I recommend it to others 21:07:33 yeah, I expect it to be a very productive week 21:07:34 we need a mattoliverau lifesize cutout 21:08:02 well maybe not life sized.. you could at least make it slimming ;) 21:08:09 :D 21:08:13 the other upcoming event, of course, is the summit in sydney. info at https://www.openstack.org/summit/ 21:08:36 the call for presentations for sydney is only open for about another week, so if you're interested in giving a talk, now's the time to submit it 21:08:50 note that the sydney summit is only 3 days long 21:09:06 :( 21:09:18 3 days is teh dumb 21:09:31 while I expect to be going, I do not expect there to be much in the way of deep, productive swift team time (ops or otherwise) 21:09:44 therefore, that is why I am encouraging all devs and ops to be at the PTG instead 21:09:53 except for visiting the beach house right? 21:09:58 notmyname: ^ 21:10:23 for most of the world, sydney is a *very* long and expensive trip for a 3 day conference that doesn't have any dedicated time for our team 21:10:43 OMG the sessions is still empty :\ 21:10:48 it's like way out in september tho? 21:10:50 mattoliverau: That is my plan. 21:11:02 july, august - it seems like we have awhile yet 21:11:05 clayg: for the PTG? 21:11:07 jungleboyj: ;) 21:11:08 yeah 21:11:18 mattoliverau: Bucket list check point, here I come. 21:11:28 any questions about the ptg or the summit? 21:12:19 all right. moving on, then 21:13:14 I wanted cschwede to share his follow-up for bug triage, but he's not able to make it today. he said he'd follow up via email 21:13:27 #topic docs migration work 21:13:43 have you been following the mailing list threads about the docs migrations? 21:14:14 the summary is that for various reasons, the stuff that is currently managed by the docs team will be moved into respective project repos 21:14:48 and the work to do so is on each team (in large part because nearly the whole docs team got laid off at the same time mattoliverau did :-( ) 21:15:04 :( 21:15:12 I was traveling last week and holiday early this week, but i'm starting to look at that migration work now 21:15:13 How about that 1.2 Gig Docs repo? 21:15:21 heh, only 760MB ;-) 21:15:43 so there's a couple of steps here 21:15:44 Yeah, after I did a build it ballooned and almost killed my VM. :-( 21:15:56 sounds so huge docs 21:16:02 first is that we'll have to import a bunch of stuff to the swift and swiftclient repos 21:16:15 *after* that, we can rearrange, de-dup, etc 21:16:17 So big! 21:16:24 but that's across all projects, right? not 3/4 gig just for swift? 21:16:33 timburke: Right. 21:16:34 timburke: correct. it's just the source repo 21:16:41 jungleboyj: have you already started looking at the docs migration work for swift? 21:16:47 notmyname: isn't the first step to start using the new openstackdocs theme? https://review.openstack.org/#/c/478402 21:16:47 cause if so... yay! 21:16:48 patch 478402 - swift - Switch from oslosphinx to openstackdocstheme 21:16:58 oic 21:17:01 notmyname: No, I was supposed to do it for Cinder. 21:17:06 jungleboyj: ah ok 21:17:23 tdasilva: perhaps. I'm getting caught back up to it. seems like that would be a simple thing to do 21:17:23 Panic'ed today when I started noticing that our links out on the web are broken. 21:17:37 yeah, I heard that happened (all the 404s) 21:17:44 notmyname: Depending on how long this takes for Cinder I could help out with Swift. 21:18:04 jungleboyj: cool. ping me when you're done with cinder, and we'll see where the swift migration is 21:18:11 notmyname: i wish, I think it really breaks the look and feel of the docs, so I think we will have to do meaningless tweaks over time just for things to look nice again :/ 21:18:15 jungleboyj: try to go "swiftly" on cinder 21:18:33 tdasilva: :-( 21:18:40 notmyname: Sure. Is anyone starting on it? 21:18:43 I am 21:19:15 notmyname: Ok, let me know if you have questions. I have found, already, that the build fails when just copying things over. 21:19:22 :-( 21:19:30 They have also removed everything from the openstack-manuals tree. 21:19:32 * notmyname was hoping for a few days of work and be done with it 21:19:40 tdasilva: what breaks? patch 478402 looks fine? 21:19:41 https://review.openstack.org/#/c/478402/ - swift - Switch from oslosphinx to openstackdocstheme 21:19:43 So you need to go back to the 'before-migration' tag. 21:20:22 clayg: builds don't break, it's very subjective, I'm just used to the old looks, so I don't like how things look in the new theme 21:20:36 clayg: compare: http://docs-draft.openstack.org/02/478402/5/check/gate-swift-docs-ubuntu-xenial/7563106//doc/build/html/associated_projects.html#associated-projects 21:20:38 oh... sorry I don't have opinions on that sort of thing 21:20:43 to https://docs.openstack.org/swift/latest/associated_projects.html#associated-projects 21:20:47 lol 21:21:11 yeah I mean both pages seems to have the same information :\ 21:21:34 I assumed since someone spent time on it they must think one is better than the other? 21:21:46 we can work on the prettiness later 21:22:05 maybe one has the denting wrong? 21:22:05 yes, this work is not insignificant, and it is being forced upon us as a team by external forces 21:22:09 notmyname: ++ 21:22:24 I was planning to just get Cinder's stuff moved and then work on cleanup afterwards. 21:22:24 it sorta looks wrong on both tho - un-even 21:22:32 but docs are important, so we need to do it 21:22:39 let's do it! 21:23:04 I'll start, and hopefully finish, without needing to bring in too many other people. but I may need to ask for help, depending on how it goes 21:23:40 but also since this is a "do it or lose your docs" sort of situation, I expect the eventual reviews to be cursory "yup, content moved, +2/+A 21:24:01 so don't freak out when a lot of new stuff gets added :-) 21:24:09 notmyname: Let me know if I can help. I am hoping to get a couple of patches up yet today for Cinder. I am traveling tomorrow but will probably spend Friday and the weekend working on the rest. 21:24:10 tdasilva: did someone figure we're all getting older and increase the font size accordingly? ;) 21:24:21 jungleboyj: thanks 21:24:22 acoles: hehe 21:24:23 acoles: yes! 21:24:27 WHO MOVED MY CHEESE 21:24:42 ok, any questions about the docs migrations? 21:24:42 * jungleboyj shakes my head 21:24:46 I refuse to get older! 21:25:22 ok, moving on 21:25:31 #topic open discussion 21:25:32 toward the cheese 21:25:48 I expect there to be swift and swiftclient releases in the next few weeks 21:25:50 jFYI 21:26:05 we had a libec/pyeclib release last week 21:26:09 I'll keep the priority reviews page updated 21:26:11 why is patch 470158 crossed off on priority reviews? 21:26:12 tdasilva: nice! 21:26:12 https://review.openstack.org/#/c/470158/ - swift - Write-affinity aware object deletion 21:26:17 working now to get rpm out for it 21:26:50 clayg: good question. I think christian did that. looks like a typo 21:26:51 I don't know. I need to go loop back around and review that 21:26:58 is onovy building .deb packages? 21:27:00 mattoliverau: do you know how len(local_primary_nodes) is calculated? 21:27:24 now uncrossed 21:27:27 yeah, using the is_local method 21:27:40 and looping thorugh the primaries.. 21:27:52 torgomatic: anything we need to know about your patch 340526 vs zaitcev's PUT+POST? 21:27:53 https://review.openstack.org/#/c/340526/ - swift - Unify Putter and MIMEPutter into a single class. 21:28:00 ahh... so it *is* per part - that's EPIC 21:28:00 it kinda does it when it needs it.. I wonder if that can be cleaned up to be in one place. 21:28:06 morning, guys 21:28:24 nothing to know yet; we'll see how things shake out 21:28:42 I had some thoughts around using different verbs for versioning instead of putting it in the path, but no code yet 21:28:50 kong: I think mattoliverau is telling us we can probably merge patch 470158 - so that's exciting! 21:28:50 https://review.openstack.org/#/c/470158/ - swift - Write-affinity aware object deletion 21:29:18 that way, we get a 405 when the object server is old, and if we need v3 or v4 or whatever, we just have to make up and implement some new verbs 21:29:25 clayg: hooray! 21:29:33 I'll go give it another whirl today, and if all good I'll give it the second +2 ;) 21:29:34 but like I said, no code yet 21:29:37 kong: ^ 21:30:05 mattoliverau: thanks, feel free to ping me at irc if you have any question 21:30:33 kong: cool, we share a similar timezone, so that works well ;) 21:30:39 ataraday_: i see ;) 21:30:41 torgomatic: seems a little weird to invent new verbs all the time instead of actually, you know, a protocol version number 21:30:45 mattoliverau: i see :) 21:31:29 * kong stops disturbing swift meeting 21:31:33 kota_: have you been following kevin & tushar on the isa-l bug? 21:31:34 notmyname: true, but this lets you do it on an operation-by-operation basis... you can come up with a new bit of stuff for object PUT, then later on do something special for GET, but you don't have to tie them together with a single version number 21:32:03 kong: you are very welcome here. you aren't disturbing anything. this meeting is for you :-) 21:32:04 acoles: kota_: do we have more exposure on this isa-l rs_vand >= 5 parity thing or not? 21:32:07 clayg: is that update recently in github issue traacker? 21:32:17 kota_: yeah pretty sure 21:32:32 clayg: ah, right. thanks for bringing that up 21:32:44 i looked at it briefly but i don't think kevin and tuchar following that 21:33:08 * kong is reading what you are saying carefully to pretend he understands everything 21:33:26 kota_: I also want to get some details on urgency of lp bug #1701472 21:33:27 Launchpad bug 1701472 in OpenStack Object Storage (swift) "ring builder can converge dead state in rebalance" [Undecided,New] https://launchpad.net/bugs/1701472 21:33:48 clayg: yeah, i'm still working on how it works 21:34:03 sorry i have been sick in the primary half of this week. 21:34:05 clayg: kota_: how should we proceed on the isa-l bug? kota_ can you ping them? do we need to ignore it and let intel deal with it (that seems bad)? 21:34:15 this is the EC bad frag decode bug https://github.com/01org/isa-l/issues/10 21:34:42 the heart of the issue that caused us to refuse to start swift if some isa-l policies were not deprecated 21:34:54 kong: please don't hesitate to ask for clarity. IRC meetings are the hardest time for non-english speakers because everything goes by so quickly 21:35:57 notmyname: sure, i will. btw, for some people who don't know, we (catalyst cloud in New Zealand) have been using Swift at production since last year. 21:36:14 we are keen to help fix bug :-) 21:36:15 wooo! wtg kong !!!! 21:36:27 kong: oh your with catalyst! 21:36:42 mattoliverau: yeah 21:36:44 kong: :-) 21:36:52 kong: say hi to bruno ;) 21:37:03 mattoliverau: hah, i promise i will 21:37:13 kong: https://bugs.launchpad.net/swift there's only 500 of them :D 21:37:24 heh 21:37:35 kong: also tell him to hire me ;P 21:37:37 i will try my best to figure out what i can help there 21:37:54 mattoliverau: hah, actually, we were talking about that in yesterday's meeting 21:38:14 kong: just kidding, all help is welcome and thanks for working on patch 470158, I remember Bruno saying that's very important to your customers 21:38:14 https://review.openstack.org/#/c/470158/ - swift - Write-affinity aware object deletion 21:38:45 tdasilva: yes it is. after merging, i will backport it to our internal repo 21:38:52 we are using a old version of swift 21:39:14 kota_: so but the ring rebalanace bug - just to understand the priority ... it is annoying ... but do you have a workaround? 21:39:19 kong: would be great to see someone from catalyst at the PTG 21:39:48 kota_: like just increase overload or add a node or something? i guess you can just use the any/all trick get the rings you want and ship it out - out-of-band change only! 21:39:57 * kong will ask Bruno to arrange some money for PTG :) 21:40:03 clayg: https://github.com/01org/isa-l/issues/10 seems to be saying that there may be an issue with 4 parity frags if num data frags > 21 21:40:06 kong: great! thanks 21:40:15 acoles: that's what scares me 21:40:21 for rebalance, it's not super high priority but working soon is better because it looks like making dead lock state to rebalance on some cases 21:40:49 kota_: I only ask because the all/any change isn't obvious to me - it seems the code did intend to do what it was saying - like it wasn't just a typo - i'm freaked out that more tests don't rub up against that condititonal? 21:40:50 i notice some ways to mitergate the dead lock by hand operation though 21:41:23 clayg: yup, changing overload or weight could help to rebalance work 21:42:16 kota_: ok, tagged, torgomatic is working around lp bug #1697543 21:42:18 Launchpad bug 1697543 in OpenStack Object Storage (swift) "Ring refuses to save even when 100% parts move" [Medium,Confirmed] https://launchpad.net/bugs/1697543 21:42:35 clayg: ok, current code is in your intention, I need to rethink the algorism because I WAS not sure if it was intended or not 21:42:46 acoles: not sure this is related, but found this: https://github.com/ceph/ceph/blob/master/src/erasure-code/isa/README#L23 and this https://github.com/ceph/ceph/blob/master/src/erasure-code/isa/ErasureCodeIsa.cc#L381 21:43:22 kota_: which is definitely a different issue - but christian pointed out that the associated change doesn't fix the ring/script in the bug - so maybe rubbing some braincells together will help 21:44:24 clayg: kk, thx. I will look at. 21:44:31 kota_: yeah... it was there to prevent picking up parts from one disk and putting them back on the same server 21:44:47 kota_: but... if the disks on the server are already not balanced!!! :\ 21:45:05 tdasilva: should we add those same constraints to libec? 21:45:38 clayg: yeah, and my simulation with current code, in some case, the gatherd parts seems to go back to same servers as reassignment. 21:45:55 which I reported in the lp bug. 21:46:11 notmyname: i'm not sure, but I definetely found interesting that they will just change the k for the user, don't allow at all 21:46:14 tdasilva: acoles: notmyname: yeah I think the rules outlined in the bug report are acctually correct 21:46:15 tdasilva: thanks. so that also shows cap of 32 data frags 21:46:36 tdasilva: yes. "interesting" 21:47:22 clayg: the rules listed on https://github.com/01org/isa-l/issues/10#issuecomment-310944022 ? 21:47:49 yeah 21:48:07 maybe our answer then is to user isa-l v2.19 then 21:48:43 last comment is funny...there's no source, i just know it! 21:49:06 i mean... that's probably a good idea - but also probably orthogonal - unless they expose a "check if these params are cool" sorta method? 21:49:21 brute force 21:49:52 but that last one is interesting - it makes it sound like a 7+6 is ok! 21:49:52 are there any other topics to bring up this week in the meeting? does anyone else have something to discuss? 21:51:13 * kota_ need to look at the k, m conditions on https://github.com/01org/isa-l/issues/10#issuecomment-310944022 to clarify 21:51:30 I think we're done, then 21:51:48 Thanks. 21:51:51 thanks for coming, everyone. thanks for working on swift 21:51:54 #endmeeting