03:01:31 #startmeeting Large Deployments Team May 2016 Meeting 03:01:32 Meeting started Fri May 20 03:01:31 2016 UTC and is due to finish in 60 minutes. The chair is VW. Information about MeetBot at http://wiki.debian.org/MeetBot. 03:01:33 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 03:01:35 The meeting name has been set to 'large_deployments_team_may_2016_meeting' 03:02:28 quick roll call 03:02:37 o/ 03:02:49 o/ 03:02:50 1 03:02:54 o/ 03:02:59 0/ 03:03:38 hope everyone didn't have too many cloud fires to put out after the summit 03:04:11 #topic summit follow up 03:04:35 so, the one thing we didn't manage to boil things down to was a new item to pick up as a team and pursue with devs 03:05:14 I think that the retiring/removing/hidding old images was as close as we got 03:05:43 but, any particular items from the LDT sessions at the summit - or any other sessions - that we need to bring back up as a team? 03:06:11 I also think the whole delivery of images to compute nodes and then the snapshot process is something we could look into? 03:06:51 yeah originally back after tokyo we had talked about trying to pick up some glance stuff. so that seems as good as any. 03:07:30 can you elaborate sorrison_laptop 03:07:39 you mean the passing of images through the glance APIs 03:07:41 or somethin gelse 03:07:44 i would also liek to see LDT as a source for info for shepherding people through the cells v1 —> v2 transition, although it’s probably a bit early to be doing much of anything real useful there. 03:08:01 yeah I like that idea mdorman 03:08:05 me too 03:08:26 sounds like the change for migrating is going to be O, right? 03:08:51 i don’t recall exactly, but that seems about right. definitely should probably be a focus for barcelona 03:08:53 WRT glance I partly mean bypassing glance to get it out of the way 03:09:21 yeah - same thing 03:09:23 yeah i think these glance things would be a good shorter term thing to do 03:09:37 cool 03:10:03 unfortunately this is not a good time for belmiro. He has looked at the deprecated image bit the most 03:11:38 basically need a flag on a glance image that nova can query on boot 03:12:03 yeah, I think so 03:12:14 was just looking back at the Glance stuff here - https://etherpad.openstack.org/p/AUS-ops-informal-meetup 03:12:29 yeah I went to that session 03:14:18 i think i missed that part. 03:14:26 looks like https://wiki.openstack.org/wiki/Glance-deactivate-image and https://review.openstack.org/#/c/132717/ are most appropriate 03:15:28 VW: with the deactivate image, i think nova rebuild will still fail 03:15:41 yeah - I think that's the curx of the issue 03:15:54 the hard part to deprecate an image in cloud is there are still some instances depend on that 03:15:54 we want to stop the new, but have tenants out there who may need to rebuild 03:16:05 exactly 03:16:48 i'm responsible for our cloud's images maintaining 03:17:21 and i would say the old images maintenance is frustrating 03:17:38 agreed 03:17:53 with a public cloud to look after, I'm not sure they ever dia :) 03:18:11 yeah unfortunately reading the commit message it doesn't look like it will do what we want 03:18:40 almost want a "bootable" flag for an image 03:18:58 that's really the only thing we want to disable 03:19:06 agreed 03:19:12 which means nova and glance interaction 03:20:10 so I guess we need to talk to some glance people about whether we can redefine what deactivated is or we need a new action 03:20:29 yes, I think so 03:20:42 but do we need to bring Nova to the conversation too 03:21:22 yeah as they would need to check that value on a boot api request? 03:21:55 really does seem this is mostly on the nova side. glace wise you could simply put some ‘bootable’ metadata key on the image, no changes needed to glance. 03:22:29 maybe a nova filter can do the trick 03:22:29 yeah true 03:22:31 sorrison_laptop: that's a good point, re the 'bootable' flag 03:23:00 VW: I'm a glance core actually :D 03:23:11 WIN! 03:23:28 would be good to discuss with nova, get a feel for how they would envision doing this. unsure if this particular piece of the issue has ever been raise with them. 03:24:09 a filter is an interesting idea 03:24:17 yeah flwang - I haven't taken off my Rackspace hat and hit Hemmanth up with anything re: LDT 03:24:18 yet :) 03:24:30 yes, a filter is 03:24:30 glance has the functionally to support admin metadata key/values 03:24:31 VW: i think the tricky part is we don't want to allow to create new instance based on the image, but we should keep all the other things, like rebuild, live migration, etc, works 03:25:00 exactly 03:25:32 there are a lot of filters in nova, if we can't reuse them, then maybe we can create a new one 03:25:49 :D 03:26:01 using a filter means that instead of the boot request not being accepted it is accepted and fails 03:26:03 yeah i like this filter idea too 03:26:20 I would prefer it got stopped at the api level 03:26:27 true 03:26:29 me too 03:26:29 i can work on that guys, recently, i'm playing with filter to support windows images on our cloud 03:26:53 sorrison: that may introduce some logic change in nova 03:27:08 yeah exactly, that is the hard way 03:27:11 and i can imagine it would be hard to be merged, just my 2 cents 03:27:37 would be good to better understand how to add custom filters though 03:27:41 the filter thing might be a good first MVP step. at least it could help stop the bleeding of using old images 03:27:55 would be interested to see what you come up with flwang 03:27:57 I'd like a flag in nova to say only allow images to boot if they have image-meta X 03:28:16 but yeah a filter may be a good first step 03:28:33 yes - in the spirit of moving things along, flwang - can you put together a spec (or an etherpad mock up there of) and circulate it 03:28:49 sorrison: i will talk with nova guys to figure out a way, i will let you guys know when there is any progress 03:29:00 great 03:29:02 VW: no problem 03:29:04 tag it with [Large Deployement Team] on the ops list 03:29:18 yes, and let me know if you need help pushing on the Nova folks 03:29:27 I can harrass a few cores myself 03:29:28 VW: let me know how large it is? 03:29:29 ;) 03:29:59 VW: i know the nova PTL very well, we were colleagues when i worked for IBM 03:30:17 yeah, and the former one works with me 03:30:58 it’s not what you know, but who you know. 03:31:14 but, this brings me back to something that we sort of talked about in Austin, but we should probably revisit 03:31:23 VW: awesome 03:31:56 VW: i discussed with brain romatia and nikhil about how it's done in Rackspace 03:32:03 do we, as a team, need to make a combined statement on where we think that Nova (since most clouds center around compute) needs to be more knowledgable of resources in the other services than it is 03:32:14 but i can't remember anything ;D 03:32:24 oh yeah - forgot that Brian was back in the Glance game 03:33:06 VW: or translate it as: the other services need to provide more way/apis to be consumed by others 03:33:21 or just more attributes 03:34:00 well, one of the concepts we were talking about flwang was things like a nova delete not getting a response from Neutron and then leaves a ports/IPs stranded 03:34:22 Nova will say - it's a Neutron problem - we sent the request to the API 03:34:23 VW: that's bad 03:34:42 VW: imagine delete a tenant from keystone :) 03:34:45 try deleting a keystone project, and see what instances get left behind :) 03:35:02 you will feel better 03:35:08 sometimes it shows we work together :) 03:35:21 * jamespd looks at all of the owner-less routers :/ 03:36:00 I think my point is for all the work to keep the services seperate - as large deployers - we can admit that most of the resources in our clouds are tied to an instance 03:36:01 jamespd: you're the king of owner-less routers 03:36:03 VW: is there a case for what you’re talking about re: Nova knowing more about other resources that doesn’t fall into the delete/orphaning scenario? 03:36:30 yeah, I've got orphaned IPs I have to go clean up on a somewhat frequent basis 03:37:23 for me, the ideal scenario would be that Nova tracks resources used by a VM in other services 03:37:33 and doesn't complete the VM delete until they are cleaned up 03:37:41 VW: as for the image deprecation, should we totally forbid user boot images from old public images? 03:38:05 yes flwang 03:38:24 on the other, maybe I'm crazy 03:38:36 VW: haha, thanks for the confirmation 03:39:07 I just get the sense from sessions - and halway talk - that nova is looking to pay less and less direct attention to other things 03:39:10 and we may not want that 03:40:53 perhaps nova should just track the resources it consumes - i.e. things a VM is using like IPs - which could at least prevent the orphans 03:41:13 yeah xavpaice - something like that 03:41:16 but maybe we need to try to raise a bug whenever we see such things rather than just fixing them in the db 03:41:32 (I know I'm unlikely to do so because it's hard to replicate) 03:42:15 I know our nova folks fixed at least one recently. I'll follow up on if it is/will be upstream 03:43:21 anyway, I think I sidetracked things. We'll keep an eye on that stuff. This was a brief part of chatter in our Thursday session and wasn't sure if folks had thoughts 03:43:34 sounds like we have a start on the deprecated image stuff 03:43:41 flwang: what's your geography 03:43:51 VW: NZ 03:43:51 will you be able to make next months meeting or do we want to revisit in July 03:44:10 same company with xavpaice and jamespd 03:44:18 ah 03:44:39 UTC+12 fwiw 03:44:58 well, if we don't cover much on the mailing lists between now and then, lets circle back in two months here 03:45:22 unless you just have a burning desire to jump in a meeting at 2:00 am your time :) 03:46:23 heh 03:46:29 before we run out of time was there anything from the cells V2 sessions we wanted to circle back on? 03:46:43 they occured after our last LDT session 03:47:01 VW: i will stay at this channel a while and update the status about image deprecation 03:47:03 spec 03:47:38 excellent 03:47:45 we'll be sure to circulate, comment, etc 03:47:54 iirc the cells v2 discussion was pretty deep in the weeds and i don’t recall anything particularly piqueing my interest 03:48:03 I'm sure the CERN folks will want to chime in when they wake up 03:48:22 me either mdorman, but I know I wasn't the smartest person in the room 03:48:36 so, I thought I'd double check with you all 03:48:41 yeah, klindgren and sorrison_laptop were in there too 03:50:30 ok - so aside from the great migration fears we all share, no specific cells V2 stuff to worry about right now then? 03:50:49 yeah that’s the sense i have 03:50:55 cool 03:51:44 don't forget sorrison_laptop, if any of your sites build local to the cell via an api at that level, the new DB introduced in M will potentially break that 03:52:09 that's the first cells V2 thing we tripped over since we build bypass apis for testing new cells 03:52:38 well, we have 8 minutes left, so... 03:52:44 #other business 03:52:55 anything else we need to cover 03:53:13 argh - sorry - late here 03:53:19 #topic other business 03:53:26 i have nothing. thanks for leading VW 03:53:33 +1 all hail VW 03:53:38 haha 03:53:42 fifieldt: ! o/ 03:53:45 who let that guy in ^ 03:53:50 oh, snap. 03:54:07 I was just about to say that I joined fifieldt and others in a good chat about the next mid-cycle 03:54:29 a good group of engaged folks is looking to make that an open and solid process 03:54:39 shameless plug: https://wiki.openstack.org/wiki/Ops_Meetups_Team 03:54:50 stay tuned for details, but it's looking strongly like it will be North America in August 03:55:15 yeah iwas just lurking during that, i was pleased as well 03:55:38 also, one of the things fifieldt challenged me on was having this team really think about what our ideal summit schedule would be like in the future with the new structure 03:56:04 that pushes the bulk of design work to different gatherings and focuses more on dev and ops interaction 03:56:11 correct me if I got that wrong fifieldt 03:56:29 seems to me that the ops midcycle becomes a whole lot more important with the new summit schedule? 03:56:51 not for this meeting, but something we should start thinking on now because it will take us a bit to get it all fleshed out 03:56:55 probably so xavpaice 03:57:10 I think ttx put out a blog post today on the perceived changes, check it out 03:57:26 oh - really 03:57:27 cool 03:58:44 #link https://www.openstack.org/blog/2016/05/faq-evolving-the-openstack-design-summit/ 03:59:06 awesome! 03:59:06 plz2join the webinar and get your opinons across :) 03:59:51 we will 03:59:55 thanks for popping in, sir 04:00:06 cheers 04:00:09 * fifieldt lurks 04:01:10 well, folks - that's about all she wrote. let's see where the glance bits go. feel free to hit the mailing list about it, flawg. Otherwise, I'll return you to your regularly scheduled shenanigans and see you all next month 04:01:43 #endmeeting