03:01:31 <VW> #startmeeting Large Deployments Team May 2016 Meeting
03:01:32 <openstack> Meeting started Fri May 20 03:01:31 2016 UTC and is due to finish in 60 minutes.  The chair is VW. Information about MeetBot at http://wiki.debian.org/MeetBot.
03:01:33 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
03:01:35 <openstack> The meeting name has been set to 'large_deployments_team_may_2016_meeting'
03:02:28 <VW> quick roll call
03:02:37 <mdorman> o/
03:02:49 <sorrison_laptop> o/
03:02:50 <gfa_> 1
03:02:54 <gfa_> o/
03:02:59 <VW> 0/
03:03:38 <VW> hope everyone didn't have too many cloud fires to put out after the summit
03:04:11 <VW> #topic summit follow up
03:04:35 <VW> so, the one thing we didn't manage to boil things down to was a new item to pick up as a team and pursue with devs
03:05:14 <VW> I think that the retiring/removing/hidding old images was as close as we got
03:05:43 <VW> but, any particular items from the LDT sessions at the summit - or any other sessions - that we need to bring back up as a team?
03:06:11 <sorrison_laptop> I also think the whole delivery of images to compute nodes and then the snapshot process is something we could look into?
03:06:51 <mdorman> yeah originally back after tokyo we had talked about trying to pick up some glance stuff.  so that seems as good as any.
03:07:30 <VW> can you elaborate sorrison_laptop
03:07:39 <VW> you mean the passing of images through the glance APIs
03:07:41 <VW> or somethin gelse
03:07:44 <mdorman> i would also liek to see LDT as a source for info for shepherding people through the cells v1 —> v2 transition, although it’s probably a bit early to be doing much of anything real useful there.
03:08:01 <sorrison_laptop> yeah I like that idea mdorman
03:08:05 <VW> me too
03:08:26 <VW> sounds like the change for migrating is going to be O, right?
03:08:51 <mdorman> i don’t recall exactly, but that seems about right.  definitely should probably be a focus for barcelona
03:08:53 <sorrison_laptop> WRT glance I partly mean bypassing glance to get it out of the way
03:09:21 <VW> yeah - same thing
03:09:23 <mdorman> yeah i think these glance things would be a good shorter term thing to do
03:09:37 <VW> cool
03:10:03 <VW> unfortunately this is not a good time for belmiro.  He has looked at the deprecated image bit the most
03:11:38 <sorrison_laptop> basically need a flag on a glance image that nova can query on boot
03:12:03 <VW> yeah, I think so
03:12:14 <VW> was just looking back at the Glance stuff here - https://etherpad.openstack.org/p/AUS-ops-informal-meetup
03:12:29 <sorrison_laptop> yeah I went to that session
03:14:18 <mdorman> i think i missed that part.
03:14:26 <VW> looks like https://wiki.openstack.org/wiki/Glance-deactivate-image and https://review.openstack.org/#/c/132717/ are most appropriate
03:15:28 <flwang> VW: with the deactivate  image, i think nova rebuild will still fail
03:15:41 <VW> yeah - I think that's the curx of the issue
03:15:54 <flwang> the hard part to deprecate an image in cloud is there are still some instances depend on that
03:15:54 <VW> we want to stop the new, but have tenants out there who may need to rebuild
03:16:05 <VW> exactly
03:16:48 <flwang> i'm responsible for our cloud's images maintaining
03:17:21 <flwang> and i would say the old images maintenance is frustrating
03:17:38 <VW> agreed
03:17:53 <VW> with a public cloud to look after, I'm not sure they ever dia :)
03:18:11 <sorrison_laptop> yeah unfortunately reading the commit message it doesn't look like it will do what we want
03:18:40 <sorrison_laptop> almost want a "bootable" flag for an image
03:18:58 <sorrison_laptop> that's really the only thing we want to disable
03:19:06 <VW> agreed
03:19:12 <sorrison_laptop> which means nova and glance interaction
03:20:10 <sorrison_laptop> so I guess we need to talk to some glance people about whether we can redefine what deactivated is or we need a new action
03:20:29 <VW> yes, I think so
03:20:42 <VW> but do we need to bring Nova to the conversation too
03:21:22 <sorrison_laptop> yeah as they would need to check that value on a boot api request?
03:21:55 <mdorman> really does seem this is mostly on the nova side.  glace wise you could simply put some ‘bootable’ metadata key on the image, no changes needed to glance.
03:22:29 <gfa_> maybe a nova filter can do the trick
03:22:29 <sorrison_laptop> yeah true
03:22:31 <flwang> sorrison_laptop: that's a good point, re the 'bootable' flag
03:23:00 <flwang> VW: I'm a glance core actually :D
03:23:11 <VW> WIN!
03:23:28 <mdorman> would be good to discuss with nova, get a feel for how they would envision doing this.  unsure if this particular piece of the issue has ever been raise with them.
03:24:09 <sorrison_laptop> a filter is an interesting idea
03:24:17 <VW> yeah flwang - I haven't taken off my Rackspace hat and hit Hemmanth up with anything re: LDT
03:24:18 <VW> yet :)
03:24:30 <VW> yes, a filter is
03:24:30 <sorrison_laptop> glance has the functionally to support admin metadata key/values
03:24:31 <flwang> VW: i think the tricky part is we don't want to allow to create new instance based on the image, but we should keep all the other things, like rebuild, live migration, etc, works
03:25:00 <VW> exactly
03:25:32 <flwang> there are a lot of filters in nova, if we can't reuse them, then maybe we can create a new one
03:25:49 <gfa_> :D
03:26:01 <sorrison_laptop> using a filter means that instead of the boot request not being accepted it is accepted and fails
03:26:03 <mdorman> yeah i like this filter idea too
03:26:20 <sorrison_laptop> I would prefer it got stopped at the api level
03:26:27 <mdorman> true
03:26:29 <VW> me too
03:26:29 <flwang> i can work on that guys, recently, i'm playing with filter to support windows images on our cloud
03:26:53 <flwang> sorrison: that may introduce some logic change in nova
03:27:08 <sorrison_laptop> yeah exactly, that is the hard way
03:27:11 <flwang> and i can imagine it would be hard to be merged, just my 2 cents
03:27:37 <xavpaice> would be good to better understand how to add custom filters though
03:27:41 <mdorman> the filter thing might be a good first MVP step.  at least it could help stop the bleeding of using old images
03:27:55 <mdorman> would be interested to see what you come up with flwang
03:27:57 <sorrison_laptop> I'd like a flag in nova to say only allow images to boot if they have image-meta X
03:28:16 <sorrison_laptop> but yeah a filter may be a good first step
03:28:33 <VW> yes - in the spirit of moving things along, flwang - can you put together a spec (or an etherpad mock up there of) and circulate it
03:28:49 <flwang> sorrison: i will talk with nova guys to figure out a way, i will let you guys know when there is any progress
03:29:00 <sorrison_laptop> great
03:29:02 <flwang> VW: no problem
03:29:04 <VW> tag it with [Large Deployement Team] on the ops list
03:29:18 <VW> yes, and let me know if you need help pushing on the Nova folks
03:29:27 <VW> I can harrass a few cores myself
03:29:28 <flwang> VW: let me know how large it is?
03:29:29 <flwang> ;)
03:29:59 <flwang> VW: i know the nova PTL very well, we were colleagues when i worked for IBM
03:30:17 <VW> yeah, and the former one works with me
03:30:58 <mdorman> it’s not what you know, but who you know.
03:31:14 <VW> but, this brings me back to something that we sort of talked about in Austin, but we should probably revisit
03:31:23 <flwang> VW: awesome
03:31:56 <flwang> VW: i discussed with brain romatia and nikhil about how it's done in Rackspace
03:32:03 <VW> do we, as a team, need to make a combined statement on where we think that Nova (since most clouds center around compute) needs to be more knowledgable of resources in the other services than it is
03:32:14 <flwang> but i can't remember anything ;D
03:32:24 <VW> oh yeah - forgot that Brian was back in the Glance game
03:33:06 <flwang> VW: or translate it as: the other services need to provide more way/apis to be consumed by others
03:33:21 <flwang> or just more attributes
03:34:00 <VW> well, one of the concepts we were talking about flwang was things like a nova delete not getting a response from Neutron and then leaves a ports/IPs stranded
03:34:22 <VW> Nova will say - it's a Neutron problem - we sent the request to the API
03:34:23 <flwang> VW: that's bad
03:34:42 <flwang> VW: imagine delete a tenant from keystone :)
03:34:45 <xavpaice> try deleting a keystone project, and see what instances get left behind :)
03:35:02 <flwang> you will feel better
03:35:08 <xavpaice> sometimes it shows we work together :)
03:35:21 * jamespd looks at all of the owner-less routers :/
03:36:00 <VW> I think my point is for all the work to keep the services seperate - as large deployers - we can admit that most of the resources in our clouds are tied to an instance
03:36:01 <flwang> jamespd: you're the king of owner-less routers
03:36:03 <mdorman> VW:  is there a case for what you’re talking about re: Nova knowing more about other resources that doesn’t fall into the delete/orphaning scenario?
03:36:30 <VW> yeah, I've got orphaned IPs I have to go clean up on a somewhat frequent basis
03:37:23 <VW> for me, the ideal scenario would be that Nova tracks resources used by a VM in other services
03:37:33 <VW> and doesn't complete the VM delete until they are cleaned up
03:37:41 <flwang> VW: as for the image deprecation, should we totally forbid user boot images from old public images?
03:38:05 <VW> yes flwang
03:38:24 <VW> on the other, maybe I'm crazy
03:38:36 <flwang> VW: haha, thanks for the confirmation
03:39:07 <VW> I just get the sense from sessions - and halway talk - that nova is looking to pay less and less direct attention to other things
03:39:10 <VW> and we may not want that
03:40:53 <xavpaice> perhaps nova should just track the resources it consumes - i.e. things a VM is using like IPs - which could at least prevent the orphans
03:41:13 <VW> yeah xavpaice - something like that
03:41:16 <xavpaice> but maybe we need to try to raise a bug whenever we see such things rather than just fixing them in the db
03:41:32 <xavpaice> (I know I'm unlikely to do so because it's hard to replicate)
03:42:15 <VW> I know our nova folks fixed at least one recently.  I'll follow up on if it is/will be upstream
03:43:21 <VW> anyway, I think I sidetracked things.  We'll keep an eye on that stuff.  This was a brief part of chatter in our Thursday session and wasn't sure if folks had thoughts
03:43:34 <VW> sounds like we have a start on the deprecated image stuff
03:43:41 <VW> flwang: what's your geography
03:43:51 <flwang> VW: NZ
03:43:51 <VW> will you be able to make next months meeting or do we want to revisit in July
03:44:10 <flwang> same company with xavpaice and jamespd
03:44:18 <VW> ah
03:44:39 <xavpaice> UTC+12 fwiw
03:44:58 <VW> well, if we don't cover much on the mailing lists between now and then, lets circle back in two months here
03:45:22 <VW> unless you just have a burning desire to jump in a meeting at 2:00 am your time :)
03:46:23 <jamespd> heh
03:46:29 <VW> before we run out of time was there anything from the cells V2 sessions we wanted to circle back on?
03:46:43 <VW> they occured after our last LDT session
03:47:01 <flwang> VW: i will stay at this channel a while and update the status about image deprecation
03:47:03 <flwang> spec
03:47:38 <VW> excellent
03:47:45 <VW> we'll be sure to circulate, comment, etc
03:47:54 <mdorman> iirc the cells v2 discussion was pretty deep in the weeds and i don’t recall anything particularly piqueing my interest
03:48:03 <VW> I'm sure the CERN folks will want to chime in when they wake up
03:48:22 <VW> me either mdorman, but I know I wasn't the smartest person in the room
03:48:36 <VW> so, I thought I'd double check with you all
03:48:41 <mdorman> yeah, klindgren and sorrison_laptop were in there too
03:50:30 <VW> ok - so aside from the great migration fears we all share, no specific cells V2 stuff to worry about right now then?
03:50:49 <mdorman> yeah that’s the sense i have
03:50:55 <VW> cool
03:51:44 <VW> don't forget sorrison_laptop, if any of your sites build local to the cell via an api at that level, the new DB introduced in M will potentially break that
03:52:09 <VW> that's the first cells V2 thing we tripped over since we build bypass apis for testing new cells
03:52:38 <VW> well, we have 8 minutes left, so...
03:52:44 <VW> #other business
03:52:55 <VW> anything else we need to cover
03:53:13 <VW> argh - sorry - late here
03:53:19 <VW> #topic other business
03:53:26 <mdorman> i have nothing.  thanks for leading VW
03:53:33 <fifieldt> +1 all hail VW
03:53:38 <mdorman> haha
03:53:42 <mdorman> fifieldt:  !   o/
03:53:45 <VW> who let that guy in ^
03:53:50 <jamespd> oh, snap.
03:54:07 <VW> I was just about to say that I joined fifieldt and others in a good chat about the next mid-cycle
03:54:29 <VW> a good group of engaged folks is looking to make that an open and solid process
03:54:39 <fifieldt> shameless plug: https://wiki.openstack.org/wiki/Ops_Meetups_Team
03:54:50 <VW> stay tuned for details, but it's looking strongly like it will be North America in August
03:55:15 <mdorman> yeah iwas just lurking during that, i was pleased as well
03:55:38 <VW> also, one of the things fifieldt challenged me on was having this team really think about what our ideal summit schedule would be like in the future with the new structure
03:56:04 <VW> that pushes the bulk of design work to different gatherings and focuses more on dev and ops interaction
03:56:11 <VW> correct me if I got that wrong fifieldt
03:56:29 <xavpaice> seems to me that the ops midcycle becomes a whole lot more important with the new summit schedule?
03:56:51 <VW> not for this meeting, but something we should start thinking on now because it will take us a bit to get it all fleshed out
03:56:55 <VW> probably so xavpaice
03:57:10 <fifieldt> I think ttx put out a blog post today on the perceived changes, check it out
03:57:26 <VW> oh - really
03:57:27 <VW> cool
03:58:44 <fifieldt> #link https://www.openstack.org/blog/2016/05/faq-evolving-the-openstack-design-summit/
03:59:06 <VW> awesome!
03:59:06 <fifieldt> plz2join the webinar and get your opinons across :)
03:59:51 <VW> we will
03:59:55 <VW> thanks for popping in, sir
04:00:06 <fifieldt> cheers
04:00:09 * fifieldt lurks
04:01:10 <VW> well, folks - that's about all she wrote.  let's see where the glance bits go.  feel free to hit the mailing list about it, flawg.  Otherwise, I'll return you to your regularly scheduled shenanigans and see  you all next month
04:01:43 <VW> #endmeeting