19:00:02 <j^2> #startmeeting operators_ops_tools_monitoring
19:00:03 <openstack> Meeting started Wed Dec  2 19:00:02 2015 UTC and is due to finish in 60 minutes.  The chair is j^2. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
19:00:07 <openstack> The meeting name has been set to 'operators_ops_tools_monitoring'
19:00:20 <j^2> #topic rollcall
19:00:25 <j^2> hey everyone!
19:00:33 <xavpaice> good morning
19:00:42 <j^2> i'll give 5-6 mins to allow everyone to filter in
19:01:02 <serverascode> o/
19:02:05 <klindgren> hi
19:02:10 <j^2> friendly ping: balajin, klindgren, mdorman
19:02:12 <j^2> :D
19:02:40 <j^2> #link https://etherpad.openstack.org/p/osops-irc-meeting-20151202
19:02:52 <j^2> that's the agenda ^^ if anyone wants to put any last min topics
19:03:00 <j^2> friendly ping: mriedem
19:03:17 <balajin> hi
19:03:30 <j^2> cool, starting in 2 mins
19:04:20 <klindgren> Does anyone have a tool for moving a vm and its associated stuff between projects?
19:04:33 <j^2> rsync? ;)
19:04:49 <xavpaice> common ask from our sysadmins
19:05:09 <balajin> mostly needs backend db updates
19:05:18 <j^2> #topic Previous Business
19:05:20 <balajin> its an ask that we have had
19:05:36 <j^2> Ok, so I had the only action items from last meeting
19:05:46 <j^2> i got the agendas and template on the wiki
19:05:57 <j^2> #link https://wiki.openstack.org/wiki/Osops
19:06:05 <j^2> #link https://wiki.openstack.org/wiki/Osops/Osops-agenda
19:06:08 <Bjoern_> Thanks J^2
19:06:16 <j^2> any objections to to these?
19:06:57 <j^2> i'm just cargo culting what i did in openstack-chef it worked well, but i'm willing to change anything if anyone has any better suggestions
19:06:58 <balajin> #agreed
19:07:12 <xavpaice> great stuff
19:07:30 <j^2> i've already put the agenda for next meeting up too
19:07:39 <j^2> so we can push things if we need to
19:07:45 <j^2> but i don't we'll need it today
19:08:01 <j^2> i'll give 1-2 mins for any other thoughts or suggestions
19:09:23 <klindgren> seems good to me
19:09:41 <j^2> awesome, the second action item was mine too
19:09:44 <j^2> j^2 to figure out how to use zuul to validate against the pep8 (python|bash) and rubocop linters
19:10:00 <j^2> yeah i totally didn't work on this, i got caught up in the board election prep and all
19:10:15 <j^2> i'll take this again for next meeting, unless someone else wants it
19:11:10 <j^2> ok, if you want to discuss it you can ping me out of band and we can talk about it
19:11:23 <j^2> #topic new business
19:11:39 <j^2> #action j^2 to figure out how to use zuul to validate against the pep8 (python|bash) and rubocop linters
19:11:53 <j^2> mriedem): The nova dev team would appreciate operator input on the problem and questions posed in this ML thread related to fixing the nova-manage db archive_deleted_rows command: http://lists.openstack.org/pipermail/openstack-dev/2015-November/079778.html
19:12:14 <j^2> is mriedem here?
19:12:32 <j^2> #link: http://lists.openstack.org/pipermail/openstack-dev/2015-November/079778.html
19:13:08 <klindgren> In general I like it.  The only thing i was unsure of when they say when archiving soft_deleted instances also archive off instance actions.  What does archiving mean exactly?
19:13:13 <klindgren> move it to the shadow table?
19:13:16 <klindgren> purge from db?
19:13:35 <j^2> it sounds like they are open to suggestions
19:13:59 <klindgren> The plan right now (at least agreed to between myself and sdague) is not
19:13:59 <klindgren> to soft delete instance actions, but to archive and hard-delete them
19:14:00 <klindgren> when archiving instances.
19:14:16 <klindgren> just read that...
19:14:22 <klindgren> so I am ok with that
19:14:28 <xavpaice> the 'who deleted my instance' scenario seems like something people would want
19:15:04 <klindgren> the issue is - only admin's can query for deleted instances - then you can give them the old uuid and they can see the actions
19:15:04 <j^2> xavpaice: i agreed, auditing is always something useful
19:15:13 * mriedem joins late
19:15:19 <j^2> mriedem: hi!
19:15:25 <j^2> we're just discussing your post
19:15:50 <klindgren> but yes - I agree - its useful.  But I know most of our users, don't track their vm uuid's
19:15:51 <mriedem> i don't even remember what that was
19:15:53 <xavpaice> I think the inability of the api to read instance-actions for a deleted instance is probably a slightly different issue to the one mriedem is asking about though?
19:16:03 <j^2> mriedem: http://lists.openstack.org/pipermail/openstack-dev/2015-November/079778.html
19:16:23 <mriedem> ok,
19:16:28 <balajin> klindgren: yeah
19:16:39 <balajin> usually the users so i had a vm which was this hostname / ip and it is missing
19:16:46 <mriedem> so what we more or less agreed to in the -dev list was when archiving instances, we also archive and hard-delete the instance actions
19:16:47 <balajin> and it is the ops who track it down
19:17:32 <mriedem> this http://lists.openstack.org/pipermail/openstack-dev/2015-November/080033.html
19:18:14 <j^2> #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/080033.html
19:18:19 <mriedem> we have to do this for bw_usage_cache also since it has a reference to instance uuid but no foreign key, and those aren't soft deleted (but only relevant if you use xen)
19:18:38 <mriedem> there is a patch up to add a fkey to bw_usage_cache though
19:19:18 <mriedem> as for reading instance actions for deleted instances, that spec was approved https://review.openstack.org/#/c/248288/
19:19:45 <j^2> #link https://review.openstack.org/#/c/248288/
19:19:47 <mriedem> so are people ok with nova archiving and hard-deleting instance actions when it archives the instances?
19:20:06 <mriedem> they aren't really usable after you've archived the instances
19:20:15 <xavpaice> seems reasonable to me
19:20:18 <mriedem> and they are breaking nova-manage db archive_deleted_rows right now
19:20:21 <j^2> mriedem: i havent clicked on any of the links you've put, but the tl;dr: where do you archive the data?
19:20:27 <mriedem> shadow tables
19:20:33 <j^2> klindgren: ^^
19:20:34 <mriedem> which is what nova-manage db archive_deleted_rows does
19:20:45 <mriedem> but is broken on the instance_actions fkey right now
19:20:57 <klindgren> kk
19:21:02 <mriedem> tl;dr is i'm trying to fix that so ops don't need their own script to archive :)
19:21:09 <j^2> mriedem: rock on
19:21:22 <balajin> so what happens to the shadow tables and when do they get cleaned up?
19:21:24 <mriedem> i think it was mfisch that shamed us on that one :P
19:21:46 <mriedem> there is no in-tree thing in nova to purge shadow tables
19:22:00 <xavpaice> so that's a custom sql anyway?
19:22:07 <mriedem> for now yeah
19:22:15 <mriedem> there is a bp for mitaka to add a purge command to nova-manage
19:22:24 <xavpaice> well, at least it's a bunch safer than the main tables
19:22:34 <mriedem> purge would hard-delete instances and their related tables
19:22:37 <mriedem> not archive
19:22:38 <balajin> okay, i thgouth nova-manage was going away in N
19:22:50 <mriedem> nova-manage is in it for the long haul
19:23:00 <mriedem> nova-manage is a hack to the nova db,
19:23:15 <mriedem> for things that nova-manage isn't needed for, where the api is sufficient, those things are removed
19:23:29 <klindgren> to -> openstackcleint - right?
19:23:34 <klindgren> client**
19:23:35 <mriedem> or novaclient
19:23:37 <mriedem> or rest api
19:23:40 <balajin> mriedem: got it, so now i recall nova service-list comment
19:23:42 <mriedem> or sdk, whatever
19:23:44 <mriedem> yeah
19:23:59 <balajin> sorry one more question on the topic
19:24:13 <mriedem> this is the purge command change btw https://review.openstack.org/#/c/203751/
19:24:17 <balajin> the archival of instances is it a time based policy or a manual setting or always on?
19:24:35 <balajin> s/manual setting/manual run of the command/
19:24:38 <mriedem> manual
19:24:53 <mriedem> so you could run that on a cron if you wanted
19:24:57 <mriedem> same with purging shadow tables
19:25:01 <balajin> mriedem: thanks
19:25:12 <balajin> makes sense
19:25:17 <mriedem> archive is also a big whacked in other ways
19:25:24 <mriedem> the max_rows thing is sort of terrible imo
19:25:29 <balajin> and i assume it supports a date parameter so i would do this for things older than n days
19:25:31 <mriedem> *bit
19:25:45 <mriedem> the new purge command supports an age parameter
19:25:52 <mriedem> archive doesn't, but probably should
19:25:54 * balajin kicks himself for not reading the ML / spec
19:26:04 <mriedem> archive has this --max-rows option, which is not how you'd expect it to work really
19:26:09 <j^2> balajin: ;) you arent the only one
19:26:19 <balajin> :(
19:26:25 <mriedem> when i run archive, i basically pass --max-rows 3000 just to make sure i'm not missing something
19:26:40 <mriedem> if you pass like --max-rows 10, and you have 10 rows in the first table it processes, it's done
19:26:47 <mriedem> not 10 rows from *all* tables
19:26:58 <balajin> yeah i feel bad because i asked for the agenda, clicked on the link and then got side tracked
19:27:12 <j^2> ha!
19:27:17 <j^2> no worries, this discussion is useful
19:27:36 <balajin> so most folks are fine with this, how about going on with this
19:27:46 <j^2> sounds good.
19:27:47 <balajin> and if people have comments, do it directly on the spec / ml?
19:27:59 <mriedem> balajin: -nova channel is also good
19:28:01 <balajin> mriedem: xavpaice: klindgren: ?
19:28:07 <klindgren> I ma good with it
19:28:08 <balajin> sure
19:28:20 <j^2> mriedem: thanks for your time and joining us here :D
19:28:40 <mriedem> np
19:28:43 <xavpaice> :)
19:28:58 <mriedem> btw,
19:29:00 <mriedem> as a nova dev,
19:29:12 <mriedem> please continue to beat us up on things that ops are having to do themselves out of the nova tree for things like this
19:29:15 <mriedem> and we need to fix
19:29:22 <mriedem> it's important to know these things for setting our priorities
19:29:25 <klindgren> mriedem, moving a vm between projects
19:29:32 <mriedem> otherwise we go off and waste time reviewing vendor x's shiny new toy
19:29:33 <j^2> klindgren: well played sir
19:29:43 <mriedem> klindgren: there was a spec for that in kilo
19:29:51 <mriedem> it wasn't fully hashed out to say the least
19:30:04 <mriedem> i could probably find it at some point (later)
19:30:24 <mriedem> how much are ops keyed into the product working group?
19:30:36 <j^2> mriedem: not too much, tbh
19:30:40 <mriedem> neither are devs
19:30:45 <klindgren> yea - I mean the issues is you also need: glance/neutron/cinder (others as well)
19:30:46 <xavpaice> ouch
19:30:48 <mriedem> i feel like it's project / product managers talking to each other
19:30:52 <j^2> mriedem: that group is run by vendors iirc too
19:30:59 <mriedem> j^2: yup
19:31:11 <mriedem> klindgren: same is true for auto delete of resources once you delete a project/user
19:31:25 <j^2> mriedem: or at least that's how i felt when i was in the UC conversations it Tokyo
19:31:40 <xavpaice> we do have a script for that - I should get the author to put it up to osops
19:31:40 <klindgren> UC?
19:31:43 <mriedem> j^2: that's the impression i got from reading the list of things in the product work group
19:31:52 <mriedem> user committee?
19:31:55 <klindgren> xavpaice, really??!?!
19:32:09 <j^2> klindgren: user committee, there's an intersection between the product wg and UC
19:32:27 <j^2> xavpaice: YES
19:32:28 <klindgren> I get asked pretty frequently to move stuff between projects.  A script would be pretty nice
19:32:38 <xavpaice> klindgren: add and remove tenant scripts, yeah - will ask to get it shared
19:32:51 <xavpaice> not moving stuff between though
19:32:57 <mriedem> you could give me an action for digging up that old spec
19:33:07 * mriedem mriedem to find old nova spec on moving vm's between projects
19:33:09 <mriedem> oops
19:33:14 <mriedem> #action mriedem to find old nova spec on moving vm's between projects
19:33:21 <j^2> #action mriedem to find old nova spec on moving vm's between projects
19:33:30 <j^2> i think only chairs can give actions iirc
19:33:37 <j^2> otherwise you have it doublely :P
19:33:47 <klindgren> I have been tempted to play aroudn in the nova db.  But I am sure that will go horribly on the next db upgrade attempt
19:34:30 <j^2> heh
19:34:57 <j^2> cool, ok, just a time check, shall i open this up for "Open Discussion" or do we want to say on these topics?
19:35:44 <j^2> BjoernT: do you have any thoughts?
19:36:01 <xavpaice> the author of our tenant create/delete scripts is good to share - did you want to put an action for me to do that?
19:36:24 <mriedem> cha ching https://review.openstack.org/#/c/105367/
19:36:25 <BjoernT> sorry, i got hijacked in a meeting, i'll be passive. Don't wait on me
19:36:42 <j^2> #action xavpaice to post the tenant scripts to osops
19:36:43 <mriedem> ^ is the transfer of ownership spec
19:36:55 <j^2> mriedem: rock on
19:37:00 <j^2> #link mriedem to find old nova spec on moving vm's between projects
19:37:16 <xavpaice> that was fast!
19:37:22 <j^2> #link https://review.openstack.org/#/c/105367/
19:37:30 <mriedem> knowing the gerrit query language is helpful
19:37:51 <balajin> should hte action to be to propose that spec for M?
19:38:03 <mriedem> you have to find someone to propose it
19:38:06 <mriedem> and it won't make M
19:38:06 <balajin> that spec shows as abondoned and also suggest to move to oslo
19:38:09 <mriedem> spec freeze is tomorrow
19:38:16 <balajin> ha okay
19:38:30 <mriedem> but, if you have working code, that goes a long way
19:38:33 <mriedem> for N
19:38:53 <mriedem> but yeah it's a cross-project spec for sure
19:39:01 <mriedem> thingee said he was herding those cats now for cross project specs
19:39:18 <j^2> impressive, cross-project specs must be crazy hard
19:39:49 <mriedem> invest product work group resources in that and maybe they'd be helpful
19:40:00 <j^2> makes sense
19:41:40 <j^2> #topic Open Floor
19:41:56 <j^2> cool, is there anything anyone would like to bring up infront of the group?
19:42:15 <j^2> i have one, but i'd like to open it for others first
19:43:44 <j^2> bueller?
19:44:18 <j^2> ok, cool
19:44:35 <j^2> I'd like to just mention that I've accepted my nomination for the OpenStack board
19:44:37 <j^2> #link https://www.openstack.org/community/members/profile/19802
19:44:49 <j^2> I'd like to ask for ya'lls support too
19:45:29 <j^2> and if anyone has any ideas on how i can win this seat i'm all ears
19:46:09 <j^2> other than than, i think we're good, we have a couple action items and somethings to follow up on.
19:46:17 <mriedem> are there any other ops people on the board nomination?
19:46:24 <mriedem> if not, that's probably your selling point
19:46:30 <j^2> mriedem: from what it seems i'm the only one
19:46:36 <mriedem> that and wild promises
19:46:42 <mriedem> channel your inner trump
19:46:50 <klindgren> haha
19:46:55 <j^2> mriedem: kissn' babies, make OPS GREAT AGAIN!
19:47:37 <ctina> i have a stupid question...how often are these meetings held?
19:47:37 * xavpaice gets a badge
19:48:09 <j^2> ctina: odd weeks
19:48:14 <xavpaice> oh, that's not on the wiki page
19:48:20 <klindgren> it is
19:48:21 <j^2> xavpaice: it is
19:48:29 <klindgren> We have an official meeting time at Every two weeks (on odd weeks) on Wednesday at 1900 UTC in #openstack-meeting-4, our agendas are posted here.
19:48:53 <xavpaice> oh yeah! d'ok...
19:49:03 <balajin> how many are around next week?
19:49:04 <balajin> i am not
19:49:09 <balajin> so i will see you in jan
19:49:09 <j^2> ctina: and that's not a stupid question at all, we havent sold ourself very well, we need to announce it better
19:49:15 <klindgren> I am
19:49:16 <j^2> balajin: sounds good, :D
19:49:20 <balajin> assume there would be none on 31st
19:49:25 <j^2> balajin: and you mean in two weeks
19:49:39 <balajin> j^2: yeah, thanks for the correction
19:49:41 <klindgren> j^2 - you can get my vote if you can promise to hae people start focusing on the bascis :-)
19:49:43 <balajin> i am out on 16th
19:50:04 <klindgren> I will settle for deleting tenants doing stuff other than leaving a mess for operators, moving stuff between tenants
19:50:14 <j^2> klindgren: to quote what i think the top proirity is: Understanding the scope of the practical use cases for OpenStack. The Board seems to be looking at the theoretical use cases not what people are actually building.
19:51:31 <j^2> cool, anything else? otherwise i'll close the meeting
19:51:42 <j^2> ill give 2 mins for anyone to speak up
19:52:33 <j^2> thanks everyone!
19:52:50 <xavpaice> thank _you_
19:53:10 <j^2> #endmeeting