21:01:19 <russellb> #startmeeting nova
21:01:20 <openstack> Meeting started Thu Apr 10 21:01:19 2014 UTC and is due to finish in 60 minutes.  The chair is russellb. Information about MeetBot at http://wiki.debian.org/MeetBot.
21:01:21 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
21:01:23 <openstack> The meeting name has been set to 'nova'
21:01:32 <russellb> hi! anyone around for the nova meeting today?
21:01:35 <mrodden1> heya
21:01:38 <dansmith> yar
21:01:41 <dripton> hi
21:01:43 <leifz> I'm around.
21:01:49 <alaski> o/
21:01:55 <russellb> #link https://wiki.openstack.org/wiki/Meetings/Nova
21:01:55 <melwitt> o/
21:02:04 <russellb> cool, hi all :)
21:02:08 <russellb> #topic icehouse-rc
21:02:15 <russellb> we put out RC2 for Nova yesterday
21:02:22 <russellb> #link https://launchpad.net/nova/+milestone/icehouse-rc2
21:02:24 <dansmith> woo!
21:02:27 <russellb> right now we have no plans for an RC3
21:02:31 <russellb> unless the sky falls
21:02:58 <russellb> basically, would take a serious show stopped at this point
21:03:05 <mriedem> hi
21:03:13 <russellb> but please let me know if anything comes up that may qualify
21:03:32 <russellb> otherwise, feel free to tag bugs with icehouse-backport-potential
21:03:40 <russellb> and fixes can be backported and go out in a stable update
21:03:46 <russellb> any comments/questions on icehouse?
21:04:17 <russellb> #topic bugs
21:04:26 <russellb> any bugs we should discuss today?
21:04:36 <russellb> #link https://bugs.launchpad.net/nova
21:04:51 <russellb> 0 open critical bugs it seems, so that's a plus.
21:05:16 <russellb> tjones: anything from the bug meeting?
21:05:37 <tjones> russellb: no nothing looked critical.  im just taking another quick look
21:05:40 <tjones> at untagged
21:05:43 <russellb> OK
21:06:06 <mriedem> sort of related, any minesweeper news?
21:06:18 <tjones> nothing jumps out as a rc blocker.
21:06:33 <tjones> sooo close....
21:06:43 <russellb> k, sounds good!
21:06:45 <tjones> not fixed thought
21:07:07 <tjones> i'll give you a better update in 5 minutes
21:07:13 <russellb> k, will move on for now then
21:07:17 <russellb> #topic blueprints
21:07:24 <russellb> blueprint review for juno in full swing
21:07:50 <russellb> please join in the review fun in the nova-specs repo
21:07:57 <russellb> any blueprints folks want to discuss in meeting?
21:08:13 <mriedem> no one wants to talk about anything today
21:08:32 <russellb> it's the "i'm tired from icehouse, leave me alone" phase
21:08:33 <russellb> that's OK.
21:08:38 <dansmith> yeah :)
21:08:44 <dansmith> mriedem: don't poke the hornet's nest :)
21:08:54 <mriedem> oh i'll poke alright
21:08:56 <russellb> #link open discussion
21:09:01 <mriedem> i had an agenda item
21:09:02 <russellb> mriedem: you had something for this at least
21:09:08 <mriedem> deprecating nova-manage subcommands
21:09:10 <mriedem> how do we do that
21:09:11 <mriedem> ?
21:09:16 <russellb> set it on fire
21:09:17 <russellb> i mean ..
21:09:24 <mriedem> put something in the command for a release and send a note to the operators list?
21:09:29 <russellb> emit a deprecated message when used, ensure it's covered in release notes
21:09:33 <russellb> and then can remove the next release
21:09:35 <mriedem> k
21:09:38 <russellb> IMO
21:09:41 <dansmith> +1
21:09:42 <mriedem> my plan was subcommand by subcommand
21:09:43 <mriedem> not all
21:09:46 <russellb> agreed
21:09:52 <mrodden> comstud had a good point about wanting to configure before bringing up the API
21:09:59 <russellb> we can go ahead and create the ReleaseNotes/Juno wiki page
21:10:05 <russellb> keep notes along the way
21:10:15 <russellb> helps with making sure we don't forget stuff like this
21:10:17 <dansmith> mrodden: I think we've mostly been going for "do it through the API unless it's impossible"
21:10:19 <mriedem> mrodden: but you don't need to configure flavors before the API comes up do you?
21:10:33 <mrodden> if you have no pre existing ones
21:10:37 <dansmith> mrodden: cells being something you can't configure after the API
21:10:53 <mriedem> mrodden: the flavors subcommand is a good example, completely covered in novaclient
21:10:58 <mrodden> yeah
21:11:02 <russellb> bringing up an API endpoint can be different from bringing up a public API endpoint
21:11:02 <mriedem> i wouldn't be touching db, service type subcommands right now
21:11:03 <russellb> so meh
21:11:12 <mrodden> yeah
21:11:12 <russellb> *admin* API endpoint i mean
21:11:26 <mrodden> API =/= public API i guess
21:11:30 <dansmith> yeah
21:11:30 <russellb> right
21:12:09 <mrodden> should be doing all the setup with config managment anyways
21:12:19 <mrodden> so it *should* be configured already
21:12:27 <mriedem> mrodden: unless your cookbooks are 3 years old :P
21:12:38 <mrodden> sounds like a personal problem... err wait
21:13:31 <russellb> so, there's some elections happening.
21:13:35 <russellb> you should vote.
21:13:38 <russellb> ends tomorrow.
21:14:03 <russellb> anything else for today?
21:14:18 <tjones> rc minesweeper - they are working on the last step to getting it back up as we speak.  fingers crossed we are hours away from it working again
21:14:22 <tjones> s/rc/re
21:14:28 <mriedem> cool
21:14:34 <mikal> There's a few third party CIs in a bad state to be honest
21:14:41 <mikal> I don't think minesweeper is alone
21:14:43 <russellb> tjones: curious what the root cause of the downtime has been?
21:14:54 <russellb> tjones: anything we should be working on in nova?  or something environment/planning specific?
21:14:59 <tjones> the upgrade from grizzly to havana.  we have a few bugs from that
21:15:08 <mriedem> mikal: yeah, i just know of a vmware fix with several +2s lined up waiting for a CI vote
21:15:15 <russellb> tjones: vmware bugs, or general nova bugs?
21:15:50 <russellb> fwiw, despite some rough spots, I'm still very happy with the 3rd party CI progress in Icehouse
21:15:51 <tjones> i can give more details on the ML - i think they are vmware bugs.  but i want to be sure.  there are 2 bugs waiting on minesweeper and they are going to do a manual run (but have been focused on getting the )(#%&*)($ thing up)
21:15:55 <russellb> it's a long way from where we were 6 months ago
21:15:59 <russellb> this stuff is really hard
21:16:12 <mikal> russellb: has anyone chased the hyper-v ci guys recently?
21:16:17 <russellb> mikal: yes
21:16:19 <mriedem> mikal: yesterday
21:16:24 <mikal> johnthetubaguy and I have pinged the Xen peoples
21:16:33 <mriedem> they were hitting zuul merge conflict problems
21:16:56 <mikal> Are the hyper-v people using zuul?
21:17:03 <mriedem> mikal: apparently
21:17:23 <mikal> Ok, that's actually harder than it looks. The zuul git repo used for merging has some confusing behaviours.
21:17:25 <dansmith> the last time this particular problem came up, they were corrupting their git somehow
21:17:33 <dansmith> which caused the merge messages, IIRC
21:17:48 <mikal> We had issues with the git repo zuul uses being in a weird state too in about January
21:17:57 <mikal> It turns out the infra team has a script they can use to work around
21:18:09 <mikal> That repo isn't meant to be clonable, if that's what they're trying to do
21:18:18 <mriedem> i know at least at one time alexpilotti was working with the infra guys on the zuul issues
21:18:31 <mriedem> so assuming hyperv team has at least the same workaround
21:18:34 <mriedem> but not certain
21:18:38 <mikal> Ok
21:18:46 <mikal> Who were you talking to about it and I'll reach out and ask
21:18:57 <mriedem> alexpilotti:
21:19:01 <mikal> OK
21:19:03 <russellb> i thought i saw it pass on a patch today?
21:19:06 <russellb> yeah, https://review.openstack.org/#/c/86643/
21:19:22 <alexpilotti> hi guys
21:19:22 <russellb> so hopefully restored now
21:19:22 <mikal> http://www.rcbops.com/gerrit/reports/nova-cireport.html is what I am looking at
21:19:34 <alexpilotti> we fixed the zuul issue today
21:19:45 <mriedem> ah cool
21:19:49 <mikal> Excellent
21:20:14 <alexpilotti> the file system on the zuul server had issues and went read only
21:20:24 <russellb> darn computers
21:20:29 <mikal> For reference, neutron seems to be having exactly the same issues with third party CI reliability, so this isn't just us
21:20:34 <alexpilotti> well, Linux :-)
21:20:39 <russellb> ha
21:20:40 <alexpilotti> (kidding :-D)
21:20:45 <russellb> well played
21:20:55 * dansmith sprays alexpilotti with the hose
21:21:19 <alexpilotti> anyway, zuul has the habit of reporting a slightly misleading error
21:21:31 <alexpilotti> about needing to rebase, whatever happens
21:21:45 <tjones> want to take a quick look at https://bugs.launchpad.net/nova/+bug/1305423 to make sure there is no regression here?
21:21:46 <uvirtbot> Launchpad bug 1305423 in nova "nova libvirt re-write broken with mulitiple ephemeral disks" [Undecided,New]
21:22:37 <alexpilotti> would it be worth syncing with the infra guys to see how to improve the error reporting in case of zuul issues?
21:23:08 <dansmith> alexpilotti: sure
21:23:09 <mikal> alexpilotti: jhesketh has done some stuff there
21:23:19 <mikal> We wanted merge fails reported separately from test fails for example
21:23:28 <mikal> So it might be worth chatting to him about your issues as well
21:23:29 <alexpilotti> mikal: we’re also working on a tool based on your ci hacks for reporting: http://91.232.152.172/
21:24:02 <russellb> tjones: hm, no indication there that it's a regression
21:24:04 <alexpilotti> wondering if it’s the case to join forces and do something together for the community
21:24:07 <comstud> computers are hard
21:24:16 <mikal> alexpilotti: have you seen the new reports we generate?
21:24:20 <russellb> tjones: also not a common use case i think, so i don't think i'd block on it at this point
21:24:23 <russellb> tjones: thanks for raising it
21:24:45 <alexpilotti> mikal: this one? #link http://www.rcbops.com/gerrit/reports/nova-cireport.html
21:24:47 <mikal> alexpilotti: http://www.rcbops.com/gerrit/reports/nova-cireport.html
21:24:50 <mikal> Yeah
21:24:52 <tjones> russellb: ok good.  wasn't sure if that used to work - it does seem like a corner case
21:25:07 <alexpilotti> yep, what we need is also the performance over time
21:25:11 <mikal> So we do graphing now too... I personally feel those line graphs are very hard to read -- we played with them for a while before giving up on them
21:25:27 <mikal> alexpilotti: oh, I see
21:25:27 <alexpilotti> with a minimum of drill up / down
21:25:42 <mikal> alexpilotti: I'd actually like to feed this data into a time series database and let people do alerting on it
21:25:56 <mikal> alexpilotti: i.e. CI authors get an email when their fail rate is higher than jenkins' for example
21:26:00 <mikal> alexpilotti: but that's a work in progress
21:26:41 <alexpilotti> mikal: what about an official db with some API where anybody can read the raw events data and build any reporting on top?
21:27:05 <alexpilotti> mikal: reading directly the zuul stream is a bit far from optimal :-)
21:27:10 <mikal> alexpilotti: well, I guess we have that in the form of gerrit now, but I agree its hard to query
21:27:53 <alexpilotti> and if your ssh connection goes down, you lose events that need to be queried again, etc
21:28:16 <mikal> alexpilotti: oh, you know that we have an on disk archive of the event stream right?
21:28:30 <mikal> alexpilotti: it lags reality by 15 minutes, but we've tried hard to make it reliable
21:29:20 <alexpilotti> mikal: can you send me some pointers? we’re now also using an historical source, have to check from where, might simply be your site :-)
21:29:41 <mikal> alexpilotti: http://www.rcbops.com/gerrit/merged/2014/4/ is this month
21:30:03 <mikal> alexpilotti: the archive goes back to May last year, which is when I started thinking about building turbo hipster
21:30:05 <alexpilotti> mikal: yep, that’s what we use :-)
21:30:26 <mikal> alexpilotti: heh, its good to have a customer!
21:30:44 <alexpilotti> heh
21:30:47 <mikal> Well, we can chat more about this offline I suppose
21:30:59 <alexpilotti> we can start a beers subscription!
21:31:32 <alexpilotti> last question: what about moving it to something.openstack.org?
21:31:53 <mikal> alexpilotti: I don't have a problem with that, but I'd need to talk to infra about if they're willing to run it
21:32:12 <dansmith> alexpilotti: do you mean a DNS entry, or actual hosting of it?
21:32:21 <alexpilotti> dansmith: both
21:32:25 <dansmith> alexpilotti: I think the infra folks could just point a name at your IP if you're interested
21:32:26 <dansmith> okay
21:32:28 <mikal> Hmmm, a DNS pointer is another option
21:32:35 <alexpilotti> mostly for the community endorsement
21:32:41 <dansmith> alexpilotti: maybe a dns name is a good way to start
21:33:00 <alexpilotti> say that the site goes down at… 3am in mikal’s timezone
21:33:02 <dansmith> awesomesauce.openstack.org -> 91.232.152.172/
21:33:13 <alexpilotti> how do we handle that?
21:33:24 <mikal> alexpilotti: we ping infra in IRC
21:33:54 <alexpilotti> mikal: meaning that they have admin access to http://www.rcbops.com/ ?
21:33:58 <mikal> But I suspect if its a thing people rely on, we shuold probably talk about if the foundation wants to host it
21:34:20 <mikal> alexpilotti: we'd have to move to different instances to do that, www.rcbops.com hosts lots of other random thigns as well
21:34:57 <alexpilotti> mikal: that’s what I’m suggesting (if the foundation agrees on supporting this of course)
21:35:11 <mikal> alexpilotti: for sure. How about we chat with thema bout it at the summit?
21:35:24 <alexpilotti> ok, great plan!
21:35:42 <alexpilotti> maybe a design session on this?
21:35:58 <russellb> it'd be an infra one, but seems reasonable
21:35:59 <alexpilotti> including data + reporting tools?
21:36:58 <mikal> Reporting is a thing, so sure
21:37:31 <russellb> any other topics for today?  otherwise we can just close the meeting
21:38:28 <russellb> alrighty, have a nice weekend everyone!
21:38:31 <russellb> #endmeeting