20:03:53 <devananda> #startmeeting tripleo
20:03:54 <openstack> Meeting started Mon May 13 20:03:53 2013 UTC.  The chair is devananda. Information about MeetBot at http://wiki.debian.org/MeetBot.
20:03:55 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
20:03:57 <openstack> The meeting name has been set to 'tripleo'
20:04:04 <devananda> #topic bugs
20:04:21 <devananda> i'm'a just follow the agenda lifeless posted on the wiki, and let others do a lot of the talking hopefully :)
20:04:28 <devananda> #link https://wiki.openstack.org/wiki/Meetings/TripleO
20:04:41 <devananda> who all is here, anyway?
20:04:51 <echohead_> \o
20:04:58 <Ng> .o/
20:05:45 * devananda pokes a few people in other channels
20:05:46 <cody-somerville> \o_
20:06:25 <devananda> so, bugs.....
20:06:29 <devananda> #link https://bugs.launchpad.net/nova/+bugs?field.tag=baremetal
20:06:30 <devananda> and
20:06:49 <devananda> #link https://bugs.launchpad.net/tripleo/+bugs
20:06:53 <devananda> there's lots of them!
20:07:00 <echohead_> a productive week.  :)
20:07:13 <anteaya> o/
20:07:19 <devananda> echohead_: you're going to fix all the bugs, right? :)
20:07:31 <Ng> sweet
20:07:53 * echohead_ grabs his flyswatter to squash all the bugs
20:08:13 <devananda> anyone want to bring up specific bugs to talk about?
20:08:37 <devananda> i haven't looked through them all yet, but a lot are marked high or critical, so i am imagining there are some things to discuss :)
20:08:57 <SpamapS> seems we have some "server gets wedged" bugs that are probably the most serious
20:09:14 <SpamapS> Heat does not deal well at all with the ERROR state or with deleted servers.
20:09:23 <SpamapS> working on that one after we get heat into boot-stack entirely.
20:09:49 <Ng> I only had one comment, which was on the text mode console kernel options one - I don't think it's going to be fixed upstream. lifeless' comments suggest that we should set it ourselves in the base element. I am going to do that unless objections arise
20:10:14 <echohead_> agreed about the 'server gets wedged' bug(s) being the most urgent atm.
20:10:17 <devananda> #link https://bugs.launchpad.net/tripleo/+bug/1178112
20:10:18 <uvirtbot`> Launchpad bug 1178112 in tripleo "baremetal kernel boot options make console inaccessible on ILO environments" [Critical,Triaged]
20:10:19 <SpamapS> +1, though we should all go +1 the bug as well
20:11:16 <devananda> Ng: ++ to fixing it
20:12:03 <devananda> Ng: in testing taht bug, what are you using to get a console?
20:12:10 <devananda> Ng: nova commands or something else?
20:12:34 <devananda> SpamapS: any particular "server gets wedged" bugs you want to point out?
20:13:14 <SpamapS> devananda: no, I'm mostly referencing things overheard
20:13:30 <Ng> devananda: so actually the easiest way to test it I've found so far, is to run an image with kvm -curses. Can't show the console in a terminal if it's graphical, however, I will test in a nova to make sure it doesn't interfere with the console log getting, but I don't think it should do, that will still be specified as the last console= on the commandline and so win upstart's heart
20:14:06 <echohead_> devananda: in the test rack, ~20% of nova boots fail.  i'm not sure that it has been narrowed down enough yet to produce a valuable bug report.
20:14:23 <echohead_> not clear yet if it is isolated to specific machines, or what.
20:14:26 <devananda> Ng: gotcha. in theory, baremetal driver has some support for textconsole... i'd be interested to know if that actually works :)
20:14:27 <cody-somerville> To help split up the work, can I suggest folks who are knowledgeable about the bug and/or nova make sure there is enough information in each bug to make them actionable?
20:14:40 <devananda> cody-somerville: ++
20:15:37 <devananda> also, if you're going to work on a bug, please assign it to yourself (shoudl go without saying... :) )
20:15:39 <Ng> devananda: huh, ok, I'll have a look at that too
20:15:53 <anteaya> don't know if this is a good idea or not, but I am working on https://bugs.launchpad.net/nova/+bug/1178378 and don't have enough experience to really get my head around it, if someone was willing to pair with me to get it done, I think I would learn a lot from that
20:15:54 <uvirtbot`> Launchpad bug 1178378 in tripleo "confused baremetal instance thinks its off, is clearly operational" [Critical,Triaged]
20:16:00 <devananda> echohead_: I have heard similar numbers from others using similar hardware.
20:16:12 <echohead_> interesting
20:16:19 <anteaya> failing that I will keep plugging away
20:16:19 <devananda> echohead_: my inclination is to guess that it's 20% of machines are bad -- or just have the wrong info
20:16:56 <echohead_> i think we should have an action item to see if that is the case, and if so, to un-enroll the problematic machines.
20:17:17 <devananda> #action echohead_ to determine if test rack failures are machine specific
20:18:03 <devananda> anteaya: I volunteer NobodyCam to help with that :) (assuming he doesn't mind)
20:18:14 <anteaya> yay NobodyCam
20:18:21 <anteaya> does that work for you?
20:18:42 * NobodyCam looks at bug
20:18:49 <anteaya> it would cut down on my aimless flailing
20:18:55 <cody-somerville> Is the 30% fail to boot thing different from lp #1178586?
20:18:56 <uvirtbot`> Launchpad bug 1178586 in nova "scheduling failures leave baremetal instances stuck in BUILDING" [Medium,Triaged] https://launchpad.net/bugs/1178586
20:18:58 <cody-somerville> *20%
20:19:33 <anteaya> NobodyCam: I planted a flag here: https://review.openstack.org/#/c/28817/
20:19:41 <anteaya> not much to it, but a beginning
20:19:41 <devananda> cody-somerville: i think that's a case where the bug doesn't have enough info for me to know what it is
20:19:50 <cody-somerville> lp #1178919 is also related if not the same bug
20:19:51 <uvirtbot`> Launchpad bug 1178919 in tripleo "instances get stuck in 'BUILDING' sometimes" [High,Triaged] https://launchpad.net/bugs/1178919
20:20:02 <echohead_> cody-somerville: the failures look a bit different, in that they are stuck in 'spawning', as opposed to 'scheduling', as in the bug.
20:20:45 <cody-somerville> echohead_: like "| 0a171cbe-0f3c-40d5-ae8d-606f1dde41ce | test-0a171cbe-0f3c-40d5-ae8d-606f1dde41ce | BUILD | spawning | NOSTATE | | "?
20:21:05 <echohead_> yep
20:21:36 <devananda> yea, 1178919 and 1178586 appear to be the same
20:22:13 <devananda> marked as sup
20:22:14 <devananda> dup
20:22:17 <cody-somerville> devananda: Just noticed that one says it's stuck in spawning and the other in scheduling.
20:22:18 <NobodyCam> devananda: anteaya and I will look into whats up
20:22:33 <anteaya> thanks NobodyCam
20:23:22 <devananda> cody-somerville: gah, thanks
20:23:25 <cody-somerville> devananda: may be the same problem (booting too many things at once maybe?) or slightly different - I see lifeless speculates that the one stuck in scheduling might not be bm related.
20:23:43 <devananda> that isn't a stuck-in-scheduling issue
20:23:55 <devananda> the scheduler gave up after trying 3 times unsuccessfully
20:24:09 <devananda> each attempt tried to perform a deploy
20:24:10 <devananda> failed
20:24:16 <devananda> and was deleted by the scheduler
20:24:31 <SpamapS> What about 'stuck in deleting' ?
20:24:50 <SpamapS> I have that right now. :-P
20:25:00 <devananda> hah
20:25:12 <echohead_> SpamapS: lifeless had success with cleanup stuck 'deleting' instances manually.
20:25:19 <SpamapS> ugh
20:26:31 <devananda> ok, http://paste.ubuntu.com/5650528/ shows the compute log for the scheduling failure
20:26:54 * SpamapS unfortunately has conflicting things now and so will just be lurking
20:28:09 <devananda> it is different indeed. the spawn failure doesn't look like it ever powered on. the schedule failure powered on then failed in deploy
20:28:50 <devananda> huh?
20:28:59 <anteaya> there goes the bot I do believe
20:29:02 <devananda> i was about to say, enough with bugs let's move on
20:29:05 <NobodyCam> grrr
20:29:12 <devananda> #topic test rack
20:29:21 <devananda> yay! bot's still alive
20:29:25 <cody-somerville> bot is good, just chanserv rejoined
20:29:31 <anteaya> cool
20:29:33 <cody-somerville> or came up
20:29:35 <anteaya> good bot
20:29:49 <echohead_> test rack is coming along nicely, i expect heat to be running there by eod, hopefully.
20:30:09 <echohead_> hopefully the nova-boot failures are specific to certain boxes, and can be removed.
20:32:05 <devananda> echohead_: you needed me to lok at some heat / t-i-e reviews, yes?
20:32:35 <echohead_> devananda: yes, that would be good.  i will be applying those pending changes on the test rack to bring up heat.
20:32:53 <devananda> #action devananda to review t-i-e heat changes
20:32:55 <devananda> k, will do
20:33:08 <devananda> anything else to discuss on the rack?
20:33:09 <echohead_> seems like getting the openstack service heat templates working is the next thing in the critical path, which can proceed in parallel in a virtual-bm environment.
20:33:51 <echohead_> also, we will need a first-boot script to write the /etc/network/interfaces appropriately for openstack nodes on the machines.
20:34:11 <devananda> echohead_: ah, is that related to the baremetal-always-does-file-injection bug?
20:34:31 <devananda> or put another way, if bm didn't do file injection, would you still need to write that script?
20:34:45 <echohead_> devananda: i think we would still need it even without the bug.
20:34:52 <echohead_> because we must configure vlan interfaces, etc.
20:35:03 <devananda> i see
20:35:48 <devananda> then i won't prioritize fixing _that_ bug
20:35:53 <echohead_> :)
20:35:56 <devananda> #topic open discussion
20:36:24 <dkehn> fyi wrapping up the quantum PXE changes
20:36:32 <devananda> awesome
20:36:38 <dkehn> for review beod
20:36:50 <dkehn> and the quantum client
20:37:05 <dkehn> will go next to the nova changes to talk to it
20:37:09 <devananda> please drop links in #tripleo for that when it's up
20:37:17 <dkehn> yes sir
20:37:37 <devananda> i can help with the nova changes
20:37:41 <devananda> once i see what it has to talk to
20:37:44 <dkehn> will be going silent for a few days at the end of the month
20:38:01 <dkehn> transistion to Europe
20:38:19 <devananda> sounds fun!
20:38:27 <dkehn> fingers crossed
20:38:41 <anteaya> dkehn: safe travels
20:38:45 <dkehn> tx
20:38:47 <dkehn> thx
20:39:14 <devananda> i've been mostly occupied with lots and lots of hacking on Ironic
20:39:21 <devananda> for the last week and all weekend
20:39:28 <devananda> and will probably continue to be so consumed :)
20:39:34 <dkehn> gotta love the name, quantuim is still battling with legal
20:39:38 <dkehn> over a name
20:39:40 <devananda> yea :(
20:39:52 <devananda> i'm actually still waiting on foundation to sign off on "Ironic"
20:40:01 <devananda> but my own research says it's not TMd in the US
20:40:14 <echohead_> Alanis Morrisette begs to differ.
20:40:19 <echohead_> oh wait, that's canada.
20:40:45 <devananda> if folks need me for things, don't hesitate to poke (not that anyone has hesitated anyway, just sayin)
20:40:46 <anteaya> do not invoke that name
20:41:01 <devananda> echohead_: i think the song title is "isn't it ironic" :)
20:41:06 <anteaya> no no no
20:41:10 <anteaya> it shall not be said
20:41:13 <devananda> :P
20:41:15 <echohead_> devananda: ok, guess we're clear then :)
20:42:03 <devananda> any other topics?
20:42:11 <devananda> or shall we wrap up early again?
20:42:29 <cody-somerville> :)
20:42:37 <dkehn> when I logged into IRC tripleo is said meeting at 2000 UTC, which time is ti
20:42:57 <dkehn> nevermind
20:43:11 <devananda> dkehn: :)
20:43:28 <devananda> well, thanks all!
20:43:32 <dkehn> devananda, so that is the real time?
20:43:40 <anteaya> great chairing devananda
20:43:42 <devananda> #endmeeting