20:01:08 <lifeless> #startmeeting tripleo 20:01:09 <openstack> Meeting started Mon Jul 22 20:01:08 2013 UTC. The chair is lifeless. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 20:01:13 <openstack> The meeting name has been set to 'tripleo' 20:01:13 <lifeless> #topic agenda 20:01:20 <lifeless> bugs 20:01:21 <lifeless> Grizzly test rack status 20:01:21 <lifeless> CI virtualized testing progress 20:01:21 <lifeless> open discussion 20:01:56 <lifeless> #topic bugs 20:02:06 <lifeless> https://bugs.launchpad.net/tripleo/ 20:02:06 <lifeless> https://bugs.launchpad.net/diskimage-builder/ 20:02:06 <lifeless> https://bugs.launchpad.net/os-refresh-config 20:02:06 <lifeless> https://bugs.launchpad.net/os-apply-config 20:02:08 <lifeless> https://bugs.launchpad.net/os-collect-config 20:03:41 <jog0> o/ 20:04:00 <SpamapS> o/ 20:04:03 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1182249 20:04:35 <SpamapS> lifeless: _almost_ ready to tackle that 20:04:42 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1183223 20:04:47 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1184484 20:04:49 <SpamapS> lifeless: once we swap in os-collect-config , should be able to realistically address it. 20:04:54 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1189385 20:04:59 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1200201 20:05:03 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1201580 20:05:09 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1201581 20:05:14 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1201584 20:05:19 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1202322 20:05:29 <lifeless> https://bugs.launchpad.net/diskimage-builder/+bug/1202612 20:05:32 <jog0> will be working on 1184484 this week 20:05:40 <lifeless> wheee we have a bunch of crits ;) 20:06:25 <SpamapS> did we lose the bug bot? 20:06:35 <lifeless> bug 1202322 20:06:39 <jog0> been cought up in rootwrap nova-network entrypoints land 20:06:41 <lifeless> appears so 20:07:54 <lifeless> ok so 20:08:04 <lifeless> the dib one is kinda worrying 20:08:13 <lifeless> since it's -really- harsh when it happens 20:08:37 <SpamapS> lifeless: https://bugs.launchpad.net/tripleo/+bug/1202322 just need to land the 2 reviews on os-collect-config and then try devtest with os-collect-config instead of heat-cfntools 20:08:40 <derekh> lifeless: this happened me quite a few times last week, but not once today .... 20:09:01 <lifeless> SpamapS: Well, lets get that done :> 20:09:13 <derekh> lifeless: If it happens again will try and track it down 20:09:25 <SpamapS> lifeless: yeah, its what I was working on, when the meeting started :) 20:09:34 <lifeless> SpamapS: 1201581 - do we need new tenant instances ? 20:09:38 <lifeless> SpamapS: or just server side ? 20:09:52 <SpamapS> lifeless: also for the di-b bug.. we can fix it by using rm --one-file-system 20:10:22 <lifeless> SpamapS: you think the bind mounted dev is still in place when we rm ? 20:10:27 <lifeless> SpamapS: say so in the bug :) 20:10:36 <SpamapS> lifeless: I think we can just deploy a newer keystoneclient in keystone's venv. 20:11:36 <lifeless> bug 1201580 is going to be dependent on the precious fs movement stuff 20:11:45 <lifeless> plus in-instance upgrade hacks 20:12:19 <lifeless> jog0: we will need your nova expertise at some point, we have this crazy idea about updating the boot ramdisk + kernel for ari+aki using flavors 20:12:30 <lifeless> jog0: (and making that work and push out to existing instances) 20:12:41 <lifeless> s/flavors/images/ 20:12:43 <jog0> lifeless: is this the rsync based thing? 20:12:54 <lifeless> jog0: not directly, but tied into it. 20:13:04 <SpamapS> lifeless: I was just thinking that we could push out a git-tree-puller and a 'pip install -U''er and an 'apt-get upgrade''er 20:13:16 <lifeless> SpamapS: ah, so a hack :) 20:13:19 <SpamapS> lifeless: as ghetto and stinky as possible 20:13:24 <SpamapS> right 20:13:26 <jog0> lifeless: have a link to this crazy idea? 20:14:03 <lifeless> jog0: no, I put a bug and etherpad up about the issue 20:14:06 <lifeless> *will* 20:14:31 <jog0> lifeless: cool 20:15:20 <lifeless> #action lifeless to ensure we have bugs surrounding the in-instance upgrade path and new ramdisks/kernels 20:15:51 <lifeless> jog0: actually what we'd like to do is to rebase an instance onto a new kernel/ramdisk and disk image - but not reboot it - trust it will redo the image contents itself. 20:16:00 <lifeless> but yeah, will write that up 20:16:38 <jog0> interesting, thanks don't fully grok how that works so a writeup would be great 20:16:44 <lifeless> SpamapS: - https://bugs.launchpad.net/tripleo/+bug/1200201 - still exists ? 20:17:29 <SpamapS> lifeless: I haven't verified it is closed yet. 20:17:35 <SpamapS> lifeless: forgot to tag it in ORC-REFACTOR 20:17:38 <lifeless> kk 20:18:10 <lifeless> https://bugs.launchpad.net/tripleo/+bug/1189385 is still pending something; we haven't seen reproduction in a while now. 20:18:13 * SpamapS assigns self 20:18:24 <lifeless> and I still owe https://bugs.launchpad.net/tripleo/+bug/1184484 some config extraction 20:18:33 <lifeless> ok 20:18:45 <lifeless> any pet bugs folk want to chat about ? 20:19:35 <lifeless> ok 20:19:46 <lifeless> #topic grizzly test rack status 20:19:53 <lifeless> so this is ticking along 20:20:06 <lifeless> I found the network node services had no upstart jobs yesterday 20:20:09 <lifeless> this had everything down 20:20:13 <lifeless> and I have NFI how/why.. 20:20:35 <lifeless> I added them using os-svc-install 20:20:37 <lifeless> but sheese. 20:20:45 <SpamapS> I think the thing is a little rickety and concerning. We realy do need at least a tiny subset rack to be able to CD to so we don't have a dead duck. 20:22:16 <SpamapS> A huge portion of what was done in the POC has been rewritten and refactored a lot since then.. no idea if it would apply to that rack now. :-P 20:22:55 <lifeless> so there are spare machines 20:23:24 <lifeless> someone needs to grab the hw list and examine the machines that are faulty and try recovery 20:23:31 <lifeless> I can offer some offline hints about that 20:23:43 <lifeless> also I have a list of 8 or so other machines that were earmarked for monty and are idle 20:23:51 <lifeless> again, someone with time needs to JFDI 20:24:19 <SpamapS> time + access ;) 20:24:29 <lifeless> so, access - good point. 20:25:17 <lifeless> This is HP hardware in a production datacentre; I don't have the authority to give control plane access to the cloud to non-HP staff, *but* any HP staff involved in tripleo should be totally fine. 20:25:51 <lifeless> # action HP tripleoers If you don't have access to the POC rack control plane. ping me/ng/spamaps - all of us should be able to add you. 20:25:55 <lifeless> #action HP tripleoers If you don't have access to the POC rack control plane. ping me/ng/spamaps - all of us should be able to add you. 20:26:17 <lifeless> huh, failbot ? 20:26:25 <lifeless> #action HP-tripleoers If you don't have access to the POC rack control plane. ping me/ng/spamaps - all of us should be able to add you. 20:26:30 <lifeless> NFI.... 20:26:57 <SpamapS> hey the bug bot is back :) 20:27:05 <lifeless> heh 20:27:14 <lifeless> so - we need to action the criticals around the rack 20:27:19 <lifeless> but we talked about that 20:27:25 <lifeless> so - next topic time ? 20:27:35 <SpamapS> are all of the criticals assigned? 20:27:45 <SpamapS> or at least, the blocking criticals? 20:27:51 <SpamapS> (may be ordering issues..) 20:28:41 <lifeless> no 20:28:43 <lifeless> they are not 20:29:27 <SpamapS> Ok well I think we can address them as criticals and just attack them one by one. 20:29:39 <lifeless> yup 20:29:41 <lifeless> #topic CI virtualized testing progress 20:29:48 <lifeless> ok, that works. da fuq 20:29:55 <lifeless> pleia2: oh hai. 20:30:03 <lifeless> pleia2: I suspect you're going to say 'nochange' :> 20:30:26 <pleia2> yeah, at oscon this week 20:31:08 <lifeless> #topic open discussion 20:32:30 <dkehn> once again all reviews have been addressed on the neutron and neutronclient, going to push for merge in next meeting 20:32:37 <lifeless> coool! 20:32:43 <SpamapS> It's worth stating here, I am overhauling os-refresh-config and replacing cfn-hup with os-collect-config .. so please do report any weirdness you see there. 20:32:53 <lifeless> wooo 20:33:09 <lifeless> I'm going to be AWL from thursday through wednesday 20:33:44 <SpamapS> Oh and 20:33:47 <SpamapS> we're like, official and stuff 20:33:55 <lifeless> I have some leave thursday/fri then tuesday doing tech @ work day in Sydney. Mon and wednesday are a combination of being not-at-home connectivity spottiness and travel. 20:34:04 <lifeless> so I need someone to run this meeting next week. 20:34:32 * SpamapS checks schedule to be sure 20:35:07 <SpamapS> lifeless: I will run it 20:35:18 <lifeless> thanks! 20:35:26 <lifeless> #action SpamapS run da meeting next week. 20:35:33 <lifeless> #help 20:36:29 <lifeless> ok 20:36:35 <lifeless> so something I think we should try and sync on 20:36:37 <lifeless> is the roadmapish 20:36:46 <lifeless> we're now at the opencloud - woo! 20:37:08 <lifeless> in my head, it's now time to take our narrow feature set and start expanding sideways 20:37:12 <lifeless> - updates 20:37:14 <lifeless> - HA 20:37:34 <SpamapS> I need to spend some time on bringing Heat up to our expectations. 20:37:40 <SpamapS> we need more failures to be retryable 20:37:56 <SpamapS> and rolling/canary updates will not make h3 if I don't start on it by next week. 20:37:57 <lifeless> - bare metal improvements (like including vendor firmware flashes in the deploy ramdisk) 20:38:06 <lifeless> - heat ^ 20:38:44 <lifeless> SpamapS: so H3 doesn't worry me too much, as long as we're not entirely blocked for 2 months - can you land it with an option to enable it or something 20:40:21 <SpamapS> lifeless: it can be marked experimental for sure. 20:40:31 <SpamapS> lifeless: it won't interfere with anything until you say "please update using canaries/rolling" 20:40:43 <lifeless> SpamapS: great 20:40:45 <SpamapS> so yeah, the manual can say "This is experimental use at your own risk" 20:40:59 <lifeless> SpamapS: well more I mean do you *need* to stress about H3 20:41:03 <lifeless> we deploy trunk today 20:41:11 <lifeless> so as long as we can get the support into trunk... 20:41:38 <SpamapS> I have a collaborator from outside tripleo who will be helping, who might care about H3 :) 20:41:42 <lifeless> kk 20:42:03 <lifeless> what else can we bifurcate onto 20:42:05 <lifeless> - performance 20:42:16 <lifeless> - monitoring [as NobodyCam is already!] 20:42:20 <SpamapS> CI/CD for di-b 20:42:22 <lifeless> - reporting 20:42:28 <lifeless> - CICD yes yes yes! 20:42:31 <SpamapS> I know a lot has been done already 20:42:40 <lifeless> basically - we've automated the stuff the POC taught us. 20:42:45 <SpamapS> seems like we're close with the offline features to being able to test it in isolation 20:42:51 <lifeless> Perhaps we need a new stretch goal to consolidate around ? 20:42:59 <SpamapS> well there is this sprint.. 20:43:40 <lifeless> its ages off, we should be finished by then :P 20:43:47 <SpamapS> true 20:44:42 <lifeless> baseline - if anyone is aimless, we have tonnes to do, ping me [or anyone else on the team] and we'll help you find a useful thing that is within reach 20:45:05 <lifeless> last call on discussion ... 20:47:15 <lifeless> ok, thanks for playing! 20:47:18 <lifeless> #endmeeting