19:00:49 #startmeeting ironic 19:00:50 Meeting started Mon Apr 14 19:00:49 2014 UTC and is due to finish in 60 minutes. The chair is devananda. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:51 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 19:00:53 The meeting name has been set to 'ironic' 19:00:55 o/ 19:01:00 o/ 19:01:00 o/ 19:01:02 o/ 19:01:03 hello o/ 19:01:03 here 19:01:03 o/ 19:01:04 #chair NobodyCam 19:01:04 Current chairs: NobodyCam devananda 19:01:05 o/ 19:01:12 o/ 19:01:13 :) 19:01:14 o/ 19:01:16 hi 19:01:22 hey hey 19:01:26 as usual, our agenda can be found at 19:01:28 \o 19:01:28 #link https://wiki.openstack.org/wiki/Meetings/Ironic 19:01:36 \o 19:01:48 glad to see so many folks here! 19:02:02 \o/ 19:02:04 #topic what's needed to deprecate nova-bm 19:02:13 * NobodyCam note I could drop at any time :-p 19:02:27 several things. and I'd like us to talk about it at the summit... 19:02:36 docs, migration script, console, jobs in the gate 19:02:50 testing 19:02:53 really - it's testing 19:02:59 yeah 19:03:01 Automated? 19:03:01 the most dificult 19:03:05 and then for the nova team to accept the nova.virt.ironic driver into their tree 19:03:12 which they wont do without automated testing 19:03:32 * dtantsur would like see those major test refactorings to land before merge with nova... 19:03:35 we have all the req's of an incubated project (docs, tests, integration with other projects, etc) 19:03:36 after nova-bm is deprecated, will the classes for nova_bm still in the nova package? 19:03:49 which are outlined here 19:03:51 #link http://git.openstack.org/cgit/openstack/governance/tree/reference/incubation-integration-requirements#n62 19:04:00 AND we have the requirement to deprecate nova-bm 19:04:11 which is both a technical and a human dependency 19:04:27 since the nova team has to approve those change(s) BEFORE we can graduate 19:04:34 regardless of how ready we are in every other way 19:04:43 The reason I am asking is that we have a xCAT baremtal driver that uses the nova_bm classes. And also uses nova_bm database tables. 19:04:46 devananda, the design session you proposed to nova to talk about it, any feedback from the nova ptl/responsible for approving it?? 19:04:57 * lucasagomes tries to find the link 19:05:12 #link http://summit.openstack.org/cfp/details/215 19:05:24 linggao: AFAIK, yes. Nova's plan is that, when they land nova.virt.ironic driver, they will mark nova.virt.baremetal deprecated for one cycle, then remove it 19:06:05 does this mean we don't leave incubation for at least one cycle after nova-bm is marked deprecated? 19:06:14 lucasagomes: none yet. I imagine mikal has been busy since his election, and there is still a week for session proposals 19:06:24 devananda, hmm, that's destructive for other home gown baremetal drivers. 19:06:33 mrda, I think we do, both will exist in the nova tree, but any new feature should go to ironic 19:06:37 mrda, instead of nova bm 19:06:46 devananda, I c 19:06:55 mrda: afaik, that's the backwards order. (ironic graduates) && (nova-bm is marked deprecated) should, AFAIK, happen at the same time 19:07:04 mrda: then a cycle later, nova-bm is actually removed 19:07:24 linggao: I believe once nova-bm is marked deprecated it will live in tree for a cycle 19:07:46 but I defer to mikal or russellb if either one are around and want to comment 19:08:02 so if nova-bm is marked deprecated in juno, it will still live in k*, and be removed in L* 19:08:20 that is my understanding (correct or not) 19:08:26 NobodayCam, is one cycle 6 month? 19:08:35 ok, cool, would be good to get ironic graduating without having to wait for nova-bm to be removed (i.e wait 6 months for nova-bm deprecation) 19:08:50 linggao: yes 19:08:59 mikal won't be online right now, it's a bit early in AUS 19:09:08 linggao: yep 19:09:25 one point to call out that we haven't discussed much 19:09:29 is integration with other projects 19:09:36 eg, horizon and ceilometer 19:09:46 as the current guidelines stand, we will need both to graduate 19:10:03 haomeng was working on the ceilometer integration 19:10:26 I think for horizon there's no directly work for us to do, since they will just use our api to register the nodes 19:10:40 #link https://review.openstack.org/#/c/72538/ 19:10:45 on the other hand ceilometer, needs ironic to collect the metrics for them 19:10:52 is there anything in tuskar that addresses ironic/horizon? 19:10:53 lucasagomes, as I remember discussion in mail list, someone wanted to do some UI 19:10:56 lucasagomes: well, someone has to create a horizon plugin to talk to ironic 19:11:16 right 19:11:18 #link https://etherpad.openstack.org/p/ironic-ui 19:11:21 but it's outisde ironic 19:11:26 rloo, afaik not yet, tuskar uses nova-bm 19:11:31 jcoufal and i jotted down some notes there ^ 19:11:35 where ceilometer needs code in ironic to get it working 19:11:52 lucasagomes: is it out ironic if we need it to graduate 19:11:55 lucasagomes: right. but we need code to land in both projects prior to our graduation, too 19:12:13 s/out/outside/ 19:12:15 so we have 5 project dependencies at this point 19:12:23 testing: devstack + tempest 19:12:25 right 19:12:29 monitoring: ceilometer 19:12:35 UI: horizon/tuskar 19:12:42 generally just doing what we do: Nova 19:12:57 and console? 19:13:01 devananda, for testing we also have a dependency to tripleo, right? 19:13:20 linggao: console is implemented in Ironic 19:13:21 ifarkas: not as far as a graduation requirement, since tripleo is not part of the integrated release 19:13:27 ifarkas: but of course we want that to work :) 19:13:38 and are working on it :-p 19:13:48 linggao: console is an internal requirement for feature parity, not an integration requirement with other projects, AFAIK 19:13:52 linggao: but yes, we do need it :) 19:14:14 s/is implemended/is been implemented/ 19:14:22 devananda, I am working on it. 19:15:09 can have horizonto start the console 19:15:18 i'll give folks a few minutes to digest / ask more questions about graduation path 19:15:21 before we move on 19:15:41 right, the ceilometer stuff, is not needed for graduation right? 19:15:49 lucasagomes: afaik, it is 19:15:51 Thank you devananda for the info 19:16:13 Is this / should this be listed somewhere? 19:16:18 So we can all track it? 19:16:31 +1 matty_dubs 19:16:33 * matty_dubs happy to set up an Etherpad if not 19:16:47 heh +1 yeah it would be useful to have an etherpad for tracking such things 19:17:01 matty_dubs: There is a document in our wiki https://etherpad.openstack.org/p/IronicGraduationDiscussion 19:17:20 +1 (wishes it wasn't another ehterpad, but is good with that option too) 19:17:22 devananda: can ironic graduate *any time* after it has satisfied the requirements, or only at the end of a cycle? 19:17:33 that document was based on an old version of the governance doc 19:17:48 devananda: so we need to update it 19:17:58 matty_dubs: if you want to refresh that etherpad based on current status and http://git.openstack.org/cgit/openstack/governance/tree/reference/incubation-integration-requirements#n62 19:18:02 matty_dubs: that would be great 19:18:13 devananda: Sure thing; will do 19:18:24 iron1_, does horizon start the console? 19:18:25 rloo: graduation is officially done by a TC vote near the end of the cycle, just before the PTL elections 19:18:48 iron1_, or it just shows the console log? 19:18:51 #link https://etherpad.openstack.org/p/IronicGraduationDiscussion 19:19:09 NobodyCam: Perhaps a meta-etherpad is needed so we can track all the etherpads :) 19:19:13 devananda: ok, is there a way to get feedback prior to that, as to whether/what might be missing. Or does the TC meet and say yay/nay and no chance to fix whatever? 19:19:16 mrda: we have that :) 19:19:20 mrda, heh we do have it 19:19:25 https://etherpad.openstack.org/p/IronicWhiteBoard 19:19:31 lines 12 - 28 19:19:31 ok, cool, thnx! 19:19:42 mrda: use https://etherpad.openstack.org/p/IronicWhiteBoard 19:19:45 for that 19:19:47 :-p 19:19:52 #link https://etherpad.openstack.org/p/IronicWhiteBoard 19:20:03 fwiw, I keep that "whiteboard" linked in the IRC room topic ;) 19:20:16 +1 19:20:18 sort of a launching point for all the other resources 19:20:27 :) 19:20:29 Horizon starts Nova vnc so we can consider starting bm console from horizon 19:21:02 ok, one more minute for graduation questions, then i'll move on 19:21:16 thanks iron1_ for the info. 19:21:38 oh - tilera. From the TC's perspective, I think we have a good case for not needing that. 19:22:03 so devananda, not sure it is related. do we need to put console support in the ironic bm driver? 19:22:03 ? oh really 19:22:06 from ironic's POV, it's a third party driver. we have no ability to test it upstream, and without the original contributor's support, can't port it. 19:22:27 ahh 19:22:39 i have reached out to NTT a while back, and they indicated that they didn't have the time / resources to port it 19:22:48 so we will need ntt to prot that? and provide testing HW 19:23:00 s/prot/port/ 19:23:20 right. if they do that, great, but it's not a graduation blocker 19:23:35 :) nice :) 19:23:40 linggao: nova will need to be able to start/stop the console and get console log, yes 19:23:43 ok -- moving on 19:23:49 #topic icehouse release 19:23:56 #link https://launchpad.net/ironic/icehouse/icehouse-rc2 19:24:04 icehouse RC2 milestone was tagged this morning 19:24:14 we have a few more days before the icehouse release is official 19:24:14 \o/ 19:24:27 i'm not aware of any more backport-potential bugs, so hopefully this will be our final version 19:25:32 one thing this means is we should (and in my opinion, we need) to maintain API compatibility 19:25:35 from this point on 19:25:56 +1 19:26:01 even though it's not an "integrated release" -- it's still a release. this version will be in downstream repos. 19:26:17 then I have a issue about console API. 19:26:19 and we need to be clear that, as a developer community, we support it 19:26:22 +1. So we need some policies about upgrading API versions 19:26:39 +1 19:26:53 linggao: what's up? 19:26:56 the https://bugs.launchpad.net/ironic/+bug/1307555 19:26:57 Launchpad bug 1307555 in ironic "get console API should not return a string" [Undecided,New] 19:27:20 The console get API v1/nodes//states/console. 19:27:36 I think it should be v1/nodes//console 19:27:52 and return {console: {"url": "blahblah", "type": "blah"}} 19:28:16 Can we mark some API's (console ones) as unstable like python-dev do it for some new API's? 19:28:17 linggao: could that be v2/nodes//console 19:28:21 the "states" there does not make sense. 19:28:35 linggao: since none of the drivers in the Icehouse release will support console, I think it's fair to mark that API as unsupported/experimental right now 19:28:45 it isn't too late to change it for icehouse? 19:28:52 rloo: it is 19:29:00 #link https://wiki.openstack.org/wiki/Ironic/ReleaseNotes/Icehouse 19:29:06 the set console state is v1/nodes/uuid/states/console. this one sets the console enablement state. 19:29:19 so I have begun jotting some release notes there 19:29:27 and will continue to fill it in as I go through the logs and the release process 19:29:41 so we can just put this console api down as a known issue? 19:29:48 our (lack of) console support clearly bears mention there 19:29:53 rloo: I think that's fine 19:30:42 +! 19:30:46 +1 19:31:06 +1 19:31:21 +1 makes sense to me 19:31:38 +! 19:31:42 =1 19:31:53 +1 19:31:55 :) 19:32:07 sorry :) 19:32:14 * devananda updates wiki 19:32:39 any other questions on the icehouse release? 19:33:37 not here 19:34:08 ok, moving on then 19:34:12 #topic integration testing 19:34:29 should I mention benchmarking here? 19:34:38 adam_g: hi! how's it going? 19:34:43 NobodyCam: any updates on triple testing? 19:35:06 TripleO update: I am still unable to get correct logging from the gate jobs 19:35:07 our scenario test in tempest is now running as part of the virutal-ironic devstack job 19:35:08 romcheg: hmm, only if the benchmark tests are running and reporting upstream :) 19:35:29 I can get it locally 19:35:37 devananda: No, they are not :) Will mention that in open discussion 19:35:41 the scenario test in the virtual-ironic devstack job is currently *not* passing. i've been trying to reproduce outside of the gate and debug, but its proving challenging. any help would be appreciated! 19:36:19 adam_g: I was dealing with those things 19:36:47 adam_g: please poke me tomorrow and we can take a look together 19:36:56 adam_g, I'd have a look from Fedora point of view, but I can have even more problems :) 19:36:57 NobodyCam: the tripleo-ironic-seed job is still passing -- and should be trusted, yes? 19:36:59 romcheg, maybe we can chat after the meeting. i've narrowed the problem down to TFTP during boot, but still stumpted 19:37:09 dtantsur, is fedora still having issues with ssh? 19:37:13 seed yes ... undercloud no 19:37:21 adam_g: let's chat after the meting 19:37:28 adam_g: you might try adding the tftp log files to infra, so that they're visible to you after the job 19:37:34 adam_g, nothing new, without two controversial patches it does not work 19:37:41 devananda, that all goes to syslog, so it should be 19:37:49 adam_g: ah, ok. 19:37:51 adam_g, for me sources of problems with TFTP: SELinux, firewall, IPv6 19:37:53 dtantsur, you might be interested in https://review.openstack.org/#/c/87355/ 19:38:12 ^ follow up from an action last week, removes ssh reconfiguration from the devstack deploy 19:38:22 adam_g I +many on this 19:39:09 anyway, there are still some blocking issues in tempest that i have patches to address that are causing unrelated tests to fail against ironic. ive been tracking on the etherpad @ https://etherpad.openstack.org/p/IronicCI 19:39:10 adam_g: nice, thanks! 19:39:34 hopefully we can get these loose ends tied up before the summit 19:39:35 adam_g, have you checked you don't have troubles with IPv6? It seems to be relatively undiscovered 19:39:43 #link https://etherpad.openstack.org/p/IronicCI 19:40:21 I also found the IPv6 issue with Fedora on devtest. 19:40:31 dtantsur, it shouldn't be an issue AFAICS.. we can chat after meeting and see what we can debug info suck out of the devstack gate with an experimental patch review 19:40:40 dtantsur, adam_g, you might be interested in this patch: https://review.openstack.org/#/c/87208/ 19:41:12 adam_g, (at least on Fedora) if you have IPv6 enabled, TFTP only listens on IPv6 endpoint 19:41:29 interesting 19:41:30 thus breaking attempts to reach it on IPv4 endpoint 19:41:57 NobodyCam: it looks like tripleo-undercloud is breaking even without ironic? 19:42:10 atm yes 19:42:32 that started late friday I believe 19:42:36 ah, k 19:43:11 adam_g, NobodyCam, all -- thanks for the updates & continued progress on testing! 19:43:22 I dont have any other questions about it 19:43:38 but if there are ways I can unblock any of your efforts, pls dont hesitate to poke me 19:44:15 any one have questions on OoO testing I can answer 19:44:21 dtantsur: any brief updates on fedora status? 19:44:42 devananda, out-of-box devstack is blocked by SSH and IPv6 issues 19:45:08 also checking RPMs that are will be in Fedora, they need more love as of today 19:45:22 devananda, as for Fedora, I also tried it with devtest and I managed to get it working there 19:45:32 would like to see the test scenario to try it 19:45:41 ifarkas: awesome. are we going to get a check-tripleo-ironic-seed-fedora job? :) 19:45:50 devananda, beside the previous ipv6 issue I found another one with requiretty: https://review.openstack.org/#/c/87253/ 19:46:01 * lucasagomes would love to see a job for fedora 19:46:02 the best way to work on Fedora for now: disable SELinux, firewall, IPv6 :) 19:46:04 devananda, that would be great! ;-) 19:46:22 +1 for fedora testa 19:46:27 tests even 19:46:30 dtantsur: hah. sounds just like the early '00s! :) 19:46:35 ifarkas, again requiretty... 19:46:54 ok - since there's again a long list of open discussion topics 19:46:59 devananda, if we have port reset to 22, we may have it out-of-box 19:47:06 i'm going to move on 19:47:28 dtantsur: i'm going look at the jenkins log for that and/or run it locally, then will vote on that patch today 19:47:40 #topic open discussion 19:47:44 I have an update about benchmarks for Ironic 19:48:06 romcheg: are we faster then nova bm? 19:48:20 NobodyCam: U need more patience :-P 19:48:25 floor's open, don't wait for me to say "go" before you jump in, folks :) 19:48:26 hah 19:48:29 devananda: I want to see if there is anyway I can help to get our agent-related patches to land. I believe most of them are here: https://review.openstack.org/#/q/status:open+project:openstack/ironic+branch:master+topic:bp/agent-driver,n,z 19:48:29 I have started implementing all required plumbings 19:48:39 ok quick q... are we going to have a meeting next monday? (it's holiday here at least) 19:48:59 one more related patch here: https://review.openstack.org/#/c/81919/ 19:49:01 oh easter 19:49:09 I except to implement a first benchmark tomorrow and show some results 19:49:25 lucasagomes: yes - 4/20 is the deadline for summit talk submissions 19:49:35 lucasagomes: so I am planning on reviewing them on 4/21 in the weekly meeting 19:49:53 romcheg: awesome. is this just API/RPC benchmarks with the 'fake' driver? 19:49:53 devananda, ack :) 19:49:55 thanks 19:50:00 devananda: yes 19:50:15 There are more cases we need to benchmrk 19:50:17 romcheg: awesome. so. Nova has something similar in their test suite 19:50:27 romcheg: we need to also have a realistically-large data set in the DB 19:50:28 Some of them are not very easy to do 19:50:30 with fake data 19:50:42 jroll: hi! so, a few things would help me 19:50:51 jroll: more time in the day, for one ;) 19:51:01 I have registered a session about that http://summit.openstack.org/cfp/details/275 19:51:06 jroll: but seriously, I need to make some time to test deploys with the agent 19:51:38 * jroll gives devananda one of his hours 19:51:51 devananda: that's what we've been doing 19:51:56 is trying to test deploys 19:51:58 jroll: in an ideal situation, I would say, we should just add a CI pipeline that switches to that driver and runs tempest on it 19:52:08 lol /me has lost two of his hours so far 19:52:12 and there's a few integration breaks we're finding, and putting up patches for those as we go 19:52:31 I agree, I'm working on throwing dwalleck under that bus 19:52:36 who seems to be not here 19:52:58 devananda: my main problem is that it's very hard to test this, when we're constantly rebasing patches 19:53:14 jroll: at a minimum (eg, without upstream testing working yet) we should at least be able to stand up a devstack env with the agent+ssh drier 19:53:23 jroll: and run tempest on it 19:53:36 Is there a reason the driver shouldn't be merged even if it's imperfect though? It seems like it's better to be in tree than for us to be in tree making things better than having us get more and more code behind the dam unmerged? 19:53:38 jroll: right. out of tree development sucks. seriously. I've had to do a lot of it, too. 19:53:51 devananda: indeed :) 19:53:57 no -- it absolutely should be merged while it is imperfect. 19:54:03 ok 19:54:10 however, we need cores (and ideally myself, too) to have time 19:54:13 to review it 19:54:14 yeah 19:54:16 I understand 19:54:21 Then why can't we do that, then file bugs on items we can improve -- like the testing scenarios you've pointed out. 19:54:26 and make sure that it's architecturally close *enough* 19:54:26 jroll: have you guys looked at the tripleO deploy-ironic element. are we going to need a deploy-ironic-agent element too? 19:54:44 NobodyCam: I haven't personally looked at it at all 19:54:57 jroll: how are you guys building the "agent ramdisk" today? 19:55:00 NobodyCam: The diskimage-builder as it exists now is insufficient to run the agent 19:55:05 ooh 19:55:16 NobodyCam: We require a true init in the ramdisk, and right now that's unsupported. 19:55:21 devananda: there's an imagebuild/coreos directory with a makefile 19:55:32 essentially we just run make on that 19:56:15 I can document the process to build the image as it exists now if that would be helpful 19:56:22 JayF, any work on the pipeline to add support to diskimage builder (at least for fedora element) to use systemd? 19:56:35 devananda: what about supporting lvm in agent? yes or no? 19:56:41 JayF, that would yes 19:56:52 lucasagomes: not as I've seen, and it basically changes behavior significantly if you're on a ramdisk or not 19:57:01 are we going to keep a golden image of the deploy-agent or require users to build? 19:57:18 vkozhukalov: i'm still of the opinion "no". I started drafting an email to the list but haven't finished it yet 19:57:26 lucasagomes: so the diskimage-builder stuff, when using a ramdisk, competely strips out the init system, and that appeared to be the same on fedora/ubuntu/etc 19:57:40 NobodyCam: I believe it should be the same plan as we have with DIB today 19:57:42 JayF, yeah, it uses a custom init script :( 19:57:43 NobodyCam: we haven't discussed that, I'm okay with a "golden image" if it's possible 19:57:57 devananda: ok, now I think it is not so big problem as i thought before 19:57:57 lucasagomes: yeah, and frankly, we want systemd or upstart or any type of init system that can respawn an agent 19:58:03 fyi: two minute beep... *BEEP* 19:58:10 NobodyCam: I don't think there's any user-specific things in that image, but there may be 19:58:11 vkozhukalov: oh? i'm glad to hear that but curious what changed :) 19:58:11 lucasagomes: we never want to be in a situation where the agent can die without being respawned 19:58:18 devananda: devtest builds each time 19:58:27 JayF, heh yeah I think that quite important as well (/me likes systemd) 19:58:47 JayF: how many of your requirements would go away if you did not assume a long-lived agent? 19:58:51 lucasagomes: it's on a list of things I'd like to do, but frankly, it's hard to spend time on it when we have another solution working and other things between us and a working prototype 19:58:59 devananda: I just realized that our use cases could be covered with md 19:59:16 JayF: eg, "we want .. an init system that can respawn an agent" -- we can also just power cycle the machine :) 19:59:26 hmmmmmm 19:59:38 < 1 minutes 19:59:39 devananda: IMO software running as pid 1 should be an init system, not an agent, and I'm not in favor of any model where the agent, or some bash script that does a little more than spawn the agent, as pid 1 19:59:49 whether the agent is short or long lived 19:59:55 JayF, I see, yeah I can figure out that it's out of the scope for u guys now... but we should at least track it somewhere, since we have been using tripleO for geneting all the images we use 20:00:08 I see only one reason for a long-lived agent, which you guys presented previously -- save 1 POST cycle 20:00:09 it's just bad operational practice, and this will be running on more machines than ironic itself, at least in the environment I invision 20:00:30 *Beep* time 20:00:31 right -- time's up! thanks everyone! let's continue in -ironic as needed 20:00:38 thanks everyone 20:00:42 #endmeeting