16:01:02 <kozhukalov> #startmeeting Fuel
16:01:03 <openstack> Meeting started Thu Mar 12 16:01:02 2015 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:05 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:08 <openstack> The meeting name has been set to 'fuel'
16:01:16 <kozhukalov> #chair kozhukalov
16:01:17 <openstack> Current chairs: kozhukalov
16:01:24 <kozhukalov> hey guys
16:01:28 <ikalnitsky> o/
16:01:34 <kozhukalov> is anyone here?
16:01:34 <xarses> \o
16:01:39 <mihgen> hi
16:01:42 <brain461> hello
16:01:59 <kozhukalov> just 4 people
16:02:03 <agordeev> hi
16:02:13 <kozhukalov> ok let's start
16:02:39 <kozhukalov> if anyone has any announcements, please share them right now
16:02:56 <kozhukalov> before we go through the agenda
16:03:21 <daniel3_> hi
16:03:22 <kozhukalov> agenda as usual
16:03:29 <xarses> merge status?
16:03:32 <kozhukalov> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:04:05 <kozhukalov> xarses: feature by feature including merge status
16:04:15 <docaedo> hi
16:04:36 <kozhukalov> ok, looks like no one has any announcements
16:04:46 <kozhukalov> let's start discussing topics
16:04:55 <kozhukalov> #topic Separating MOS from upstream ubuntu (vparakhin)
16:05:09 <brain461> hello
16:05:10 <brain461> spec: https://review.openstack.org/#/c/148279/35/specs/6.1/separate-mos-from-linux.rst
16:05:11 <kozhukalov> brain461: could you please share the status
16:05:17 <brain461> Code is ready for review: https://review.openstack.org/#/c/161222/
16:05:21 <brain461> We're testing both features (separation + external ubuntu) on a set of cloned jobs in product CI (iso + bvt), there's an iso with latest patches.
16:05:26 <brain461> Currently, ddmitriev is working on implementing changes to system tests to consume external repos.
16:05:31 <brain461> Bad news: we're blocked by issues in vrouter feature.
16:05:35 <brain461> We're expecting to have vrouter fixes on review this evening.
16:06:15 <mihgen> vrouter fixes are in master?
16:06:22 <ikalnitsky> mihgen: nope
16:06:26 <ikalnitsky> they are on review
16:06:39 <mihgen> how comes you are blocked then?
16:06:52 <ikalnitsky> mihgen: default gateway
16:06:57 <kozhukalov> ikalnitsky: please give a link for vrouter
16:07:14 <ikalnitsky> currently, we have vrouter gateway on controllers, and that's an issue
16:07:32 <ikalnitsky> #link https://review.openstack.org/#/c/163408/
16:07:33 <ikalnitsky> here's the fix
16:07:42 <mihgen> currently we don't have such a thing as vrouter, there is public gateway propagated from UI, is not it?
16:07:59 <ikalnitsky> nope.
16:08:01 <mihgen> here we go, so it's in master
16:08:42 <ikalnitsky> in master, we already have vrouter feature and it works with issues, since it sets default gateway to vip__management, not to br-ex
16:08:56 <ikalnitsky> on controllers, which is wrong
16:09:11 <ikalnitsky> controllers MUST have br-ex as default gateway
16:09:14 <mihgen> yeah I see. We might want to revert it if it blocks from split repo / ubuntu thing
16:09:47 <ikalnitsky> we definetely should apply the fix above (which is revert, yeah)
16:10:01 <kozhukalov> it is better to fix this than to revert it at all
16:10:10 <ikalnitsky> revert is a fix
16:10:14 <ikalnitsky> it should be so
16:10:23 <ikalnitsky> it was wrong that the patch was applied
16:10:41 <xarses> ikalnitsky: why must controllers have a IP on br-ex:?
16:10:51 <xarses> this is bad from security side
16:10:58 <xarses> and I intend to have it removed
16:11:05 <mihgen> xarses: public gateway must be not from management
16:11:12 <mihgen> as I undestand it became now
16:11:18 <ikalnitsky> mihgen: true
16:12:01 <mihgen> ok folks you better voice sync with xenolog / vrouter lead and make a change.. I really don't want your feature to be blocked
16:12:21 <ikalnitsky> mihgen: we already do
16:12:21 <mihgen> we have to make local mirror script, etc.
16:12:51 <brain461> mihgen: yes, I started to work on it
16:13:47 <kozhukalov> ok, and what we need to make it fixed? reviewers? another voice call? anything else?
16:14:47 <kozhukalov> ok, let's move on
16:15:01 <kozhukalov> #topic Consume external ubuntu mirrors (ikalnitsky)
16:15:15 <ikalnitsky> today i managed to deploy cluster with external ubuntu and separate repos.
16:15:20 <ikalnitsky> obviously, it was required to apply vrouter fixes manually.
16:15:25 <ikalnitsky> ostf tests were passed.
16:15:32 <ikalnitsky> i've built an iso with all patches (including vrouter ones) and i want to ask qa to test it.
16:15:59 <ikalnitsky> so i think we should start merging some fixes one-by-one, in order to do not break master
16:16:05 <angdraug> can you post a link to the iso?
16:16:08 <ikalnitsky> though, we don't test inird downloading
16:16:20 <ikalnitsky> angdraug: yep, sure
16:16:35 <ikalnitsky> #link http://jenkins-product.srt.mirantis.net:8080/view/Repo%20Separation/job/6.1.all.clone/30/
16:16:41 <mihgen> can you share patch which downloads initrd pls?
16:17:01 <kozhukalov> mihgen: there couple of them on review
16:17:22 <kozhukalov> #link https://review.openstack.org/#/c/163599 script itself
16:17:25 <ikalnitsky> yeah, it really hard to find them right now.
16:17:34 <ikalnitsky> here's the link to working etherpad https://etherpad.openstack.org/p/consume-external-ubuntu
16:17:47 <kozhukalov> #link https://review.openstack.org/#/c/161640/  package for this script
16:18:08 <ikalnitsky> my iso includes all patches (even inird), so i don't know how it works. just going to figure out it.
16:18:15 <kozhukalov> #link https://review.openstack.org/#/c/161208/ download task
16:18:36 <mihgen> thx
16:18:50 <kozhukalov> ikalnitsky: how close we are to merging pre-provision tasks ?
16:19:22 <ikalnitsky> actually, i was thinking about merging it today.. but it didn't happen :(
16:19:30 <ikalnitsky> i've built and iso with these three patches
16:19:39 <ikalnitsky> but centos bvt has been failed
16:19:43 <ikalnitsky> we have some issues
16:19:44 <ikalnitsky> in master
16:19:48 <ikalnitsky> not in our patches
16:20:03 <ikalnitsky> but i want to pass them both bvts
16:20:17 <kozhukalov> but master is green, isn't it?
16:20:23 <ikalnitsky> nope
16:20:48 <ikalnitsky> oh, it's now green.
16:20:52 <salmon_> hi
16:20:54 <ikalnitsky> few hours ago, it wasn't
16:21:55 <mihgen> thx ikalnitsky. Any blockers here?
16:22:05 <mihgen> what is left to be done?
16:22:17 <kozhukalov> so, we can build another iso with pre-provision tasks and check them again, right?
16:22:43 <mihgen> can't we just go shortcut here, take green ISO and inject your changes?
16:22:54 <ikalnitsky> mihgen: i don't see any blockers. i think we should test the latest iso and start merging things.
16:22:55 <mihgen> verifiy if it works, and then merge?
16:23:09 <kozhukalov> mihgen: we have merging plan which is on this etherpad ^^^
16:23:19 <ikalnitsky> kozhukalov: that's right. i'll run it again.
16:23:30 <ikalnitsky> mihgen: sorry, what do you mean?
16:23:46 <ikalnitsky> first of all, we should merge pre-provision tasks
16:23:52 <ikalnitsky> they shouldn't break master
16:24:04 <ikalnitsky> they will just build provisioning images on master node
16:24:27 <ikalnitsky> patches were reviewed and tested manually
16:24:37 <ikalnitsky> we need just get bvt passed and merge them
16:24:54 <kozhukalov> ok, i think the details of the merging order is up to us, just need to keep master green
16:25:06 <mihgen> ok thanks guys
16:25:09 <bookwar> to get bvt passed we need patches to fuel-qa to be merged as well, don't we?
16:25:21 <brain461> sure
16:25:40 <kozhukalov> moving on?
16:25:41 <ikalnitsky> nope
16:25:44 <ikalnitsky> not for this patches
16:25:46 <ikalnitsky> bookwar: ^
16:26:06 <fletcher> any of the bandit peeps around?
16:26:06 <bookwar> ikalnitsky: ok
16:26:08 <mihgen> bookwar: hi, do we have mirrors of ubuntu prepared in our infra in every location?
16:26:27 <bookwar> mihgen: yes, upstream mirrors are ready
16:26:37 <mihgen> cool
16:27:14 <kozhukalov> #topic IBP (agordeev)
16:27:41 <agordeev> hi
16:27:50 <kozhukalov> agordeev: hi
16:27:53 <agordeev> merge status of blueprints:
16:27:55 <agordeev> 1 was landed https://blueprints.launchpad.net/fuel/+spec/ibp-build-ubuntu-images and the remaining 2 was not.
16:27:57 <agordeev> https://blueprints.launchpad.net/fuel/+spec/ibp-reconnect : 1 patch is waiting to be reviewed and merged https://review.openstack.org/157090
16:27:59 <agordeev> https://blueprints.launchpad.net/fuel/+spec/ibp-image-checksums : 2 patches are waiting to be reviewed and merged https://review.openstack.org/#/c/161318/ https://review.openstack.org/#/c/161759/
16:28:01 <agordeev> I'll ask for getting FFE for 1d for them right after the meeting ends
16:28:03 <agordeev> bugs:
16:28:05 <agordeev> 1 critical was filed https://bugs.launchpad.net/fuel/+bug/1428759 looks like it was just h/w issue
16:28:07 <openstack> Launchpad bug 1428759 in Fuel for OpenStack "[IBP] degraded RAID prevents operating system from boot" [Critical,Confirmed] - Assigned to Fuel provisioning team (fuel-provisioning)
16:28:07 <agordeev> backports to 6.0:
16:28:09 <agordeev> I'm insisting on backporting of ibp-reconnect. Even IBP for 6.0 is in experimental status. Additionaly it closes on of customer-found bugs
16:28:11 <agordeev> https://review.openstack.org/#/c/161721/
16:28:13 <agordeev> https://review.openstack.org/#/c/161722/
16:28:15 <agordeev> other:
16:28:17 <agordeev> it's time to start re-implementing ibp-build-ubuntu-images as a pure python tool.
16:29:56 <kozhukalov> agordeev: thanks and yet another issue about ibp
16:30:02 <mihgen> thanks agordeev. anything prevents us from merging reconnects to master now?
16:30:44 <kozhukalov> agordeev: provision progress bar does not work and taking into account that we are going to spend ~10mins building images just before starting provisioning
16:30:45 <agordeev> mihgen: i think that nothing is blocking it. Just needs to be reviewed and merged.
16:31:02 <mihgen> about backport to stable/6.0: I think we need to take a closer look on it. I see dependency introduced (urllib3), so I'm trying to understand how risky it can be
16:31:09 <kozhukalov> we definitely need progress bar to be working
16:31:26 <mihgen> we generally don't backport such things of course, I'm just trying to see if we can make exception here
16:31:35 <mihgen> and if we really need to
16:32:43 <kozhukalov> guys, please take part in reviewing IBP patches if you have some time
16:32:55 <kozhukalov> patches are pretty small
16:33:18 <agordeev> mihgen: that dependecy is already installed into bootstrap image. As far as netchecker tool requires it.
16:33:21 <kozhukalov> we'd really appreciate this if you had a chance to take a look at them
16:34:19 <kozhukalov> ok, if there are no other q, moving on then
16:34:35 <kozhukalov> #topic Plugins (evgeniyl)
16:34:41 <evgeniyl___> Hi guys.
16:34:48 <kozhukalov> evgeniyl___: hi
16:34:56 <evgeniyl___> All plugins related features are merged, currently we are working on the docs.
16:35:06 <evgeniyl___> Today we’ve done plugins migration guide.
16:35:12 <evgeniyl___> #link https://wiki.openstack.org/wiki/Fuel/Plugins#How_to_migrate_plugins_from_1.0.0_to_2.0.0_package_version
16:35:43 <evgeniyl___> Also I’ve just had a discussion with guys who develop melanox plugin, they really need plugable bootstrap image, we’ve figured out a solution, I’m not sure if it will be able to implemented it in the current release.
16:36:17 <mihgen> Q: about an issue with 2+ plugins, what is the order nailgun chooses to install those?
16:36:19 <evgeniyl___> Another story is ability to define plugins execution order, I need approve to start working on it, if we are going to do it as a feature freeze exception.
16:36:41 <mihgen> evgeniyl___: yep, this is my question
16:37:16 <evgeniyl___> mihgen: are you asking about execution order? Afaik currently it depends on the name.
16:37:30 <mihgen> evgeniyl___: name of what?
16:37:35 <evgeniyl___> mihgen: of plugin
16:37:54 <evgeniyl___> mihgen: we don't have proper mechanism to define plugins execution order.
16:37:58 <mihgen> so there is a workaround, right ;)
16:38:28 <evgeniyl___> mihgen: Oh, I don't think that we should call it a workaround.
16:38:42 <mihgen> where is the issue comes from?
16:38:53 <mihgen> contrail + zabbix = wrong order?
16:39:12 <mihgen> according to you, zabbix would be installed after contrail, so it should be just fine
16:39:14 <evgeniyl___> mihgen: it's not general solution
16:39:28 <evgeniyl___> mihgen: if we want to solve specific case, yes we can use workaround
16:39:51 <mihgen> I get that. So what's your estimate and approach to possibly resolve it in 6.1. ?
16:39:52 <evgeniyl___> mihgen: yeah, we should check it, but it should work this way
16:40:03 <angdraug> solving it would require metadata format change (exec priority field)
16:40:19 <angdraug> wouldn't it be better to make such change early rather than later worry about backwards compatibility?
16:40:24 <evgeniyl___> mihgen: one week including spec, fpb fixes, nailgun fixes, and merging process with all of the reviews.
16:41:30 <mihgen> this is pretty serious gap, actually, frankly speaking…
16:42:06 <mihgen> if we don't want people to name their roles like "A_Zabbix", "B_Contrail", etc., we need to fix it..
16:42:06 <kozhukalov> evgeniyl___: if you have managed to find a solution on pluggable bootstrap, can you share you opinion about this? how is it going to look like?
16:42:10 <evgeniyl___> It's my current velocity, if we take into account that the most of the day I spend on side tasks, reviewes, docs, meetings
16:43:01 <angdraug> are you the only one who can do this?
16:43:10 <angdraug> bus factor alert...
16:43:38 <evgeniyl___> kozhukalov: plugins can share some hooks, and when node is booted, it can iterate through plugins hooks and execute them. In one of the hook guys from mellanox can add kernel module loading.
16:43:56 <evgeniyl___> angdraug: I'm not the only person who can do that
16:44:14 <evgeniyl___> agordeev: for example Vladimir Sharshov in the current release implemented reboot tasks from end to end.
16:44:36 <xarses> evgeniyl___: mihgen https://bugs.launchpad.net/fuel/+bug/1428427
16:44:38 <openstack> Launchpad bug 1428427 in Fuel for OpenStack "Priority of plugins execution can't be defined" [Medium,Invalid] - Assigned to Fuel Python Team (fuel-python)
16:44:56 <mihgen> thanks evgeniyl___. if you are available after this meeting, I'd like to do a short call, together with dpyzhov_
16:45:08 <evgeniyl___> mihgen: np
16:45:17 <dpyzhov_> ok
16:45:23 <xarses> it was marked invalid
16:45:33 <evgeniyl___> xarses: yes, because it's not a bug.
16:45:38 <xarses> sort of
16:45:39 <evgeniyl___> xarses: it's a feature request
16:45:42 <xarses> it's a defect
16:45:51 <evgeniyl___> xarses: it's known constraint
16:45:53 <kozhukalov> a flaw
16:46:26 <kozhukalov> evgeniyl___: thanks a lot
16:46:34 <xarses> kozhukalov: same meaning effectively
16:46:35 <kozhukalov> moving on to open discussion
16:46:45 <kozhukalov> xarses: sure
16:47:25 <kozhukalov> xarses: i just started to think we are going to list synonyms to the word bug )
16:47:42 <kozhukalov> #topic Open discussion
16:47:50 <salmon_> I have something
16:47:53 <salmon_> Small update for 200 nodes bp
16:47:53 <salmon_> there are two patches not merged yet.
16:47:53 <salmon_> One is about allowing to fail nodes during provision. I'm refactoring it after evgeniyl___ comments.
16:47:53 <salmon_> Second one is network verification. There is some dissagreement about backward compatibility https://review.openstack.org/#/c/138760/6 :( I think that we should just add warrning that it's not backward compatible and desribe in docs what and how to upgrade to fix it.
16:48:49 <evgeniyl___> salmon_: I agree to merge it, but lets create a bug, and start the discussion in ML to solve the problem, that we can break it in the future releases
16:49:13 <agordeev> salmon_: just curious to know, have your guys tried 200 nodes against IBP ?
16:49:16 <evgeniyl___> salmon_: also "Roman said that backward compatibility is not a issue for him" doesn't look like a valid argument
16:49:22 <xarses> why cant we fix it in a backward way?
16:49:25 <evgeniyl___> salmon_: user care about it
16:49:26 <salmon_> agordeev: no, only on 100
16:49:57 <mihgen> salmon_: I'm curios if we can make it backward compatible by 'small blood' ?
16:49:59 <salmon_> evgeniyl___: well, the qustion is if user will ever notice it
16:50:05 <evgeniyl___> salmon_: and I didn't say that I don't want to merge it just because, I provided a technical solution how to solve the problem
16:50:33 <salmon_> evgeniyl___: yeah, this is why I gave a link to patch
16:50:35 <evgeniyl___> salmon_: I think user can easily notice that network checker is broken for old releases
16:50:47 <salmon_> mihgen: if user updates some packeges it will work
16:51:03 <evgeniyl___> mihgen: but he should manually get required packages
16:51:10 <salmon_> yup :/
16:51:12 <evgeniyl___> mihgen: or we should backport it to all previous releases
16:51:48 <mihgen> folks, technically we can make everything backward compatible. But if it takes from us SO many resources to get done, I'm inclined to think that we should just document it / come up with any other workaround
16:51:49 <evgeniyl___> mihgen: which is not the case I think, because it brakes the previous interface, and it doesn't look like a good candidate for backport
16:52:09 <mihgen> so here, if it's easy to make backward compatible, why not to do so?
16:52:37 <evgeniyl___> mihgen: guys, we implement backward compatibility on almost every layer now, I don't see problems to add the same stuff into MCagents.
16:52:44 <salmon_> it takes some time and make everything look uglier. Updating package is simpler for me :)
16:53:28 <salmon_> I changed mcollective-agent so user has to updata package on each node
16:53:54 <mihgen> frankly UX doesn't look like as great one. Let's say you had Fuel master with N nodes in bootstrap
16:54:08 <mihgen> then upgraded fuel master, and added more bootstraps
16:54:26 <salmon_> mihgen: if you have bootsrap nodes, just reboot them
16:54:36 <mihgen> and now your net-verify fails for some of the nodes, and you are lost, as you get unclear error messages
16:55:48 <salmon_> evgeniyl___: can we get agent version? If yes it's easy to make it bacward compatible
16:55:50 <kozhukalov> i think salmon_ is right, if you have bootstrap nodes, just reboot them. why can't we reboot those bootstrap nodes just after upgrade?
16:56:09 <salmon_> good idea
16:56:28 <salmon_> ot just add a notice that is good idea to reboot
16:56:32 <salmon_> *or
16:56:47 <mihgen> problem is that no one reads all those long release notes. .
16:57:00 <evgeniyl___> salmon_: I don't know, we should check it and look at the api. But anyway as I said, there is another workaround, just implement "version" method and if it's not there, it means that it's old version.
16:57:09 <xarses> so you plan on making a feature to reboot un-used discovered nodes when we update the master, or the base image?
16:57:14 <salmon_> mihgen:  it can be just above error message after faile verification
16:57:17 <kozhukalov> 3 minutes
16:57:43 <mihgen> salmon_: really if it's easy to make, we need backward compatibility layers anyway, please consider it
16:58:05 <mihgen> if you see it takes too long .. then just release notes ..
16:58:06 <salmon_> ok, I will start discussion on ML
16:58:10 <mihgen> thx
16:58:30 <evgeniyl___> salmon_: thanks
16:58:36 <kozhukalov> ok, guys, thanx everyone for attending
16:58:42 <kozhukalov> time to end
16:58:48 <salmon_> thx
16:59:02 <kozhukalov> #endmeeting