16:00:34 <xarses> #startmeeting fuel
16:00:35 <xarses> #chair xarses
16:00:35 <xarses> Todays Agenda:
16:00:35 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:00:35 <xarses> Who's here?
16:00:37 <maximov> hi
16:00:38 <openstack> Meeting started Thu Jan 28 16:00:34 2016 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:39 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:40 <mwhahaha> hi
16:00:41 <openstack> The meeting name has been set to 'fuel'
16:00:42 <openstack> Current chairs: xarses
16:00:46 <salmon_> hi
16:01:10 <ashtokolov> hi
16:01:12 <rmoe> hi
16:01:28 <mihgen> hi
16:01:28 <kozhukalov> hi
16:01:44 <fzhadaev> Hi!
16:01:56 <SheenaG> \o/
16:01:57 <akislitsky_> hi
16:02:18 <xarses> lets get started then
16:02:33 <yottatsa> hi
16:02:35 <xarses> #action items from last meeting
16:02:44 <xarses> doh
16:02:52 <xarses> #topic action items from last meeting
16:02:58 <xarses> SheenaG will follow up with GregE regarding desired UI elements from NFV features work in 9
16:03:15 <mattymo> hi
16:03:27 <SheenaG> Done - met with Dmitry and DP today to discuss, following up with DP on Monday re: minimum UI possible
16:03:35 <SheenaG> Dmitry will provide estimates of dates for API availability
16:03:45 <SheenaG> Then UI team will scope work and determine if it's reasonable or not
16:03:48 <warpc> hi
16:03:53 <bgaifullin_> hi all.
16:04:16 <xarses> thanks
16:04:19 <xarses> SheenaG bookwar will follow up on creating a scenario/policy for vendor specific code
16:04:44 <SheenaG> Had a meeting this morning, most of the details are ironed out - follow up for QA on who will monitor BVT failures on vendor specific ISO
16:05:41 <sbog> hi
16:06:01 <SheenaG> xarses: next
16:06:05 <xarses> so we have identified who will create the scenario/policy?
16:06:41 <SheenaG> xarses: bookwar_ has it all outlined, will send to mos-dev with follow up for QA
16:07:13 <xarses> ok, thanks
16:07:15 <xarses> ogelbukh will start moving configdb code over to openstack
16:07:23 <bookwar_> xarses: we working on details with vkramskikh and ikalnitsky
16:08:25 <xarses> ogelbukh: ?
16:08:32 <xarses> thanks bookwar_
16:08:56 <xarses> moving on then
16:09:19 <xarses> #topic UI Team status (vkramskikh)
16:09:27 <vkramskikh> Hi! This week there was only 1 High UI bug for 8.0 ( https://bugs.launchpad.net/fuel/+bug/1538757 ). It's not fixed yet, but is going to be fixed soon.
16:09:28 <openstack> Launchpad bug 1538757 in Fuel for OpenStack 8.0.x "[fuelmenu] Default gateway set to None on 'Apply' action" [High,Incomplete] - Assigned to Matthew Mosesohn (raytrac3r)
16:09:28 <vkramskikh> We've started working on 9.0 features:
16:09:28 <vkramskikh> 1) https://blueprints.launchpad.net/fuel/+spec/converge-to-eslint-config-openstack - mostly done, some minor patches are expected. The spec is ready to be merged.
16:09:28 <vkramskikh> 2) https://blueprints.launchpad.net/fuel/+spec/redesign-of-node-roles-panel - in progress, probably we'll change the scope to introduce role groups for better UX and fix long-living bug-feature https://bugs.launchpad.net/fuel/+bug/1375750
16:09:30 <openstack> Launchpad bug 1375750 in Fuel for OpenStack mitaka "Controller CPU/RAM are counted as cloud's capacity" [Medium,Confirmed] - Assigned to Fuel UI Team (fuel-ui)
16:09:30 <vkramskikh> 3) https://blueprints.launchpad.net/fuel/+spec/network-requirements-popup - development is in progress
16:09:32 <vkramskikh> 4) https://blueprints.launchpad.net/fuel/+spec/node-display-ip-address - design is in progress
16:09:35 <vkramskikh> 5) Ability to show all network groups - part of https://blueprints.launchpad.net/fuel/+spec/multirack-in-fuel-ui which wasn't done in 8.0 - development is in progress
16:09:38 <vkramskikh> 9.0 features on hold:
16:09:42 <vkramskikh> 1) https://blueprints.launchpad.net/fuel/+spec/remove-vendor-code - waiting for infra
16:09:44 <vkramskikh> 2) https://blueprints.launchpad.net/fuel/+spec/separate-fuel-ui-repo - waiting for kozhukalov, who is busy with docker removal
16:09:47 <vkramskikh> 3) NVF features - scoping is still in progress
16:09:49 <vkramskikh> Questions?
16:10:17 <angdraug> can you comment on remove-vendor-code?
16:10:30 <vkramskikh> what do you want to know?
16:10:43 <vkramskikh> as bookwar_ said, we're still discussing branching strategy
16:11:03 <angdraug> do I understand it right that it's waiting on conclusion of the policy discussion between you, ikalnitsky, and bookwar?
16:11:13 <vkramskikh> yes
16:11:23 <angdraug> ok, making sure there's nothing else pending from infra here
16:11:40 <angdraug> thanks
16:12:06 <xarses> #topic Enhancements Team status (ashtokolov)
16:12:19 <ashtokolov> We are working on:
16:12:34 <ashtokolov> 1. Polishing task-based deployment engine and we are going to merge it next week
16:12:40 <ashtokolov> 2. Gracefully stop Deployment and further restart it
16:12:46 <ashtokolov> 3. Deployment Tasks idempotency with Fuel Mixed Team
16:13:07 <ashtokolov> That's all
16:13:31 <ashtokolov> Questions?
16:14:03 <maximov> what about bugs related to task based deployments
16:14:04 <xarses> is the new task engine enabled by default in 8?
16:14:39 <ashtokolov> maximum, there are 3 bugs, we are going to fix all of them on this week
16:14:52 <maximov> ashtokolov: ok
16:15:00 <angdraug> gracefully stop: is that referring to the problem of stopping a deployment of changes over an existing env?
16:15:05 <ashtokolov> xarses, nope, 8.0 - experimental only, 9.0 - by default
16:15:30 <ashtokolov> angdraug, no, it's only for task-based and LCM
16:16:12 <ashtokolov> Graceful stop can be enabled by task idempotency
16:16:13 <xarses> does this mean that we are also going to be able to (eventually) resume after the first failure?
16:16:31 <angdraug> #link http://lists.openstack.org/pipermail/openstack-dev/2016-January/084700.html
16:16:54 <angdraug> please make sure you take the thread ^ into account
16:17:23 <ashtokolov> xarses, it means that we will rerun puppets w/o reprovisioning for all node, which were stopped gracefully
16:17:44 <ashtokolov> and re provision nodes with failed tasks
16:18:11 <ogelbukh_> ashtokolov: re-run from the point where it was stopped?
16:18:16 <ashtokolov> angdraug, there is no way to fix it with granular deployment
16:18:57 <ashtokolov> ogelbukh_ from the beginning
16:19:13 <xarses> re-provison failed nodes? wait what?
16:19:18 <ogelbukh_> ashtokolov: good, thank you
16:19:22 <angdraug> ashtokolov: just make sure you don't break it further :)
16:19:52 <angdraug> if you can't fix it, stop button should remain disabled for production envs
16:20:06 <angdraug> moving on?
16:20:14 <xarses> #topic idempotency (ashtokolov)
16:20:36 <ashtokolov> xarses, if puppet finished with error, there is no way to be sure that rerun can help
16:21:24 <ashtokolov> as I mentioned above we are working on Deployment Tasks idempotency with Fuel Mixed Team
16:22:09 <xarses> moving then?
16:22:10 <mwhahaha> well it might be able to be fixed if someone typoed a setting or ip address, but there's no guarantee that rerunning would fix it
16:22:28 <ashtokolov> first step is to rerun all tasks twice and checks how it can brake the env
16:23:08 <xarses> #topic NFV status and specs (yottatsa)
16:23:16 <yottatsa> I’ve finished with specs for SR-IOV and DPDK features, as well as skolekonov done spec for NUMA/CPU pinning. Pls review. Design is almost established.
16:23:16 <yottatsa> Packaging is done and we’re ready for merging it in 9.0.
16:24:00 <xarses> yottatsa: do we need to backport anything for 9-kilo?
16:24:30 <yottatsa> smatov is working on backport question
16:25:09 <xarses> ok, thanks
16:25:17 <xarses> #topic Telco Team Status (fzhadaev)
16:25:23 <fzhadaev> Fuel Telco Team Status:
16:25:23 <fzhadaev> 1. One of the main activities is fixing of bugs for 8.0
16:25:23 <fzhadaev> In progress: 2
16:25:23 <fzhadaev> Done: 7
16:25:23 <fzhadaev> 2. Actions related to 9.0:
16:25:24 <fzhadaev> 2.1 We're continue scoping NFV-related features. For now the spec for NUMA/CPU Pinning is in review state[1] and the spec for Huge Pages is in progress (soon will be on review).
16:25:24 <fzhadaev> 2.2 We've started scoping one of tech-debt items related to implementing support of cgroup containment for OpenStack and other services in order to not affect other services. There is no blueprint for this activity yet..
16:25:25 <fzhadaev> [1] https://review.openstack.org/#/c/273043/4/specs/9.0/support-numa-cpu-pinning.rst
16:25:25 <fzhadaev> That's all. Any questions?
16:26:37 <xarses> I'm assuming your working with yottatsa on the NFV features?
16:27:18 <yottatsa> xarses fzhadaev yep
16:27:29 <fzhadaev> yes )
16:28:13 <angdraug> moving?
16:28:14 <xarses> #topic UCA Mitaka status (mattymo)
16:28:24 <mattymo> oh I thought I put bugfix first. Let's do UCA
16:28:48 <mattymo> I sent out a mail today regarding successful deployment of Mitaka b1 through bvt
16:29:01 <mattymo> http://lists.openstack.org/pipermail/openstack-dev/2016-January/085190.html
16:29:11 <angdraug> nice work!
16:29:16 <xarses> WOOO!
16:29:29 <mattymo> Some challenges are apt pins because there are haproxy and ceph packages there. Our config doesn't work with newer ceph, unfortunately
16:29:39 <ogelbukh_> that's awesome, mattymo
16:30:05 <xarses> o.O
16:30:08 <mattymo> The only openstack bug we haven't fixed is one that reproduces in devstack. nova service-list reports extra services that don't actually report any status. osapi_compute and metadata. The services really do work, but they report XXX state. rpodolyaka is going to work on fixing that upstream
16:30:45 <mattymo> bookwar_, will set up a regular job so we can gate against UCA Mitaka starting next week to make sure we don't introduce regressions
16:30:48 <xarses> mattymo: please send me a bug on the ceph issue
16:31:00 <angdraug> have you overcome the apt pin challenges for haproxy and ceph or is that still a problem?
16:31:15 <mattymo> xarses, I don't have one yet... I was more racing to get it deployed than to figure out exactly how it broke. I have time now to give you a good report, however. Expect it tomorrow
16:31:23 <mattymo> the pins aren't hard to set up, no
16:31:38 <xarses> ok thanks, this is fantastic anyways
16:31:38 <angdraug> ok good, so you're not blocked on that
16:31:55 <mattymo> b1 landed on Jan 20, midway through testing, so it was funny to see it break
16:31:56 <angdraug> cores, please get this reviewed and merged asap
16:32:07 <angdraug> you mean b2?
16:32:33 <mattymo> https://review.openstack.org/#/q/topic:puppet-mitaka iberezovsky is coordinating all the puppet fixes, actually. I had a lot of help from him. We're hoping to land all the mitaka manifests by tomorrow
16:34:14 <mihgen> great work guys, thank you
16:34:24 <xarses> please keep in mind, we want to avoid patching the openstack module
16:35:03 <mattymo> xarses, there are very few of those
16:35:08 <mattymo> We can move on to bugfixing I believe
16:35:13 <xarses> #topic Bugs team status (mattymo, akislitsky) https://etherpad.openstack.org/p/fuel-bugs-status
16:35:21 <mattymo> I won't spam you this week. Breakdown of open bugs for Fuel 8.0 is here: https://etherpad.openstack.org/p/fuel-bugs-status
16:35:40 <mattymo> There are 39 critical/high bugs open. The majority is in Python team. In Library side (where I am), bugfix team owns only a handful of what's open, so we're mainly working on bugs confirmation and then medium 9.0 bugs.
16:35:52 <mattymo> We have 2 very serious library side bugs. One is regarding ntpdate and the other regarding openstack client and keystone availability.
16:36:02 <mattymo> https://bugs.launchpad.net/fuel/+bug/1533082 which sbog is working on
16:36:03 <openstack> Launchpad bug 1533082 in Fuel for OpenStack 8.0.x "[BVT] Deployment failed with Failed to execute hook 'sync_time' command: cd / && ntpdate -u" [Critical,In progress] - Assigned to Stanislaw Bogatkin (sbogatkin)
16:36:44 <mattymo> this is the other: https://bugs.launchpad.net/fuel/+bug/1539117
16:36:45 <openstack> Launchpad bug 1539117 in Fuel for OpenStack mitaka "Keystone api sporadically stops answering" [Critical,Confirmed] - Assigned to MOS Puppet Team (mos-puppet)
16:36:45 <mihgen> holser_: I thought you merged a fix to this one ^^^ ?
16:36:57 <mattymo> the second is keystone api availability is really quite poor, even on a single controller deployment
16:37:06 <holser_> The fix didn’t help a lot
16:37:06 <mattymo> it causes random deployment failures
16:37:11 <aglarendil> mihgen: this is completely diffferent bug
16:37:13 <holser_> though we produced another fix
16:37:36 <mihgen> I meant for 1533082
16:38:13 <aglarendil> okay, so it did not help much, I changed some of the args of timeout command to make it kill ntpdate in a less gentle manner
16:38:19 <aglarendil> mihgen: ^^
16:38:37 <mihgen> :(
16:38:55 <mihgen> it's still regression folks, as we had may be not ideal but better situation before
16:39:03 <mihgen> so may be we increased a load to VMs?
16:39:23 <mihgen> may be we can mitigate it now just by increasing resources for CI? And switch to sntp in 9.0?
16:39:53 <holser_> yep for #1533082
16:39:56 <aglarendil> mihgen: we do not have RCA right now. it seems an ntpdate bug which might have been triggered by underlying OS changes, e.g. security or kernel updates
16:40:39 <aglarendil> mihgen: it is just the start of the deployment, there is no significant load so far
16:40:53 <mihgen> (( let's keep troubleshooting then...
16:41:01 <xarses> moving?
16:41:35 <xarses> #topic review queue bit.ly/1RD6JLR (mihgen)
16:41:45 <mihgen> Just wanted to highlight this. For fuel-specs repo, holser, ikalnitsky, can you take an action item and ensure that subject matter experts review those?
16:41:45 <mihgen> nurla - please take a look at fuel-qa patches
16:42:02 <ikalnitsky> mihgen: sure
16:42:03 <mihgen> kozhukalov: there is a patch to fuel-agent too
16:42:12 <nurla> mihgen: sure
16:42:25 <mihgen> that's it for this one
16:42:33 <xarses> #topic Fuel/Solar integration workshop: https://etherpad.openstack.org/p/fuel-solar-workshop-poznan-jan-16 (mihgen)
16:42:43 <mihgen> It is hard to tell in a few words what have been discussed. We'd still need to make some of a decisions. So overall we've discussed:
16:42:49 <mihgen> 1) Issues which we have with current Fuel data flow (astute.yaml, can't easy extend/modify data, etc.)
16:42:50 <mihgen> 2) ConfigService proposal and how it may solve issues. To my view, it's in many ways similar to Solar resources + plugable Data Processors
16:42:58 <mihgen> 3) Solar concepts and appoach to solving issues outlined presented
16:42:58 <mihgen> 4) Problems with the approach identified, which needs to be resolved -
16:43:06 <mihgen> a) If we consider Fuel Managers, aka Network Manager, Volume Manager, etc. as having their own DBs, then there will be problem of syncronization of data with SolarDB. Or, we'd have to acknoledge that there will be no way to rollback configuration of environment, and we can move only forward with changes.
16:43:07 <mihgen> b) it's not yet fully clear what component and how will be creating Solar resources (we call it Policy Engine, but there is no clear definition for it yet)
16:43:34 <mihgen> so it's still early to have any final decisions, but I think we are doing a good progress overall
16:43:37 <angdraug> no rollback sounds like a no-go to me
16:43:47 <mihgen> in understanding how Solar+Fuel would work together
16:44:00 <mihgen> angdraug: it may be a hard-to-believe reality ;)
16:44:52 <mihgen> #link  https://etherpad.openstack.org/p/fuel-solar-workshop-poznan-jan-16
16:44:59 <aglarendil> mihgen: distributed transactional coordination engine is a must have in this architecture
16:45:02 <mihgen> so you can take a look for some notes there
16:45:03 <YorikSar> angdraug: rollback will have to be implemented as some external script. No single component (even Solar) can contain enough knowledge to handle rollback.
16:45:06 <aglarendil> it is hard to write such things
16:45:11 <ogelbukh_> no rollback is ok as long as there's restore
16:45:22 <xarses> it won't be
16:45:29 <aglarendil> but it is what we need to do
16:45:36 <xarses> it will have to be integrated
16:45:40 <angdraug> +1 to aglarendil
16:45:59 <aglarendil> although some of things are not rollbackable
16:46:00 <xarses> getting rollback in the normal workflow is a blocker
16:46:10 <aglarendil> e.g. if you actually zero out data on the disk
16:46:16 <aglarendil> there is no way back
16:46:34 <xarses> reprovision the node with the old settings =)
16:46:39 <aglarendil> but IP allocation or simpler things are important to get rollbacked at least from business logic point of view
16:46:41 <xarses> yes, data is gone
16:46:48 <YorikSar> aglarendil: We need to see user stories that are left uncovered before introducing distributed transactions.
16:46:55 <salmon_> rollback sometimes may be possible. It depends from where we get a data which needs to be reverted
16:46:56 <ogelbukh_> that's not rollback, xarses :) that's revert or restore
16:47:22 <aglarendil> YorikSar: I frankly cannot understand the link between them
16:47:25 <angdraug> backup/restore is ok as a fallback, not as the only way to undo a change of a single line in nova.conf
16:47:30 <xarses> as far as the operator is concerened, they have to be able to un-do or go back to old settings
16:47:35 <pigmej> aglarendil: even with distributed transactional engine you may end with a state that you can't rollback. With multiple sources of data you're in troubles anyway
16:47:35 <aglarendil> I think we need to move this hot topic to ML
16:47:44 <xarses> aglarendil: +1
16:47:56 <mihgen> we'll try to collect all the pros/cons
16:47:58 <aglarendil> pigmej: I may in troubles out of the box even with one source of data
16:48:12 <pigmej> yeah but then it's different story
16:48:14 <xarses> lets move on then
16:48:18 <xarses> #topic Mixed Team Status (asaprykin)
16:48:19 <mihgen> it's gonna be quite hard to discuss over email, frankly, as it's complicated even near whiteboard
16:48:38 <asaprykin> Mixed team is working on bugfixing and reviews.
16:48:38 <asaprykin> Currently we have 3 high priority bugs related to 8.0.
16:48:38 <asaprykin> For https://bugs.launchpad.net/fuel/+bug/1538526 (zynzel) patch is ready and on review. It's being tested by QA team and will be merged after it's done.
16:48:39 <asaprykin> https://bugs.launchpad.net/fuel/+bug/1536314 and https://bugs.launchpad.net/fuel/+bug/1538226 are on review as well.
16:48:41 <asaprykin> Estimated time when patches will be merged is Monday \ Tuesday.
16:48:41 <openstack> Launchpad bug 1538526 in Fuel for OpenStack mitaka "Openstack-config missing relationships for new options" [High,In progress] - Assigned to Bartosz Kupidura (zynzel)
16:48:42 <openstack> Launchpad bug 1536314 in Fuel for OpenStack mitaka "Verify network fails after backup-reinstall-restore Fuel" [High,In progress] - Assigned to Peter Zhurba (pzhurba)
16:48:43 <openstack> Launchpad bug 1538226 in Fuel for OpenStack "Master node migration failed with on 'lvcreate os -n root -L 9,77g'" [High,In progress] - Assigned to Peter Zhurba (pzhurba)
16:49:06 <asaprykin> Questions?
16:49:09 <xarses> #action aglarendil raise rollback support in solar to the ML for further discussion
16:49:12 <ogelbukh_> mihgen: I guess this cannot be discussed in an abstract manner, we need to be much more concrete on what we gonna do and how
16:50:03 <ogelbukh_> 1536314 - does it relate to dockerctl backup/restore?
16:50:09 <ogelbukh_> asaprykin: ^^
16:51:32 <asaprykin> Yes, it is related to dockerctl backup/restore
16:51:57 <ogelbukh_> does it even make sense to fix if we are dropping docker in 9.0?
16:52:26 <mihgen> 8.0 still have docker
16:52:28 <asaprykin> It should be backported to 8.0
16:52:39 <ogelbukh_> it says mitaka in bug report
16:52:43 <ogelbukh_> k, I see
16:53:10 <xarses> #topic Network Team status (alex_didenko)
16:53:53 <ogelbukh_> we have troubles with it in 7.0 too, are you going to backport it there as well?
16:54:19 <alex_didenko> We're working on bugs. Also we're in the process of deprecating and removing old "nodes" list. This list does not take into account network roles and templates so pulling information from it is not reliable. We should pull info from "network_metadata" instead.
16:54:20 <alex_didenko> It's already removed from fuel-library, all the corresponding functions are updated as well. Removing it from nailgun is in TODO.
16:54:20 <alex_didenko> We're also working on a bunch of tricky bugs, running a lot of discussions on them in order to find the best solution. Some of those discussions are on ML already so feel free to join :)
16:55:20 <alex_didenko> That's it, thanks
16:55:28 <xarses> #topic Upgrades team status (ogelbukh)
16:55:34 <asaprykin> ogelbukh_: AFAIK it's about 8.0 currently. Need to clarify this information.
16:55:44 <xarses> alex_didenko: thanks!
16:56:04 <ogelbukh_> we've completed backup part, working on restore on 8.0
16:57:00 <ogelbukh_> some issues in astute's puppet manifest, nothing blocking
16:57:33 <ogelbukh_> I also wanted to ask for revivews on bunch of upgrade specs for 9.0 and further
16:57:44 <angdraug> links?
16:58:12 <mihgen> ogelbukh_: you'd need to coordinate with holser_ & ikalnitsky on who should be assigned from subject matter experts..
16:58:18 <ogelbukh_> got some excellent feedback already from xarses and mwhahaha
16:58:38 <ogelbukh_> mihgen: yes, we do, and will go on
16:59:17 <xarses> 1 min
16:59:27 <angdraug> configdb?
16:59:36 <xarses> #topic Configdb status (ogelbukh, ytaraday)
16:59:45 <ogelbukh_> angdraug: https://review.openstack.org/#/q/status:open+project:openstack/fuel-specs+owner:%22Oleg+Gelbukh+%253Cogelbukh%2540mirantis.com%253E%22 - spec links
16:59:52 <YorikSar> We've been and will be discussing it with Solar team here.
17:00:13 <YorikSar> It's unclear if it's still needed if we target Solar, that's yet to be discussed..
17:00:39 <angdraug> thanks, and time
17:00:39 <tmcpeak> o/
17:00:41 <xarses> and thats time, thanks for playing folks
17:00:48 <xarses> @endmeeting
17:00:50 <xarses> #endmeeting