16:01:08 <xarses> #startmeeting fuel
16:01:09 <xarses> #chair xarses
16:01:09 <openstack> Meeting started Wed Dec 23 16:01:08 2015 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:09 <xarses> Todays Agenda:
16:01:09 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:01:09 <xarses> Who's here?
16:01:10 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:12 <openstack> The meeting name has been set to 'fuel'
16:01:13 <openstack> Current chairs: xarses
16:01:18 <mwhahaha> hi2u
16:01:20 <mihgen> hi
16:01:23 <ikalnitsky> o/
16:01:23 <rmoe> hi
16:01:34 <dpyzhov> hi
16:01:36 <maximov> hi
16:01:36 <fzhadaev> Hi!
16:01:52 <agordeev> o/
16:02:01 <xarses> thanks to the ML2 team taking the week off  we can use our normal room =)
16:02:07 <angdraug> o/
16:02:27 <xarses> #topic Action Items from last meeting
16:02:34 <xarses> monester and bookwar will work on getting the add reviewers by MAINTAINERS data script
16:03:05 <yottatsa> hi
16:03:29 <SheenaG> o/
16:03:37 <bookwar> we have job in testing for fuel-main repo https://ci.fuel-infra.org/job/auto-add-reviewers/14/console, but there are still issues
16:03:39 <jaranovich> hi
16:04:01 <bookwar> so in progress
16:04:16 <xarses> bookwar: thanks
16:04:23 <xarses> xarses find a meeting room from 12/23, and announce meeting schedule - done
16:04:33 <xarses> dpyzhov will announce changes in swarm-blocker tag
16:04:39 <dpyzhov> tbd tomorrow
16:04:46 <xarses> thanks
16:04:52 <xarses> #topic Mixed team status (sslypushenko)
16:05:49 <xarses> #topic Telco Team Status (fzhadaev)
16:06:02 <fzhadaev> The main activity of Telco team is fixing bugs.
16:06:03 <fzhadaev> For now we have 3 open bugs with high priority
16:06:03 <fzhadaev> Fixes for two of them are on review, and the last one is in progress.
16:06:16 <fzhadaev> Your questions? :)
16:06:43 <xarses> how are we doing income wise?
16:06:53 <holser_> what does it mean 3 bugs?
16:06:59 <xarses> and what are your plans for the next iteration?
16:07:20 <fzhadaev> holser_: 3 high bugs on our team
16:07:23 <holser_> we have plenty of them
16:07:37 <holser_> Are you going to grab more bugs?
16:08:15 <holser_> what’s a purpose to separate bugs? all teams have same plate with bugs
16:08:28 <fzhadaev> xarses: there is no certain plan for now. but as probably activity we have BP about ubunut bootstrap improvements
16:08:35 <ikalnitsky> holser_: +1
16:08:49 <sslypushenko> xarses: Mixed team status:  development of feature for changing OpenStack configs is finished. Bugs - fixed. Autotests is done and are in green status in swarm
16:09:27 <angdraug> fzhadaev: we are behind on the bugfixing for 8.0, until the situation is under control we need every team to prioritize fixing High and Critical bugs
16:09:36 <fzhadaev> holser_: sure, we're taking another bugs. there is the current state of our team. and there are bugs for 8.0
16:09:45 <holser_> k
16:09:56 <angdraug> thanks
16:11:14 <xarses> #topic UI Team status (vkramskikh)
16:11:24 <jaranovich> Hi, let me post ui team status:
16:11:27 <jaranovich> We're finishing our work on bugfixing. As for High bugs, we have 2 bugs left: https://bugs.launchpad.net/fuel/+bug/1391481 and https://bugs.launchpad.net/fuel/+bug/1528224, fixes for which we're going to merge right after merge freeze is lifted. We're also working on UI part for https://bugs.launchpad.net/fuel/+bug/1518993 and plan to finish everything soon.
16:11:27 <jaranovich> After dealing with bugs, we plan to continue our work on Fuel UI repo separation, vendor code separation and solving our tech debt.
16:11:27 <jaranovich> Questions?
16:11:27 <openstack> Launchpad bug 1391481 in Fuel for OpenStack "There are no descriptions for network parameters" [High,In progress] - Assigned to Andrey Shestakov (ashestakov)
16:11:29 <openstack> Launchpad bug 1528224 in Fuel for OpenStack "[UI] Offloading options are not fitted on a page" [High,In progress] - Assigned to Kate Pimenova (kpimenova)
16:11:30 <openstack> Launchpad bug 1518993 in Fuel for OpenStack "Plugins metadata (keys in hash with settings) gets corrupted" [Critical,In progress] - Assigned to Vitalii Myhal (xmig)
16:11:40 <SheenaG> xarses: also, if there was an action item for me that I missed, it is done - I met with DP, Fuel UI team has added more Network descriptions (thanks vkramskikh) and DP is reviewing and refining them now (and adding a couple descriptions that were missing)
16:11:47 <SheenaG> (and thanks jaranovich)
16:12:10 <ikalnitsky> jaranovich: what about plugins' metadata get corrupted? is it done?
16:12:26 <ikalnitsky> i didn't pay much attention for this activity last days
16:12:26 <jaranovich> both nailgun and ui requests under code review
16:12:34 <jaranovich> iso for testing was builded
16:12:35 <ikalnitsky> have it been tested?
16:13:02 <jaranovich> we plan to test this custom iso in a few days in parallel wit code review
16:13:15 <ikalnitsky> well, ok. if you need any help from my side - just ping me.
16:13:20 <jaranovich> ok, sure
16:13:24 <jaranovich> thanks)
16:13:34 <ikalnitsky> no problem ;)
16:14:16 <xarses> #topic Bugs team status (dpyzhov) https://etherpad.openstack.org/p/fuel-bugs-status
16:14:23 <dpyzhov> hi agail
16:14:27 <dpyzhov> *again
16:14:36 <dpyzhov> I've updated numbers in etherpad
16:15:03 <dpyzhov> we have pack of degradations and even had broken master and bvt
16:15:18 <dpyzhov> right now my team is fully loaded with bugs
16:15:29 <dpyzhov> we have no bugs in queue for python
16:15:50 <dpyzhov> we have some bugs in our queue for library but we feel confident with them
16:16:06 <dpyzhov> still waiting for more bugs from QA
16:16:17 <mihgen> did we fix bvt / fully unblocked qa?
16:16:27 <dpyzhov> we hope so
16:16:47 <dpyzhov> bvt is green
16:16:55 <mihgen> I don't see any good ISO over here: https://ci.fuel-infra.org/
16:17:30 <mihgen> dpyzhov: can you please take an action item and figure out with the team what's up here.. ?
16:17:40 <dpyzhov> bookwar: any news about these jobs?
16:17:49 <angdraug> mihgen: https://ci.fuel-infra.org/job/8.0.fuel_community.ubuntu.bvt_2/79/
16:17:57 <angdraug> there was a green iso earlier today, now it's back to red
16:18:39 <dpyzhov> angdraug: is it because of SCF?
16:18:44 <angdraug> dpyzhov: bookwar: is ^ a problem with jobs?
16:18:59 <bookwar> new bvt is in progress https://ci.fuel-infra.org/job/8.0.fuel_community.ubuntu.bvt_2/81/console
16:19:03 <mihgen> yes, 182. BTW can we remove BVT1 from there? As it's not in use anymore and always red there
16:19:10 <bookwar> there was an issue with fuel-qa code today
16:19:34 <bookwar> which was fixed in both stable/8.0 and master branches, we wait for new test results
16:20:24 <dpyzhov> mihgen: do we have an owner for our community bvt?
16:20:26 <mihgen> bookwar: how hard would it be to remove BVT1 from that table and fix color? (to make it green if build & bvt2 is fine
16:20:46 <bookwar> it can be removed as for 7.0
16:21:01 <bookwar> but maybe we replace it with a test for centos computes
16:21:04 <bookwar> ?
16:21:11 <bookwar> once it will be available
16:21:50 <mihgen> bookwar: I think we need to remove it now, and add new test when it's available
16:21:55 <mihgen> now it's just confusing..
16:22:11 <mihgen> dpyzhov: I don't know one.. could your team be the one?)
16:22:43 <bookwar> mihgen: i'll take a look
16:22:51 <bookwar> and the bvt1 removal
16:22:54 <dpyzhov> mihgen: let me think about it
16:24:30 <xarses> moving on
16:24:40 <xarses> #topic Enhancements Team Status (ashtokolov)
16:26:34 <dpyzhov> he'll be available in 3 min
16:26:41 <xarses> moving again then
16:26:44 <xarses> #topic Serious issue with provisioning (agordeev) https://etherpad.openstack.org/p/multiple-reboots-of-provisioned-nodes
16:27:33 <agordeev> it looks that there's a critical issue related with provisioning (or at least its consequences observed in provisioning part)
16:27:33 <agordeev> symptoms: 2 or 3 unexpected reboots of provisioned node happen within 1 min at cloud-init stage of provisioning flow.
16:27:33 <agordeev> sometimes it could break cloud-init scripts, hence provisioning will be timed out.
16:27:33 <agordeev> the issue is 100% reproducible according to the oldest BVT logs i was able to find. The oldest BVT logs timestamped at 15'th of Dec.
16:27:33 <agordeev> We're investigating the root cause.
16:27:33 <agordeev> #link https://etherpad.openstack.org/p/multiple-reboots-of-provisioned-nodes
16:28:58 <holser_> interesting
16:29:38 <agordeev> so, even if BVT is green, then the issue is still here. It just magically passed cloud-init scripts
16:29:39 <yottatsa> could it be a host-node based OOM, where all the VMs is starting to consume memory at one moment?
16:30:17 <mattymo> agordeev, dpyzhov and I were discussing this earlier today. It's probably related to how mcollective restarts
16:30:37 <mattymo> or nailgun-agent retriggers mcollective and multiple fuel checkins cause unexpected behavior
16:30:59 <mattymo> or cloud-init too :(
16:31:28 <agordeev> mattymo: so, could mcollective' restart reboot nodes multiple times?
16:31:46 <mattymo> good question. I don't have the answer
16:32:16 <xarses> there is the fallback over ssh
16:32:31 <xarses> maybe that code is running, but it would only reboot the node 1 extra time
16:33:08 <xarses> moving?
16:33:35 <xarses> #topic Enhancements Team Status (ashtokolov)
16:33:41 <ashtokolov> xarses, folks, sorry for delay
16:33:55 <ashtokolov> Enhancements Team is going to start feature development for 9.0 tomorrow
16:34:04 <ashtokolov> We are working on backlog with Product Management
16:34:20 <ashtokolov> Preliminary our main goals are:
16:34:29 <ashtokolov> 1. Enable task-based deployment engine as a default
16:34:35 <ashtokolov> 2. Enhance nailgun serializators
16:34:44 <ashtokolov> 3. Implement task-based progress bar
16:34:51 <ashtokolov> 4. Enhance plugin framework
16:34:57 <ashtokolov> 5. Improve partitioning for master and slave nodes
16:35:03 <ashtokolov> 6. Improve shotgun log collection
16:35:09 <ashtokolov> 7. And many other small enhancements
16:35:23 <ashtokolov> As usual our backlog is availible in Trello https://trello.com/b/JlWmptJN/enhancements-team
16:35:33 <ashtokolov> #link https://trello.com/b/JlWmptJN/enhancements-team
16:35:44 <bookwar> the order is time based? i am concerned about 1) and how it affects infra
16:36:12 <mihgen> etckeeper plugin?
16:36:15 <ashtokolov> bookwar, no it's just a unsorted list
16:36:35 <ashtokolov> mihgen, etckeeper plugin is already under development
16:36:57 <angdraug> what's the status of the UCA plugin that mwhahaha was working on?
16:37:13 <ashtokolov> plugin release is not connected to 9.0 release
16:37:31 <mattymo> angdraug, what is UCA?
16:37:39 <mwhahaha> using ubuntu cloud archive packages
16:37:43 <mwhahaha> UCA
16:38:17 <ashtokolov> mwhahaha, could you please provide an update?
16:38:18 <angdraug> #link http://lists.openstack.org/pipermail/openstack-dev/2015-November/078936.html
16:38:19 <mwhahaha> i've been slowly working on some of the incompatibilies in fuel as it relates to not using MOS packages
16:38:33 <mwhahaha> specifically Bug 1526938
16:38:33 <openstack> bug 1526938 in Fuel for OpenStack "read_timeout not supported by pymysql causes deployment failures" [High,In progress] https://launchpad.net/bugs/1526938 - Assigned to Alex Schultz (alex-schultz)
16:39:05 <mwhahaha> as part of the UCA plugin POC i used a custom iso with that issue addressed
16:39:19 <mwhahaha> so i'm working on making it configurable based on the os_package_type fact
16:39:28 <mwhahaha> so it will continue to work with MOS and be able to work with UCA
16:39:44 <mwhahaha> after that is addressed the next big item would be to address the forked HAproxy from fuel/mos
16:40:10 <mwhahaha> that's pretty much it on the UCA front
16:40:15 <angdraug> there was some talk about ditching haproxy altogether, is that feasible in 9.0?
16:40:52 <angdraug> #link https://qa.debian.org/developer.php?login=zigo
16:41:02 <mwhahaha> I have not had time to investigate further. We had mentioned possible alternatives of nginx or lvs
16:41:13 <angdraug> zigo already has mitaka-1 packages in Debian/experimental, what's the state of mitaka in UCA?
16:41:35 <angdraug> is it worth investigating as an option for 9.0?
16:42:09 <angdraug> ashtokolov: apologies about hijacking your report :)
16:42:29 <ashtokolov> angdraug: it's ok)
16:42:38 <mwhahaha> i think so but i would like some others input
16:42:48 <angdraug> mihgen: ^ ?
16:42:56 <angdraug> anyone else has an opinion?
16:43:21 <xarses> Id like to see research in lvs
16:43:34 <xarses> there is a nice article on it's performance
16:44:32 <xarses> moving though?
16:44:40 <mihgen> angdraug: let's connect offline
16:44:48 <angdraug> xarses: yes, lets move on
16:44:50 <xarses> #topic Mixed team status (sslypushenko)
16:45:21 <xarses> sslypushenko: I know you posted it already, but we where in another topic
16:46:39 <xarses> <sslypushenko> xarses: Mixed team status:  development of feature for changing OpenStack configs is finished. Bugs - fixed. Autotests is done and are in green status in swarm
16:46:56 <xarses> moving on then
16:47:00 <xarses> #topic Upgrade team status (ogelbukh)
16:47:59 <angdraug> ogelbukh: around?
16:48:19 <xarses> was a few min ago
16:48:44 <xarses> ok, will jump forward then
16:48:49 <xarses> #topic soft code freeze (angdraug)
16:49:32 <angdraug> in case you missed the announcement on ML, we in SCF now:
16:49:33 <angdraug> #link http://lists.openstack.org/pipermail/openstack-dev/2015-December/082939.html
16:49:33 <angdraug> #link http://lists.openstack.org/pipermail/openstack-dev/2015-December/082941.html
16:49:33 <angdraug> bookwar: how are we doing on stable/8.0 and CI for 9.0?
16:49:54 <bookwar> Fuel CI is updated
16:50:03 <bookwar> stable/8.0 branch created
16:50:13 <bookwar> we are working on merging version bumps
16:50:31 <mattymo> bookwar, is there a gerrit topic we can follow for monitoring that progress?
16:50:34 <bookwar> we will build 9.0-liberty iso from fuel master and mos 8.0
16:51:03 <bookwar> #link https://review.openstack.org/#/q/status:open+branch:master+topic:8.0-scf
16:51:18 <angdraug> any estimates on when we can lift merge freeze?
16:51:39 <bookwar> we have custom iso build from these patches, so we going to send it through simple test
16:51:44 <bookwar> ETA about an hour
16:51:48 <maximov> do we plan to run SWARM on stable/8.0 and master ?
16:52:17 <bookwar> we will run swarm on stable/8.0 normally, but we are going to find a window for swarm on master
16:52:26 <bookwar> less frequent then stable/8.0 probably
16:52:28 <ogelbukh> xarses angdraug sorry, missed it
16:52:45 <xarses> ogelbukh: will be back in just a min =)
16:52:46 <bookwar> it is still on discussion with qa team
16:52:49 <maximov> bookwar: how often ?
16:53:04 <bookwar> maximov: no exact data yet
16:53:29 <maximov> well, we opened master for merges, so there will be regressions
16:53:41 <maximov> for example, docker removal
16:53:52 <maximov> fuel ui separation from fuel-web
16:54:12 <maximov> we will have to find a way to run swarm on master daily
16:54:38 <bookwar> we are going to estimate resources
16:54:56 <bookwar> and try to fit both in one day
16:55:06 <maximov> ok, thanks
16:55:50 <bookwar> but qa team have no resources for deep analysis of master swarm failures for now, so we'll get less detailed results anyway
16:56:20 <xarses> moving?
16:56:24 <holser_> yep
16:56:30 <xarses> #topic Upgrade team status (ogelbukh)
16:56:42 <holser_> ogelbukh: ^^^
16:56:51 <ogelbukh> so
16:57:11 <holser_> you have 4 minutes
16:57:13 <ogelbukh> we completed the backup via fuel-upgrade + shootgun
16:57:14 <holser_> hurry up
16:57:24 <ogelbukh> and started restore part
16:57:48 <ogelbukh> it will be also fuel-upgrade command
16:57:54 <ikalnitsky> ogelbukh: so shotgun isn't an alternative way for backuping?
16:58:06 <ogelbukh> yes, we decided to stick to it
16:58:15 <ikalnitsky> i just saw in your spec "shotgun" in alternative section
16:58:19 <ikalnitsky> should be probably updates?
16:58:22 <ogelbukh> since in fuel-upgrade backup capability is not flexible enough
16:58:25 <ikalnitsky> s/updates/updated
16:58:27 <ogelbukh> yes, will update
16:58:59 <ogelbukh> restore will replace certain files on normally installed system
16:59:13 <xarses> 1 min
16:59:19 <ogelbukh> upload db, run migrations, rebuild containers and upload fixtures via fuel-update
16:59:34 <xarses> ogelbukh: please send out a mail about the last topic
16:59:36 <ogelbukh> eta is next week
16:59:38 <ogelbukh> sure
16:59:49 <ogelbukh> that's all from me
17:00:03 <holser_> good
17:00:21 <xarses> #endmeeting