16:00:12 <xarses> #startmeeting fuel
16:00:12 <xarses> #chair xarses
16:00:12 <xarses> Todays Agenda:
16:00:12 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:00:12 <xarses> Who's here?
16:00:15 <mihgen> hi
16:00:18 <openstack> Meeting started Thu Mar 10 16:00:12 2016 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:19 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:20 <maximov_> hi
16:00:22 <openstack> The meeting name has been set to 'fuel'
16:00:24 <openstack> Current chairs: xarses
16:00:35 <angdraug> o/
16:00:50 <mwhahaha> hi
16:01:07 <aignatov> \o/
16:01:12 <dpyzhov> hi
16:01:41 <zynzel> hi!
16:01:41 <vsakharov> hi
16:01:44 <asvechnikov> hi
16:01:48 <evgenyl> hi
16:01:53 <ogelbukh_> greetings
16:01:54 <ashtokolov> o/
16:02:10 <fzhadaev> Hi!
16:02:14 <mattymo`> hi
16:02:32 <ikutukov> hi!
16:02:36 <xarses> ok, lets get going then
16:02:40 <xarses> #topic Action items from last meeting
16:02:57 <xarses> xarses to follow up with aspiers on RA convergence
16:03:27 <xarses> this isn't done I still need to follow up with him
16:03:50 <xarses> moving on to the regular agenda
16:04:05 <xarses> #topic Fuel upgrade status (ogelbukh)
16:04:24 <salmon_> hi
16:05:44 <xarses> moving on then
16:05:50 <ogelbukh_> so, we stubled upon couple more things in fuel installation
16:05:54 <xarses> nvm
16:06:00 <ogelbukh_> which we need to work around
16:06:21 <ogelbukh_> specifically, issue with nailgun admin password not syncing back to astute.yaml
16:06:55 <ogelbukh_> which is understandable, but breaks our flow
16:07:33 <ogelbukh_> we run another round of tests today and tomorrow, and be ready by monday
16:07:48 <ogelbukh_> I will send announces to MLs
16:08:32 <xarses> ok
16:08:32 <ogelbukh_> as for configdb, we work on adding repo in openstack/ space for extension, extension itself is working with Nailgun, starting integration from tomorrow
16:08:40 <ogelbukh_> that's basically all
16:09:04 <angdraug> anyone remembers why we didn't add octane to fuel repos in governance?
16:09:15 <ogelbukh_> nope
16:09:16 <angdraug> if not, I think we should add it and the new repo ogelbukh_ is creating
16:09:35 <ogelbukh_> please, assign me an AI on it
16:09:45 <ogelbukh_> xarses: ^^
16:09:50 <xarses> ok
16:09:55 <xarses> anything else?
16:09:57 <ogelbukh_> thank you
16:10:30 <xarses> #action ogelbukh_ will add octane and repo for configdb to fuel governance
16:10:33 <xarses> #topic concerns on puppet-openstack upstream master HEAD usage (aglarendil)
16:10:55 <aglarendil> \o
16:11:05 <aglarendil> so far, folks. I expressed several concerns on this
16:11:20 <aglarendil> 1) build is not reproducible
16:11:37 <aglarendil> 2) there is no duty process set up currently
16:11:51 <aglarendil> 3) issue with puppet-openstack modules and/or their integration
16:12:05 <aglarendil> affects other developers who do not actually need newer puppet-openstack modules
16:12:12 <aglarendil> at least for another couple of weeks
16:12:19 <angdraug> (1) is partially addressed by mwhahaha's commit to record sha1's of puppet modules:
16:12:21 <angdraug> #link https://review.openstack.org/286310
16:12:37 <mwhahaha> wrong link :D
16:12:48 <xarses> I was gonna say
16:12:51 <mwhahaha> #link https://review.openstack.org/#/c/288768/
16:12:54 <aglarendil> I think we need to address all the concerns first and enable the process after
16:13:07 <angdraug> right you are, bad paste
16:13:17 <aglarendil> currently, if something fails, there is no guarantee that I will be able to debug things
16:13:17 <ogelbukh_> could we work off the latest milestone tags?
16:13:21 <angdraug> (2) I've described the duty process:
16:13:26 <angdraug> #link https://wiki.openstack.org/wiki/Fuel/CI/Puppet_OpenStack_CI_duty
16:13:28 <aglarendil> angdraug: it is not set up
16:13:42 <aglarendil> moreover, other concerns: process of using upstream commits
16:13:45 <aglarendil> is implicit
16:14:16 <aglarendil> e.g. we do not control things - they are sliding below us without ability for us
16:14:20 <aglarendil> to intervent manually
16:14:27 <holser_> o/
16:14:36 <aglarendil> let alone, the job to monitor puppet-openstack has been disabled since March 5th, AFAIK
16:14:50 <bookwar> aglarendil: it wasn't
16:15:03 <aglarendil> okay, than I misinterpreted maximov_'s words
16:15:05 <maximov_> re 2 we do have duty, last week bugfix team was monitoring
16:15:06 <aglarendil> sorry for that
16:15:18 <maximov_> this week, enhancement team on duty
16:15:35 <aglarendil> and lastly, having a duty for a thing that can be easily automated is ridiculous
16:15:57 <aglarendil> we are wasting people time that can be put for better use
16:16:05 <bookwar> aglarendil: please check duty description
16:16:09 <aglarendil> and, frankly, look into our backlog
16:16:09 <angdraug> now you are contradicting yourself
16:16:15 <xarses> aglarendil: "moreover, other concerns: process of using upstream commits"
16:16:20 <xarses> this isn't clear
16:16:48 <aglarendil> moreover, majority of features of Fuel do not actually require newest openstack
16:16:53 <aglarendil> while having risk of being blocked
16:16:59 <angdraug> you're complaining that it's ridiculous to do manually what can be automated, and yet you're objecting to implicitly following upstream modules instead of manually updating them
16:17:18 <xarses> we've broken ourselves nearly 10x more than puppet-openstack
16:17:21 <aglarendil> angdraug: I am objecting to do it in implicit uncontrollable way
16:17:28 <aglarendil> xarses: this is a loose argument
16:17:40 <aglarendil> this actually means that we need less sources of failures
16:17:41 <aglarendil> not more
16:17:53 <angdraug> can't parse that last one
16:18:02 <mihgen> particule suggestions how we can get the process controllable?
16:18:02 <aglarendil> angdraug: if we fail our CI
16:18:08 <aglarendil> it is simple
16:18:18 <aglarendil> as openstack upstream does with 3rd party libraries
16:18:29 <mihgen> so that only certain people are working on failure, and the rest is doing their other stuff and not blocked
16:18:30 <aglarendil> run a test nightly
16:18:32 <angdraug> puppet modules are not 3rd party libraries for us
16:18:40 <aglarendil> nope, they are
16:18:46 <angdraug> openstack upstream projects don't do that between themselves
16:18:51 <xarses> yes, and check voting on puppet-openstack will make this problem even more non-existent
16:18:56 <aglarendil> please tell me the percentage of features directly dependant on newest openstack modules
16:18:58 <aglarendil> LCM?
16:19:01 <aglarendil> SR-IOV?
16:19:03 <aglarendil> DPDK?
16:19:27 <aglarendil> which one of blockers except for Mitaka support?
16:19:30 <bookwar> aglarendil: we have risk of getting hundred of patches in upstream which we need to integrate before the Fuel Mitaka release. To address this risk we need to integrate patches one by one, as it is easier to debug. Yes, there is another risk of "blocking" development, but this are blocks which we should address before it is to late to integrate properly
16:19:36 <iberezovskiy> new openstack code, that should support, requires updates which are coming from puppet modules
16:19:40 <ogelbukh_> I think xarses makes the right point
16:19:57 <aglarendil> okay, we did not get to the point
16:19:58 <xarses> aglarendil: so you are making yourself available to update the manifests because they are out of date?
16:20:00 <ogelbukh_> not only fuel should be checked against puppet modules, but vice versa as well
16:20:07 <aglarendil> of how make it satisfactory for everyone
16:20:25 <aglarendil> nightly test which checks a set of commits
16:20:37 <aglarendil> a set of commits are being proposed to Puppetfile each day
16:20:51 <aglarendil> if they pass CI, we merge them and know exactly which commits are used
16:21:01 <xarses> again, that moves the responsibility to too narrow of a group
16:21:04 <aglarendil> in majority of cases we will have a lag of 1 day
16:21:08 <alex_didenko> should we vote or something?
16:21:08 <mwhahaha> why add a complex checking system with tag moving rather than just providing visible votes upstream?
16:21:11 <iberezovskiy> aglarendil, who's gonna to prepare adapt patch in case of failures?
16:21:16 <aglarendil> we do not move tags
16:21:19 <aglarendil> we specify refs
16:21:43 <xarses> if we are checking fuel on puppet-openstack changes, this problem is moot
16:21:43 <aglarendil> the team responsible for puppet-openstack modules support in Fuel, obviously
16:21:46 <angdraug> by my calculations, CI duty for tracking upstream modules is a lot less effort than what it replaces in major migrations to newer upstream modules in every Fuel release
16:21:55 <mwhahaha> i don't understand the call to add more complexity to the system for something that is not that risky?  You claim we should be using resources elsewhere, then lets stop trying to build complex systems
16:22:09 <xarses> mwhahaha: +1
16:22:29 <mihgen> let's take a step back and observe ideal picture for a second
16:22:29 <mwhahaha> add visibility to fuel ci on upstream
16:22:30 <mwhahaha> problem solved
16:22:38 <xarses> besides, we have entered FF
16:22:45 <mihgen> to me the one is to have fuel ci putting -1 on puppet-openstack if there is a bug
16:22:53 <xarses> both sides should be calm
16:22:56 <aglarendil> which they will ignore easily
16:23:00 <mihgen> which is 100% cases should show bug in puppet-openstack
16:23:05 <aglarendil> o, btw, last time I checked, this -1 was silent
16:23:08 <xarses> aglarendil: no, by practice they don't
16:23:10 <mihgen> and easy to debug
16:23:13 <alex_didenko> guys, so two consequential runs of, let's say, noop tests could be different, because we've pulled different puppet-openstack manifests.. and two ISOs built within a hour could be different to. is it ok?
16:23:25 <mwhahaha> if fuel -1's upstream, that's a break in backwards compatibility and upstream takes CI failures very seriously
16:23:26 <aglarendil> there is no guarantee, that they will not ignore it
16:23:42 <angdraug> aglarendil: bookwar is enabling public comments from Fuel CI today
16:23:44 <mwhahaha> more seriously then we do i might add
16:23:48 <aglarendil> it is advisory
16:23:53 <xarses> aglarendil: if we don't respond, yes they will ignore it
16:24:17 <angdraug> alex_didenko: yes, it is ok
16:24:32 <holser_> guys let’s move HEAD offline
16:24:36 <holser_> let’s move on
16:24:38 <xarses> but we've added lots of cross project CI in puppet-openstack, and respect it as best as possible, even when non-voting
16:24:39 <angdraug> holser_: +1
16:24:47 <angdraug> this is a debate that should be held on ML
16:24:52 <holser_> I’ll talk to aglarendil with the actual analysis I did yesterday
16:25:07 <angdraug> holser_: please also post it on ML
16:25:16 <aglarendil> angdraug: oh, common, than we have nothing to discuss here. I suggest, angdraug that you do troubleshooting by yourselves and see what the real pitfall is
16:25:28 <holser_> aglarendil: let’s move on
16:25:34 <holser_> we have +5 topics
16:25:47 <holser_> so feature leads have to provide the status
16:25:58 <xarses> #action holser_ to update ML on fuel-lib ci
16:25:59 <mihgen> aglarendil: I suggest that you don't treat puppet-openstack as something we can't change
16:26:08 <xarses> #action aglarendil to update ML on fuel-lib CI
16:26:34 <aglarendil> mihgen: I suggest that we do not use 'fuzzy' things when we should verify and be certain
16:26:49 <xarses> moving on
16:26:51 <xarses> #topic cross-project liaisons for Fuel https://wiki.openstack.org/wiki/CrossProjectLiaisons (angdraug)
16:27:07 <angdraug> I've registered our Infra cross-project liaisons in the official wiki:
16:27:07 <angdraug> #link https://wiki.openstack.org/wiki/CrossProjectLiaisons#Infra
16:27:07 <angdraug> I think we should add more liaisons, at least for Puppet OpenStack project
16:27:07 <angdraug> I propose mwhahaha as our liaison to Puppet OpenStack
16:27:07 <angdraug> and I think degorenko can be Puppet OpenStack's liaison to Fuel,
16:27:08 <angdraug> unless EmlienM would rather delegate this to someone else
16:27:10 <angdraug> objections? other projects we need liaisons with?
16:27:55 <xarses> angdraug: I thought we needed to reduce overlap with trippleo's API
16:28:03 <xarses> so someone there?
16:28:14 <angdraug> good point. any volunteers?
16:28:20 <xarses> and what about Baron?
16:28:31 <angdraug> we have a topic for austin about our own API and versioning
16:28:52 <angdraug> ikalnitsky: you're listed as session lead for the API session
16:29:38 <angdraug> I think investigating possibilities for reducing overlap with TripleO should be part of the prep for this session
16:29:51 <angdraug> xarses: you mean Bareon?
16:30:03 <xarses> probably
16:30:10 <ikalnitsky> angdraug: i didn't get quite well. tripleo and api?
16:30:24 <ikalnitsky> angdraug: sorry, i missed a lot of conversation due to red master and trying to make it green
16:30:41 <angdraug> in short, when Fuel's Big Tent application was discussed in TC,
16:30:58 <angdraug> it was brought up that our Nailgun API has a functional overlap with Tuskar API from TripleO
16:31:12 <angdraug> we should investigate how much overlap there really is and see what we can do to reduce it
16:31:36 <xarses> in that they can overlap in implmentation
16:31:39 <xarses> but not in spec
16:31:49 <ikalnitsky> angdraug: ok, got it. thank you for pointing that
16:31:52 <ikalnitsky> i'll investigate that
16:31:54 <ogelbukh_> btw, is the sessions submission for summit open already?
16:32:01 <ogelbukh_> I mean design summit, not conference
16:32:41 <xarses> ogelbukh_: we went over them weeks ago https://etherpad.openstack.org/p/fuel-austin-agenda
16:32:59 <angdraug> I've submitted our ask for space based on the link ^
16:33:01 <xarses> I think we've already asked for space for a short list of this
16:33:50 <angdraug> before we move on, anyone wants to comment about liaisons for Bareon?
16:34:04 <szaher> $ git branch
16:34:05 <szaher> * (detached from origin/stable/kilo)
16:34:21 <ogelbukh_> OK, need to follow the MLs closer :(
16:34:21 <szaher> sorry !
16:34:28 <xarses> evgenyl: ?
16:34:57 <evgenyl> xarses: sorry, what is the question?
16:35:08 <angdraug> I have a feeling Bareon team is too small for now to bother with official liaisons between them and Fuel
16:35:10 <xarses> "<angdraug> before we move on, anyone wants to comment about liaisons for Bareon?"
16:36:26 <evgenyl> I'm leading this activity.
16:37:00 <angdraug> evgenyl: do you mind being listed as our official liaison to Bareon?
16:37:02 <mihgen> what is eta of switching to bareon in fuel btw?
16:37:15 <evgenyl> angdraug: np
16:37:40 <evgenyl> mihgen: as usual, the work will be done in 10.0 timeframe.
16:38:03 <mihgen> I hope not 1 day before FF :P
16:38:20 <evgenyl> mihgen: we will do our best to make if faster.
16:38:28 <angdraug> there's no official newton schedule yet, but based on previous cycles FF is going to be in August
16:38:41 <mihgen> xarses: angdraug folks many things in the agenda
16:38:46 <angdraug> lets move on
16:39:00 <xarses> #topic PTL & CL Elections http://lists.openstack.org/pipermail/openstack-dev/2016-March/088609.html (xarses)
16:39:14 <xarses> This is just a note, that the elections are starting
16:39:29 <xarses> Nominations for PTL start tomorrow
16:39:56 <xarses> #topic bit.ly/1RD6JLR - stuck reviews. 16 of them are new features-related (mihgen)
16:40:27 <mihgen> highlighting your attention to this. angdraug - your help is needed here to assign people to review specs
16:40:43 <mihgen> a few patches to fuel-devops, nurla  - can you help to move them forward?
16:41:06 <ikalnitsky> mihgen: i think we need to ask ddmitriev to watch fuel-devops
16:41:12 <ikalnitsky> he's an active core and contributor
16:41:13 <mihgen> there are several from other repos. Please take a look folks and poke people around to whether move them further or abandon
16:41:36 <mihgen> that's it on this topic
16:41:45 <xarses> #topic UI Team status (vkramskikh)
16:41:55 <vkramskikh> Hi! Here is our status for 9.0 features, only 2 of them aren't finished:
16:41:56 <vkramskikh> 1) https://blueprints.launchpad.net/fuel/+spec/remove-vendor-code - no vendor code left in the upstream; as for downstream, we're fighting with CI issues - tests won't pass because of https://bugs.launchpad.net/fuel/+bug/1549750 . Any help from python guys would be appreciated.
16:41:56 <openstack> Launchpad bug 1549750 in Fuel for OpenStack "Async tasks (cluster deletion, resetting, snapshot generation) sometimes fail" [Critical,Confirmed] - Assigned to Alexander Kislitsky (akislitsky)
16:41:56 <vkramskikh> 2) NFV stuff - implementations for node attributes (for Nova and DPDK CPU pinning) and NUMA topology representation are merged, though we've got FFE and decided to make some visual improvements and refactoring there; interface screen changes are on review - we expect them to be merged by Monday.
16:41:56 <vkramskikh> We still haven't switch to bugfixing and plan to start from the next week.
16:41:58 <vkramskikh> Questions?
16:43:30 <xarses> thanks vkramskikh
16:43:45 <xarses> #topic Fuel-mixed team status (zynzel/bkupidura)
16:43:53 <zynzel> Fuel-mixed team is working on fuel-library ensurability/idempotency and multipath feature
16:44:00 <zynzel> Idempotency, 2 changes left in review https://review.openstack.org/#/c/290340/ https://review.openstack.org/#/c/290915/
16:44:03 <zynzel> We need to work on CI/jenkins part.
16:44:07 <zynzel> Multipath feature was merged, but we found a bug https://bugs.launchpad.net/fuel/+bug/1555664 currently we are investigating
16:44:07 <openstack> Launchpad bug 1555664 in Fuel for OpenStack "provision with multipath device fails randomly" [High,Confirmed] - Assigned to Sergey Slipushenko (sslypushenko)
16:44:24 <zynzel> That's all folks
16:44:51 <xarses> zynzel: thanks, do you have a path forward on that bug?
16:45:18 <zynzel> no, currently guys are trying to found root cause
16:45:38 <xarses> #topic Enhancements Team status (ashtokolov)
16:45:45 <ashtokolov> Hi
16:45:52 <ashtokolov> We are working on:
16:45:54 <ashtokolov> 1. Custom graph execution - Good Progress (core part - merged or on review, API + CLI - WIP)
16:46:01 <ashtokolov> 2. Versioning storage for serialised cluster data and cluster settings - In progress
16:46:07 <ashtokolov> 3. Data-driven decision which tasks should be run during redeployment (YAQL) - In progress
16:46:15 <ashtokolov> 4. Store Deployment Tasks Execution History in DB - Good Progress (core part - on review, API + CLI - WIP)
16:46:21 <ashtokolov> 5. Deployment Tasks idempotence with Fuel Mixed Team - Good Progress
16:46:42 <ashtokolov> That's all
16:47:26 <xarses> ashtokolov: in good shape for meeting the FFE deadline?
16:47:58 <ashtokolov> yes, we are on track
16:48:15 <xarses> thanks
16:48:17 <xarses> #topic FFE exception update for Enable UCA repositories for deployment (mattymo)
16:48:53 <holser_> 10 minutes ;)
16:48:59 <mattymo`> Hi xarses
16:49:40 <angdraug> mattymo`: you've asked for FFE until today
16:49:48 <mattymo`> I didn't manage with the holidays and unstable UCA repo state to get a fully passed custom system test. The code really works, but we've got intermittent CI failures all over fuel web and fuel library now
16:49:48 <angdraug> are all your commits merged?
16:50:03 <mattymo`> so I would like to report, that my feature will not land in 9.0 as a result
16:50:23 <mattymo`> it's ready, but because of CI, it will not land and I will wait until SCF and resubmit for 10.0
16:50:25 <mihgen> :(
16:50:46 <angdraug> CI ate your commits. got it.
16:51:06 <angdraug> xarses: lets move on
16:51:08 <xarses> #topic Fuel network team status (alex_didenko)
16:51:19 <alex_didenko> Fuel network team status per feature:
16:51:19 <alex_didenko> Allow any VIP: bugfixing
16:51:19 <alex_didenko> External LB: working on another plugin for automated tests under swarm runner
16:51:19 <alex_didenko> SR-IOV: all the patches except UI are merged already. We're still checking into some possible issues/bugs though
16:51:19 <alex_didenko> DPDK: all the needed patches are on review
16:51:20 <alex_didenko> We're not expecting any delays in the current schedule/FFE
16:51:43 <xarses> alex_didenko: thanks
16:52:08 <xarses> #topic Fuel Telco Team Status (fzhadaev)
16:52:12 <fzhadaev> My update will be short.
16:52:12 <fzhadaev> Our two main activities for now are:
16:52:12 <fzhadaev> 1) Finishing work on NFV features (features status was provided by alex_didenko)
16:52:12 <fzhadaev> 2) Fixing bugs
16:52:15 <fzhadaev> NFV features, which was not presented yet:
16:52:15 <fzhadaev> NUMA+CPU Pinning - all patches are on review (2)
16:52:15 <fzhadaev> Huge Pages - all patches are on review (4)
16:52:23 <fzhadaev> Do you have any questions?
16:52:37 <xarses> are you on track to make your FFE deadline?
16:52:46 <fzhadaev> I think yes
16:52:54 <xarses> good to hear, thanks
16:53:19 <xarses> #topic Bugfix team status (dpyzhov)
16:54:03 <dpyzhov> hi
16:54:36 <dpyzhov> I have only several minutes left so I'd like to highlight that we have growing number of tech-debt bugs related to tests and CI
16:54:51 <dpyzhov> it is the main concern and we should focus on this
16:55:36 <dpyzhov> Other things: we are pretty good with high priority bugs
16:55:49 <dpyzhov> And have no progress with medium bugs
16:56:12 <mihgen> are we tagging somehow ci-related bugs?
16:56:20 <mihgen> so to see them all quickly?
16:56:23 <dpyzhov> tech-debt, high/critical priority
16:56:36 <mihgen> not all tech-debt are ci related
16:57:00 <dpyzhov> you are right. we have no special filter for tests
16:57:13 <xarses> dpyzhov: can we add a tag then
16:57:13 <dpyzhov> but most of high priority tech debts are about tests
16:57:29 <dpyzhov> xarses: do we really need one?
16:57:42 <xarses> do we have a area-ci one?
16:57:48 <xarses> we can combine the two then?
16:58:18 <dpyzhov> we have only 16 bugs in area-python with tech-debt tag
16:58:22 <bookwar> xarses: we have area-ci tag, but i doubt dpyzhov means the same by 'ci-related bugs' here
16:58:34 <dpyzhov> I think it is still manageable to find test-related bugs here
16:58:59 <xarses> yes, but the tag helps with reporting later
16:59:15 <xarses> lets think about it
16:59:17 <xarses> anything else?
16:59:29 <dpyzhov> Actually, we should start watching our tests health
16:59:39 <dpyzhov> But it is rather big topic
16:59:45 <dpyzhov> nothing else from my side
16:59:53 <xarses> thanks
17:00:05 <xarses> #endmeeting