16:00:15 <xarses> #startmeeting fuel
16:00:15 <openstack> Meeting started Thu Sep 10 16:00:15 2015 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:16 <xarses> #chair xarses
16:00:16 <xarses> Todays Agenda:
16:00:16 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:00:16 <xarses> Who's here?
16:00:17 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:19 <openstack> The meeting name has been set to 'fuel'
16:00:20 <openstack> Current chairs: xarses
16:00:25 <angdraug> o/
16:00:27 <akislitsky> hi
16:00:29 <mwhahaha> o/
16:00:31 <rvyalov> hi
16:00:33 <rmoe> hi
16:00:52 <teran> hi
16:00:54 <salmon_> hi
16:00:54 <aglarendil> hi
16:00:55 <maximov> рш
16:00:57 <maximov> hi
16:00:58 <alex_didenko> hi
16:01:08 <agordeev> hi
16:01:12 <ikalnitsky> o/
16:01:21 <salmon_> does etherpad work for you?
16:01:26 <angdraug> no
16:01:27 <xarses> its very slow
16:01:28 <maximov> nope
16:01:35 <maximov> issue with srt infrastructure
16:01:37 <xarses> I have the agenda up
16:01:47 <xarses> it was bad in the U.S. too
16:01:52 <angdraug> can't blame srt for this, can't open it in US, either
16:02:13 <bgaifullin> Hi all.
16:02:19 <angdraug> xarses: lets start?
16:02:23 <xarses> #topic community builds from master (someone from Infra team?)
16:02:38 <xarses> angdraug: ^?
16:02:39 <angdraug> bookwar: ^
16:02:41 <bookwar> hi
16:02:44 <xarses> =)
16:02:57 <mihgen> hi
16:02:58 <bookwar> we have builds enabled, available here https://ci.fuel-infra.org/view/ISO/
16:03:23 <angdraug> are they using Liberty packages?
16:03:28 <bookwar> they are built from current up-to day mos8.0 repo of ubuntu packages
16:03:40 <bookwar> which means that most of the packages are still from kilo
16:04:03 <bookwar> but as packaging team does their job they add packages from liberty to this repo
16:04:08 <mihgen> bvt2 actually runs against these builds from master there?
16:04:19 <bookwar> so we expect that this built will start to fail soon
16:04:22 <bookwar> yes
16:04:22 <xarses> when do you see that we will have the packages in place?
16:04:56 <bookwar> that a question for packaging team, i don't have enough data
16:05:14 <mihgen> bookwar: thanks bookwar. It's great to see that we enabled master builds very soon after we've created stable/7.0
16:05:45 <mihgen> question about liberty packages
16:05:57 <mihgen> will we keep kilo builds running?
16:06:19 <angdraug> yes until liberty builds stabilize
16:06:36 <mihgen> angdraug: great, thanks
16:06:37 <angdraug> Fuel CI will stay with kilo until then, too
16:06:45 <bookwar> yes, we keep 8.0-kilo builds in place, and we will use them for fuel-library ci
16:07:04 <bookwar> you can see that currently fuel-library ci uses 8.0-kilo-1 iso
16:07:23 <bookwar> status is here https://ci.fuel-infra.org/ in the Status section
16:08:16 <xarses> moving on?
16:08:22 <mihgen> thank you guys
16:08:24 <angdraug> yes please. great job bookwar!
16:08:30 <xarses> #topic elections of nailgun, fuel-library component leads; PTL (mihgen)
16:08:57 <mihgen> I just wanted to reiterate this. There was email to openstack-dev about code review process changes
16:09:11 <mihgen> no one objects of having elections for component leads
16:09:33 <mihgen> and we have two largest pieces in Fuel, which are nailgun & fuel-library
16:09:55 <mihgen> so I'd like to hear your opinions guys whether we can move forward with it
16:10:10 <mihgen> it was also suggested to run elections at the same time as for PTL
16:10:17 <ikalnitsky> how we're going to elect component leads? from the cores?
16:10:19 <mihgen> using standard openstack procedures
16:10:46 <xarses> So nominate your self and share your platform?
16:10:50 <aglarendil> so, we have the same nomination deadlines and voting deadlines as in upstream OSt ?
16:10:58 <angdraug> announcement of the PTL elections on openstack-dev:
16:11:02 <angdraug> #link http://lists.openstack.org/pipermail/openstack-dev/2015-September/074037.html
16:11:16 <angdraug> aglarendil: yes
16:11:19 <ikalnitsky> what if no one among cores wants to be a component lead? xD
16:11:20 <mihgen> component leads - from cores, I suppose. Frankly, I'd still need to read how openstack does it currently in other projects
16:11:42 <mihgen> ikalnitsky: that's gonna be bad, that's what we want to avoid )
16:12:00 <angdraug> I think just following the PTL process should work for component leads, too
16:12:08 <angdraug> no need to reinvent the bicycle
16:12:10 <mihgen> two component leads will basically get freedom from feature-work assignments and can influence arhitecture of the component
16:12:37 <mihgen> angdraug: that's my assumption as well. So I think we'd need to learn more, and propose it in openstack-dev
16:12:53 <mihgen> (I'd need to learn more at least)
16:13:08 <angdraug> and fast, nomination period opens tomorrow
16:13:26 <mihgen> nomination period where?
16:13:29 <angdraug> voting period is September 17-24
16:13:34 <angdraug> #link https://wiki.openstack.org/wiki/PTL_Elections_September_2015
16:13:36 <xarses> #link https://wiki.openstack.org/wiki/PTL_Elections_September_2015
16:14:10 <angdraug> so we have until 17th to persuade as many Fuel cores as possible to nominate themselves :)
16:14:50 <mihgen> oh. then we'd need to try to get in the same schedule
16:14:51 <salmon_> so one core will not work on features?
16:15:02 <mihgen> salmon_: yes.
16:15:12 <aglarendil> folks, one question, it seems that OSt has its own voting platform
16:15:15 <xarses> salmon_: component lead should not work on features
16:15:16 <angdraug> salmon_: yes except that will be doing specs/design review for all features for that component
16:15:25 <aglarendil> did we arrange all the details that we will have the same platform used?
16:15:39 <aglarendil> or will we conduct votese calculation on ourselves?
16:15:45 <salmon_> nailgun is to big for it
16:16:09 <mihgen> salmon_: too big for what?
16:16:27 <aglarendil> for having one Lead, says salmon_
16:16:30 <salmon_> mihgen: to have on lead
16:16:36 <salmon_> *one
16:16:39 <alex_didenko> are we going to re-elect component leads on a periodic basis?
16:16:51 <aglarendil> alex_didenko: according to OSt workflow - yes
16:16:54 <aglarendil> each six months
16:17:03 <mihgen> aglarendil: I don't know frankly, not sure if angdraug is aware. As I said, we'd need to analyze how it works now in openstack
16:17:29 <aglarendil> mihgen: angdraug we would need to rush with this
16:17:30 <angdraug> at the very least we can reuse the voting system, anyone can use that
16:17:38 <angdraug> aglarendil: true
16:18:04 <xarses> angdraug: can I give you an action to follow up on that?
16:18:04 <maximov> mihgen what is your expectations from component lead. bullet points...
16:18:13 <mihgen> salmon_: well, may be. But with modularization we've started a work on, I'd expect more and more modules to become independent expertise areas, where we may grow other leads
16:18:31 <salmon_> I hope so :)
16:18:43 <angdraug> xarses: yes
16:18:46 <xarses> #action angdraug will follow up on usage of upstream voting system
16:19:03 <mihgen> maximov: you can find it in my email about code review...  and we'd need another one with more data
16:19:22 <mihgen> in short - architecture, quality, making it easy for others to contribute the code
16:19:43 <xarses> mike, lets create a new mail thread back in the ML and request candidates
16:19:52 <mihgen> catch majority of bugs on review/gating, not a month after code merge
16:19:58 <maximov> mihgen ok, but what about community work. will component lead drive communication w/ community?
16:20:11 <mihgen> maximov: yes.
16:20:12 <maximov> I think we missed this part
16:20:26 <mihgen> it's going to be way more important for PTL
16:20:43 <mihgen> but PTL would rely on component leads for specific areas
16:21:05 <xarses> #action mihgen will create a new ML covering duties, duration, and nomination pool for component leads and request candidates
16:21:07 <mihgen> for instance, fuel-library component lead will have to work closely with puppet-openstack, ansible-openstack if needed, etc.
16:21:25 <ikalnitsky> and what about nailgun lead?
16:21:29 <ikalnitsky> we don't have community
16:21:41 <xarses> thats not true
16:21:57 <mihgen> ikalnitsky: we will be building one + there are other areas of Fuel, where integration is needed
16:22:13 <xarses> also they would need to work with the current compute project so that the API's are aligned. I forgot their name
16:22:14 <angdraug> plugin developers are part of nailgun community
16:22:30 <angdraug> xarses: tuskar
16:22:31 <mihgen> all these integration questions with whatever other components / pieces, plugable architecture falls into this category
16:22:32 <angdraug> ?
16:22:41 <xarses> angdraug: thanks, yes
16:23:49 <mihgen> there should be a balance, common sense in prioritization of work
16:24:20 <maximov> mihgen I only have one concern, component lead will be doing a lot of review, communication, architecture etc.. but what about writing code, what would be a trigger  for component lead to contribute to the code?
16:24:46 <xarses> maximov: mostly, they won't be, they won't really have time
16:24:57 <mihgen> angdraug: are we waiting for your message or that was question mark for something else?
16:25:12 <angdraug> angdraug: that was question to xarses, he replied already
16:25:17 <mihgen> maximov: that's ok if component lead will write way less of code
16:25:29 <xarses> lets go back to the ML and discuss further, we need to move along through the schedule for today
16:25:30 <maximov> xarses well, don't you think that in this case component lead will loose his expertise..
16:25:42 <mihgen> but it's important to write code which defines framework for others to work within
16:25:51 <aglarendil> folks, sorry to interrupt, can we discuss details of elections and roles in the ML? we spent almost 25 minutes on that
16:25:57 <xarses> maximov: by reviewing, it shouldn't be much of an issue
16:26:06 <mihgen> let's move on
16:26:16 <xarses> #topic backlog of small enhancements (gaps in functionality) to work on: http://bit.ly/1Nk9rmI (mihgen)
16:26:27 <mihgen> #link http://bit.ly/1Nk9rmI
16:26:33 <mihgen> that's what we have in our launchpad
16:26:40 <mihgen> bugs with "feature" tag
16:26:55 <mihgen> they in fact form our backlog of small enhancements
16:27:08 <mihgen> so my proposal is to continue using launchpad for small feature gaps
16:27:21 <angdraug> +1
16:27:22 <mihgen> and let enhancements team go through this backlog
16:27:26 <maximov> +1
16:27:28 <mihgen> in kanban-like model
16:27:28 <alex_didenko> one small note - we have many different tags, "fueature-*"
16:27:39 <mihgen> alex_didenko: nope those are different
16:27:43 <alex_didenko> so it's not very convenient to filter all of them
16:27:54 <angdraug> feature-* is about bugs introduced by a feature
16:27:56 <mihgen> those are for taxonomy of bugs
16:28:17 <xarses> alex_didenko: those should be for bugs that relate only to that feature (usually because it caused them)
16:28:19 <angdraug> "feature" is for bugs which require a requests for enhancement
16:28:36 <aglarendil> mihgen: there should be more of them. we need to go thoroughly through bugs and retriage all of them
16:28:41 <alex_didenko> I deffinitely saw new feature requests in bugs with feature-* tags
16:28:42 <aglarendil> some of them miss feature tag
16:28:47 <aglarendil> and some are duplicates
16:29:07 <mihgen> then we'd need to do triaging, and mark accordingly
16:29:30 <xarses> alex_didenko: we'd need to kick them back to 'feature' then
16:29:37 <alex_didenko> and inform out engineers about tags
16:29:50 <mihgen> but I hope there are no objections on using launchpad. We already have many feature requests in there, and introducing something else like trello would be an overkill really
16:29:50 <alex_didenko> we need to gather info about tags in some single place, like wiki
16:30:01 <mihgen> alex_didenko: +1
16:30:01 <maximov> ashtokolov is working on the process, how we will organize this work
16:30:29 <salmon_> I think there is wiki page for tags, at least for nailgun
16:30:42 <xarses> salmon_: thats what I thought
16:31:07 <xarses> who will take action for preparing this mail since it still causes confusion?
16:31:08 <ashtokolov> maximov mihgen I'm working on it
16:31:26 <mihgen> ashtokolov: since you are working on it, can you please then let everyone know in openstack-dev about "feature" tag, etc. ?
16:31:56 <ashtokolov> mihgen ok I got it
16:31:57 <alex_didenko> salmon_: I think it should be somewhere on this page https://wiki.openstack.org/wiki/Fuel/How_to_contribute#Test_and_report_bugs and for whole fuel
16:32:10 <mihgen> xarses: action item for ashtokolov .. ?
16:32:14 <angdraug> alex_didenko: +1
16:32:16 <salmon_> alex_didenko: it's on our internal wiki :/
16:32:30 <angdraug> should be on wiki.openstack.org/Fuel
16:32:34 <alex_didenko> yeah, we should remove such infro from internal resources and update public wiki
16:32:39 <xarses> #action ashtokolov will send out ML about usage of 'feature' tag; will start conversation about usage of other tags
16:32:51 <mihgen> great. moving on?
16:33:03 <mihgen> sorry guys for abusing half of the meeting on my topics
16:33:23 <xarses> #topic https://bugs.launchpad.net/mos/7.0.x/+bug/1491576 (holser)
16:33:26 <openstack> Launchpad bug 7 in Launchpad itself "Need help for novice translators" [High,Fix released] - Assigned to Данило Шеган (danilo)
16:34:18 <aglarendil> folks, I need to get off the desktop - I will be online on the smartphone. Please do not forget to mention me
16:34:18 <mihgen> sgolovatiuk  ^^
16:34:23 <xarses> Looks like the issue is resolved already. Is there anything else to discuss sgolovatiuk ?
16:34:39 <mihgen> we still use holser for sgolovatiuk )
16:34:54 <sgolovatiuk> I wouls say it's workarounded
16:34:57 <sgolovatiuk> :)
16:35:11 <mihgen> randomized logrotate time?
16:35:14 <xarses> mihgen: it would work fine if he has his irc client set with the nickname
16:35:25 <maximov> is it backported to stable/7.0 ?
16:35:27 <sgolovatiuk> the problem exists - sometimes mod_wsgo seg fault on USR1 signal
16:35:35 <alex_didenko> let's use copytruncate and do not reload apache at all
16:35:46 <sgolovatiuk> randomized logrotate helped
16:36:14 <sgolovatiuk> so we will never restart all controllers at the same time
16:36:37 <sgolovatiuk> however, alex_didenko proposed to remove postinstall and replace it with copytruncate
16:36:49 <sgolovatiuk> in this case apache won't be restarted at all
16:37:12 <mihgen> this would be certainly better, but I'd leave it for master only. let's stop for stable/7.0..
16:37:21 <sgolovatiuk> aglarendil is still investigating scale up/down issue with apache and keystone
16:37:27 <sgolovatiuk> k
16:37:30 <mihgen> we'd need to create a bug for it though so not to forget about it
16:37:38 <sgolovatiuk> fo 7.0 we won't be adding anything
16:37:54 <sgolovatiuk> we stopped playing with fantom bugs
16:38:03 <xarses> phantom
16:38:14 <sgolovatiuk> random is enough for now
16:38:17 <sgolovatiuk> thnx xarses
16:38:21 <mihgen> sgolovatiuk: so do we have bug for 8.0 about copytruncate?
16:38:32 <sgolovatiuk> we have bug for 8.0
16:38:40 <mihgen> ok great then
16:38:45 <mihgen> thanks sgolovatiuk
16:38:52 <xarses> #topic https://bugs.launchpad.net/fuel/7.0.x/+bug/1493483 (smakar)
16:38:55 <openstack> Launchpad bug 7 in Launchpad itself "Need help for novice translators" [High,Fix released] - Assigned to Данило Шеган (danilo)
16:39:46 <mihgen> what is Данило Шеган ?)
16:39:58 <mwhahaha> the url parsing is bad
16:39:58 * sgolovatiuk shrugs
16:40:01 <xarses> but in bot
16:40:14 <mwhahaha> it's getting fuel/7.0.x/... and thinking it's bug #7
16:40:15 <openstack> bug 7 in Launchpad itself "Need help for novice translators" [High,Fix released] https://launchpad.net/bugs/7 - Assigned to Данило Шеган (danilo)
16:40:16 <xarses> I just poked infra guys about it
16:40:34 <mihgen> :D
16:40:39 <mwhahaha> bug #1493483
16:40:41 <openstack> bug 1493483 in Fuel for OpenStack "Pacemaker service is unable to be started in time when zabbix plugins enabled" [Critical,Incomplete] https://launchpad.net/bugs/1493483 - Assigned to Ksenia Demina (kdemina)
16:41:17 <ikalnitsky> why it's assigned on Ksenia?
16:41:27 <mihgen> aglarendil: dilyin do you guys know context here? ^^
16:41:38 <maximov> we asked her to retest this case
16:41:45 <ikalnitsky> or, miss that
16:41:45 <maximov> without zabbiz plugin
16:42:12 <mihgen> so it seems like more incomplete.. ?
16:42:19 <maximov> well, yes
16:42:47 <mihgen> bug log clearly shows error in deployment
16:42:55 <mihgen> which is unlikely to be related to zabbix..
16:43:28 <maximov> but we have similar tests in swarm
16:43:31 <maximov> which works
16:43:42 <mihgen> from smakar there: "It has just been reproduced again"
16:43:49 <maximov> the only difference is zabbix
16:43:57 <mihgen> "the root cause that pacemaker could not start on node-13 node-14"
16:44:09 <mihgen> it might be a problem which is not always reproduced
16:44:30 <aglarendil> This is an old update
16:44:31 <sgolovatiuk> mihgen: that lab was broken also
16:44:54 <sgolovatiuk> I proposed to retry on clean env with recent ISO
16:45:11 <maximov> yes. at least with 288
16:45:19 <mihgen> ok. We need a lab I think guys
16:45:29 <mihgen> so if she reproduces a bug, we need someone to take a look...
16:45:47 <mihgen> someone experienced in our HA..
16:45:57 <sgolovatiuk> k
16:46:00 <mihgen> logs don't seem to be enough..
16:46:23 <mihgen> ok.. thanks for the update on this bug. moving on?
16:46:29 <sgolovatiuk> yep
16:46:32 <xarses> #topic https://bugs.launchpad.net/fuel/+bug/1493390 (ikalnitsky)
16:46:35 <openstack> Launchpad bug 1493390 in Fuel for OpenStack 8.0.x "Deployment failed on connectivity check to http://mirror.fuel-infra.org/mos-repos/ubuntu/7.0/ after stop deployment action " [High,Confirmed] - Assigned to Fuel build team (fuel-build)
16:46:57 <ikalnitsky> actually, guys, the issue is pretty simple. sometimes we face a connectivity issue in our system tests before deployment
16:47:01 <ikalnitsky> unfortunately, we don't even log a response status code, and after env revert (obviously) everything's working fine.
16:47:04 <ikalnitsky> i remember we had a similar issue last release.
16:47:07 <ikalnitsky> so i've prepared a patch that adds 3 retries and logs response status code
16:47:10 <ikalnitsky> #link https://review.openstack.org/#/c/222172/
16:47:14 <ikalnitsky> but i'm not sure about backporting to stable/7.0
16:47:17 <ikalnitsky> i think it's medium if it's not often
16:47:20 <ikalnitsky> is there any QA around? could someone comment how often we face it?
16:47:23 <ikalnitsky> that's it :)
16:47:29 <mihgen> ikalnitsky: I'd say it's medium
16:47:55 <ikalnitsky> so mark it as won't fix for 7.0  ?
16:48:18 <ddmitriev> ikalnitsky: we faced it about once a week, maybe a little more often
16:48:23 <mihgen> more over, if it is reproduced only at certain time (like stop during certain phase), then I can't really say it's high
16:48:38 <aglarendil> It very much could be the issue with default route that we fixed
16:48:45 <mihgen> to me, frankly, stop of deployment in general is not a blocking from release feature
16:48:52 <ikalnitsky> mihgen: we do have this check only in before deployment checks.. when you add new nodes
16:49:06 <ddmitriev> mihgeh: if our repositories are failed for customers - it isn't looks like production ready
16:49:13 <ikalnitsky> but if it fails, you can retry it manually again
16:49:35 <aglarendil> Folks Can we try to reproduce it with the latest ISO?
16:49:50 <mihgen> if there is an easy workaround - it's not Critical then for sure
16:50:03 <mihgen> and now we don't accept anything but critical bugfixes to stable/7.0
16:50:36 <mihgen> so I don't know why we need to even think about backporting, if it's not a critical bug ..
16:50:44 <ikalnitsky> mihgen: agree
16:50:55 <xarses> why cant we propose it to stable/7.0
16:51:02 <xarses> then if we cut a new iso it's there
16:51:10 <xarses> otherwise don't cut a new iso just for this
16:51:14 <mihgen> xarses: because we are in HCF
16:51:20 <xarses> I think it's important to have retries here
16:51:32 <mihgen> xarses: I'd be rather strict
16:51:32 <xarses> one shot is, well quite lame
16:51:39 <mihgen> and follow agreement
16:51:53 <mihgen> we have enough backports,which introduce regressions sometimes
16:51:57 <aglarendil> It could be fixed by l3 clear route patch
16:52:15 <mihgen> xarses: it can also be targeted to updates
16:52:21 <angdraug> xarses: +1 to mihgen, even a trivial change introduces a chance of regressions, we have to be very conservative
16:52:24 <mihgen> and delivered in updates channel after release
16:52:30 <ikalnitsky> +1 here
16:52:40 <ikalnitsky> let's move it to updates, if someone think it's important
16:52:45 <ikalnitsky> but i'd remove from 7.0 at all
16:53:13 <mihgen> ok great.
16:53:17 <mihgen> moving on?
16:53:29 <xarses> #topic https://bugs.launchpad.net/fuel/+bug/1494314 (idv1985)
16:53:31 <openstack> Launchpad bug 1494314 in Fuel for OpenStack "Add of detached keystone node to existing cluster failed with Call cib_apply_diff failed (-205): Update was older than existing configuration" [High,Confirmed] - Assigned to Dmitry Ilyin (idv1985)
16:55:10 <dilyin> this bug is because we have not refactored haproxy_ocf to use pacemaker wrappers as all other services do. Because of that cluster-haproxy task is failing when it's being run on non-controller nodes as detached keystone plugin does
16:55:58 <dilyin> we can either remove hardcoded primary-controller role from the task or finish the task refactoring
16:56:16 <aglarendil> dilyin: I suppose we just fix haproxy tasj
16:56:34 <mihgen> I'd vote for proper fix in master, and don't backport anything to stable/7.0 now
16:56:39 <dilyin> actually i'm going to do both
16:56:42 <aglarendil> It is a simple similar change that can be automatically fixed by our ci
16:56:42 <mihgen> since it's not critical..
16:56:43 <angdraug> looks intrusive to me, do we really want to make commits like that in 7.0 before HCF?
16:56:57 <aglarendil> It is actually critical mihgen
16:57:10 <aglarendil> It breaks detached services feature
16:57:13 <mihgen> "before HCF" ?
16:57:23 <angdraug> err, I meant after HCF of course
16:57:31 <mihgen> aglarendil: for all services or just keystone?
16:57:47 <aglarendil> And the change is similar to the same for other services
16:58:11 <dilyin> i'll post the quick fix to make it work and we'll be able to inish refactoring on the next release
16:58:43 <mihgen> I'd consider delivering of a fix in updates channel
16:59:01 <xarses> 2 min
16:59:05 <xarses> 1
16:59:06 <mihgen> this feature is fairly complex and requires a lot of additional work anyway
16:59:26 <angdraug> time
16:59:41 <xarses> #endmeeting