16:00:21 <xarses> #startmeeting fuel
16:00:23 <xarses> #chair xarses
16:00:23 <openstack> Meeting started Thu Aug  6 16:00:21 2015 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:25 <xarses> Todays Agenda:
16:00:26 <openstack> The meeting name has been set to 'fuel'
16:00:27 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:00:28 <openstack> Current chairs: xarses
16:00:29 <xarses> Who's here?
16:00:31 <mattymo_> hi!
16:01:14 <xenolog13> \~/
16:01:19 <angdraug> o/
16:01:20 <mihgen> hi
16:01:25 <akasatkin> hi
16:01:43 <vkramskikh> hi
16:01:46 <rvyalov> hi
16:01:48 <amaksimov> hi
16:02:02 <xarses> ok, lets get started
16:02:05 <xarses> #topic SCF Status (angdraug)
16:02:11 <angdraug> 7.0 soft code freeze will be in effect at 12AM tonight
16:02:11 <angdraug> today is the last day to merge anything that is not a fix for High or Critical bug
16:02:11 <angdraug> we're still in the red zone with the number of bugs
16:02:20 <angdraug> dixi :)
16:02:50 <angdraug> questions? moving on?
16:02:54 <mihgen> angdraug: rationale is to focus on high and critical now, right?
16:03:00 <angdraug> yes
16:03:03 <amaksimov> question:
16:03:04 <mihgen> so to converge by 3rd of Sep
16:03:26 <amaksimov> we will not merge anything related to Med bugs after SCF,right ?
16:03:32 <angdraug> yes
16:03:35 <mihgen> angdraug: and I think if there is minor work to be completed in some low / medium bugs in progress, those can still be merged after SCF
16:03:43 <mihgen> let's say for another week
16:03:52 <amaksimov> ok
16:04:07 <amaksimov> our review stat might affected
16:04:13 <angdraug> if there is medium bug or lower priority related work still pending, now is the time to speak up and ask for merge or exception
16:04:29 <xarses> mihgen: how will we accomidate that? SCF exceptions?
16:04:50 <angdraug> reviews affected by code freeze should be marked workflow-1 until HCF when we create stable/7.0 branches
16:05:05 <vkramskikh> what if there is a review request which closes 1 high and 2 medium bugs?
16:05:22 <angdraug> vkramskikh: that doesn't need an exception
16:05:34 <angdraug> as mihgen said, the motivation is to focus on fixing high bugs
16:05:35 <amaksimov> this is not going to work.. we will launch auto abandon script which will abandon reviews with -1
16:05:59 <xarses> amaksimov: its not supposed to hit workflow -1
16:06:27 <xarses> If some one sees it do so, please speak up so we can fix the script
16:06:33 <amaksimov> sure? I think we need to check it with devops.
16:07:05 <amaksimov> tiran ?
16:07:09 <mihgen> So I would say there are two major things with SCF: 1) to focus on High/ Criticals to meet deadlines; 2) to lower the risk of introducing regressions by fixes of low/medium. The less code change is, the lesser risk is
16:07:13 <angdraug> #link https://review.fuel-infra.org/#/c/10080/3/requests-abandon/abandon_old_reviews.sh
16:07:13 <mihgen> teran: ^^
16:07:33 <mihgen> amaksimov: script should not touch workflow -1, if it does - then it has to be fixed
16:07:39 <angdraug> it only abandons reviews with -2
16:08:06 <teran> mihgen: angdraug AFAIR it works already such way
16:08:11 <teran> we could check it out
16:08:21 <angdraug> yes, that's my impression as well
16:08:38 <angdraug> amaksimov: does that answer your question?
16:08:42 <amaksimov> yes
16:08:43 <akasatkin> some mediums just need to be checked. what to do with them?
16:08:55 <mihgen> akasatkin: like what?
16:09:12 <akasatkin> https://bugs.launchpad.net/fuel/+bug/1399004
16:09:12 <openstack> Launchpad bug 1399004 in Fuel for OpenStack "Running network verification on running env may break network" [Medium,Confirmed] - Assigned to Fuel Python Team (fuel-python)
16:09:14 <uvirtbot> Launchpad bug 1399004 in fuel "Running network verification on running env may break network" [Medium,Confirmed]
16:09:15 <uvirtbot> Launchpad bug 1399004 in fuel "Running network verification on running env may break network" [Medium,Confirmed] https://launchpad.net/bugs/1399004
16:09:33 <mihgen> this sounds rather High
16:09:45 <akasatkin> okay
16:09:45 <angdraug> +1
16:10:15 <xarses> ya, thats high
16:10:19 <mihgen> break of deployment in general sounds critical issue, if it can be reproduced only in certain cases - then it's high
16:11:05 <amaksimov> it doesn't break deployment
16:11:09 <amaksimov> just verification
16:11:12 <amaksimov> sometimes
16:11:14 <angdraug> to reiterate, the reason we're announcing soft code freeze earlier is to spend more time on high and critical bugs
16:11:49 <mihgen> amaksimov: nope according to the bug, you break working env
16:11:53 <mihgen> you load kernel module
16:11:56 <mihgen> and ovs breaks
16:12:10 <mihgen> moving on?
16:12:11 <xarses> ok, lets move along
16:12:15 <xarses> #topic puppet-librarian (angdraug)
16:12:50 <angdraug> puppet-librarian commits are being merged according to plan:
16:12:50 <angdraug> #link http://lists.openstack.org/pipermail/openstack-dev/2015-August/071347.html
16:12:50 <angdraug> to honor soft code freeze, I propose to merge the firewall module one day early
16:12:50 <angdraug> cinder module will need a rebase over https://review.openstack.org/209945
16:12:50 <angdraug> we should discuss an SCF exception for it next week
16:13:07 <angdraug> thoughts? objections?
16:13:29 <xarses> I think its minor, and we need to move forward with it
16:13:30 <xarses> +1
16:13:52 <angdraug> #link https://wiki.openstack.org/wiki/Fuel/Library_and_Upstream_Modules
16:13:55 <mattymo_> We had 1 custom patch to firewall module for docker stuff, but that isn't relevant today
16:14:12 <mattymo_> actually... now that I think about it, it was just a backport of a more modern version
16:14:24 <mattymo_> carry on, please continue with firewall module
16:14:29 <angdraug> thanks!
16:14:57 <angdraug> apache is already merged, this means we'll also merge apt and firewall today, and leave cinder until next week
16:15:17 <angdraug> that's it from me
16:15:20 <xarses> thanks
16:15:24 <mihgen> angdraug: cinder until HCF
16:15:36 <sgolovatiuk> Hi Fuelers...
16:15:57 <angdraug> mihgen: most likely, unless there's strong consensus in favor of SCF exception
16:16:01 <xarses> #topic Open discussion (bugs, reviews that need attention)
16:16:06 <mihgen> If we enter SCF I'm not sure if we want to continue doing exceptions of any kind
16:16:28 <amaksimov> if we want to release on time - no exceptions please
16:16:33 <angdraug> fair enough
16:16:40 <mattymo_> xarses, now that we're trying to focus on high priority bugs, we should make an effort to not "hog" all the bugs
16:16:41 <mihgen> amaksimov: +1
16:17:02 <mattymo_> If you hold a high priority bug and you're not working on it, pass it back to fuel-{python,library,etc}
16:17:08 <mattymo_> one that needs to be fixed for 7.0
16:17:12 <mattymo_> that way others can pick them up
16:17:14 <xarses> mattymo_: are we having problems with bugs assigned and not being worked again?
16:17:21 <amaksimov> yes
16:17:25 <amaksimov> +1 mattymo_
16:17:37 <mattymo_> our statistics are bad, but the # in fuel-library queue are not so high
16:17:45 <mihgen> mattymo_: +1
16:18:09 <angdraug> can we get a report on number of assigned open bugs per person?
16:18:12 <mihgen> we need to do daily triage passes over all bugs left
16:18:16 <IvanKliuk> https://bugs.launchpad.net/fuel/+bug/1365368 is tricky. I've been trying to fix it but I'm facing the problems with syncronization.
16:18:16 <openstack> Launchpad bug 1365368 in Fuel for OpenStack 7.0.x "Cannot update ip_ranges for management and storage networks" [High,In progress] - Assigned to Ivan Kliuk (ivankliuk)
16:18:16 <uvirtbot> Launchpad bug 1365368 in fuel/7.0.x "Cannot update ip_ranges for management and storage networks" [High,In progress]
16:18:17 <uvirtbot> Launchpad bug 1365368 in fuel/7.0.x "Cannot update ip_ranges for management and storage networks" [High,In progress] https://launchpad.net/bugs/1365368
16:18:46 <mattymo_> mihgen, triage of all bugs owned by individuals? that's a tricky challenge. better to solve it in a bug squashing session where we can get everyone together
16:18:53 <amaksimov> angdraug in our KPI report
16:19:11 <mihgen> mattymo_: I mean get everyone together, and go over all bugs
16:19:16 <mattymo_> +1
16:19:32 <xarses> IvanKliuk: wha't ca we do to help?
16:20:01 <xarses> It seemed to work fine for networks of type ip_range, doesn't work for type 'cidr'
16:20:15 <IvanKliuk> xarses: No help needed, thanks
16:20:20 <angdraug> mihgen: "going over" 100 bugs is going to take all day, doing it with the whole team will take up all day of the whole team, not sure about that
16:20:48 <mattymo_> angdraug, we manage to do it every release, usually twice :)
16:20:53 <IvanKliuk> I just cannot estimate when it's going to be fixed
16:21:02 <xarses> ok
16:21:09 <IvanKliuk> because it's not trivial fix
16:21:29 <xarses> any one else have CR or bugs to discuss?
16:21:35 <mihgen> I managed it before in this way: I was going over bugs, and doing ping-pong of folks working in the area
16:21:42 <IvanKliuk> I'll let you know about the results by the end of the day via email
16:21:51 <mihgen> to whether provide update, ask if they work on a bug or not, etc.
16:22:23 <angdraug> I like that approach better than "get everyone together":)
16:22:35 <mihgen> however one or two passes by whole team are still needed
16:22:51 <mihgen> to see the whole picture and hear concerns from everyone
16:22:55 <angdraug> amaksimov: found it, thanks. we can use that to identify who's hogging bugs and make sure they don't just sit on any of them
16:22:59 <mihgen> around certain bugs
16:23:30 <angdraug> sounds like a good agenda for the next IRC meeting
16:23:41 <mihgen> how are we doing with upgrades?
16:23:55 <angdraug> I hear last bits were merged yesterday
16:24:01 <amaksimov> yes
16:24:09 <amaksimov> I talked with ogelbukh today
16:24:18 <amaksimov> all ISO related stuff were merged
16:24:42 <amaksimov> but scripts still not ready
16:24:54 <amaksimov> but they are not part of ISO
16:25:22 <mihgen> ok..
16:25:38 <mihgen> at least we merged all ISO-related code
16:25:58 <amaksimov> yes
16:26:00 <mihgen> what about centos 6.6 for master node support?
16:26:08 <amaksimov> not yet merged
16:26:30 <angdraug> staging was blocked earlier today
16:26:38 <angdraug> as soon as it's green we can merge it
16:27:02 <mihgen> why was it red?
16:27:03 <angdraug> the cinder commit I mentioned earlier is expected to fix staging bvt:
16:27:17 <angdraug> #link https://review.openstack.org/209945
16:27:31 <angdraug> a regression introduced by a cinder package
16:28:05 <mihgen> ok
16:28:26 <mihgen> so do we have deadline set for 6.6 ?
16:28:58 <angdraug> today
16:29:15 <amaksimov> btw shouldn't we discuss the necessity to merge centos 6.6 ?
16:29:33 <angdraug> amaksimov: didn't we already have enough discussions about it?
16:29:33 <amaksimov> we had several conversations inside
16:29:46 <amaksimov> maybe we also should discuss it here?
16:30:04 <angdraug> it's not a fuel change, it's a mos change
16:30:22 <mihgen> it's Fuel relevant I'd say
16:30:31 <mihgen> Fuel itself has to support it
16:30:34 <amaksimov> we will update kernel
16:30:57 <mihgen> do we update bootstrap as well?
16:31:13 <angdraug> yes
16:31:17 <amaksimov> upgrade script will become more complicated
16:31:29 <amaksimov> it will require reboot of master node
16:31:35 <mihgen> amaksimov: who will be changing upgrade script?
16:32:07 <amaksimov> this is additional task which we didn't plan
16:32:13 <amaksimov> that's why I am against it
16:32:14 <mihgen> do we have patchsets on review for it?
16:32:21 <amaksimov> nope
16:32:36 <mihgen> do we have anyone here who is working on it?
16:32:36 <amaksimov> this is not just bugfix
16:32:39 <amaksimov> this is feature
16:32:49 <mihgen> this is feature; I agree
16:32:59 <amaksimov> so I was against it
16:33:06 <amaksimov> because it is too late
16:33:12 <amaksimov> for such big changes
16:33:25 <mihgen> angdraug: do you know if vkozhukalov/dpyzhov was consulting about upgrade scripts
16:33:27 <amaksimov> and the fact that we are removing resource from fuel 7.0
16:33:31 <angdraug> amaksimov: mihgen: please invite people actually working on centos6.6 if you want to have a meaningful discussion of it here
16:33:41 <mattymo_> why can't we just tell the user to reboot after?
16:34:40 <amaksimov> well, we can.. but again, we need to think about UX etc
16:35:08 <amaksimov> this should be tracked as feature. not just like an ordinary bug
16:35:09 <angdraug> good form would have been to add it to agenda yesterday and invite the feature lead
16:35:10 <mihgen> ok guys we can't discuss anything here since relevant people are not here. Let's follow up on this
16:35:38 <mihgen> angdraug: good form would be to bring yourself to the meeting if you are hacking features in fuel
16:35:51 <angdraug> amnk: please comment on centos 6.6 impact on upgrade script?
16:36:15 <amnk> angdraug: it has no impact to my knowledge
16:36:43 <angdraug> amaksimov: what makes you think there will be an impact?
16:36:45 <amaksimov> how we can upgrade kernel w/o reboot amnk ?
16:36:49 <mihgen> amnk: could you please update your name in IRC settings?
16:36:49 <evgenyl> I'm not sure if upgrade script should restart the nodes, it's dangerous action and we agreed that it should be user's conscientious decision.
16:37:04 <angdraug> evgenyl: +1
16:37:10 <mihgen> evgenyl: +1
16:37:25 <angdraug> amnk is Aleksandr Mogylchenko, feature lead for centos 6.6
16:37:40 <amnk> amaksimov: well, it is your decision to reboot node or not. It will work in any case
16:38:15 <angdraug> if we are not going to amend upgrade script to reboot the node, and I agree that we shouldn't, what other impact is there?
16:38:26 <amaksimov> amnk but you will have to notify user about it
16:38:32 <amnk> amaksimov: why?
16:38:37 <mattymo_> evgenyl, +1 to user initiated reboot
16:39:46 <ogelbukh> currently upgrade script tell user to run 'yum update; docker destroy all; docker start all'
16:40:17 <mihgen> angdraug: as you've been in the discussion, do we have committment to be on the hook to fix bugs / support upgrade procedure if needed?
16:40:17 <ogelbukh> adding 'reboot' here doesn't seem to add too much manual action :)
16:40:35 <angdraug> amnk has committed to that, yes
16:40:51 <mihgen> ok that's good
16:40:56 <angdraug> amnk: right?
16:41:03 <amnk> yes
16:41:06 <mihgen> amnk: did you guys check scale up story after upgrade?
16:41:30 <mihgen> let's say you have env with openstack, then you upgrade fuel, and now you want to add couple of nodes to old env
16:41:56 <mihgen> I'm about centos-based env specifically
16:42:24 <amnk> mihgen: this is for 7.0 - there are no centos-based envs there.
16:42:49 <angdraug> amnk: mihgen is talking about a 6.1 env deployed by a 6.1 fuel before upgrade to 7.0
16:44:09 <amnk> as I said in the email already - I could not get upgrade feature to work on stock 6.1. And others confirmed that it is not stable. So I do not have reliable test case to rely on
16:44:41 <mihgen> upgrade of Fuel?
16:44:49 <amnk> yes.
16:44:54 <mihgen> nurla: ^^ ?
16:45:19 <mihgen> can you please comment on stability of upgrade procedure? I've not heard about major issues for 6.0 -> 6.1
16:45:39 <mihgen> regardless, it doesn't mean that we just drop this from our testing matrix
16:45:45 <mihgen> and declare that we don't support it
16:45:46 <angdraug> has anyone tested 6.1-7.0 yet?
16:46:11 <mihgen> if we are about to deprecate a feature, it has to be done properly
16:46:18 <amnk> I just said that upgrade feature can not be used as reliable test case.
16:46:31 <amnk> I'm not saying we should deprecate it
16:46:40 <angdraug> amnk: doesn't have to be reliable, you only have to prove that you can get it to work
16:46:49 <mihgen> this is not a test case itself. angdraug +1
16:47:03 <mihgen> it
16:47:30 <mihgen> user use case is simple, I had 6.1 with centos; now I've heard 7.0 is cool and has kilo support
16:47:42 <mihgen> so I upgrade my fuel keeping old env on centos
16:47:53 <mihgen> and now I try to scale up old centos-based env
16:48:03 <mihgen> or do any other operations on that one
16:48:42 <amnk> well, if that user is able to get it working there will be no problems after we merge our custom build of kernel.
16:48:46 <mihgen> If we can't make it as such, then we would better not to even recommend upgrade to 7.0, and instead provide manual precedure
16:49:37 <angdraug> amnk: that's exactly what mihgen wants you to confirm
16:49:50 <amnk> angdraug: I confirm
16:50:20 <mihgen> amnk: the problem can be in cobbler snippets for example
16:50:53 <mihgen> if you update any of those as part of 6.6 work, ensure that it stays backward compatible
16:51:07 <amnk> mihgen: if there is a problem - I'll fix it.
16:52:08 <angdraug> I'm concerned that no one answered my question about testing 6.1-7.0 upgrade
16:52:27 <bookwar> "I could not get upgrade feature to work on stock 6.1" - we have upgrade tests which test ubuntu deployments after 6.1 to 7.0 upgrade, and they are green. Afaik Centos-based are work in progress by qa team
16:52:45 <angdraug> thanks
16:53:33 <angdraug> nurla: can you update on status of testing centos envs after upgrade to 7.0?
16:54:21 <angdraug> amnk: if the auto test of upgrades to 7.0 is green, you should be able to test it manually
16:54:59 <angdraug> we're running out of time, lets sum up
16:55:23 <angdraug> 1) there's no impact on Fuel, including upgrade script, but its message should be updated to recommend Fuel node reboot
16:56:08 <angdraug> 2) upgrades from 6.1 to 7.0 are verified by swarm and are green, so amnk is not blocked with testing 6.1 centos env scale up post-upgrade
16:56:24 <angdraug> any other blockers?
16:57:05 <amaksimov> just common sense - big change after FF
16:57:09 <sgolovatiuk> swarm doesn't check customized environments or environments with plugins
16:58:06 <mihgen> this is a big change, and risks have to be taken seriously
16:58:10 <amnk> please stop calling it a "big change". It is not.
16:58:23 <bookwar> sgolovatiuk: that's generic issue, for any kind of upgrade
16:58:38 <sgolovatiuk> amnk: it's a big change
16:58:41 <mihgen> yes it is - there are many things which can be affected
16:58:58 <angdraug> we already have centos 6.6 in the fuel docker containers
16:59:03 <mihgen> and we don't have any thorough test matrix for it
16:59:04 <amnk> sgolovatiuk: it was done silently inside docker containers, and nobody even noticed.
16:59:40 <mihgen> amnk: no one would notice change centos -> debian in containers
16:59:41 <mihgen> but not here
16:59:50 <angdraug> biggest part of it all is kernel upgrade
16:59:52 <sgolovatiuk> Do we test such scenarios? Do we test upgrades on production (live) environments? Do we have any --noop run?
16:59:53 <mihgen> when you change bootstrap as well, and other things
17:00:04 <mihgen> are affected, such as upgrade of fuel
17:00:14 <tmcpeak> o/
17:00:16 <hyakuhei> o/
17:00:18 <angdraug> but if a kernel upgrade to a different patch level within the same major version is a feature for us we have a bigger problem
17:00:23 <angdraug> time folks
17:00:28 <mihgen> xarses: closing
17:00:36 <IvanKliuk> bye! thanks!
17:00:49 <xarses> #endmeeting