16:00:43 <kozhukalov> #startmeeting Fuel
16:00:43 <openstack> Meeting started Thu Nov 27 16:00:43 2014 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:43 <ikalnitsky> o/
16:00:44 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:47 <openstack> The meeting name has been set to 'fuel'
16:00:53 <kozhukalov> who is here?
16:01:11 <kozhukalov> agenda as usual
16:01:15 <mihgen> hi all
16:01:17 <kozhukalov> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:01:19 <sambork> hi
16:01:23 <akislitsky_> hi
16:01:40 <kozhukalov> #topic Announcements (mihgen)
16:02:00 <agordeev> hi
16:02:03 <mihgen> hi all, our focus is 5.1.1
16:02:06 <meow-nofer> sup
16:02:08 <dpyzhov> hi
16:02:18 <mihgen> main question now about mellanox patch
16:02:39 <mihgen> don't see mlnx folks here though.. (
16:02:56 <mihgen> #link https://review.openstack.org/#/c/137421
16:03:16 <mihgen> this is the patch. it blocks 5.1.1 HCF now. Let's review and move it forward
16:03:39 <mihgen> folks, any other blockers?
16:04:34 <mihgen> https://bugs.launchpad.net/fuel/+bug/1393771 - this one is going to be updated that it's very hard to reproduce (2 out 80 tries)
16:04:37 <uvirtbot> Launchpad bug 1393771 in fuel/5.1.x "HA neutron environment  - there is no connectivity to the instance" [Critical,Confirmed]
16:05:04 <mihgen> kozhukalov: this is on you. https://bugs.launchpad.net/fuel/+bug/1395761 - is it a blocker for 5.1.1 ?
16:05:05 <uvirtbot> Launchpad bug 1395761 in fuel/6.0.x "docker containers built twice during upgrade tarball " [High,In progress]
16:05:27 <nurla> we are waiting test's results for 1393771
16:05:45 <nurla> Andrey will provide result
16:06:12 <mihgen> ok, thx
16:06:17 <nurla> upgrade story we've verified yesterday
16:06:25 <nurla> for 5.1-5.1.1
16:07:48 <kozhukalov> mihgen: there is patch https://review.openstack.org/#/c/137415/
16:08:14 <mihgen> yeah but do we need it in 5.1.1P
16:08:46 <kozhukalov> it is not a blocker
16:09:15 <dpyzhov> https://bugs.launchpad.net/fuel/+bug/1395761 moved to 5.1.2
16:09:17 <uvirtbot> Launchpad bug 1395761 in fuel/6.0.x "docker containers built twice during upgrade tarball " [High,In progress]
16:09:32 <mihgen> ok, good to know. Then the major thing left is mlnx patch
16:09:52 <mihgen> now about 6.0
16:10:04 <mihgen> we are targeting for HCF on Monday, Dec 1
16:10:11 <mihgen> how ready are we?
16:10:25 <mihgen> dpyzhov: what about python team?
16:10:46 <nurla> we haven't staging env for statistic, isn't it?
16:10:49 <dpyzhov> Web team: we are pretty ready. We have 3 bugs in progress and all of them will be merged before HCF
16:10:58 <mihgen> nurla: we don't I think
16:11:46 <mihgen> teran_: what do we need to make staging for stats?
16:11:49 <dpyzhov> But we have a bug with versioning of cobbler snippets. It affects our intension to update kernel
16:12:11 <mihgen> dpyzhov: evgeniyl__ can you guys tell more about the bug?
16:12:38 <evgeniyl__> mihgen: there is a but for glusterfs plugin https://bugs.launchpad.net/fuel/+bug/1396996
16:12:40 <uvirtbot> Launchpad bug 1396996 in fuel "[Plugins] Wrong task yaml for glusterfs plugin is used" [High,In progress]
16:13:06 <dpyzhov> mihgen: one moment
16:13:22 <evgeniyl__> mihgen: another one is a bug in fpb https://bugs.launchpad.net/fuel/+bug/1396234
16:13:24 <uvirtbot> Launchpad bug 1396234 in fuel "fpb, shows incorrect message if task doesn't have timeout" [High,In progress]
16:13:25 <mihgen> evgeniyl__: how is it related to cobbler versioning ?
16:13:38 <dpyzhov> here it is: https://bugs.launchpad.net/fuel/+bug/1395052
16:13:40 <uvirtbot> Launchpad bug 1395052 in fuel "/var/lib/cobbler/kickstarts/ is not copied on upgrade" [High,Confirmed]
16:14:33 <mihgen> dpyzhov: how would it affect us?
16:15:12 <dpyzhov> mihgen: we can end up with broken ability to add new nodes to the 5.1 environments
16:15:46 <mihgen> so the scanario is the following: we upgrade 5.1.1 -> 6.0, then scale up old env on 5.1.1 (add more nodes), and new nodes contain new kernel version?
16:16:04 <dpyzhov> mihgen: exactly
16:16:37 <mihgen> at the same time, puppet modules will be used from 5.1.1, right?
16:16:43 <nurla> it is bad :(
16:16:44 <mihgen> so it won't have https://review.openstack.org/#/c/133341 piece?
16:17:30 <mihgen> ok, how hard is it to fix the bug and ensure that we deploy same kernel version as in 5.1.1 ?
16:17:31 <dpyzhov> we have ability to patch manifests for old environments. But it kind a risky idea
16:18:00 <nurla> mihgen: we'll make test for it
16:18:09 <mihgen> dpyzhov: do you mean apply the patch https://review.openstack.org/#/c/133341 during the upgrade, and new node will live with new kernel then?
16:18:38 <dpyzhov> mihgen: yes. as a possible solution
16:18:40 <mihgen> nurla: thanks, yeah it's needed...
16:18:55 <dpyzhov> but it’s better to fix snippets versioning or to postpone kernel update
16:19:03 <dpyzhov> every solution is risky
16:19:24 <mihgen> how hard is it to fix snippets versioning?
16:21:40 <dpyzhov> mihgen: about 2 days
16:21:41 <kozhukalov> it is quite hard
16:21:42 <xarses> do we need to version all snippits or just the ones like this?
16:22:11 <kozhukalov> we need have versioning for snippets for preseed for cobbler distro
16:22:34 <kozhukalov> my opinion is kinda week
16:22:39 <mihgen> xarses: what do you mean?
16:23:07 <dpyzhov> is it possible to check release version in preseed?
16:23:17 <dpyzhov> we can add a hack there
16:23:21 <kozhukalov> xarses: of course we don't need versions for all snippets
16:23:36 <mattymo> it's easier to just version all than some
16:23:46 <kozhukalov> xarses: but it is much easier to have versions for all
16:25:57 <dpyzhov> we can add ‘micro-versioning’ for snippets. just choose right kernel for each fuel release
16:26:14 <kozhukalov> we have to write something like 'if version == 5.1 then include foo_snippet_5.1'
16:26:59 <kozhukalov> for me it is not very good idea to implement that in 6.0
16:27:17 <mihgen> how much would it take to test the approach?
16:27:23 <kozhukalov> maybe much better to focus on removing cobbler in 6.1
16:27:52 <kozhukalov> mihgen: one day
16:28:01 <mattymo> maybe maybe not, but it's not like these tasks contained in snippets will magically vanish
16:28:41 <xarses> why can't this be solved by versioning the repo path?
16:28:52 <nurla> how we are going to support 5.0.x in this approach?
16:29:06 <nurla> kozhukalov: dpyzhov ^
16:29:35 <mihgen> xarses: looks like cobbler needs concrete kernel version
16:30:05 <mattymo> because of tftpdata
16:30:13 <mattymo> per profile
16:30:19 <kozhukalov> mihgen: it is not correct for definite kernel version
16:30:36 <kozhukalov> we can have different distros and profiles
16:31:09 <mattymo> actually, correction... the kernel for anaconda/preseed is fixed per cobbler profile (child of distro), then the kernel installed is in the versioned repo
16:31:40 <mattymo> this is fine for centos, but I'm not 100% sure on ubuntu
16:32:36 <kozhukalov> ok, guys, it looks like we need to discuss that somewhere else
16:32:43 <kozhukalov> moving on
16:33:04 <mihgen> aglarendil: can you provide the status on fuel-library in general?
16:33:07 <mihgen> readiness for hcf?
16:33:09 <aglarendil> yep
16:33:17 <aglarendil> for 5.1.1 it seems we have only mellanox issue
16:33:29 <mihgen> for 6.0
16:33:39 <aglarendil> for 6.0 we have only several bugs we are going to close today or tomorrow:
16:33:54 <aglarendil> most of them already have commits on review
16:34:11 <aglarendil> https://review.openstack.org/#/c/134964/
16:34:25 <aglarendil> this review is a bugfix for pacemaker providers
16:34:34 <nurla> Friday will be black
16:34:56 <aglarendil> also, we have replacement for ntp ready for merge
16:34:57 <nurla> merge party in library
16:35:07 <kozhukalov> #topic 3.13 kernel merge status (msemenov)
16:35:10 <aglarendil> and 3.13 kernel also
16:35:20 <mihgen> aglarendil: I don't think we are ready for ntp merge frankly speaking
16:35:25 <msemenov> hi
16:35:27 <mihgen> I'd postpone it
16:35:44 <msemenov> we have a bunch of chrequests on 3.13 kernel
16:35:44 <msemenov> https://review.openstack.org/#/c/133341
16:35:44 <msemenov> https://review.openstack.org/#/c/134520
16:35:44 <msemenov> https://review.openstack.org/#/c/135677
16:35:44 <msemenov> https://review.fuel-infra.org/#/c/937
16:36:03 <msemenov> only first 2 of them have some +1/+2
16:36:12 <msemenov> but we need to merge them asap
16:36:29 <msemenov> that's the update
16:36:47 <mihgen> msemenov: you saw the conversation above .. we will need to think about upgrade and support of older envs
16:36:52 <kozhukalov> msemenov, according to above discussion, we can not merge them w/o preseed versioning
16:36:54 <msemenov> about 10 mos-linux bugs are related to 3.13 update
16:37:01 <ykotko> ntp is not ready
16:37:09 <mihgen> aglarendil: ntp ^^^
16:37:27 <kozhukalov> guys, topic is 3.13
16:37:45 <mihgen> msemenov: we can merge but if we don't fix 5.1.1 scale up after upgrade then it's not gonna work out
16:38:54 <mihgen> ok folks let's review and test patches for kernel
16:39:13 <mihgen> kozhukalov: let's move on
16:39:48 <msemenov> who is going to work on preseed versioning?
16:39:50 <kozhukalov> #topic ntp
16:39:54 <msemenov> mos-linux?
16:40:19 <nurla> what issues on ntp, Egor could you clarify ?
16:40:25 <nurla> ykotko: ^
16:40:47 <kozhukalov> msemenov: no, python team and library
16:40:58 <msemenov> kozhukalov: ok thanks
16:41:33 <aglarendil> so, we have a custom ISO with ntp-dev packages that passes both BVTs
16:41:55 <aglarendil> and it fixes almost 4 bugs related to old ntpdate and ntp binaries which we install into our environments
16:42:04 <mihgen> aglarendil: bvts are good. but most of the issues we catch when we work with reverts of ceph nodes
16:42:21 <aglarendil> mihgen: it is related to our configuration of libvirt
16:42:21 <ykotko> ntp sync is freeze on computes after revert env from snapshot
16:42:31 <aglarendil> mihgen: to the system timer, I guess
16:42:42 <sbog> ykotko: there is some bug about it?
16:42:46 <Tatyanka_Leontov> Guys are we talked about external ntp feaature or about ntp issue that was found?  :)
16:42:58 <aglarendil> Tatyanka_Leontov: about update of ntp packages
16:42:59 <mihgen> ntp-dev, not external ntp
16:43:04 <aglarendil> Tatyanka_Leontov: not about extrenal ntp
16:43:23 <Tatyanka_Leontov> it good, because I thin ykotko tell us about feature)
16:43:30 <ykotko> yep
16:43:30 <nurla> yep)
16:43:32 <Tatyanka_Leontov> think
16:44:05 <aglarendil> okay, ykotko is it compute-specific?
16:44:05 <kozhukalov> ok, moving on
16:44:10 <sbog> ykotko, nurla: Can you provide link, cause I don't see any?
16:44:11 <aglarendil> because it should not be
16:44:23 <aglarendil> let's investigate it and move it to followup
16:44:24 <kozhukalov> #topic image based provisioning (agordeev)
16:44:25 <mihgen> it was about the feature, so it's not related
16:44:28 <agordeev> hi!
16:44:34 <kozhukalov> hi
16:44:46 <agordeev> 2 high priority bugs were found since the last weekly meeting.
16:44:48 <agordeev> 1 bugfix had been landed https://bugs.launchpad.net/fuel/+bug/1394617
16:44:50 <uvirtbot> Launchpad bug 1394617 in fuel "ntpd is stopped for all nodes" [High,Fix released]
16:44:50 <agordeev> 1 bugfix is still in progress and is on review. https://bugs.launchpad.net/fuel/+bug/1396078
16:44:52 <uvirtbot> Launchpad bug 1396078 in fuel "Kernel Panic - not syncing: UFS: Unable to mount root fs on uknown-block (0,0)" [High,In progress]
16:44:53 <agordeev> link https://review.openstack.org/#/c/137070/
16:45:04 <agordeev> Additionally, few concerns appeared such as forgotten snippets. At least ntp with ntp-dev package replacement and mellanox snippet.
16:45:06 <agordeev> ntp-dev already has patchsets on review. Related bugs are https://bugs.launchpad.net/fuel/+bug/1336748 and https://bugs.launchpad.net/fuel/+bug/1314958
16:45:08 <agordeev> link https://review.openstack.org/#/c/136829/
16:45:10 <agordeev> link https://review.openstack.org/#/c/136830/
16:45:12 <uvirtbot> Launchpad bug 1336748 in fuel "Need for sntp with '--uctimeout' option in repos" [High,In progress]
16:45:12 <agordeev> also it is not clear is ntp-dev a bug with high priority or just medium. Its status was changed just few hours ago.
16:45:14 <agordeev> mellanox snippet has not adopted into cloud-init boothooks yet.
16:46:19 <mihgen> agordeev: thx
16:46:20 <mihgen> agordeev: kozhukalov what about feature parity, if we replace cobbler with image-based, anything is missing?
16:47:20 <kozhukalov> mihgen: looks like nothing is going to be missed
16:47:34 <mihgen> ok. excellent!
16:47:42 <kozhukalov> agordeev: just implemented ntp feature
16:48:27 <agordeev> mihgen: i think a lot of hacks from snippets or pmanager.py could be missed somehow
16:48:49 <kozhukalov> agordeev: explain please
16:49:06 <mihgen> do you mean pmanager.py is not in use for image-based?
16:49:43 <kozhukalov> pmanager is one big hack
16:50:04 <agordeev> kozhukalov: things like 512b sector for xfs. or additional arguments for lvm utilities. We need to be very careful with this
16:50:20 <mihgen> ok, thx
16:50:21 <kozhukalov> my opin is not to backport all those sleeps from pmanager to fuel-agent
16:50:33 <kozhukalov> ok
16:50:39 <agordeev> mihgen: right, pmanager/snippets are not used for image-based
16:50:56 <kozhukalov> looks like time for open discussion
16:51:04 <kozhukalov> #topic open discussion
16:52:08 <kozhukalov> looks like no one is interested in having talk
16:52:12 <kozhukalov> closing
16:52:16 <nurla> thank you guys
16:52:16 <xarses> agordeev: yes, i know angdraug has ideas about writing tests to ensure there are few gaps from pmanager -> image prov
16:52:17 <kozhukalov> thanx everyone
16:53:03 <kozhukalov> #endmeeting