16:01:16 <vkozhukalov> #startmeeting Fuel
16:01:17 <openstack> Meeting started Thu Apr 24 16:01:16 2014 UTC and is due to finish in 60 minutes.  The chair is vkozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:18 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:20 <openstack> The meeting name has been set to 'fuel'
16:01:34 <vkozhukalov> Who is here. Checking in.
16:01:44 <christopheraedo> Hello!
16:01:51 <holser_> Hi!
16:02:07 <vkozhukalov> agenda is here
16:02:12 <vkozhukalov> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:02:15 <mattymo> Hi!
16:02:20 <evgeniyl> Hi!
16:02:31 <vkozhukalov> #topic Announcements
16:02:48 <vkozhukalov> mihgen was going to say some words
16:03:05 <vkozhukalov> #topic current status (overall), bugs statistics
16:03:14 <mihgen> hi all. we are in a phase of acting bug squashes
16:03:29 <mihgen> with still a few exceptions on mandatory for 5.0 features
16:03:38 <mihgen> so let's talk about bugs first.
16:04:06 <mihgen> after consuming icehouse, I expected even more bugs, frankly, but looks like we are pretty good
16:04:19 <mihgen> over last week there were 53 income bugs
16:04:26 <mihgen> and 94 went away
16:04:38 <vkozhukalov> great
16:04:50 <mihgen> it means that we are doing good job on squashing them. the bad side of the story is that we still have a lot of bugs unresolved
16:05:07 <mihgen> closed 266 in 5.0, still open - 174
16:05:09 <vkozhukalov> is there a  link where we can see the summary?
16:05:18 <mihgen> #link http://fuel-launchpad.mirantis.com/project/fuel/bug_table_for_status/Open/5.0
16:05:33 <mihgen> I'm using this to see stats, it gathers stats from launchpad
16:05:47 <vkozhukalov> and a plot
16:05:51 <vkozhukalov> #link http://fuel-launchpad.mirantis.com/project/fuel/bug_trends/5.0
16:06:06 <mihgen> another thing is that we must still support stable/4.1 branch
16:06:25 <aglarendil> hi, everyone
16:06:29 <mihgen> folks, please propose changes there, which have tag "backports-4.1.1" in LP
16:06:42 <mihgen> and by proposing them try to no to break it :)_
16:07:00 <mihgen> we broke stable/4.1 last night, I hope the fix is on the way
16:07:04 <mihgen> aglarendil: right?)
16:07:21 <aglarendil> mihgen: yep. there was some miscommunication
16:07:23 <mihgen> angdraug: there were no rabbit3, but we merged puppet module manifests relying on it
16:07:53 <mihgen> looking forward for it being fixed soon.
16:08:02 <mihgen> Ok, about exceptions
16:08:06 <angdraug> yeah, I missed the part about ha-mode in the rabbitmq3 ttl commit
16:08:31 <vkozhukalov> #chair vkozhukalov
16:08:32 <openstack> Current chairs: vkozhukalov
16:08:34 <mihgen> folks will provide exact status, but we have basically fuel upgrades feature still incomplete and few things around it
16:08:50 <mihgen> that's it for update.
16:08:56 <vkozhukalov> forgot  to set a chair )
16:09:04 <mihgen> anything else I should mention about?
16:09:20 <mihgen> vkozhukalov: let's proceed otherwise
16:09:27 <vkozhukalov> moving on
16:09:43 <vkozhukalov> #topic Activities updates & questions
16:09:49 <mihgen> and thanks everyone for such heroic work on squashing bugs and staying late, but ensuring that ISO works!!!
16:09:57 <vkozhukalov> #topic fuel master upgrade scripts status
16:10:02 <mihgen> evgeniyl: your time)
16:10:08 <mattymo> evgeniyl, you first then I'll talk
16:10:17 <evgeniyl> ok
16:10:42 <vkozhukalov> mattymo: please wait before I change a topic
16:11:00 <vkozhukalov> *till
16:11:57 <evgeniyl> So, I had a lot of problems like https://github.com/dotcloud/docker-py/pull/200 with docker-py, it's a binding for docker, today I had working nailgun/nginx/astute, and had a problem with rabbitmq, then I took new iso and images, suddenly there was broken postgresql, I fixed it, not I'm trying to continue to work on rabbit..
16:12:20 <evgeniyl> s/not/now/
16:12:43 <mihgen> evgeniyl: how far are you from finishing this?
16:12:55 <dpyzhov> broken psql?
16:13:27 <evgeniyl> dpyzhov: postgresql container was broken, I had to take oldest one.
16:14:09 <evgeniyl> mihgen: I hope today it will work with additional unmerged patches for docker-py
16:14:27 <mihgen> evgeniyl: ok, looking forward to see it working ....
16:14:36 <mihgen> anything else you want to add?
16:14:48 <meow-nofer_> evgeniyl, still using new postgresql version, right?
16:15:12 <evgeniyl> meow-nofer_: it's not related with new postgresql
16:15:19 <evgeniyl> meow-nofer_: there was just broken container
16:15:38 <vkozhukalov> evgeniyl: thanks for the status
16:15:44 <vkozhukalov> moving on?
16:16:03 <vkozhukalov> #topic master node containerization current issues and caveats
16:16:08 <mattymo> caveats :)
16:16:09 <vkozhukalov> mattymo: your turn
16:16:13 <mattymo> We had a number of recurring bugs that really needed to get addressed by the QA team which required fast iso building and distribution, so we split up into two halves this week.
16:16:28 <mattymo> I built containers using early scripts and got containers out to QA and found bugs while adidenko and holser_ worked on makefile improvements and polishing container workflow.
16:17:03 <mattymo> bugs discovered during this time include intermittent network issues, masquerading of source ips (affects rsyslog), and other intermittent connection issues
16:17:09 <mattymo> Accomplished items this week include puppet flow bugs with OSTF, yaml placement and symlink issues, and we found a workaround for issues with docker inter-container communication via iptables.
16:17:41 <mattymo> and on behalf of adidenko and holser_, we have functional scripts that integrate well into our Fuel ISO preparation which run puppet inside a container for each service, then export and compress in a 280mb tar.lrz file and bundle it in the iso and deploys beautifully (But a bit slowly)
16:18:30 <mihgen> mattymo: does it mean that we actually have everything in place for building ISO, but just need to fix bugs?
16:18:43 <mattymo> I believe our latest work items include testing the workaround for docker ICC breakage and log rotation
16:18:57 <mattymo> yes I believe we're mergable today
16:19:00 <aglarendil> ICC = inter-container communication
16:19:22 <mihgen> mattymo: ok, that's cool
16:19:30 <mihgen> let's get folks to review
16:19:50 <mihgen> mattymo: and when evgeniyl's code is ready, we will simply replace bash scripts with his code?
16:20:03 <meow-nofer_> actually I would review some Go code, if you have it :)
16:20:55 <mattymo> no
16:20:58 <mattymo> his code relies on mine
16:21:19 <vkozhukalov> mattymo: any other details? moving on?
16:21:20 <mattymo> evgeniyl's system currently does orchestration, I believe, to migrate data from 1 container to another, or to simply replace
16:21:50 <mattymo> dockerctl still gets used for starting, stopping, and doing all the pre/post hooks necessary to get each service up. It ought to be rewritten in python completely
16:21:56 <mihgen> evgeniyl: I thought it's foe bootstrap of master too
16:23:11 <vkozhukalov> ok, moving on
16:23:13 <evgeniyl> mihgen: we can use my code, but it's not stable enough, and I don't think that we will have tim to test it properly
16:23:25 <alex_didenko> guts, I have a question on make iso
16:23:34 <alex_didenko> s/guts/guys/
16:23:43 <mihgen> alex_didenko: go ahead
16:24:20 <alex_didenko> do we need an option to make iso with docker and without? Something like variable USE_DOCKER=true/false
16:24:24 <mihgen> evgeniyl: I meant you code to be used in bootstrap_admin_node.sh
16:24:48 <alex_didenko> so if we export USE_DOCKER=false and run "make iso" - it builds the old-style Fuel iso
16:24:58 <mihgen> I'm not sure
16:25:04 <mihgen> dpyzhov: do we need it?
16:25:10 <evgeniyl> mihgen: why not?
16:25:19 <evgeniyl> mihgen: it looks like a good idea
16:25:23 <dpyzhov> alex_didenko: we have such flag
16:25:23 <angdraug> +1
16:25:36 <mihgen> well ideally I'd like to avoid additional axis in matrix of things which should work
16:25:53 <mihgen> so to work always on production ISO
16:25:56 <mattymo> I think we ought to leave a fallback route just in case we find a critical bug we can't overcome
16:25:59 <mattymo> but default to docker
16:25:59 <dpyzhov> right now we use PRODUCTION=prod for old-fashion isos and PRODUCTION=docker for docker
16:26:07 <mihgen> unless there are very good points saying no
16:26:14 <mattymo> dpyzhov, has the route I wanted to use too
16:26:20 <mattymo> just because it's a precedent
16:26:27 <mihgen> mattymo: that's the only exception I can see so far
16:26:38 <mihgen> but moving forward, I don't think we need it
16:26:55 <mihgen> if we work on non-containers mode, and release in containers, we are always with risk
16:26:55 <angdraug> I think we should remove it when docker option becomes stable and proven
16:27:00 <mihgen> of breaking stuff
16:27:10 <mihgen> angdraug: ok, then I'd agree
16:27:24 <aglarendil> but we are having some issues with iso building performance
16:27:26 <christopheraedo> +1 to "docker only" once it's stable
16:27:28 <mihgen> alex_didenko: how much resources does it take to support both
16:27:34 <aglarendil> and bootstraping of master node AFAIK
16:27:53 <alex_didenko> mihgen: it should not take much time
16:27:59 <alex_didenko> few hours maybe
16:28:10 <mihgen> how much do we pay now for containers?
16:28:21 <aglarendil> a lot, I guess
16:28:23 <mihgen> how much time does it take to build ISO, and install master node
16:28:38 <mihgen> in comparison with what we had before
16:28:50 <holser_> it’s a huge difference
16:28:50 <mattymo> to build all containers is an extra 30 minutes.. maybe more or less
16:29:06 <alex_didenko> building ISO with docker containers on 1 CPU VM with 2G ram: Took 54 min
16:29:06 <mattymo> if we had a build system with AUFS support it would be much faster
16:29:19 <mattymo> but we build and deploy with devicemapper which is far slower
16:29:53 <mihgen> mattymo: what if I update just nailgun
16:30:01 <mattymo> install of master node is bottlenecked in docker load. it replays each layer in an lxc env, dumps it, adds new layer, etc...
16:30:02 <mihgen> mattymo: would not I have to rebuild all containers?
16:30:26 <mattymo> mihgen, it's hard to tell what code is "just nailgun" with a basic utility
16:30:46 <mihgen> what do you mean under basic utility?
16:30:49 <mattymo> but yes if you make changes to nailgun code you need a new nailgun container
16:31:06 <mihgen> mattymo: then I would need to rebuild only nailgun container, right?
16:31:10 <dpyzhov> mattymo: is it possible to update pre-built container?
16:31:13 <mihgen> and can download all other existing
16:31:17 <mattymo> or if you change nailgun::venv puppet class, that only affects nailgun container. but if you change nailgun::packages or nailgun::supervisor class, you have to rebuild all
16:31:18 <alex_didenko> please also not that you can build ISO with prebuilt containers - it will take much less time since it just outs archive with containers on ISO
16:31:25 <mihgen> dpyzhov: that's the question which I had too
16:31:27 <alex_didenko> s/not/note
16:31:33 <mattymo> alex_didenko, but it won't update the package
16:32:01 <mihgen> ok, we need to talk about it separately
16:32:08 <mihgen> we would need to come up with improvements
16:32:18 <mihgen> anyway, otherwise we slow up development
16:32:24 <mattymo> dpyzhov, yes if we script it to, for sure
16:32:45 <mihgen> mattymo: anything else to add?
16:33:38 <mattymo> mihgen, nope. Let's move on
16:33:41 <vkozhukalov> #topic  icehouse support
16:33:49 <mihgen> aglarendil: your topic ;)
16:34:12 <aglarendil> I am here
16:34:21 <aglarendil> we have almost everything done, except several bugs
16:34:38 <aglarendil> fortunately almost all of them are related to packages
16:34:55 <mattymo> nice!
16:35:03 <mihgen> unfortunately for rvyalov )
16:35:19 <aglarendil> 1) neutron gre datapath is not working in centos 6.5 - IPGRE demux is compiled in and blocks OVS
16:35:51 <aglarendil> 2) mysql reconnect patches from oslo are still not in the main code for some projects so we are waiting for hardening team to do this
16:36:20 <aglarendil> there have been also several bugs with neutron that we have successfully fixed
16:36:40 <aglarendil> I hope we can fix everything by the end of the week
16:37:03 <mihgen> aglarendil: cool
16:37:05 <aglarendil> that's all, I think
16:37:07 <vkozhukalov> aglarendil: optimist -)
16:37:11 <mihgen> nurla: do you have anything to add?
16:37:22 <mihgen> regarding icehouse?
16:37:49 <nurla> mihgen: i'll hope  "no"
16:37:54 <mihgen> )
16:38:04 <vkozhukalov> moving on
16:38:04 <mihgen> thanks. vkozhukalov - let's move on
16:38:21 <vkozhukalov> #topic status of versioning for master upgrade
16:38:35 <akasatkin> Upgrade part for Nailgun (propagation of versions and orchestrator parameters) as about to be merged. Some additional validation and cosmetic fixes are to be made – can be added as separate PR or included into this one. Doc: https://etherpad.openstack.org/p/upgrades-orchestrator-data. It will be merged when library part (dilyin) is ready.
16:39:10 <mihgen> akasatkin: can you share links to patchsets involved?
16:39:19 <mihgen> akasatkin: we really should move asap with this
16:39:37 <mihgen> mattymo: evgeniyl: can we survive without these fixes ?
16:39:40 <akasatkin> Nailgun: https://review.openstack.org/#/c/87722/
16:40:00 <akasatkin> dilyin started library part today
16:40:19 <mihgen> he is not here unfortunately
16:40:24 <mihgen> how hard is his part?
16:40:40 <mihgen> did he provide any estimates?
16:40:58 <mihgen> aglarendil: are you aware about dilyin's part there? ^^
16:41:00 <akasatkin> I suppose it shouldn't be hard. See etherpad
16:41:08 <aglarendil> mihgen: yep
16:41:12 <akasatkin> for details
16:41:21 <akasatkin> https://etherpad.openstack.org/p/upgrades-orchestrator-data.
16:41:53 <mihgen> ok. vkozhukalov - let's move on
16:42:03 <vkozhukalov> #topic vcenter status
16:42:22 <mihgen> nurla: will you provide status or someone else on ^^
16:43:08 <eshumakher> i can provide some updates
16:43:29 <mihgen> ykotko: are you around? what about testing of vcenter integration?
16:43:36 <mihgen> eshumakher: please go ahead
16:43:58 <nurla> mihgen: we have problems with our env IT problems, but Egor already create some issues
16:44:38 <eshumakher> what about Ubuntu? Are you gonna test it?
16:44:40 <nurla> as I catch we also should check ubuntu
16:44:47 <eshumakher> and VNC?
16:45:16 <mihgen> I see an issue that VMs do not get IPs
16:45:33 <eshumakher> the problem is in environment setup
16:45:35 <nurla> yes, this problems with vlans
16:46:12 <mihgen> does it mean, that we can only test that deploy phase passes?
16:46:29 <mihgen> we still able to create VMs, right? we should be able to get VNC access still
16:46:35 <mihgen> even if network is down for VM
16:47:05 <nurla> yes, we able to create instance
16:47:06 <mihgen> by listening traffic, we can still try to see if there are DHCP requests trying to get out of vmware host
16:47:31 <mihgen> ok, any more updates on this? are there any items we still have to add?
16:47:38 <mihgen> (except docs)
16:48:02 <eshumakher> what abot HA?
16:48:29 <eshumakher> are we gonna leave it the way it is done now?
16:48:30 <nurla> eshumakher: we don't get Roman point
16:48:36 <nurla> about HA
16:48:51 <mihgen> I sent an email with a question how we gonna handle
16:49:06 <mihgen> there were no response, let's create bug for disabling it then
16:49:14 <mihgen> we can revert it any time otherwise
16:49:27 <mihgen> it's not gonna be HA, so I'm strongly against enabling it
16:49:54 <mihgen> this "HA" version was about having nova-network only on one controller, which is SPoF obviously
16:49:56 <eshumakher> +1
16:50:15 <mihgen> eshumakher: will you create a bug? or please ask someone to do so..
16:50:27 <eshumakher> ok, i will
16:50:34 <mihgen> eshumakher: thanks
16:50:39 <eshumakher> will Egor test Ubuntu and VNC?
16:50:47 <mihgen> ykotko: will you?)
16:51:13 <mihgen> eshumakher: anything else to discuss?
16:51:19 <eshumakher> nope
16:51:27 <mihgen> nurla: I hope ykotko will find time to test ubuntu & vnc ..
16:51:36 <mihgen> vkozhukalov: let's move on
16:51:38 <vkozhukalov> Do we need to open 'open discussion' topic?
16:51:51 <mihgen> I have a few bugs which I'd like to raise here
16:51:57 <mihgen> and clarify status / get folks involved
16:52:05 <vkozhukalov> #topic Open discussion
16:52:14 <mihgen> but I would first listen if someone else has any other topic
16:52:26 <mihgen> so please put "?" and let us know if you do
16:53:01 <mihgen> no one so far? Ok. #1277844 Corosync doesn't stop during the primary controller deployment  #link https://bugs.launchpad.net/fuel/+bug/1277844
16:53:03 <mihgen> aglarendil: ^^
16:53:15 <mihgen> please take a look what's our plan over it
16:53:20 <mihgen> angdraug: xenolog you too
16:53:27 <aglarendil> I am not quite aware if we can fix this bug in this release
16:53:47 <aglarendil> it requires rewriting of start sequence and usage of pacemaker master control plugin
16:53:59 <aglarendil> also it is not quite frequently reproducible
16:54:16 <aglarendil> i suggest to move it to 5.1 as soon as Centos 7.0 is available
16:54:19 <mihgen> ok, understood. we will likely slip it then
16:54:24 <aglarendil> then we can easily
16:54:34 <aglarendil> move to dependency-based initialisation
16:54:46 <mihgen> thanks. next - we have bunch of disk -related issues
16:54:59 <mihgen> vkozhukalov:
16:55:00 <mihgen> #1296985 ceph-deploy osd prepare failed. GenericError: Failed to create 1 OSDs #link https://bugs.launchpad.net/fuel/+bug/1296985
16:55:03 <mihgen> this for example
16:55:13 <mihgen> vkozhukalov: will you do it or we should pass it to someone else?
16:55:32 <mihgen> 1306491 no disk information leading to error during node allocation #link https://bugs.launchpad.net/fuel/+bug/1306491
16:55:40 <mihgen> there are more I believe
16:55:57 <angdraug> is #1296985 reproducible?
16:56:10 <meow-nofer_> mihgen, I believe ikalnitsky is now working on disk issue
16:56:44 <mihgen> angdraug: should we close #1267937 No warning for HA with OSD and MON roles on same nodes #link https://bugs.launchpad.net/fuel/+bug/1267937, as there is bp about it https://blueprints.launchpad.net/fuel/+spec/fuel-ceph-roles ?
16:56:54 <mihgen> meow-nofer_: I'm not sure about that
16:57:01 <ikalnitsky> yes, the fix on review already: #link https://review.openstack.org/#/c/89813/ (it's about disks)
16:57:14 <vkozhukalov> mihgen: will take a look at https://bugs.launchpad.net/fuel/+bug/1296985
16:57:41 <mihgen> ikalnitsky: meow-nofer_ that's completely from another story, but it's good to see this being fixed too
16:57:57 <mihgen> vkozhukalov: please de-assign all issues you don't work on them
16:58:07 <angdraug> mihgen: the fuel-ceph-roles BP doesn't fix the docs part of #1267937
16:58:09 <mihgen> vkozhukalov: we will try to find folks to fix.. .
16:58:23 <mihgen> angdraug: ok then should we keep a bug as is in 5.0?
16:58:38 <mihgen> or we need to close it, create separate for docs? or simply move to 5.1?
16:58:45 <vkozhukalov> mihgen: ok
16:58:56 <mihgen> angdraug: we should do something with bug in 5.0 :)
16:59:02 <vkozhukalov> 2 minutes
16:59:03 <angdraug> I think we should fix it in the docs for 5.0
16:59:05 <meow-nofer_> mihgen, this is exactly workaround for this issue)
16:59:13 <angdraug> that's why I keep it around and assigned to me
16:59:32 <mihgen> aglarendil: 1274756 Expired Keystone tokens should be cleaned up regularly #link https://bugs.launchpad.net/fuel/+bug/1274756
16:59:38 <vkozhukalov> let's move to #fuel-dev
16:59:43 <mihgen> aglarendil: what are we up to with this?
16:59:57 <vkozhukalov> thanks everyone
16:59:57 <aglarendil> we are going to implement memcache as a backend for keystone tokens
17:00:06 <evgeniyl> thanks
17:00:11 <vkozhukalov> #endmeeting Fuel