16:01:10 <kozhukalov> #startmeeting Fuel
16:01:11 <openstack> Meeting started Thu Aug 13 16:01:10 2015 UTC and is due to finish in 60 minutes.  The chair is kozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:13 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:15 <openstack> The meeting name has been set to 'fuel'
16:01:17 <bswartz> sorry fuel guys, we'll get out
16:01:20 <kozhukalov> #chair kozhukalov
16:01:21 <mihgen> hi
16:01:21 <openstack> Current chairs: kozhukalov
16:01:23 <prmtl> hey
16:01:24 <amaksimov> hi
16:01:28 <angdraug> \o
16:01:30 <kozhukalov> hey everyone
16:01:30 <warpc__> hi
16:01:31 <evgenyl> hi
16:01:31 <alex_didenko> hi
16:01:32 <mwhahaha> hi
16:01:34 <dmitryme> рш
16:01:36 <dmitryme> hi
16:01:39 <nurla> hi
16:01:41 <SheenaG> Hi
16:01:46 <mattymo_> aloha
16:01:59 <kozhukalov> #topic Ubuntu-based bootstrap (msemenov)
16:02:04 <msemenov> hi all
16:02:12 <msemenov> Ubuntu bootstrap feature needs more testing right now. I've just written email to openstack-dev about it.
16:02:20 <msemenov> Currently, we have 2 bugs from scale team:
16:02:21 <ashtokolov> hi
16:02:24 <xenolog13> \~/
16:02:28 <msemenov> https://bugs.launchpad.net/mos/+bug/1482049
16:02:28 <openstack> Launchpad bug 1482049 in Mirantis OpenStack "Ubuntu boostrap uses wrong network interface to retrive the root filesystem image" [High,In progress] - Assigned to Alexei Sheplyakov (asheplyakov)
16:02:28 <uvirtbot> Launchpad bug 1482049 in mos "Ubuntu boostrap uses wrong network interface to retrive the root filesystem image" [High,In progress]
16:02:29 <uvirtbot> Launchpad bug 1482049 in mos "Ubuntu boostrap uses wrong network interface to retrive the root filesystem image" [High,In progress] https://launchpad.net/bugs/1482049
16:02:35 <msemenov> https://bugs.launchpad.net/fuel/+bug/1481721
16:02:35 <openstack> Launchpad bug 1481721 in Fuel for OpenStack "Only 100 of 200 nodes booted successfully with Ubuntu based bootstrap" [High,In progress] - Assigned to Alexei Sheplyakov (asheplyakov)
16:02:41 <msemenov> and we have added bvt_with_ubuntu_botstrap to SWARM:
16:02:48 <msemenov> http://jenkins-product.srt.mirantis.net:8080/job/7.0.system_test.ubuntu.bvt_ubuntu_bootstrap/
16:02:49 <uvirtbot> Launchpad bug 1481721 in fuel "Only 100 of 200 nodes booted successfully with Ubuntu based bootstrap" [High,In progress]
16:02:50 <uvirtbot> Launchpad bug 1481721 in fuel "Only 100 of 200 nodes booted successfully with Ubuntu based bootstrap" [High,In progress] https://launchpad.net/bugs/1481721
16:03:00 <msemenov> so, we need to test more, please help us.
16:03:19 <sgolovatiuk> hi
16:03:26 <msemenov> on the next week we want to switch swarm tests to the new ubuntu bootstrap
16:03:44 <mihgen> msemenov: bugs sound pretty serios
16:03:50 <kozhukalov> msemenov, do you mean we need manual testing?
16:03:50 <msemenov> becasue we have only 2 weeks left to decide whether to amke Ubuntu bootstrap by default or not
16:03:59 <mihgen> first one is perhaps fixed?
16:04:10 <msemenov> mihgen: but unreproducible right now
16:04:39 <mihgen> I'd vote for enabling it by default then if we can't reproduce bugs and there are no known blockers
16:04:55 <msemenov> kozhukalov: yes, we need everyone just try to deploy (switch to new bootstrap) on the different hardware. Especially partners and plugin developers.
16:04:58 <mihgen> which would prevent us from testing other functionality
16:05:24 <angdraug> mihgen: then -> when?
16:05:32 <mihgen> the only way is to just force everyone by making it default )
16:05:35 <msemenov> mihgen: yes, only by switching is to default right now can give us big amount of testing
16:05:56 <msemenov> mihgen: so, if you approve, Alexei will prepare a chrequest
16:06:10 <mihgen> angdraug: ? I'm saying that we should rather switch now
16:06:20 <mihgen> I'd like to hear other opinions
16:06:39 <angdraug> how much will it disrupt swarm?
16:06:39 <mihgen> any objections?
16:06:44 <kozhukalov> +1 for switching right now, early - better
16:06:56 <angdraug> anyone from QA?
16:07:24 <kozhukalov> nurla, around?
16:07:38 <msemenov> I had a conversation with nurla today. Main concern from her side is partners and plugin devs
16:07:46 <angdraug> I don't object, as long as it doesn't cause bvt disruption or swarm regression
16:07:51 <msemenov> so, we decided to post it to openstack-dev
16:08:01 <angdraug> do we have plugins that rely on centos based bootstrap?
16:08:02 <nurla> I prefer to use by default in 7.0 centos bootstrap and make switch only in 8/.0
16:08:37 <kozhukalov> angdraug, probably mlnx
16:08:41 <nurla> we have other team that use custom centos bootstrap
16:08:57 <msemenov> I can ping mlnx guys
16:09:03 <mihgen> which team? they still have time to switch
16:09:03 <msemenov> any others?
16:09:08 <kozhukalov> angdraug, they built their custom bootstrap, as far as i remember
16:09:13 <mihgen> also, we don't remove centos, do we?
16:09:22 <mihgen> so it would be easy to enable if someone needs it
16:09:26 <msemenov> mihgen: correct
16:10:07 <angdraug> nurla: what about bvt and swarm? what's the expected impact?
16:10:14 <mihgen> I don't know why would we slow down a change .. we bettter try and then see who complains
16:10:22 <msemenov> +1
16:10:25 <mihgen> we don't even know who could be broken now
16:10:36 <nurla> I need talk with my engineer
16:11:20 <mihgen> so folks - what's the decision? we've got to move on
16:11:20 <nurla> we should write code for switching, actually we haven't such functionality in swarm
16:11:26 <kozhukalov> ok, it looks like we are pretty done on this topic
16:11:30 <kozhukalov> moving on?
16:11:50 <mihgen> nurla: what code?
16:11:59 <mihgen> why do you need to write anything in swarm?
16:11:59 <msemenov> nurla: but from customer's side, we already wrote a script for switching
16:12:12 <msemenov> and also you can do it in fuelmenu
16:12:12 <nurla> for switching from centos to ubuntu bootstrap
16:12:17 <mihgen> swarm should not be even aware if it's centos or anything else
16:12:31 <mihgen> you just pxeboot and get what's given by default
16:12:51 <nurla> we use Alexey's patch for switching, it isn't production way
16:13:40 <msemenov> nurla: mihgen but it seems, we are speaking now about making Ubuntu bootstrap default.
16:13:43 <mihgen> kozhukalov: msemenov Do we have code in master for switching on live fuel master?
16:13:45 <angdraug> one impact I can think of is more time to generate the bootstrap image for each master deploy test
16:13:57 <msemenov> mihgen: yes
16:14:22 <msemenov> steps are the following
16:14:23 <msemenov> 1. Make sure the master node can access Ubuntu (http://archive.ubuntu.com/ubuntu) and MOS (http://mirror.fuel-infra.org/mos-repos) APT repositories.
16:14:23 <msemenov> 2. Run the following command on the master node:
16:14:23 <msemenov> fuel-bootstrap-image-set ubuntu
16:14:23 <msemenov> 3. Just in a case restart dnsmasq:
16:14:25 <msemenov> dockerctl shell cobbler service dnsmasq restart
16:14:27 <mihgen> angdraug: we could pregenerate one for our tests.. but this is where it will take some QA resources
16:14:27 <msemenov> 4. Reboot the slave nodes.
16:14:29 <msemenov> just for information
16:14:45 <angdraug> actually I think we already abosrbed this impact
16:14:49 <angdraug> absorbed
16:14:56 <kozhukalov> mihgen, live fuel master? i don't even understand this
16:15:14 <angdraug> correct me if I'm wrong: we're generating the ubuntu bootstrap even though it's not used by default
16:15:32 <msemenov> angdraug: yes
16:15:32 <angdraug> kozhukalov: he means, change it after master node is deployed
16:15:37 <kozhukalov> angdraug, correct
16:15:51 <angdraug> folks, 15 minutes -- lets move on
16:15:56 <msemenov> and yes, we can change it after deploy - steps are above
16:16:05 <nurla> feature with Critical priority should not affect GA in whole
16:16:20 <msemenov> ok, so Alexei will prepare a request for making ubuntu-based bootstrap by default
16:16:35 <msemenov> and I'll write to mos-dev about merge
16:16:38 <msemenov> ok?
16:16:42 <kozhukalov> this image is built during master node deployment, as far as i understand, after deployment we can again run fuelmenu and choose another bootstrap flavour
16:16:50 <kozhukalov> and then redeploy
16:17:18 <kozhukalov> it'll changed cobbler configuration so it points out on ubuntu profile
16:17:29 <kozhukalov> ubuntu bootstrap profile
16:17:49 <kozhukalov> moving on
16:18:03 <kozhukalov> #topic CentOS 6.6 based Fuel Master (amnk)
16:18:17 <kozhukalov> this patch has been merged
16:18:22 <amnk> yep
16:18:26 <angdraug> is there any outstanding impact or cleanup that still needs to be done?
16:18:30 <amnk> no
16:18:47 <angdraug> that was quick :)
16:18:47 <kozhukalov> so, we are on 6.6 now
16:18:55 <angdraug> moving on?
16:18:55 <amnk> yep, we are on 6.6
16:18:57 <mihgen> amnk: please check documentation... I'm sure there refs to 6.5 still
16:19:08 <mihgen> amnk: thank you for this work
16:19:09 <angdraug> +1
16:19:10 <amnk> mihgen: ok, I'll double-check it
16:19:37 <kozhukalov> #action amnk checks references in docs on 6.5
16:19:43 <kozhukalov> moving on
16:19:56 <kozhukalov> #topic SSL support status (sbog)
16:20:28 <kozhukalov> sbog, around?
16:20:41 <kozhukalov> looks like he's not here
16:20:44 <kozhukalov> moving on
16:20:51 <mihgen> sbog: ^^
16:21:09 <kozhukalov> #topic Trove plugin plans (bmidgley)
16:21:15 <flamebot> hi all
16:21:24 <kozhukalov> hi
16:21:26 <flamebot> sorry I came in with a different id, I’m bmidgley
16:21:30 <flamebot> not a bo
16:21:31 <flamebot> bot
16:21:40 <mihgen> :)
16:21:57 <flamebot> so I saw mihgen in https://bugs.launchpad.net/fuel/+bug/1403384 saying that trove in fuel would be entirely customer driven
16:21:57 <openstack> Launchpad bug 1403384 in Fuel for OpenStack "Create Trove plugin for Fuel" [Medium,Invalid] - Assigned to sbartel (samuel-bartel)
16:21:57 <uvirtbot> Launchpad bug 1403384 in fuel "Create Trove plugin for Fuel" [Medium,Invalid]
16:21:58 <uvirtbot> Launchpad bug 1403384 in fuel "Create Trove plugin for Fuel" [Medium,Invalid] https://launchpad.net/bugs/1403384
16:22:06 <flamebot> is that the current place we are at?
16:22:21 <flamebot> we need to build it in our MOS cloud, investigating options
16:22:51 <angdraug> do you need trove, or dbaas?
16:23:09 <flamebot> we want trove so we get the db management too
16:23:10 <mihgen> flamebot: sounds like it is. Anyone from partners team?
16:23:13 <angdraug> iirc dbaas can be achieved via murano
16:23:40 <flamebot> we are actually investigating that in a parallel track but people are asking for trove management
16:24:14 <mihgen> I have not heard if anyone else wants to work on Trove plugin...
16:24:35 <Irina_p> flamebot: this is the 1st time I hear about this plugin
16:24:52 <mihgen> so looks like it's yours :) Do you have any issues with trying to build it yourself?
16:25:00 <flamebot> several paths are set out
16:25:13 <flamebot> plugin, package
16:25:13 <mihgen> looks like there would be needed a trove package and some deployment puppet code
16:25:16 <SheenaG> flamebot: first I'm hearing too, although Irina_p usually knows before I do ;-)
16:25:27 <flamebot> ipackage is preferred for upstream inclusion?
16:26:04 <mihgen> ipackage?
16:26:49 <mihgen> so the question is do you need trove package to be in fuel/mos packages mirrors?
16:26:58 <flamebot> typo, just making sure uncertified plugin was the ideal target
16:27:09 <mihgen> you could create package yourself and deliver it as part of package
16:27:22 <mihgen> and you can actually certify plugin afterwards
16:27:42 <mihgen> Irina_P: we don't have any limitations on whether package has to be on our mirrors or as part of the plugin, do we?
16:27:52 <angdraug> mihgen: we don't
16:28:01 <angdraug> plugin can include its own package repo
16:28:31 <Irina_p> flamebot: we could do code review and check if everything is okay..then you could put it into DriverLog or to fuel-infra
16:28:38 <mihgen> flamebot: so I think for this type of extensition, pluggable architecture just fits the best
16:28:47 <Irina_p> flamebot: and sure - get this validated by us
16:29:21 <flamebot> ok we are parsing this :)
16:29:22 <kozhukalov> Irina_p, what about testing?
16:29:52 <kozhukalov> are we going to run tests on mos ci?
16:30:01 <Irina_p> kozhukalov: we could put this into DriverLog/fuel-infra if flamebot at least runs some baseline tests
16:30:23 <Irina_p> kozhukalov: plugins should have their own CI or at least manual testing performed
16:30:43 <kozhukalov> Irina_p, thanx for clarifiction
16:30:50 <kozhukalov> moving on?
16:31:05 <kozhukalov> #topic RabbitMQ (dmitryme)
16:31:15 <mihgen> flamebot: feel free to ask questions in openstack-dev if any..
16:31:20 <Irina_p> flamebot: feel free to ask in fuel-dev then about details, will be happy to help
16:31:28 <dmitryme> hello guys, should I tell you our current status?
16:31:28 <flamebot> thanks
16:31:30 <mihgen> yeah or in #fuel-dev IRC
16:31:35 <dmitryme> * status on RabbitMQ
16:31:51 <kozhukalov> dmitryme, yes, please
16:31:57 <kozhukalov> if you have
16:32:00 <dmitryme> sure
16:32:50 <dmitryme> RabbitMQ 3.5.4 was merged last Saturday. Before that we ran tests with custom ISO with new RabbitMQ on 50-nodes lab and didn’t spot any degradation of behaviour
16:33:26 <dmitryme> so, basically that is it. Starting from this week all new ISOs contain new RabbitMQ
16:33:42 <SheenaG> Have there been any QA issues related to this merge that you know of?
16:34:12 <kozhukalov> what are the main pros of using new Rabbit?
16:34:26 <kozhukalov> HA?
16:34:50 <dmitryme> the main advantage is that we get the freshest fixes + we community support
16:35:07 <dmitryme> community is not really interested in discussing issues in old RabbitMQ
16:35:22 <dmitryme> * we _get_ community support
16:35:29 <kozhukalov> dmitryme, ok, i see
16:35:42 <kozhukalov> thanx for your update
16:35:49 <kozhukalov> any q here?
16:35:53 <kozhukalov> moving on
16:36:05 <SheenaG> mihgen: discussion yesterday re: RMQ, do we need to follow a SCF exception process next time?
16:36:25 <mihgen> SheenaG: yes we certainly need to ask for exception next time.
16:36:35 <mihgen> no major pkgs upgrades should be done after FF
16:36:43 <dmitryme> mihgen: we will definitely do that
16:36:45 <SheenaG> Okay, noted.  Anything after FF we will raise an exception to openstack-dev.
16:36:52 <mihgen> thanks.
16:37:08 <mihgen> I hope that our HA will become even better with new MQ :)
16:37:09 <sgolovatiuk> RMQ 3.5.4 is much more stable
16:37:27 <kozhukalov> #topic Keystone wsgi issues
16:37:33 <sgolovatiuk> many many fixes were added, and it's durable queues works as expected by design
16:37:35 <mihgen> I'm looking for some conrete numbers on downtime in sec, etc...
16:37:59 <sgolovatiuk> Guys, we have bunch of problems with WSGI
16:38:26 <kozhukalov> sgolovatiuk, bugs, links?
16:38:31 <sgolovatiuk> we have process based WSGI, which creates huge amount of connections to MySQL and memcache
16:38:38 <mihgen> I remember we had to revert it in 6.1...
16:38:54 <sgolovatiuk> all reviews are linked https://review.openstack.org/#/c/209589/3
16:38:56 <mihgen> sgolovatiuk: was it discussed with Keystone team / puppet-openstack?
16:39:00 <sgolovatiuk> bugs are included
16:39:23 <sgolovatiuk> also wsgi blocks apache, sometimes it doesn't want to stop
16:39:31 <sgolovatiuk> just hanging
16:40:07 <sgolovatiuk> we made an amount of reviews, that should fix apache probles
16:40:25 <sgolovatiuk> I am testing all of them on custome ISO
16:40:40 <sgolovatiuk> Over weekend scale team will test it on scale
16:40:40 <angdraug> +1 to mihgen, who did you talk to about this?
16:41:03 <sgolovatiuk> if there no any improvements, we'll be reverting it back to eventlet
16:41:28 <sgolovatiuk> mihgen: I am working with A. Makarov and B. Bobrov
16:41:38 <sgolovatiuk> they made fix for WSGI module
16:41:48 <sgolovatiuk> added critical section
16:42:02 <mihgen> sgolovatiuk: +1 on a plan but only if you guys discuss it with keystone upstream
16:42:11 <sgolovatiuk> it looks like helped with threaded based model as well as apache stop operations
16:42:38 <sgolovatiuk> anyhow, I am working with them to provide stable keystone service under Apache WSGI
16:43:15 <sgolovatiuk> I will have more results after rully and destructive tests we are going to do on scale lab over weekend
16:43:31 <sgolovatiuk> rally*
16:45:00 <kozhukalov> i think mihgen means that your cooperation with those guys should be reflected in ml or at least in irc logs
16:45:01 <mihgen> sgolovatiuk: thank you. I hope you guys nail it down..
16:45:12 <sgolovatiuk> I hope so :)
16:45:19 <kozhukalov> ok, moving on
16:45:40 <kozhukalov> #topic Keystone openrc issues (LP: #1479879)
16:45:41 <openstack> Launchpad bug 1479879 in Fuel for OpenStack "fix auth_url in keystone providers for v3/v2.0 api split" [High,In progress] https://launchpad.net/bugs/1479879 - Assigned to Matthew Mosesohn (raytrac3r)
16:45:43 <uvirtbot> Launchpad bug 1479879 in fuel "fix auth_url in keystone providers for v3/v2.0 api split" [High,In progress] https://launchpad.net/bugs/1479879
16:45:44 <uvirtbot> Launchpad bug 1479879 in fuel "fix auth_url in keystone providers for v3/v2.0 api split" [High,In progress]
16:45:47 <kozhukalov> ohh, keystone again
16:46:00 <alex_didenko> :)
16:46:23 <angdraug> mattymo_: around?
16:46:39 <amaksimov> let me ping him
16:46:45 <mattymo_> angdraug, yes sorry for delay
16:46:49 <mattymo_> amaksimov, no worries
16:47:31 <mattymo_> so it seems that there is a healthy debate on openstack-dev and on the puppet-keystone review for how we should approach openrc/env vars being set and then trying to create keystone_* resources
16:48:21 <mattymo_> several folks believe that internalurl and publicurl should be set in the service catalog with no version in keystone... but that so far doesn't seem to work either
16:48:52 <mattymo_> so in the interests of meeting HCF on time, I think we should fork from upstream for this commit until they've decided what is best and what meets the most people's needs
16:49:18 <mattymo_> at the same time, I will continue pushing this patch upstream since it seems to be necessary
16:49:56 <kozhukalov> mattymo, thanx for the status
16:49:56 <mihgen> mattymo_: +1 to fork it, we need to test it before HCF..
16:50:27 <mattymo_> it's very tested for Fuel's puproses, mihgen , but not enough for acceptance tests upstream
16:50:34 <mattymo_> https://review.openstack.org/#/q/topic:bug/1479879,n,z for those of you who want to see the patches
16:50:38 <mattymo_> purposes*
16:51:18 <kozhukalov> moving on?
16:51:22 <mattymo_> yes, please kozhukalov
16:51:39 <kozhukalov> #topic default network allocation for CEPH-based installations (xenolog)
16:51:48 <xenolog13> hi, guys!
16:52:04 <xenolog13> I want ask about storage network and its usage
16:52:14 <xenolog13> In the 7.0 we have admin, management, public, storage networks by default.
16:52:25 <xenolog13> If CEPH-based deploy was chosen, storage network for ceph replication
16:52:25 <xenolog13> and management for ceph-public (access from VM to NBD) will be used.
16:52:25 <xenolog13> Do we need change this default scheme in 7.0?
16:53:11 <xenolog13> Some people complain, why storage traffic goes through management network
16:53:54 <angdraug> can we now have 2 storage networks in 7.0?
16:54:04 <xenolog13> not
16:54:13 <angdraug> so what exactly can we change?
16:54:37 <xenolog13> We can leave all as is.
16:54:51 <xenolog13> Or change default scheme
16:54:53 <angdraug> moving RBD client traffic to storage network and combining it with RADOS replication traffic isn't recommended by Ceph
16:55:04 <xenolog13> yes.
16:55:06 <mihgen> can we have at least example templates which would do the right schema
16:55:24 <alex_didenko> there was a problem with cluster/rabbit/HA glitches due to overload of management network by VM to RDB traffic
16:56:04 <angdraug> in the most typical case with dedicated compute and ceph-osd nodes,
16:56:33 <angdraug> computes don't need rados replication so they can use storage for rbd traffic
16:56:39 <kozhukalov> why can't we have two storage networks?
16:56:47 <angdraug> ceph-osd nodes don't need public, so they can use that for rbd traffic
16:57:11 <angdraug> can we provide a template that does different role allocation for compute and osd nodes?
16:57:24 <xenolog13> May be "move ceph public traffic to the storage network" checkbox in the settings tab is a solution for 7.0 ?
16:57:26 <angdraug> different network _role_ allocation...
16:57:50 <angdraug> xenolog13: that's stealing from Peter to feed Paul...
16:58:27 <kozhukalov> 2 minutes
16:58:30 <angdraug> +1 to kozhukalov's question, why can't we add a new network type yet? is it out of scope of network templates in 7.0?
16:59:09 <angdraug> and time :(
16:59:11 <xenolog13> because SCF was done
16:59:31 <mihgen> yeah I'd say it is too late for any major changes
16:59:37 <kozhukalov> ok, let's move out discussion somewhere else
16:59:42 <mihgen> so I vote for providig better docs, and template example..
16:59:51 <mihgen> so people can use templated networking if needed
16:59:55 <kozhukalov> thanx everyone for attending
16:59:59 <kozhukalov> closing
17:00:08 <kozhukalov> #endmeeting