16:01:09 <mihgen> #startmeeting fuel
16:01:10 <openstack> Meeting started Thu Jun 12 16:01:09 2014 UTC and is due to finish in 60 minutes.  The chair is mihgen. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:11 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:14 <openstack> The meeting name has been set to 'fuel'
16:01:19 <mihgen> hi Fuelers
16:01:32 <mihgen> #topic Announcements
16:02:00 <mihgen> we've released 4.1.1 this week, congratulations to everyone
16:02:06 <mihgen> many HA fixes were backported
16:02:24 <mihgen> due to my long trip I've not set tags in repo yet, but will do soon
16:02:39 <mihgen> Folks, I'm still worried about bugs in master
16:02:46 <mihgen> we need bug squashing day I think
16:03:11 <mihgen> I propose Tu next week to bring a team together and squash bugs
16:03:11 <holser> Hi
16:03:20 <bogdando> hi
16:03:31 <mihgen> we have 300 of them with many in non-confirmed/triaged state
16:03:52 <mihgen> any objections? ideas?
16:04:12 <salmon_> bug squashing day would be nice
16:04:17 <angdraug> bug squashing day sounds good, as long as it's next week :)
16:05:15 <sbog> hi all
16:05:16 <mihgen> cool
16:05:18 <mihgen> #action mihgen to send an email to ML about bug squashing day on Tu next week
16:05:34 <ikalnitsky> after bug squashing day we need a review-day
16:05:48 <mihgen> Actually I think we actually need to combine
16:05:55 <mihgen> ikalnitsky: so I agree
16:06:06 <mihgen> and some folks can do bug confirming /triaging
16:06:10 <mihgen> some will do fixing
16:06:15 <mihgen> and some - reviews
16:07:05 <mihgen> I suggest to not invent a wheel here and follow openstack practices - will provide in email, if it doesn't work for us - we will come up with adjustcemements
16:07:31 <mihgen> Ok. next topic - let me provide ideas regarding our focus for 5.1
16:08:13 <mihgen> After going through design docs, talking to the team, we have identified how many folks can work full time and how much we can cover
16:08:27 <mihgen> obviously not everything as we wish
16:09:05 <mihgen> so Fuelers, let's concentrate our efforts on a few essential for 5.1 things, and the rest if possible:
16:09:19 <mihgen> 1) upgrade 5.0 -> 5.0.1, 5.0.1 -> 5.1; 2) patching of openstack
16:09:31 <mihgen> 3) authx master node: https://blueprints.launchpad.net/fuel/+spec/access-control-master-node; 4) ML2 support
16:09:37 <mihgen> 5) deploy openstack from master
16:09:58 <mihgen> these are user-facing things
16:10:04 <salmon_> 5?
16:10:20 <mihgen> we also have a few essential sustaining things, e.g. fixes to HA
16:10:44 <mihgen> I've updated blueprints priorities to reflect it - you will see all with essential status
16:10:46 <angdraug> salmon_: 5 is essential for our ability to have 6.0 with juno code before is release
16:10:54 <angdraug> before it is released..
16:10:56 <salmon_> ah, master branch
16:10:57 <salmon_> ok
16:11:19 <angdraug> mihgen: https://blueprints.launchpad.net/fuel/+spec/multiple-cluster-networks ?
16:11:30 <mihgen> #link https://launchpad.net/fuel/+milestone/5.1 blueprints for 5.1
16:11:38 <mihgen> angdraug: nope, decreased priority
16:12:11 <mihgen> reason is simple: let's make Fuel work ideally with <50 nodes first, and multiple cluster network is actually when you care about hundreds
16:12:22 <mihgen> or at least > 2 racks of servers
16:12:57 <mihgen> it is still High priority and intersects with refactoring of nailgun networking stuff I assume
16:13:20 <mihgen> any other blueprints I'm missing here which should be essential you think?
16:13:24 <xarses> mihgen: we are going to get alot of push back about that, several large groups are very interested
16:13:32 <angdraug> zabbix, too
16:13:51 <rmoe> yes, zabbix
16:13:59 * odyssey4me agrees
16:14:00 <angdraug> just like multiple-cluster-networks, it is almost ready for merge
16:14:01 <mihgen> xarses: I understand. but do you agree that it's after those 5 at least?
16:14:39 <xarses> mihgen: it could be #6, ya
16:14:41 <angdraug> priority wise, yes, but each of those 5 is a big and disruptive piece of work
16:14:45 <mihgen> rmoe: angdraug: zabbix is High. My expectation that we will make it in experimental mode first. It's literally almost ready for merge
16:14:46 <odyssey4me> is #3 such a high priority?
16:15:52 <mihgen> angdraug: if multiple-cluster-networks are almost ready, I've nothing against. However, remember that for production readiness we need to add many things to it, like acceptance testing (automated with devops), docs, etc.
16:16:08 <angdraug> odyssey4me: it's security. should be #1 except #1 and #2 are things that were on the top of our priorities for 3 releases now
16:16:14 <mihgen> we need that code anyway, but we can skip claiming it as a feature for now
16:16:34 <angdraug> experimental mode it is :)
16:16:43 <mihgen> angdraug: odyssey4me: yep, security..
16:16:54 <rmoe> for multiple-cluster-networks there are also more changes beyond what's in the first review
16:16:56 <mihgen> actually users keep complaining it's insecure
16:17:11 <mihgen> however you can always workaround by placing master node in separate L2
16:17:25 <mihgen> but looks like it's time to finally fix it..
16:18:00 <odyssey4me> alright - it's simple enough to manage in other ways, but I do acknowledge that it's a good to-do... I'm only questioning whether it's essential to push for it this release.
16:18:08 <mihgen> The thing is that 3) authx requires us not only put authx in place, but also fix all iptables rules, etc., to securing everything
16:18:35 <odyssey4me> yup - it's a big, big job when you go beyond just putting some sort of auth in place
16:18:57 <mihgen> so we will need to put deployment engineers to fix it, and additional python resources perhaps
16:19:34 <mihgen> salmon_: so actually we would need to execute stage #3 & 4 from your design in 5.1, do you think it's possible to parallelize?
16:19:42 <mihgen> if add resources there?
16:20:03 <salmon_> mihgen: 4) ML2 support ?
16:20:24 <salmon_> grr, stae 3 & 4?
16:20:30 <mihgen> salmon_: nope, I'm about authx
16:20:41 <salmon_> don't think so. There are some open questions there
16:21:16 <mihgen> salmon_: ok. will need to discuss then...
16:21:42 <mihgen> #action mihgen to request details on whether we could parallelize the work on authx and complete it fully in 5.1
16:21:57 <salmon_> mihgen: stage 4 is about LDAP, many users groups... Why do we need yhis now?
16:22:00 <xarses> salmon_: neutron ml3 plugin
16:22:11 <mihgen> salmon_: I don't think we need LDAP in 5.1
16:22:22 <mihgen> I'll review you it once again
16:22:28 <salmon_> mihgen: ok
16:23:09 <mihgen> Ok fuelers let's get a status on essential blueprints in the first order
16:23:32 <mihgen> 1) upgrade of Fuel 5.0 -> 5.0.1
16:23:55 <mihgen> eveniyl is in charge, but it's holiday
16:24:06 <mihgen> ikalnitsky: do you know anything about it by any chance?
16:24:49 <ikalnitsky> as far as i know, it almost done. in last few days evgeniyl just changed some parts in more convenient way
16:25:12 <ikalnitsky> i even had a successful upgrade from 5.0 to 5.0.1
16:25:21 * mihgen forgetting about using IRC commands
16:25:31 <mihgen> #topic upgrade of Fuel feature status
16:25:50 <mihgen> ikalnitsky: cool. Do you know about QA automation for it?
16:26:03 <mihgen> anyone from QA team here?
16:26:11 <ikalnitsky> unfortunately, i know nothing about qa
16:27:13 <mihgen> ok, thanks for what you know. Let's move on
16:27:25 <mihgen> #topic Patching of OpenStack feature status
16:27:32 <mihgen> ikalnitsky: back to you again )
16:28:05 <ikalnitsky> today i have finally run the tarball with upgrades to patch openstack and it was done successfully.
16:28:17 <ikalnitsky> unfortunately, i've met some problems
16:28:43 <ikalnitsky> afaik, some of them are known and will be fixed by library team.
16:28:50 <mihgen> ikalnitsky: sounds that it is very close to be fully ready
16:28:57 <ikalnitsky> some of them, i'll fix tomorrow
16:29:07 <mihgen> do you see any risks to complete it in time ensuring a great quality?
16:29:27 <mihgen> we can't break users workloads.. otherwise they will hate us forever )
16:29:30 <ikalnitsky> for now, i don't see such things.
16:29:45 <ikalnitsky> there's a one thing with rollbacking patch
16:29:55 <ikalnitsky> i need to discuss it with Dmitry I.
16:30:07 <ikalnitsky> there's a manifest and i don't know who have to generate it
16:30:23 <ikalnitsky> that's all
16:30:40 <mihgen> ok. sounds like a missing piece in design. dilyin is on holiday..
16:31:08 <mihgen> ikalnitsky: > run the tarball with upgrades to patch openstack and it was done successfully  - did you run upgrade tarball to get patching done?
16:31:11 <mihgen> can you please clarify
16:31:30 <mihgen> my expectation is that you run tarball to upgrade / add new release info
16:31:37 <mihgen> and then you run patching from Fuel UI
16:32:03 <mihgen> like you have existing env, and on Actions tab you choose what version of OpenStack you would like to update it to
16:32:35 <ikalnitsky> yeah, sure. but this tarball has additional stuff: puppet manifests/modules, repos and so one. this stuff have to be installed.
16:33:20 <mihgen> ikalnitsky: yep, that's my understanding. Very well, thanks.
16:33:24 <ikalnitsky> i mean that i've finally install this stuff with some fixes and run upgrade from ui
16:33:25 <mihgen> what about system tests for this?
16:33:36 <mihgen> cool
16:33:40 <ikalnitsky> i didn't run it yet
16:33:53 <xarses> ikalnitsky: do we only have new puppet because we don't have puppet upstream yet?
16:33:57 <mihgen> who is assigned to do QA of design /implementation?
16:34:12 <xarses> ie if we have puppet upstream, do you need it in the tar?
16:34:49 <mihgen> Tatyana is QA but looks like she is not here
16:34:59 <holser> good question …
16:35:14 <ikalnitsky> xarses: i'm not sure, but as far as i know we need it in tarball to make different upgrades. for example: 5.1 -> 6.0 or 5.1 -> 5.2
16:35:30 <ykotko> Tatyana or Andrey (who is not here too)
16:35:41 <mihgen> ykotko: thanks
16:35:52 <mihgen> hmm I don't quite follow the question about puppet
16:36:24 <mihgen> xarses: do we have it in upgrade package always, is that what you mean?
16:36:46 <xarses> i thought we where talking about patching openstack
16:36:57 <mihgen> xarses: yes, we are
16:36:58 <xarses> didn't think we needed alot of manifest around it
16:37:13 <alex_didenko> xarses: we may need them
16:37:18 <mihgen> the thing is that we need to upgrade Fuel first to introduce packages
16:37:20 <alex_didenko> so we need to be prepared
16:37:23 <mihgen> and possilby manifests
16:37:24 <xarses> I only thought it would be needed more when we upgrade fuel version
16:37:26 <mihgen> to make patching
16:37:48 <xarses> mihgen: ya, i figured that out
16:38:23 <mihgen> ok folks let's get details in #fuel-dev, we need to move on now, ikalnitsky - thanks for status, looks like we are in good shape
16:38:35 <mihgen> the only thing unclear to me is QA, but I'll figure it out
16:39:12 <mihgen> #topic Securing Fuel master node, authentication for UI and REST API
16:39:21 <mihgen> #link https://blueprints.launchpad.net/fuel/+spec/access-control-master-node
16:39:28 <salmon_> there is still some disqusion about blueprint, but in the meantime I'm working on installing/using keystone. Vitaly prepared GUI part, Matt prepared code for integration with fuelmenu. Stage I should be ready soon.
16:40:38 <mihgen> salmon_: very good. I'll provide my answers regarding stage 3/4 soon
16:40:49 <mihgen> any risk items?
16:41:23 <salmon_> mihgen: for now I don't see any
16:41:30 <mihgen> ok, thanks
16:41:33 <mihgen> #topic ML2 support in Neutron
16:41:48 <xarses> research into ml2 plugin is done, design doc is nearly done (today), started sorting out differences between upstream puppet module https://github.com/xarses/puppet-neutron/compare/fuel-neutron?expand=1
16:42:09 <mihgen> xarses: cool. Where is the design doc?
16:42:19 <mihgen> omg diff is huge
16:42:28 <xarses> it will get smaller still
16:42:33 <xarses> but it was twice as big
16:42:56 <holser> xarses, We have task to update our manifests to upstream so it’s related to ml2 too
16:43:01 <mihgen> ok.. how risky do you think to play this game with syncing upstream manifests?
16:43:28 <mihgen> holser: my expectation is that xarses is actually doing exactly this
16:43:37 <holser> I see
16:43:43 <mihgen> syncing upstream neutron to get upstream ml2 support
16:43:46 <angdraug> mihgen: holser: yes, that's what he's doing
16:43:51 <mihgen> and fix remaining issues
16:43:56 <xarses> looking at it, most of our changes are deprecated, the only main risk i think is with keeping HA support
16:44:28 <mihgen> xarses: whoops.. we need to start thinking about it in parallel. Did you talk to xenolog about it?
16:44:50 <mihgen> just static analysis of code might help ..
16:44:55 <xarses> I was speaking with puppet-openstack folks yesterday, and we agree that it should probably be in an outside wrapper module. I also found out that we don't need to use pacemaker for DHCP-agent HA
16:45:15 <xarses> holser: yes, pulling upstream module is the goal
16:46:00 <xarses> mihgen: no, I'll start now that the design doc is mostly done
16:46:00 <mihgen> xarses: ok. please sync with xenolog on regular basis.. dhcp agent ha might be there for a reason. all this ha stuff is always tricky
16:46:34 <mihgen> Ok thanks for status. Looking forward to see it in action..
16:46:45 <mihgen> moving on
16:46:50 <xarses> mihgen: we don't need to do it with pacemaker since grizzly, dhcp-agent is internally HA ready
16:47:19 <mihgen> xarses: hmm ok. I trust you guys in such details :)
16:47:27 <mihgen> #topic deploy openstack from master
16:47:41 <mihgen> rvyalov is in charge, he is holiday though
16:48:10 <mihgen> I know a bit about it. the idea is to modify our build system so the Fuel should be able to build openstack packages on the fly
16:48:20 <mihgen> using our spec files and upstream openstack code
16:48:47 <mihgen> so it's slow progressing, gerrit / CI for it are still in design phase I believe
16:49:10 <mihgen> moving on
16:49:18 <brain461> mihgen: I'm working on this feature. It's still on the PoC stage, I plan to get PoC working tomorrow
16:49:40 <mihgen> oh dude that's cool
16:49:54 <mihgen> brain461: sorry actually you are assignee for that blueprint
16:50:05 <brain461> yep
16:50:37 <mihgen> brain461: very good. We really need this feature for our community presence and for  full CI/CD with openstack upstream
16:50:55 <mihgen> brain461: I hope you guys gonna use Infra's stuff to get gerrit up and running for it
16:51:09 <mihgen> ok, moving on, not much time left..
16:51:12 <mihgen> #topic Galera HA and other HA improvements
16:51:21 <mihgen> holser: can you provide the status pls
16:51:44 <mihgen> Galera HA improvements I meant :)
16:52:15 <holser> We had meetings regarding Rabbit so I was helping bogdando
16:52:40 <holser> Galera is still in progress, thinking to finish it by end of week and commit
16:53:10 <holser> also it requires cs_shadow removal
16:53:11 <mihgen> good. Will it fix the issue of rebuild in case of power outage of whole cluster?
16:53:54 <holser> didn’t have chance to talk to Dmitry Illyn regarding that
16:54:13 <holser> I ripped out all cs_shadow for now
16:54:25 <holser> to be able to test Galera
16:54:28 <mihgen> holser: ok, please do.. power outage may happen in small installs..
16:54:45 <holser> k
16:54:55 <holser> that’s all from me
16:55:01 <mihgen> ok thanks. moving on
16:55:03 <mihgen> #topic Mellanox features support
16:55:14 <mihgen> do we have anyone from Mellanox around?
16:55:29 <aviramb> yes
16:55:50 <mihgen> aviramb: hi, can you provide a short update on where we are ?
16:56:04 <mihgen> did you get reviews for design doc?
16:56:12 <aviramb> mihgen: yes thanks
16:56:28 <aviramb> mihgen: any more reviewers are welcome
16:56:48 <mihgen> yep, I think we need to get someone with networking background..
16:56:59 <xarses> aviramb: looks nice
16:57:13 <mihgen> besides reviews, what are the next steps now?
16:57:18 <aviramb> mihgen: that would be great, we are at the end of our POC based on the master and ML2 plugin (manually)
16:57:22 <xarses> I was confused, do you need neutron ML2 plugin. the title implies you do
16:57:31 <xarses> ahh ok
16:57:45 <mihgen> ok that was my question too
16:57:48 <aviramb> mihgen: next steps - finish automating drivers installation (made custom small package of MLNX_OFED)
16:57:59 <xarses> mihgen: there is a 4.1 iso with it
16:58:01 <mihgen> actually we need to mark blueprint then as dependent on ml2
16:58:01 <aviramb> mihgen: and integrate to fuel-library
16:58:03 <xarses> and a nice write up
16:58:16 <aviramb> mihgen: yes
16:58:22 <mihgen> aviramb: will it require any nailgun/UI changes?
16:58:30 <xarses> #link http://community.mellanox.com/docs/DOC-1449?et=watches.email.document
16:58:49 <aviramb> mihgen: yes, new section for installing mellanox SRIOV plugin or just drivers
16:59:10 <aviramb> mihgen: and iSER support (iSCSI over RDMA)
16:59:26 <mihgen> need to wrap up the meeting.. I have a few more questions but will add them in the design doc then
16:59:30 <aviramb> mihgen: checkbox, near iSCSI (in storage section)
16:59:45 <mihgen> aviramb: ok, I see. Thanks for the status
17:00:05 <mihgen> folks as usual any unaddressed things can be continued in #fuel-dev
17:00:14 <mihgen> thanks everyone for participation!
17:00:18 <mihgen> #endmeeting