16:00:07 <xarses> #startmeeting fuel
16:00:07 <xarses> #chair xarses
16:00:07 <xarses> Todays Agenda:
16:00:07 <xarses> #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda
16:00:07 <xarses> Who's here?
16:00:07 <openstack> Meeting started Thu Mar 17 16:00:07 2016 UTC and is due to finish in 60 minutes.  The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:08 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:10 <openstack> The meeting name has been set to 'fuel'
16:00:12 <openstack> Current chairs: xarses
16:00:12 <maximov_> hi
16:00:17 <HeOS> hi!
16:00:21 <romcheg> o/
16:00:37 <mihgen> hi
16:00:39 <angdraug> \o
16:01:11 <akislitsky_> hi
16:01:13 <evgenyl> hi
16:01:13 <rvyalov_> hi
16:01:18 <akscram> hi
16:01:19 <holser_> hi
16:01:42 <dklenov> o/
16:01:58 <monester> hi
16:02:14 <ikalnitsky> o/
16:02:20 <kozhukalov> hi
16:02:47 <xarses> #topic Action items from last meeting
16:03:10 <xarses> ogelbukh will add octane and repo for configdb to fuel governance
16:03:35 <yottatsa> o/
16:03:36 <mwhahaha> hi
16:03:57 <xarses> ogelbukh_:
16:04:00 <ashtokolov> hi
16:04:41 <mattymo_> aloha
16:04:58 <xarses> holser_ to update ML on fuel-lib ci
16:05:02 * angdraug prods ogelbukh_
16:05:20 <holser_> ok
16:05:47 <holser_> I am not sure I sent email
16:05:57 <holser_> let me follow up on this
16:06:03 <holser_> let’s move on
16:06:17 <xarses> aglarendil to update ML on fuel-lib CI
16:06:19 <xarses> thanks
16:06:57 <ogelbukh_> hi
16:07:04 <angdraug> looks like there weren't updates on that thread since last week
16:07:25 <angdraug> #link http://lists.openstack.org/pipermail/openstack-dev/2016-March/087862.html
16:07:37 <xarses> yep
16:07:45 <angdraug> back to ogelbukh_'s update?
16:08:04 <holser_> yep
16:08:13 <xarses> ogelbukh_ will add octane and repo for configdb to fuel governance
16:08:31 <ogelbukh_> update is brief, changes are in review https://review.openstack.org/293255
16:09:03 <ogelbukh_> all relevant signoffs are there, waiting for infra team to merge them
16:09:07 <angdraug> wrong link?
16:09:23 <ogelbukh_> there were some confusion around name of the project and release schedule
16:09:30 <ogelbukh_> but they seem to be resolved now
16:09:32 <angdraug> #link https://review.openstack.org/291380
16:09:41 <ogelbukh_> https://review.openstack.org/291380
16:09:52 <ogelbukh_> correct, that's the governance patch
16:10:13 <angdraug> tick tock, lets move on
16:10:24 <angdraug> ogelbukh_: don't walk away far, you've also got status report to give
16:10:42 <xarses> #topic Commit messages need to be more descriptive (Xarses)
16:10:42 <ogelbukh_> https://review.openstack.org/286137 and this is the project repo itself, should be merged first
16:11:17 <xarses> Hi I wanted to remind people that short commit messages are not useful. If there isn't enough to understand why this change is needed, then it isn't long enough. this is usually 3 or more sentences.
16:11:32 <angdraug> +1 to that
16:11:38 <mattymo> What are you changing, why, and some references as needed :) It's not hard
16:11:47 <maximov_> +1
16:11:48 <ikalnitsky> xarses: +1
16:11:57 <angdraug> just link to a bug with no comment is not enough, bug description can change and usually not that useful to begin with
16:12:15 <angdraug> core reviewers, please -1 at will for bad commit messages
16:12:32 <xarses> I'd like to remind reviewers to help here, if you can't easily tell why the change is needed, please speak up
16:12:45 <angdraug> looks like we have a strong consensus in favor, moving on?
16:12:54 <xarses> thanks
16:12:58 <xarses> #topic Fuel-mixed team status (asaprykin)
16:13:06 <asaprykin> Fuel-mixed team is working on fuel-library ensurability/idempotency and multipath feature
16:13:07 <asaprykin> - Idempotency: All changes are merged. QA in progress.
16:13:07 <asaprykin> - Multipath: Work on bug https://bugs.launchpad.net/fuel/+bug/1555664 is in progress
16:13:08 <asaprykin> - Working on bugfixing as well.
16:13:08 <asaprykin> That's all. Than you.
16:13:08 <openstack> Launchpad bug 1555664 in Fuel for OpenStack "provision with multipath device fails randomly" [High,In progress] - Assigned to Sergey Slipushenko (sslypushenko)
16:13:37 <mihgen> folks, have you been working on idempotency for master node installation tasks.. ?
16:14:17 <holser_> asaprykin: take to consideration https://bugs.launchpad.net/fuel/+bug/1558607
16:14:18 <openstack> Launchpad bug 1558607 in Fuel for OpenStack "Wrong configuration of Cinder Multi-backend" [High,Confirmed] - Assigned to Denis Egorenko (degorenko)
16:14:18 <asaprykin> no, only fuel library connected to environment nodes.
16:14:24 <kozhukalov> mihgen: are they not idempotent?
16:14:41 <alex_didenko> mihgen, idempotency is needed for LCM feature
16:14:52 <alex_didenko> it's not related to puppet-master
16:15:06 <mihgen> I don't know. that's why I ask. Can you update master node IP, rerun puppet, and be sure that your cobbler will pick up new IP?
16:15:17 <mattymo> that's never worked!
16:15:17 <mihgen> I'm talking about Fuel Master, not related to puppet master or anything like that
16:15:24 <alex_didenko> oh, I see
16:15:30 <mattymo> never ever. it's going to break nailgun
16:15:33 <mihgen> that's why I'm asking if we have plans to fix it
16:15:48 <alex_didenko> but Fuel master is not covered by LCM, only openstack nodes
16:15:56 <ogelbukh_> kozhukalov: they are not idempotent, yes
16:15:57 <angdraug> alex_didenko: why?
16:15:58 <mihgen> if we even have a backlog item for it. We are making config mgmt for openstack, but not for fuel
16:16:03 <alex_didenko> so I don't think we need to make sure puppet manifests for master are idempotent
16:16:15 <angdraug> Fuel node is not special, we need to be able to manage it, too
16:16:21 <xarses> angdraug: +1
16:16:22 <mihgen> +1
16:16:30 <alex_didenko> what parts exactly?
16:16:30 <ogelbukh_> we need to make sure that there's no manifests used in installation of Fuel Master
16:16:33 <kozhukalov> ogelbukh_: could you please give an example?
16:16:41 <ogelbukh_> sure, let me fetch some
16:16:53 <angdraug> alex_didenko: all parts
16:17:05 <mihgen> I gave a very specific example with IP change & cobbler. this is just one of the use cases why we need it
16:17:24 <xarses> it sounds like these where out of scope
16:17:27 <alex_didenko> but some parts are configured by bootsrap_admin_node.sh, should it be idempotent as well?
16:17:34 <angdraug> we're not talking a 9.0 FFE for this, no, but we should scope it and plan it for Newton
16:17:34 <mihgen> can someone take an action item, file a bug /blueprint on this?
16:17:40 <xarses> and we need to plan to visit this area in 10
16:17:41 <kozhukalov> mihgen: you gave an example, but it was a question, not statement
16:17:59 <mihgen> ideally to investigate /scope it..
16:18:15 <xarses> kozhukalov: and mattymo replied that it never worked right
16:18:31 <kozhukalov> if it is a statement that they are not indempotent, then i'd like to have an example
16:18:46 <mihgen> bootsrap_admin_node.sh - yes it should be idempotent as well. Beacuse we need to get rid of it ideally
16:19:13 <alex_didenko> well, yes, then we should get rid of shell scripts and move those parts to manifests
16:19:14 <mihgen> kozhukalov: if everything is idempotent, then great. do we have a test which covers it?
16:19:33 <xarses> ok, who's got the action? we need to move along
16:19:59 <sbog> Certificate creation for nginx UI is not idempotent, for example. All openssl puppet module is not idempotent, btw if you need an example
16:20:11 <kozhukalov> mihgen: no, we don't have tests, we certainly need them
16:20:30 * angdraug is shocked by the lack of volunteers
16:20:52 <ogelbukh_> one example here https://bugs.launchpad.net/fuel/+bug/1539577
16:20:54 <openstack> Launchpad bug 1539577 in Fuel for OpenStack "[fuel-library] Astute service not restarted upon changes in config file" [High,Fix committed] - Assigned to Oleg S. Gelbukh (gelbuhos)
16:21:10 <kozhukalov> sbog: nice, thanx
16:21:13 <angdraug> folks, lets stop discussing it here and volunteer someone to lead this offline
16:21:32 <xarses> #action asaprykin will follow up on fuel master node idempotentcy for 10
16:21:36 <mattymo> angdraug, xarses I can take care of fuel master idempotency ---- changing fuel master IP, however, is a tech challenge that needs to originate from a python eng
16:21:46 <xarses> team lead assigned
16:21:54 <xarses> they may change owner
16:22:08 <xarses> #topic Fuel-telco team status (mpolenchuk)
16:22:16 <pma_> Brief status of fuel-telco team: CGroups:   demo has been delivered, minor bugfixing NUMA/CPU pinning/Hugepages:   all patches are merged   (except 2 patches w/ validation, we've time until EOW to complete it) QA is in progress for all the features above.
16:22:25 <pma_> thanx
16:22:43 <xarses> pma_: what is the scope of these two patches?
16:22:44 <angdraug> how are those validation patches doing?
16:22:49 <xarses> are the available already?
16:22:54 <pma_> just validation
16:23:03 <pma_> yes, i review
16:23:09 <pma_> yes, on review
16:23:48 <angdraug> I've checked them yesterday and one was ready for merge, the other in need of a rebase
16:23:58 <angdraug> it all looked like both could be merged today
16:24:09 <dklenov> one patch with +2 in CR and ready for merge
16:24:18 <dklenov> second one has -1 from CI
16:24:24 <angdraug> #link https://review.openstack.org/#/c/286495/
16:24:31 <angdraug> #link https://review.openstack.org/#/c/285282/
16:24:50 <angdraug> lets not wait until the last moment and then have to ask for another extention
16:25:16 <dklenov> yes, we are working to have them merged ASAP
16:25:18 <angdraug> xarses: moving on?
16:25:21 <xarses> #topic Fuel Network team status (alex_didenko)
16:25:46 <alex_didenko> Fuel network team status per feature:
16:25:47 <alex_didenko> Allow any VIP: bugfixing
16:25:47 <alex_didenko> External LB: working on tests automation along with fuel-qa team (new plugin introduced for this purpose)
16:25:47 <alex_didenko> SR-IOV: bugfixing
16:25:47 <alex_didenko> DPDK: library part is merged, nailgun and UI patches are still on review:
16:25:47 <alex_didenko> #link https://review.openstack.org/#/q/status:open+branch:master+topic:bp/support-dpdk
16:25:48 <alex_didenko> We're still optimistic and do not expect any delays in the current schedule/FFE for DPDK (unless some major outage with BVT/CI).
16:25:48 <alex_didenko> Also we're working on Fuel bugs related to network (bugs with team-network tag).
16:26:14 <alex_didenko> That's all
16:26:24 <angdraug> does the link above contain _all_ remaining patches?
16:26:33 <angdraug> or do you expect more to be pushed?
16:26:59 <alex_didenko> all patches, new patches may appear only as bugfixes
16:27:15 <angdraug> tbh doesn't look too good, 5 out of 6 patches have -1 from CI
16:27:39 <alex_didenko> yep, they had -1 from CI the most of time, right now there're some merge conflicts
16:28:19 <angdraug> is that related to the deadlocks problem ikalnitsky raised on ML yesterday?
16:28:27 <alex_didenko> we had a lot of problems with bvt and master during this week, as you know
16:28:35 <angdraug> yup
16:28:37 <ikalnitsky> angdraug: i need to check each patch
16:28:44 <angdraug> no easy way to tell?
16:29:17 <angdraug> between that and bvt regressions, we're going to have a lot of trouble landing all these patches in time
16:29:26 <angdraug> ...and we still have bugs to fix
16:30:02 <angdraug> who's working on the deadlocks problem?
16:30:29 <ikalnitsky> no one
16:30:30 <holser_> romcheg:
16:30:33 <xarses> fuel-library has been quite unstable too
16:30:58 <akislitsky_> romcheg: and me are working on the deadlocks issues
16:31:07 <ikalnitsky> really?
16:31:22 <ikalnitsky> that's good to hear. however, what exactly you're doing folks?
16:32:42 <akislitsky_> we've found that main problem is in postgresql configuration. due to logs we shouldn't have locks on executed queries
16:32:42 <romcheg> ikalnitsky: we actually discuvered, that there seems to be no deadlocks
16:33:02 <xarses> we can circle back, or move this to the ML guys
16:33:09 <ikalnitsky> how that could be possible if postgresql tells you 'deadlock' ?
16:33:14 <ikalnitsky> xarses: agree
16:33:18 <ikalnitsky> moving on
16:33:35 <angdraug> akislitsky_: romcheg: please follow up on ML with your findings
16:33:40 <xarses> #action akislitsky_: romcheg to update on fuel-web deadlocks issue
16:33:52 <xarses> #topic Enhancements Team status (ashtokolov)
16:33:53 <mattymo> +1 I want to read about what's going on with nailgun deadlocks
16:34:06 <ashtokolov> 1. Custom graph execution - Good Progress (core part + API - merged or on review, CLI - WIP)
16:34:12 <ashtokolov> 2. Versioning storage for serialised cluster data and cluster settings - In progress
16:34:18 <ashtokolov> 3. Data-driven decision which tasks should be run during redeployment (YAQL) - core part - in progress
16:34:24 <ashtokolov> 4. Store Deployment Tasks Execution History in DB - Good Progress (core part + API + CLI - on review)
16:34:29 <ashtokolov> 5. Unlock settings tab with UI team - In progress
16:34:34 <ashtokolov> 6. Deployment Tasks idempotence with Fuel Mixed Team - Merged
16:35:06 <mihgen> when can you show first demo folks?)
16:35:19 <holser_> I am interested in Demo also
16:35:26 <mihgen> I really want to see something and spread the word
16:35:28 <xarses> +1
16:35:38 <angdraug> any links to code currently on review?
16:35:45 <angdraug> any code left to write that's not yet on review?
16:36:04 <ashtokolov> our deadline is 3/24, so I think 3/28 or 29
16:36:23 <mihgen> you mean demo 3/28?
16:36:25 <angdraug> deadline 3/24 doesn't mean that you should wait until very last moment to show stuff
16:36:32 <ashtokolov> yes 3/28 for demo
16:36:37 <angdraug> what if you get feedback? :p
16:36:55 <xarses> it goes to 10?
16:37:11 <mihgen> well I'm not asking for full end-to-end demo ;) ok let's take it offline from here, we can figure out what could be shown before I hope
16:37:21 <mihgen> it goes to 9
16:37:31 <ashtokolov> please review our specs, it will help us to get your feedback
16:38:03 <ashtokolov> ok, I think we can show the custom graph and deployment history next week
16:38:25 <xarses> ok, moving on
16:38:26 <holser_> xarses: Can you make action item?
16:38:35 <xarses> #topic Fuel Octane team status (ogelbukh)
16:38:56 <xarses> #action ashtokolov will follow up on scheduling demo for enhancements team features
16:39:38 <xarses> ogelbukh_: ^
16:39:54 <ogelbukh_> configdb status:
16:39:54 <ogelbukh_> 1. API extension in works, already working with Nailgun, recorded some simple demo
16:39:54 <ogelbukh_> now implementing relevant API handlers and automating tests
16:39:54 <ogelbukh_> code is still available from Github, project in opesntack/ is pending addition
16:39:55 <ogelbukh_> specs in review, all comments addressed
16:39:55 <ogelbukh_> https://review.openstack.org/284109
16:39:56 <ogelbukh_> https://review.openstack.org/286012
16:39:56 <ogelbukh_> 2. Deployment task for data upload is in progress in Puppet Master plugin source code tree
16:39:57 <ogelbukh_> 3. Auth mechanisms for Puppet Master plugin in review:
16:39:57 <ogelbukh_> https://review.openstack.org/#/q/status:open+branch:master+topic:bp/upload-deployment-facts-to-configdb
16:41:03 <ogelbukh_> data-driven upgrade:
16:41:03 <ogelbukh_> acceptance in progress, bugs still being found and fixed
16:41:03 <ogelbukh_> Fuel admin password change issues
16:41:03 <ogelbukh_> - https://bugs.launchpad.net/fuel/+bug/1557563
16:41:05 <ogelbukh_> - https://bugs.launchpad.net/fuel/+bug/1555145
16:41:05 <openstack> Launchpad bug 1557563 in Fuel for OpenStack mitaka "[fuel-bootstrap] fuel-bootstrap activate ID fails if nailgun password were changed" [Medium,In progress] - Assigned to Arthur Svechnikov (asvechnikov)
16:41:05 <ogelbukh_> Default timeout in Puppet is insufficient for DB migrations:
16:41:05 <ogelbukh_> - https://review.openstack.org/#/c/292243/
16:41:06 <openstack> Launchpad bug 1555145 in Fuel for OpenStack 8.0.x "[upgrade][8.0] Upgrade does not works if admin password was changed after initial setup " [High,Confirmed] - Assigned to Sergey Abramov (sabramov)
16:42:16 <mihgen> data-driven upgrade - it's not relying on custom graphs yet, right?
16:42:34 <ogelbukh_> it's upgrade of Fuel Master node
16:42:48 <ogelbukh_> it doesn't have anything to do with custom graph
16:42:53 <mihgen> demo link
16:42:55 <mihgen> #link https://asciinema.org/a/87v363mz6gn3oxmu1zw3b0s3f
16:43:09 <ogelbukh_> we will explore the new function when start working on upgrade of OpenStack envs to 9.0
16:43:32 <mihgen> how do we do upgrade of master node? I thought we just backup db, install new master, restore from backup
16:43:39 <ogelbukh_> because it seem to be exactly what we need to perform proper upgrade with minimal external logic
16:43:49 <ogelbukh_> yes, but that's more than that
16:44:14 <mihgen> woops :( where can I learn more?
16:44:18 <ogelbukh_> we need to update astute.yaml, restart all containers, run puppet in host, etc
16:44:24 <xarses> ogelbukh_: https://review.openstack.org/#/c/292243/ is an example of way to short of a commit message, I got more information from your sentence about it. It also has no bug id
16:44:45 <ogelbukh_> xarses: that's true
16:45:07 <ogelbukh_> parses: we have a conversation with Sergey about that just couple days ago, in fact
16:45:18 <ogelbukh_> I will enforce the commit message policy
16:45:42 <ogelbukh_> *had a conversation, sorry
16:45:46 <xarses> thanks
16:46:46 <xarses> ogelbukh_: upgrade to 8?
16:46:50 <ogelbukh_> yes
16:47:20 <mihgen> if we had it all idempotent... in theory it should not require any tweaks from you, ogelbukh_
16:47:38 <xarses> #topic fuel-octane upgrade to 8
16:47:40 <ogelbukh_> the problem is, we have loads of different stuff happening in master node
16:47:54 <ogelbukh_> like bootstrap images building
16:48:09 <ogelbukh_> they have to be rebuilt with restored SSH keys
16:48:43 <ogelbukh_> we need to reload the openstack.yaml into DB by hand after we complete the DB migrations
16:49:01 <xarses> ogelbukh_: anything on this topic?
16:49:04 <ogelbukh_> and different tools have different ways to do the same things
16:49:26 <ogelbukh_> well, I guess I covered the status above
16:49:35 <xarses> next one then?
16:49:38 <ogelbukh_> need to add that we're week behind our initial schedule ATM
16:49:59 <xarses> #topic Discuss the update enchantment for this bug report: https://bugs.launchpad.net/fuel/+bug/1558655 (akscram)
16:50:01 <openstack> Launchpad bug 1558655 in Fuel for OpenStack "There is no possible to reassign nodes during upgrade without re-provisioning" [Undecided,New]
16:50:22 <akscram> Guys, I wanna ask about the reconsideration to land an enhancement for the cluster_upgrade extention in Nailgun in the 9.0 release.
16:51:00 <holser_> ogelbukh_: SSH keys can be stored outside of bootstrap image
16:51:13 <mihgen> akscram: link?
16:51:16 <akscram> It was declined for FFE but it's needed for one and a half of our customers.
16:51:21 <holser_> it should be built in the same way we do for image base provisioning
16:51:23 <mattymo> holser_, or regenerate bootstrap image
16:51:25 <akscram> https://review.openstack.org/#/c/280067/
16:51:38 <holser_> cloud-init will do the job
16:51:53 <ogelbukh_> holser_: let's move it to fuel-library maybe?
16:51:58 <holser_> k
16:52:07 <mihgen> akscram: what happens if we don't merge it
16:52:27 <akscram> It's actively reviewing and has a good progress.
16:52:31 <angdraug> is this a regression from 8.0?
16:52:55 <angdraug> the bug linked above was created 31 minutes ago
16:52:55 <akscram> Otherwise we should downstream it.
16:53:14 <angdraug> there's no reference to the blueprint that didn't get FFE
16:53:23 <akscram> angdraug: yep, you are right about the bug, it was created a few minutes ago.
16:53:32 <mihgen> it's actually 95% of code that goes to upgrade extension
16:54:12 <ogelbukh_> unfortunately, we didn't have capacity in this cycle to extract the extension as we should have
16:54:18 <mihgen> and few lines of code which goes to node.py. I'd suggest to ikalnitsky take a look and understand what we lose and what we get..
16:54:25 <angdraug> akscram: you're 2 weeks too late to ask for an FFE now
16:54:33 <ikalnitsky> ok
16:54:47 <ikalnitsky> akscram: could you please bring more details in the bug?
16:54:48 <akscram> angdraug: I know, we discussed it with you to weeks ago.
16:55:03 <akscram> ikalnitsky: Yes, I will.
16:55:09 <angdraug> yes, and the answer back then was "no FFE"
16:55:12 <ikalnitsky> because there's abstract "we can't since 9.0" withour explanation. and there's no info why you need it
16:55:21 <angdraug> lets resurrect the FFE thread on ML and have more details there, and in the bug
16:55:25 <ikalnitsky> akscram: thx
16:55:51 <mihgen> could it be reimplmeneted to separate extension, and as such to be installed as separate package?
16:56:09 <akscram> angdraug: thanks, I will do it.
16:56:12 <mihgen> in this case you could deliver this separately to those who need it
16:56:29 <xarses> moving on then? we have two more adn 5 min
16:56:35 <akscram> mihgen: It can but not in the current stage, I mean FFE.
16:56:36 <xarses> s/adn/and
16:57:07 <xarses> #topic UI Team status (vkramskikh)
16:57:12 <vkramskikh> Hi! Here is our status for 9.0 features:
16:57:12 <vkramskikh> 1) https://blueprints.launchpad.net/fuel/+spec/remove-vendor-code - no vendor code left in the upstream; downstream part is still not merged, since we decided to change some upstream code to minimize amount of downstream changes. We plan to finish it this week.
16:57:13 <vkramskikh> 2) NFV stuff - only DPDK enablement is still not merged. We plan to merge it as soon as backend part is merged.
16:57:13 <vkramskikh> 3) https://blueprints.launchpad.net/fuel/+spec/unlock-settings-tab - help was requested with that feature. This week we finalized the design and going to start implementation soon.
16:57:15 <vkramskikh> After completion of #1 and #2 we're switching to bugfixing. There are 28 UI bugs, 5 of them are High.
16:57:17 <akscram> mihgen: in the current state the cluster_upgrade extention is highly relates on the internals
16:57:18 <vkramskikh> Questions?
16:57:45 <xarses> #action akscram to update ML and bug regarding FFE for in-place upgrade code
16:58:11 <angdraug> vkramskikh: nice and concise update, thanks!
16:58:54 <xarses> vkramskikh: thanks
16:58:57 <xarses> #topic fuel-web/docs to fuel-docs move (bookwar)
16:58:57 <xarses> #link http://lists.openstack.org/pipermail/openstack-dev/2016-March/089355.html
16:59:03 <bookwar> just short announcement: fuel-web/docs content was moved to fuel-docs/devdocs/ Please send all new patches there, http://lists.openstack.org/pipermail/openstack-dev/2016-March/089355.html
16:59:35 <xarses> thanks everyone
16:59:44 <xarses> #endmeeting