16:00:09 #startmeeting Fuel 16:00:10 Meeting started Thu Jul 10 16:00:09 2014 UTC and is due to finish in 60 minutes. The chair is vkozhukalov. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:11 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:13 The meeting name has been set to 'fuel' 16:00:24 hey everyone 16:00:27 hey 16:00:29 hi 16:00:33 o/ 16:00:35 hi 16:00:36 hi doctor bob 16:00:39 #topic Greetings, announcements 16:00:51 hi 16:00:52 as usual agenda is here 16:00:57 hi 16:01:01 #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda 16:01:02 hello 16:01:27 hi 16:01:30 Ok let me provide short status 16:01:33 #topic feature freeze status 16:01:46 #chair vkozhukalov 16:01:47 Current chairs: vkozhukalov 16:01:51 Feature Freeze was moved by one week and it's planned for today 16:02:00 well it is still planned for today :) 16:02:12 as far as I can tell we are not in ideal state, but in pretty good one 16:02:34 first of all, I can say that Fuel CI now verifies patches correctly for ~10 hours already 16:02:59 for FF date, we will need to make a few exceptions 16:03:20 +1 for working CI. It's great. 16:03:27 Those which I'm aware of: ML2, Mellanox 16:03:39 will talk about it more later during the meeting 16:03:49 mihgen: ok 16:03:53 let me know if we have anything else to request exceptions for 16:04:01 vkozhukalov: that's short update from me 16:04:02 moving topic to 5.0.1? 16:04:04 yep 16:04:19 #topic 5.0.1 release status 16:04:30 #link https://bugs.launchpad.net/fuel/+milestone/5.0.1 16:04:45 5.0.1 is mostly ready for code freeze 16:04:59 we're still waiting for a few bugs: 16:05:05 https://bugs.launchpad.net/mos/+bug/856764 16:05:07 Launchpad bug 856764 in oslo.messaging "RabbitMQ connections lack heartbeat or TCP keepalives" [Critical,In progress] 16:05:15 https://bugs.launchpad.net/fuel/+bug/1319451 16:05:18 Launchpad bug 1319451 in fuel/5.1.x "AMQP Channel errors in orchestrator logs" [High,In progress] 16:05:20 https://bugs.launchpad.net/fuel/+bug/1339236 16:05:21 Launchpad bug 1339236 in fuel/5.1.x "DockerUpgrader: failed to upgrade: Failed to run services" [Medium,Confirmed] 16:05:34 mihgen: ^ is Medium priority, but you mentioned that it must be fixed, can you clarify? 16:05:58 764 is painful thing, though it doesn't always break stuff 16:06:01 as you can see at the milestone link above, there's a couple High priority New bugs 16:06:18 evgeniyl: https://bugs.launchpad.net/fuel/+bug/1339236 doesn't look like Medium to me 16:06:21 it's rather critical 16:06:24 fix for 764 is in testing since a few hours ago, no? 16:06:39 as far as I understand it fails upgrade procedure pretty often 16:06:52 upped 236 to Critical 16:07:26 Yes, it has medium priority because it affects specific tarball 16:07:40 I'm making a call to wait fix for #link https://bugs.launchpad.net/mos/+bug/856764 16:07:41 Launchpad bug 856764 in oslo.messaging "RabbitMQ connections lack heartbeat or TCP keepalives" [Critical,In progress] 16:08:03 I'd like HA to be better. As far as I know fix is under testing now 16:08:17 anyone can update on https://bugs.launchpad.net/bugs/1338573 ? 16:08:18 evgeniyl: reproduced on 112 tarball 16:08:18 Launchpad bug 1338573 in fuel/5.1.x "Incorrect status of progress bar on Ubuntu" [High,Confirmed] 16:08:34 and https://bugs.launchpad.net/bugs/1340237 ? 16:08:35 Launchpad bug 1340237 in fuel "Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install iptables-persistent=0.5.3ubuntu2' returned 100" [High,New] 16:08:41 akasatkin: could you please provide env? 16:08:41 aglarendil: also I'm wondering if #link https://bugs.launchpad.net/fuel/+bug/1323277 could be anyhow fixed in 5.0.1 ? 16:08:43 Launchpad bug 1323277 in fuel "vip__management recovered with error" [Critical,Confirmed] 16:08:51 akasatkin: I would like to check if it is the same issue 16:09:10 evgeniyl: akasatkin let's keep as critical then before we find out 16:09:10 mihgen: 277 is related to rabbitmq ocf script 16:09:23 there is no way to fix it in 5.0.1 completely 16:09:30 :( 16:09:31 but oslo.messaging may help a lot 16:09:57 ok it must go to release notes then.. 16:10:27 evgeniyl: 172.18.194.24 : 5599 16:10:32 angdraug: I think #link https://bugs.launchpad.net/bugs/1338573 is not that important and we could ignore it for 5.0.1 16:10:34 Launchpad bug 1338573 in fuel/5.1.x "Incorrect status of progress bar on Ubuntu" [High,Confirmed] 16:10:44 what is the expectation about date of 5.0.1? 16:11:05 vkozhukalov: July 15th 16:11:20 not sure abotu 573, a very in-your-face problem 16:11:27 it may change if we are stuck with 856764 16:11:35 I faced 573 many times 16:11:41 and facing it right now 16:11:58 progress bar is hanging until the reload 16:12:05 vkramskikh: did you guys manage to reproduce the issue? ^^ 16:12:42 let's move on folks, take this topic out of here unless angdraug has more info 16:12:54 #topic Features 16:13:01 too many things to discuss still, sorry 16:13:06 #topic Zabbix status 16:13:31 mihgen: we didn't unfortunately :( 16:13:38 Zabbix is tested on custom ISO in HA and multi-node modes 16:13:49 vkramskikh: you may need real deployments 16:13:51 now it is merged into master 16:13:57 akislitsky: all merged? 16:14:06 mihgen, all 16:14:08 great 16:14:09 excellent 16:14:16 any critical bugs you aware of ? 16:14:22 #topic Install OpenStack from upstream 16:14:51 brain461 around? 16:15:07 yes, but there's still question to akislitsky 16:15:15 i have seen problems with nova networks and may be zabbix specific errors will be found in QA cycle 16:15:41 i guess nova will be fixed in library in short time 16:15:48 no worries if someone not replied, those can put "?" here to request for waiting 16:16:04 akislitsky: ok, thanks 16:16:16 brain461: go ahead 16:16:20 rpm building code is merged into master 16:16:37 according to devops, public gerrit is setup, so this unblocks the whole migration process 16:16:47 we prepared the repository with build dependencies for components from master 16:16:57 we prepared specs that could be built against the current master for most of the openstack components, but some of them are still on review 16:17:26 brain461: very good. Do you have any architecture / specs / slides for the setup? 16:17:50 so we could get more info on how it's gonna be implemented and work all together 16:17:52 tomorrow on demo I'll be demonstrating the build process 16:18:17 specs are merged into fuel-specs, btw 16:18:58 #link http://docs-draft.openstack.org/68/101868/3/check/gate-fuel-specs-docs/74f9fd7/doc/build/html/specs/5.1/openstack-from-master.html ? 16:19:19 sorry I was on vacation and didn't have a chance to look into :) will do 16:19:25 ok brain461 thanks 16:19:31 moving 16:19:33 spec looks very short ( 16:19:35 no more questions from me 16:19:37 thanks brain461 16:19:52 #topic ML2 16:19:59 +1 to angdraug … expected to see some diagrams 16:20:06 will provide my feedback later 16:20:15 aglarendil: xarses you go folks 16:20:26 so. 16:20:40 we had a meeting today with xarses and mihgen 16:20:53 yep results are in openstack-dev mailing list 16:21:21 and we decided to merge xenologs implementation right now 16:21:28 to unblock other teams 16:21:41 is it merged or not yet?) 16:21:48 #link http://lists.openstack.org/pipermail/openstack-dev/2014-July/039851.html 16:21:49 I am waiting for the final test result 16:21:54 and we will do extensive tests for Andrew's request 16:22:03 if it passes the test, we will rebase it and merge 16:22:16 along with rebase of Mlnx and NSX features 16:22:28 I know that xenolog did some patches to xarses's request 16:22:32 for this purpose we decided to extend FF a little bit for these particular features 16:22:34 while he was sleeping in PST 16:22:45 yep, he posted some stuff into change requests 16:22:51 but he is offline now 16:23:01 so, the ball goes to xarses again 16:23:02 xarses: are you aware of that? what's the status? 16:23:32 he's already posted another patchset on top of xenolog's changes, so probably aware 16:23:43 mihgen: yes, I've spoken with xenolog, the last changed failed due to missed coma, and its back up for testing 16:23:45 should be just minor fixes from xenolog, he did not have much time today.. 16:23:53 ok 16:24:01 I've almost finished rebuilding my test env with the 307 iso 16:24:11 and will start retesting locally 16:24:19 so I hope it will be tested more or less by EOD on Monday MSK 16:24:41 and we will merge it after that, so extending FF for this ML2 implementation 16:24:53 any more stuff on this topic? 16:24:59 I didn't see mail about test plan 16:25:07 did I miss it? 16:25:12 it's in ML2 blueprints thread 16:25:18 ok 16:25:18 on openstack-dev 16:25:29 I've posted link above 16:25:40 nurla: can we have test plan for ML2 ? 16:25:53 to test xarses implementation? 16:26:28 mihgen: no, we actually should check regression 16:26:51 nurla: we need more than just regression I think. We need to play with params, should not use defaults 16:27:03 and then make sure all IP ranges, etc. propagated properly 16:27:12 regression may appear in random place 16:27:33 even for regression, we need a plan so we can all follow to test xarses ISO once it's ready 16:27:39 can you please prepare one for us? 16:28:38 for xenolog patch i sure our checks that we already have more enough 16:29:13 moving on? 16:29:26 moving 16:29:31 nurla: I'm not about xenolog patch, ok will follow up on what we need 16:29:32 ok, for xarser iso we'll create testplan 16:29:36 vkozhukalov: let's move onn 16:29:37 #topic VMware 16:29:42 ? 16:30:01 NSX+KVM 16:30:02 blueprint #link https://blueprints.launchpad.net/fuel/+spec/neutron-nsx-plugin-integration 16:30:03 fuel-web part #link https://review.openstack.org/#/c/105438/ has two +1 and ready to merge 16:30:03 fuel-library part #link https://review.openstack.org/#/c/86329 has -1 related to ML2 16:30:24 vCenter features 16:30:24 blueprint #link https://blueprints.launchpad.net/fuel/+spec/vcenter-hv-full-scale-support 16:30:26 VMDK support 16:30:27 fuel-library part #link https://review.openstack.org/#/c/104942/ merged 16:30:29 fuel-web part #link https://review.openstack.org/#/c/104944/ merged 16:30:30 HA support #link https://review.openstack.org/#/c/104130/ merged 16:30:31 multi cluster support 16:30:32 fuel-library part #link https://review.openstack.org/#/c/104197/ has one +1 and -1 from CI 16:30:33 fuel-web part #link https://review.openstack.org/#/c/104927/ merged 16:30:51 +100500 for preparing a paste in advance! 16:30:57 #link https://review.openstack.org/#/c/105438/ akasatkin, evgeniyl can we merge? 16:31:35 thanks AndreyDanin 16:31:40 how much is it all ready to go? 16:31:47 any tested by anyone? 16:32:09 As far as I know we merged a lot of VMWare stuff today once master was fixed 16:32:09 we tested it by hands 16:32:10 mihgen: looks good. but didn't test it 16:32:46 AndreyDanin: -1 from CI in multi-cluster - should be on your side or we just need to retrigger 16:32:55 Fuel CI should show correct results today 16:33:05 We have got 6 servers and going to setup a test lab. 16:33:30 so we need to decide what to do with FF for multi cluster support 16:33:43 how much time do we need to finish with merging this stuff? 16:33:49 I would give it a chance 16:34:00 until the end of week 16:34:10 this should be some easy fix 16:34:24 aglarendil: ok.. what about https://review.openstack.org/#/c/86329 ? 16:34:40 how time will it need after ML2 is introduced? 16:35:01 hmmmm... anyway as we are going to give xarses' ML2 a chance 16:35:06 aglarendil: how both things are isolated, i.e. how is it risky to merge for core features? 16:35:11 let's give it to ML2 implementation 16:35:20 also, NSX is isolated and should not affect master 16:35:23 aglarendil: but after xenolog's code, it should work, right? 16:35:32 so AndreyDanin can rely on xenolog's code for now? 16:35:32 ML2+NSX will require a couple additional days 16:35:43 and then rebase? Or what's the plan? 16:35:55 yep, changes for NSX +ML2 should be similar for xenolog's and xarses' code 16:35:58 mihgen: I didn’t test it with xenolog code. 16:36:48 aglarendil: if NSX+ML2 enters from /etc/astute.yaml quantum_settings L2 settings then there will not need to be any code change between the two 16:36:51 aglarendil: xarses If you guys see it's pretty well isolated, we could give an exception for these almost ready stuff up until next Th, but no later 16:37:21 mihgen: I need day or two to rebase to check xenolog/xarses code, make fix and test it 16:37:32 I mean NSX + ML2 & multi cluster support 16:37:47 anyway, we are waiting for MLNx guys until Thursday. 16:37:51 so I would give it a try 16:38:30 it doesn't make a sense to make exception for all, then it would mean slipping the date. But a few isolated small things are fine 16:38:55 I agree 16:38:59 xarses, angdraug: are you agree on NSX + ML2 & muleticluster ? 16:39:17 I agree 16:39:42 yes 16:39:46 we have the majority features to be merged in 24 hours, so, I would feel ok with simple isolated features to be slipeed a little bit 16:39:48 ok, cool. thanks. Let's move on 16:40:06 seems like we are done on this topic 16:40:17 #topic Mellanox 16:40:39 AndreyDanin: do you have any info about this activity? 16:40:52 if not let's pass this topic 16:40:54 Are there any guys from Mellanox? 16:41:00 and move on 16:41:19 I was speaking with aviramb yesterday 16:41:24 It seems I have nothing to share. 16:41:26 I think Dmitry Ilyin can provide some update 16:41:28 looks like there are no mellanox guys 16:41:32 they will fix their entrypoint into ML2 16:41:42 I'll call him 16:41:46 they are waiting also for qemu 2.0 16:41:51 yes, i'm here 16:42:03 which will allow them to use either puppet module 16:42:05 dilyin: you have just had a call with Mellanox guys 16:42:24 dilyin: could you provide a short update on the status of Mellanox code ? 16:42:58 meanwhile angdraug what about documentation for upgrade 5.0 -> 5.0.1 ? 16:43:04 are you aware of the status on this? 16:43:19 dilyin: I assume you are typing meanwhile.. 16:43:55 vkozhukalov: let's move on 16:44:01 we don't have much time 16:44:10 #topic OpenStack patching 16:44:10 documentation for upgrade is in progress, there's a review out 16:44:11 folks be prepared in advance for staus 16:44:18 angdraug: ok thanks 16:44:21 #link https://review.openstack.org/105324 16:44:34 reviews are appreciated 16:44:42 the Puppet part of the code is mostly ready. at least I have refactored all bad styles and not it's passing our CI without mellanox part enabled. I have not tested deployment with mellanox enabled yet 16:45:10 dilyin: ok sound very good, thanks. 16:45:17 ikalnitsky: are you here for status on patching? 16:45:18 ikalnitsky: around? 16:45:25 yep, i'm here 16:45:32 ok, it looks like openstack patching works fine. we have two known bugs and both are not critical, as for me. 16:45:38 the first one is related to node status which doesn't marked as error when update fails 16:45:43 #link https://bugs.launchpad.net/fuel/+bug/1332494 16:45:44 Launchpad bug 1332494 in fuel "[Updates] Nodes do not marked by error status when update fails" [Medium,Confirmed] 16:45:50 the second is related to our openstack init scripts which in some cases may lead to the situation when update process hangs (i mentioned it in previous demo) 16:45:56 #link https://bugs.launchpad.net/fuel/+bug/1333292 16:45:57 Launchpad bug 1333292 in fuel "nova-api hangs during openstack updating" [Critical,In progress] 16:46:00 the fix is already done and has been tested by qa, but it still doesn't merged. 16:46:01 On Mellanox, there are a lot of things to do still, and in agreement FF is extended for them by next Th 16:46:17 and that's it :) 16:46:29 ikalnitsky: very good 16:46:38 nurla: can you confirm QA side of it? 16:47:06 are there any publicly available test cases for this feature? 16:47:59 mihgen: Tatiana Leontovich is responsible for testig this feature. Unfortunately, she is ooo today. 16:48:11 #topic Sustaining features: galera, rabbitmq, pacemaker improvements, manifests merge 16:48:14 nurla: i can confirm, we already have cases and check all important reviewa 16:48:26 guys we have just several minutes 16:49:03 aglarendil: you were going to give some details on that topic 16:49:13 nurla: so what about test cases? 16:49:20 so, we finally merged ha related blueprints 16:49:26 with rabbitmq and galera improvements 16:49:32 they have some bugs 16:49:33 wait 16:49:37 is galera there? 16:49:46 I meant galera ocf script 16:49:47 I thought we reverted it  16:49:59 oh ok, so new package is not in master yet 16:50:06 I am going to merge these modifications tomorrow 16:50:17 with all the newer packages and xtrabackup and so on 16:50:33 this will allow us to make deployment of controllers in parallel 16:50:34 aglarendil: what's the status on testing of it? 16:50:40 which part? 16:50:44 xtrabackup? 16:50:45 galera packages 16:50:50 and xtracackup 16:50:58 we will see tomorrow, I tested it and it should work 16:51:07 as soon as we modify configs correspondingly 16:51:15 this is about galera 16:51:25 ok, thanks, it would be cool to see it master. Please note, if we don't merge it tomorrow - no way for 5.1 for it 16:51:32 no more exceptions on this 16:51:46 vkozhukalov: let's move on, a few things left still 16:51:47 rabbitmq ocf script has some partitioning bugs, but after some poking it reassembles cluster just fine 16:52:00 along with galera script we have a very resilient HA architecture 16:52:09 I tested it just now and it works fine 16:52:17 aglarendil: it's great to hear that our HA finally gets right attention 16:52:24 #topic access-control-master 16:52:24 though, there are some bugs with memcache and some pacemaker improvements 16:52:25 and becomes better and better :) 16:52:50 sambork: you go 16:53:18 We are on good way to finish all tasks according to blueprint stages I and II (http://docs-draft.openstack.org/29/96429/11/gate/gate-fuel-specs-docs/2807f30/doc/build/html/specs/5.1/access-control-master-node.html). Still 7 patches waiting for merge but 4 of them have enought +1 . Only fake tests and update waiting for review https://review.openstack.org/#/q/status:open+branch:master+topic:bp/access-control-master-node,n,z .Tests are going 16:53:41 sambork: very good 16:53:55 out of those 4, all of them into fuel-library? 16:54:24 the plan from aglarendil was to merge ML2, and if it's all fine, then merge access control to fuel-library 16:54:25 3 of them are in fuel-lib one from fuel-main 16:54:50 mattymo: can you confirm we are in a good shape in fuel-library? 16:54:54 yes 16:54:55 We had a false report of a regression with setting a custom password, but I couldn't reproduce. The remainder all good ready to merge. The only holdbakc is we're not enabling auth by default. It's a matter of updating 2 lines in Fuel Library to turn it on. 16:55:10 It's all beautiful. 16:55:15 what about system tests? 16:55:22 unit? all ready for this feature? 16:55:25 Andrey Sledzinskiy said he was running thos etoday 16:55:33 we have no unit tests for puppet 16:55:33 I hope sys tests won't be broken after merge 16:55:41 I told him to run first before full merge 16:55:49 but nothing will be possibly broken until we enable real auth 16:56:01 nurla: are you aware of changes in sys test ? 16:56:06 mihgen: I run only 'one_node_deploy' sys_test with enabled authorization and it passed 16:56:08 5 minutes 16:56:10 because of auths 16:56:19 asledzinskiy: oh that's cool 16:56:24 now I'm checking ha deploys 16:56:24 perfect, thanks guys 16:56:27 https://review.openstack.org/#/c/104131/ 16:56:36 #topic nailgun plugins 16:56:39 let's keep pushing to merge remaining patches 16:56:50 sup 16:57:08 plugins are under active reviewing process 16:57:28 the bad thing is we didn't merge new SQLAlchemy yet 16:57:37 and it is required for plugins to work 16:58:11 then sorry for 5.1.. 16:58:25 will merge your part in master once it's reopened 16:58:30 but still, there are two requests, https://review.openstack.org/#/c/104608/ and https://review.openstack.org/#/c/97827/ 16:58:33 too late for sqlalchemy update now 16:58:40 and they are both ready to merge 16:58:50 #topic image based provisioning 16:58:54 but folks stuff meow-nofer is doing is extremely important 16:58:57 it almost works 16:59:07 we have 6 pull requests 16:59:15 #link https://blueprints.launchpad.net/fuel/+spec/image-based-provisioning 16:59:24 some of them +1 16:59:25 so let's put attention to that, and review thoroughly, especially in terms of design. It's gonna be example on how to do further plugins in nailgun 16:59:39 and it does not break anything 16:59:50 ending 16:59:53 vkozhukalov: then it looks like we can merge if it has enough reviews 16:59:58 thanx everyone 16:59:59 thanks vkozhukalov 17:00:07 #endmeeting