16:00:04 #startmeeting fuel 16:00:05 Meeting started Thu Jul 16 16:00:04 2015 UTC and is due to finish in 60 minutes. The chair is xarses. Information about MeetBot at http://wiki.debian.org/MeetBot. 16:00:05 #chair xarses 16:00:06 Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. 16:00:10 The meeting name has been set to 'fuel' 16:00:11 Current chairs: xarses 16:00:17 hi all 16:00:17 Todays Agenda: 16:00:17 #link https://etherpad.openstack.org/p/fuel-weekly-meeting-agenda 16:00:20 hi 16:00:22 hi 16:00:27 Who's here? 16:00:37 hi! 16:00:38 o/ 16:01:00 hi 16:01:09 hi 16:01:09 hi 16:01:15 hi 16:01:17 hi 16:01:18 hi 16:01:30 ok, lets get started 16:01:33 o/ 16:01:40 #topic Node role as a plugin (ikalnitsky) 16:01:47 hey guys 16:02:14 last week i showed the demo, and it seems the feature works fine. though, not all patches are in master 16:02:33 we have finally merged the patch that combined deployment tasks with core ones 16:02:50 so feel free to use it, you now can inject some deployment tasks for existing roles 16:03:00 the most important patches are: 16:03:05 #link https://review.openstack.org/#/c/200129/ 16:03:10 #link https://review.openstack.org/#/c/199573/ 16:03:28 we have worked on improving them and covering with tests 16:03:43 now, they are readym and should me reviewed and merged 16:03:57 when it's done, i think we could say "the feature is in master" 16:04:05 ikalnitsky: are they any blockers, or issues to raise? 16:04:12 nope 16:04:15 do we have an agreement who will help with reviews? 16:04:34 i think it would be as always - python team 16:04:48 i think sebasian, alex kislitsky, and co 16:05:18 and yeas, the patches were tested by qa 16:05:23 we are closer and closer to FF, everyone is busy 16:05:35 so that's why you guys all need to be in sync.. who helps where and when 16:05:44 i'd kindly ask them :) don't worry, they are actually small.. the most lines are tests 16:05:52 ok :) 16:06:10 thanks! This would be real step forward with plugable arch 16:06:16 will* 16:06:47 xarses: moving on? 16:06:48 xarses, moving one? 16:06:56 s/one/on 16:06:58 #topic Aligning LP groups with bugfixing confirmation queues (adidenko) 16:07:27 We need to align LP fuel-* groups with our teams and bug confirmation queues/duties. For instance https://launchpad.net/~fuel-astute/+members#active group has only 2 members, so there could be problems with bugs assigned to such groups. 16:07:34 that's my concern :) 16:07:53 yeah I saw email at openstack-dev, let's drop this group 16:08:17 if we have any other which you feel needs to be modified - let's just go ahead and propose in openstack-dev 16:08:22 and will assign bugs related to astute to fuel-python, right? 16:08:23 are there others that have this problem? 16:08:34 then if we do it once consensus is reached 16:08:36 and this also applies probably to the MOS sustaining Fuel build teams 16:08:37 yep, fuel-python 16:08:53 it's not ideal but we can rename it later (there is some ruby code still ).. 16:09:00 these teams are new, and it is not clear how to assign milestones for backports 16:09:09 would be nice to see a wiki update 16:09:18 build is a bit harder though, if it's about make system 16:09:29 alex_didenko: what about fuel-provisioning? it has only two members too. 16:09:30 xarses: osci maybe? are we going to replace it with build group? 16:09:32 we don't have many who can undertand make 16:09:43 yeah, fuel-provisioning too 16:09:52 agordeev: I'd also kill it too, we can use tags instead 16:09:56 like "ibp" 16:10:17 in order to allow filtration for SMEs 16:10:31 well, I see no clear rules of milestones assignments for the bug fixes committed to the master of fuel-library. We should clarify teams, members, membership rules here 16:10:37 and assignment criteria 16:10:48 so as you are ibp SME, I'd expect you to look for "ibp" tagged bugs frist 16:10:57 I like the groups because they help know that it's un-assigned, but tags would be best 16:11:31 a one might forgot to assign a tag, but we always will assign a group 16:11:42 so why it's not one group - just because we want to split responsibiltiy between teams, right? And also avoid emails spam 16:11:54 yep 16:11:56 if you work on puppet, you unlikely want to see UI related bugs 16:12:14 so let's just go from here, and suggest adjustments gradually 16:12:14 if we assign a bug to such "phantom" group, it may be lost for some time 16:12:43 true, that's why let's drop fuel-astute and fuel-provisioning then at first? 16:12:58 +1 16:14:09 +1 lets go from there, but it sounds like we don't need the groups any more 16:14:20 lets sum this up and take it back to the ML 16:14:41 let's separate what we +1 here, and "don't need groups anymore" 16:14:50 I don't agree with your this last statement 16:15:00 #action alex_didenko sum up dropping fuel-astute and fuel-provsioning to ML 16:15:39 moving on? 16:15:42 yep 16:15:49 #topic removing classic provisioning (agordeev) 16:16:25 Hello folks, 16:16:26 Since the last week, few blockers have been appeared for QA. 16:16:30 This one already has been fixed https://bugs.launchpad.net/fuel/+bug/1473419 16:16:30 Launchpad bug 1473419 in Fuel for OpenStack "Upgrade failed with version error" [High,Fix committed] - Assigned to Alexander Kislitsky (akislitsky) 16:16:31 And the another one is still here https://bugs.launchpad.net/fuel/+bug/1474883 16:16:32 Launchpad bug 1474883 in Fuel for OpenStack "Upgrade from 6.1->7.0 failed with error" [High,Triaged] - Assigned to Alexander Kislitsky (akislitsky) 16:16:32 Looks like master node upgrade to 7.0 isn't operatable right now. 16:17:21 agordeev, even more, I going to convert upgrade tarball into upgrade package 16:17:39 and it is going to change slightly UX and use cases 16:17:58 wait why do you combine two things 16:18:08 I've sent a letter to nurla asking for QA eng to help me here 16:18:16 upgrade tarball or package should not related to drop of classical proc 16:18:22 provisioning* 16:18:34 nurla is on vacation 16:18:36 mihgen, yes, it does not relate 16:18:47 just want to inform about this 16:19:00 agordeev: so you say that once we merged your code we've got these regressions in upgrade? 16:19:02 mihgen: in order to test the feature, upgrade should be performed 16:19:19 mihgen: nope, it doesn't related with my code. 16:19:47 agordeev: so those blockers are not related to your removal of classic prov? 16:20:11 why then you posted then in your update, you guys both confused me 16:20:15 mihgen: those block QA. 16:20:41 Ok. the feature (removal) is basically merged, but we can't verify it 16:20:42 this issue blocks testing of case: 6.1 with classic prov -> upgrade to 7.0 + add node to 6.1 cluster 16:20:42 upgrade does not work at the moment, right? 16:20:48 is that what you are saying? 16:21:25 mihgen: yeah, we can't be sure that we could add a new nodes to old env provisioned with classic. 16:21:54 kozhukalov_: right, it's non working 16:22:05 ok. I see there is a work on upgrade tarball… I'm quite surprised though that we seem to be just discovered it 16:22:12 after these are fixed, do we expect any other blockers? when do you expect these to be fixed? 16:22:26 while it was failing since HCF of 6.1, I believe 16:22:31 we definitely need to fix it (upgrade), and it looks the best way to fix it is to convert it into rpm and change UX, because anyway we need to change build script for it 16:23:04 kozhukalov_: I'm not sure about even RPM. If we spend so much effort on this, I question the whole approach 16:23:19 may be we should just back up DB, keys, whatever data we have on master 16:23:23 xarses: no other blockers are expected, if upgrage will work 16:23:28 then reinstall new version from scratch 16:23:37 what is wrong with converting it into rpm? 16:23:42 and then just restore DB, run migration and that's it 16:24:22 question is why do we need to support in place upgrade, and spend so much effort on keep it running 16:24:52 let's just think about it. RPM is better anyway than tarball, but I'm just even going further 16:24:52 anyway, there is a script which makes backups and then restores and does something else, this script can be easily delivered as rpm 16:25:13 mihgen: are you talking about getting rid of upgrades completely? 16:25:19 And I don't understand how RPM would help in case of the blocker failures we've got 16:25:22 dpyzhov: no 16:25:26 ok, lets move along here. Can we sumarize what the next steps to fixing this to the ML? 16:25:26 mihgen, there is a mail thread in openstack-dev 16:25:35 about this change 16:25:47 I explained workflow of upgrade, manual, not in-place automated 16:25:48 let's summarize possible approaches there 16:25:55 ok let's move it to openstack-dev… 16:26:06 kozhukalov_: can you take this back to ML? 16:26:17 it is even better if we don't need this upgrade script at all 16:26:30 yeah let's move on guys sorry I'm holding this for too long 16:26:40 #topic flexible networking (akasatkin) 16:26:49 spec: https://review.openstack.org/#/c/195109/ 16:26:50 We have to start 3 more tasks in Nailgun/CLI. 16:26:50 Hopefully, start them tomorrow and/or Monday. 16:26:50 Other stuff is on review/merged. 16:26:50 Library stuff is mostly merged and other is on review. 16:26:50 So, implementation performance in Nailgun is under question. 16:26:50 kozhukalov_: let's continue in openstack-dev 16:26:58 We had discussions with other teams about some changes that they need in our implementation (in VIPs and network roles) so additional time maybe required for that stuff. 16:26:58 There are problems with ISOs quality which affect our testing performance also. 16:26:59 e.g. https://bugs.launchpad.net/fuel/+bug/1475335 was discovered today (fix merged already, test in progress) 16:26:59 Feature demo is planned on Tuesday. 16:26:59 Launchpad bug 1475335 in Fuel for OpenStack "Environment provisioning fails, can't create boot image: 'Command: debootstrap --verbose --no-check-gpg --arch=amd64 trusty /tmp/tmpCVnqMk.fuel-agent-image http://mirror-pkgs.vm.mirantis.net/pkgs/ubuntu-2015-07-15-170127/' returns 1" [Critical,Fix committed] - Assigned to Aleksandr Gordeev (a-gordeev) 16:27:25 #action kozhukalov_ to update ML with next steps for get upgrades to work now that classic prov is removed 16:28:19 why do we have regressions like https://bugs.launchpad.net/fuel/+bug/1475335 ? Let's work on regressions 16:28:19 Launchpad bug 1475335 in Fuel for OpenStack "Environment provisioning fails, can't create boot image: 'Command: debootstrap --verbose --no-check-gpg --arch=amd64 trusty /tmp/tmpCVnqMk.fuel-agent-image http://mirror-pkgs.vm.mirantis.net/pkgs/ubuntu-2015-07-15-170127/' returns 1" [Critical,Fix committed] - Assigned to Aleksandr Gordeev (a-gordeev) 16:28:36 we can't allow master being broken as it immediatelly blocks a number of people.. 16:29:07 akasatkin: what do you need under additional time? How close we are to have all the code required on review? 16:29:31 I see quite a number of patches on review still: https://review.openstack.org/#/q/topic:bp/templates-for-networking+status:open,n,z 16:29:47 mihgen, it is because we removed unnecessary fuel-image package and some of its dependencies were needed by fuel-agent, we added this dependencies into fuel-agent package 16:30:20 we need 3 more patches on review , hopefully, tomorrow or Monday 16:30:41 what are those exactly about? 16:30:47 who will work on them? 16:31:13 CLI, VIPs in tepmlates, net-checker 16:31:39 why do we keep feature in "green" then this week…. ? It should be rather yellow if not red as from what I hear .. 16:31:40 Ivan Kliuk, Ryan Moe, Alex Saprykin 16:32:08 If we don't have time for net-checker, then we will have to sacrifice it 16:32:27 and keep net-checker working only for old schema 16:32:40 yes, it's the biggest part, apparently 16:33:05 are they any other blockers / issues ? 16:33:08 ideally to fix net-checker as soon as possible afterwards, and possible to put a package someone in our repos, so users can try it later 16:33:26 does it look like we will be done in time for FF? 16:33:48 if CLI is ok for exception from FF, then VIPs are not much I would say 16:34:13 can we focus on VIPs then to make it 100% to master by FF 16:34:22 looks, we should be ok with other stuff (excluding net-checker). cli and vips are rather small tasks 16:34:40 yes 16:34:54 akasatkin: it is so important to get it by FF, we've got a tone of bugs to fix too…. 16:34:59 what about library part? 16:35:39 all those network roles, etc.? 16:35:45 xenolog: xarses: ? 16:35:48 mihgen: alot is on review, nova compute may be at risk 16:35:54 xenolog works on nova/migration task which is absolutely required, 16:35:57 we need help to get reviewed and merge 16:36:18 #link https://review.openstack.org/#/c/194438/ 16:36:31 this one hangs over there for about a month 16:36:42 we need to put other tasks aside and finish this 16:37:03 it was a eary hack that has been updated to current scheme in the last 1.5 week 16:37:22 I needed to show the interface to someone else, which is why it was so old 16:37:38 also I had wrong tag on it 16:37:49 so when do we expect to finish this work on ceph? 16:38:01 today, monday at the latest 16:38:04 any other role at threat of not being done by FF? 16:38:46 no, apparently 16:39:14 let's ensure we do extensive testing on what was separated… to ensure there is no surprises :) 16:39:40 xarses: thanks sounds good, we will need to get it reviewed and merged though.. 16:39:42 sure 16:39:48 ok thanks akasatkin 16:40:12 moving on? 16:40:34 yep 16:40:36 #topic sorters and filters in UI (jkirnosova) 16:40:43 Hi all! just wanted to mention we've merged Fuel UI features: node compact view and node list sorting and filtering. So, any of your feedback is desirable. This will help us (UI team) to improve the features UX. You can write it directly to #fuel-ui chat or create tickets in Launchpad 16:40:51 tha's all i wanted to say) 16:40:55 that's * 16:41:17 jkirnosova: I saw the new layout it looks great 16:41:20 jkirnosova: thanks!! 16:41:29 do you have someone from QA going over Fuel UI manually? 16:41:55 yep, Nastya Palkina tested custom ISOs before we merged the features 16:42:01 ok cool, thanks 16:42:24 xarses: moving? 16:42:25 #topic re-enable rabbitmq admin plugin to support fixing LP#1383258 (mwhahaha) 16:42:38 We have a bug about not properly restoring non-default users/vhosts/etc in rabbitmq (LP#1383258) which may require re-enabling the admin management plugin for rabbitmq. 16:42:45 #link https://bugs.launchpad.net/fuel/+bug/1383258 16:42:45 Launchpad bug 1383258 in Fuel for OpenStack 7.0.x "RabbitMQ do not keep non-default users/vhosts/etc after failover" [High,In progress] - Assigned to Alex Schultz (alex-schultz) 16:42:49 It was previously disabled to address a possible security issue related to this plugin which seemed to be an html issue. 16:42:53 #link https://review.openstack.org/#/c/179032/ 16:42:57 Would it be acceptable to re-enable it but limit access to only localhost? There does not appear to be another way to dump/restore rabbitmq users without this plugin. It should be noted that we still have this plugin enabled on the fuel-master. 16:43:01 Or is anyone aware of an alternative way to backup and restore these items for rabbitmq? 16:43:49 mwhahaha: I would be fine with localhost only 16:44:11 if it's simplest thing to do to help in such cases, my opinion is why not 16:44:15 I suggest to re-enable this plugin and address the sec. issue by restricting access to the plugin management UI from external nets 16:44:36 bogdando: just localhost only should be just fine, right? 16:44:37 or to only localhost, indeed 16:44:42 as we aren't using it anywhere i think the safest is to limit to localhost until a time where we need more access from elsewhere 16:44:55 mwhahaha: +1 16:44:56 i will proceed with that unless anyone else has a better idea or any issues with this approach 16:45:11 solved :) xarses - let's move ) 16:45:19 got it, thanks. 16:45:25 #topic SSL feature status (sbogatkin) 16:45:29 4 things left to discuss 16:45:31 We were ready to merge last SSL commit yesterday (had had some +1s from the team already), but it didn't pass OSTF with enabled SSL. OSTF team found a problem and created fix for that. Today some wonderful stuff was merged twice already and that actions broke last SSL commit every time. I resolved merge conflicts and created new ISO with both new SSL patchset and OSTF team tests fix. If it will pass CI then those commits could (and will, I 16:45:31 hope) be merged today and SSL feature will be mainly closed. 16:45:38 Some tests from QA team is although on the way. 16:46:08 sbog_: sounds challenging but under control 16:46:20 of course 16:46:26 what about Fuel UI/rest api under SSL? 16:46:32 do you cover those as well? 16:46:51 sbog_: are there any other issues, or blockers? 16:47:07 mihgen: Already merged, but disabled by default. Need to create one more patch to enable it. 1 line of code. 16:47:18 xarses: seems that no 16:47:36 sbog_: cool. Let's ensure that we document somewhere how to disable it if people need to 16:47:44 Okay 16:48:02 sbog_: I assume you are in sync with QA for their framework 16:48:08 #topic proposal to add fuel to openstack projects - status update (angdraug) 16:48:20 mihgen: yep 16:48:32 sbog_: cool, thx 16:48:47 our proposal was received positively, but TC has voted it down for now: 16:48:47 #link https://review.openstack.org/199232 16:48:47 two primary concerns are: 16:48:47 1) forked copies of puppet-openstack modules 16:48:47 2) fuel commits must be verified and gated on openstack infra 16:48:52 the most promising approach to addressing the first problem is using puppet-librarian to manage upstream modules: 16:48:53 #link https://docs.google.com/document/d/13aK1QOujp2leuHmbGMwNeZIRDr1bFgJi88nxE642xLA/edit 16:48:53 I think we should start doing it now, one module at a time 16:48:55 for example, we should convert the ironic module commit to librarian: 16:48:59 #link https://review.openstack.org/194184 16:49:01 we have a lot of syntax and unit tests that can be moved to openstack infra 16:49:03 same with component integration tests such as fuel ui tests based on nailgun fake-ui 16:49:05 that should be our next step with the Fuel CI 16:49:07 running deploy tests (fuel-qa) on openstack infra is a much bigger problem, but also not impossible 16:49:10 we don't have time to do that in 7.0, but we should start researching this for 8.0 16:49:12 thoughts? 16:49:49 angdraug: yeah the problem is time and resources now 16:50:04 however, some simple changes can be done even now I think, in 7.0 16:50:10 exactly my point 16:50:17 like switching to re-using some of the tests 16:50:22 we can't un-fork all modules at once 16:50:33 but using librarian to add new modules would not add significant effort 16:50:45 angdraug: this one as well 16:50:52 +1 for new module format 16:50:59 angdraug: ya, I don't think we can start on any of this until after SCF. We can try some minor things then and maybe in a fork of fuel-library to isolate effect 16:51:33 we should vet out all of the technical issues there and then bring it into master once we branch from 7.0 16:51:35 we are getting ironic puppet module in 7.0, why don't we get in new way 16:51:51 right from the very beginning and see how it fly 16:51:56 exactly 16:51:57 mihgen: there is no tooling technicaly vetted yet 16:52:08 the commit I linked is still on review 16:52:11 so I'm worried, we may need to fall back to the old way 16:52:15 if it doesn't fly as expected,we can always switch to old approach 16:52:26 we can fall back to the old way if we provide a hard limit on how long to spend to port to librian 16:52:32 xarses: fall back is easy, so why don't we try 16:52:43 I think the hard limit should be end of day monday 16:52:46 well we have untill thursday to get it in? 16:52:49 is that reasonable? 16:52:51 so tuesday 16:52:57 if we find it takes more than a few days, the existing method of porting is already there w/ librarian because it'll check it out in the directory tree 16:53:26 yes, let's see how hard it is. If it's hard, old way to go for now 16:53:48 And another thing is tests, including simplest stuff like pep8, do we run those on Fuel CI? 16:53:55 on the CI: replacing noop with jobs that re-use the unit tests we already have should be relatively trivial 16:54:14 we do run pep8 in some repositories 16:54:15 we can enable in non-voting first openstack infra CI 16:54:27 then enable in voting, then disable Fuel CI 16:54:30 yes, that's the plan 16:54:49 question is who has spare cycles to work on it ) 16:55:03 start with trivial syntax checks like pep8, then move on to unit tests 16:55:12 spare cycles? what are you talking about? :p 16:55:26 we'll pray to the code fairy 16:55:38 angdraug: can we write this up to the ML and move from there? 16:55:44 xarses: +1 16:55:47 of course 16:56:06 as long as there are no objections here, I will write it up on openstack-dev 16:56:18 moving on? 16:56:21 #topic Granular deployment and separation from controller role (mattymo) 16:57:03 #action angdraug will write up to ML about moving simple tests to Openstack CI 16:57:16 mattymo: ? 16:57:25 #topic Enable auto-abandon (mihgen) 16:57:38 hey guys you saw my email and participated 16:57:44 can we just go ahead? 16:57:49 what do we need to enable it? 16:57:57 #action angdraug to write up ML about moving new puppet module to librarian and timeline 16:58:09 angdraug: is it something your team can help with ? 16:58:33 yes if there's consensus 16:58:37 Question to all core reviewers - did we go through patches and marked -1 where we need feedback? 16:58:40 there was dissent from sbog_ and I agree with him 16:58:49 and where we don't need anything, did we just merge? 16:58:58 #action mwhahaha will set up librarian with puppet-ironic as a start before FF (monday) 16:59:19 our current stats are terrible guys 16:59:28 the goal is not to fix the stats 16:59:35 the goal is to fix the review process 16:59:54 note: FF is Thursday 17:00:04 stats represent where we are with the process 17:00:08 o/ 17:00:10 #link http://stackalytics.com/report/reviews/fuel-group/open 17:00:14 wor the reviews, we need to ensure that they are touched before going 17:00:18 but we need to close 17:00:18 folks, time! 17:00:21 ok let's move it to openstack-dev 17:00:26 thanks all 17:00:28 * Daviey serves eviction notice to fuel 17:00:31 #endmeeting