16:01:14 <ihrachys> #startmeeting neutron_ci
16:01:15 <openstack> Meeting started Tue May  2 16:01:14 2017 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:15 <ihrachys> o/
16:01:16 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:18 <openstack> The meeting name has been set to 'neutron_ci'
16:01:45 <ihrachys> jlibosva, seems like we are two huh ;)
16:01:45 <jlibosva> \o
16:01:47 <ihrachys> anyhoo
16:01:57 <ihrachys> #topic Action items from prev meeting
16:01:57 * jlibosva ¯\_(ツ)_/¯
16:02:09 <ihrachys> first was "jlibosva to revise TESTING doc to update about test_install_flood_to_tun gate status with UCA used for Pike"
16:02:12 <ihrachys> I think that happened
16:02:23 <jlibosva> yep, I think so too
16:02:47 <ihrachys> this: https://review.openstack.org/457722
16:03:14 <ihrachys> next was "ihrachys to figure out why gerrit dashboard seems to not show some gate-failure fixes"
16:03:33 <ihrachys> that is a classic pebak
16:03:56 <ihrachys> the review dashboard hides patches that you reviewed, and since I reviewed most if not all of them when I was checking, I couldn't seem much there
16:04:16 <ihrachys> next was "ihrachys to report a bug for fullstack connectivity failures"
16:04:36 <ihrachys> instead of that, jlibosva figured that was kernel ovs module not compiled after UCA switch
16:04:50 <ihrachys> fixed in all branches, master fix: https://review.openstack.org/458591
16:05:06 <ihrachys> I think the job still shows bad behaviour, but a different one; we will have a look later.
16:05:15 <ihrachys> next was "jlibosva to report a bug for scenario failures"
16:05:23 <ihrachys> the scenario jobs are close to 100% still
16:05:33 <ihrachys> jlibosva, have you got a chance to look at those or report a bug?
16:05:52 * jlibosva scratches his head
16:05:54 <jlibosva> I think I didn't
16:06:48 <jlibosva> no, I don't see any bug reported by me, I somehow forgot
16:07:13 <ihrachys> ok, np. will you have time for that?
16:07:17 <ihrachys> or I can have a look
16:08:16 <jlibosva> it seems that vlan interface in trunk test can't get dhcp address
16:08:26 <clarkb> can you not stop compiling ovs if the version is newer in uca than what you need?
16:08:36 <ihrachys> clarkb, for fullstack, we need 2.6.x
16:08:40 <ihrachys> and it's 2.5.x in UCA
16:08:43 <jlibosva> I'm thinking about disabling port security for trunk tests
16:08:51 <clarkb> ihrachys: so the check for 2.5.1 was just buggy?
16:08:54 <ihrachys> we stopped compilation in functional since 2.5.x is enough there
16:09:01 <ihrachys> clarkb, yeah
16:09:27 <jlibosva> clarkb: ihrachys what? it worked, no?
16:09:39 <jlibosva> once we got newer, we stopped compiling
16:09:46 <ihrachys> jlibosva, whatever reflects actual usage of the api. but is it a must to have port sec disabled?
16:09:51 <clarkb> jlibosva: right but you needed to continue compiling
16:10:01 <clarkb> jlibosva: so the bug was in stopping compiling even though you needed to compile
16:10:11 <clarkb> sounds like if the check was >=2.6 would be fine
16:10:21 <jlibosva> ihrachys: it is until I fix ovs fw which needs some ovs patches that currently doesn't work with dvr.
16:10:30 <jlibosva> ihrachys: so until then, I would be fine by disabling port-security
16:10:42 <ihrachys> jlibosva, then let's reflect that with a caveal like a bug mentioned somewhere with TODO
16:10:59 <ihrachys> you think it will help?
16:11:06 <jlibosva> clarkb: we have two kind of tests, functional and fullstack. For functional we needed 2.5.1 userspace and for fullstack, we still do compile kernel module
16:11:13 <jlibosva> not userspace anymore tho
16:11:33 <ihrachys> clarkb, right, what jlibosva said, we also reduced the scope of compilation
16:12:07 <jlibosva> ihrachys: yep, we can enable once the firewall works for trunk use-case
16:12:24 <jlibosva> ihrachys: but it wasn't 100% before
16:12:26 <ihrachys> jlibosva, I smell you are going to post a patch?
16:12:28 <jlibosva> so maybe it's not the firewall
16:12:47 <jlibosva> ihrachys: I should, I have 4 days then I'm gone for 3 weeks :-/
16:13:15 <ihrachys> jlibosva, oh I see. we may cancel next meetings till you are back.
16:13:29 <ihrachys> #action jlibosva to post a patch to disable port sec for trunk scenarios
16:13:42 <jlibosva> ihrachys: ack
16:13:51 <ihrachys> if you don't, or it doesn't help much, I will have a look at the job next weke.
16:13:54 <ihrachys> *week
16:13:57 <ihrachys> and thanks
16:14:08 <ihrachys> next was "ihrachys to clean up gate-failure tagged bugs list"
16:14:18 <ihrachys> I did, closed some, dupped others...
16:14:26 <ihrachys> next was "ihrachys to look at what the functional spike yesterday was"
16:14:35 <ihrachys> but I did not because...
16:14:40 <ihrachys> next was "ihrachys to propose voting for functional job"
16:14:47 <ihrachys> and jlibosva was kind to post the patch instead of me
16:15:02 <ihrachys> https://review.openstack.org/460205
16:15:04 <ihrachys> and it's merged
16:15:10 <ihrachys> so now functional is in both voting queue
16:15:12 <ihrachys> *queues
16:15:20 <ihrachys> which is a huge progress I must say
16:15:26 <jlibosva> and yay to so far results, although I think we didn't have that many patches going in over weekend and bank holidays in some countries
16:15:27 <ihrachys> and it seems like it's going smooth
16:15:55 <ihrachys> ok that's all AIs we had
16:16:01 <ihrachys> #topic Grafana
16:16:04 <ihrachys> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:16:08 <ihrachys> several major issues there
16:16:30 <ihrachys> one is api job going up, currently at 35% but I believe it's going to 100%
16:16:45 <ihrachys> yamamoto posted a fix: https://review.openstack.org/#/c/461661/
16:16:52 <ihrachys> that's related to keystone v2 disablement
16:17:06 <ihrachys> I am not sure about the fix proposed. seems like it just skips tests.
16:17:12 <ihrachys> instead of making it work in new environment
16:17:45 <ihrachys> unless yamamoto gets back in next hours, I will have a look at reworking the patch so that it passes those test cases for v3
16:18:05 <ihrachys> there is also rally job failure trend going up
16:18:09 <ihrachys> for basically same issue
16:18:16 <ihrachys> that should already be fixed with https://review.openstack.org/#/c/461686/
16:18:27 <ihrachys> (again yamamoto was rapid to tackle it)
16:18:33 <jlibosva> maybe we could merge the skip with todo to enable for v2 to unblock the gate?
16:19:00 <ihrachys> jlibosva, that's possible, gotta make it Related-Bug I guess and add some todos
16:19:41 <ihrachys> I will take care
16:19:46 <ihrachys> and that's for grafana
16:19:51 <ihrachys> #topic Python3 strategy
16:20:09 <ihrachys> jlibosva, so, considering you are going offline this week, do we have anything at this point?
16:20:24 <jlibosva> nope :( I suck
16:20:36 <ihrachys> no you don't
16:20:39 <ihrachys> that's ok :)
16:20:53 <ihrachys> I guess it will be on me to get something ready
16:21:03 <ihrachys> jlibosva, btw do you go to the summit?
16:21:09 <jlibosva> I'll be at the summit next week so I hope I'll talk to relevant people there
16:21:27 <jlibosva> ihrachys: yes :) so I won't be totally offline, I hope some work will be done from my side
16:21:33 <ihrachys> ok, cool. yeah. maybe talk to kevinbenton to understand what's the plan there, and who's going to take care.
16:22:04 <ihrachys> I don't feel at this point I will have time for that in the end. but we'll see.
16:22:26 <ihrachys> I mean, not actually implementing the thing. I can write up and rant ofc.
16:22:59 <ihrachys> #action jlibosva to talk to higher summit beings on python3 gate strategy for Pike
16:23:08 <jlibosva> I think it should be somehow similar to what other projects do
16:23:38 <ihrachys> could be worth checking what's the minimal we can do to tick it
16:23:39 <jlibosva> We'll need to consider the amount of jobs we run and the fact that we might lose some of resources due to osic shutdown
16:23:55 <jlibosva> if we already haven't
16:24:45 <ihrachys> cool, I believe in your ability to get to the bottom of it. we can sync later next week.
16:24:51 <ihrachys> #topic Other patches on review
16:24:58 <ihrachys> there is not much that I am aware
16:25:14 <ihrachys> one thing is stable branches are currrently all failing because of the cinder breakage with lvm
16:25:18 <ihrachys> that is fixed in master
16:25:23 <ihrachys> I requested backports here: https://review.openstack.org/#q,I4f40a1984fe828c8ff965033f7e25b1d7516ab1e,n,z
16:25:33 <ihrachys> once those are in, we should be able to recheck all failed patches
16:26:18 <smcginnis> ihrachys: Hoping to have those approved shortly.
16:26:29 <ihrachys> smcginnis, thanks for quick review
16:26:38 <smcginnis> ihrachys: No problem
16:26:50 <ihrachys> and that's about it I guess. I am not aware of any patches of particular interest.
16:27:35 <ihrachys> ok I guess that's it for the meeting. have fun on the summit. I will definitely cancel the next meeting, and probably two more unless someone shows a particular interest in having those.
16:27:43 <jlibosva> I have one update re fullstack
16:27:48 <ihrachys> #action ihrachys to cancel next team meetings
16:27:49 <ihrachys> jlibosva, shoot
16:27:57 <jlibosva> I spent some time looking at the trunk failure
16:27:58 <ihrachys> #topic Open discussion
16:28:20 <jlibosva> and I don't understand the root cause yet but basically when we remove a port from trunk bridge, (ovs-vsctl del-port) it removes the bridge itself
16:28:30 <jlibosva> which sounds like an issue in ovs
16:28:51 <ihrachys> :-o
16:28:51 <jlibosva> then later, agent thinks that trunk bridge was cleaned up and leaves patch ports on integration bridge behind
16:29:16 <jlibosva> I'm trying to create a minimal reproducer outside of fullstack tests to report bug to openvswitch if confirmed that's it's really the issue
16:29:23 <ihrachys> do we have a bug reported for that? would make sense to capture those details somewhere.
16:29:33 <jlibosva> a simple add-br, add-port and following del-port doesn't reproduce it
16:29:56 <jlibosva> no bug reported yet, I'll report one once I have the reproducer
16:30:30 <ihrachys> may make sense to report one just for the sake of gate breakage tracking
16:30:39 <jlibosva> and that's about fullstack
16:30:48 <ihrachys> #action jlibosva to report fullstack trunk failure bug once he has a reproducer
16:30:49 <jlibosva> the ovs bugs are reported to ovs-discuss mailing list :-/
16:31:03 <ihrachys> jlibosva, oh. have a link handy for the ovs-discuss?
16:31:45 <jlibosva> so far I have this: http://paste.openstack.org/show/608616/
16:31:58 <jlibosva> ihrachys: that's a ML but I haven't reported the bug there yet
16:32:17 <jlibosva> https://mail.openvswitch.org/mailman/listinfo/ovs-discuss
16:32:19 <jlibosva> ihrachys: ^^
16:32:34 <ihrachys> ah I misunderstood
16:32:39 <ihrachys> thought you raised something to the ML
16:33:02 <jlibosva> about the paste - I'm able to reproduce by simple del-br ovs-vsctl command. So I need to figure out what the bridge needs to be removed by ovs.
16:33:22 <ihrachys> you mean by del-port
16:33:23 <jlibosva> ah, no, I haven't yet. But I hope tomorrow I'll come up with reproducer script
16:33:29 <jlibosva> yeah, del-port :)
16:33:38 <ihrachys> :) otherwise it would make sense.
16:33:43 <ihrachys> ok sounds scary.
16:33:47 <ihrachys> wonder if we use it correct.
16:33:53 <ihrachys> though nothing pops
16:34:06 <ihrachys> ok thanks for the update, really interesting.
16:34:15 <jlibosva> that's all from me
16:34:26 <ihrachys> enjoy the rest of the hour, and summit, and PTOs, and lifed
16:34:28 <ihrachys> *life
16:34:30 <ihrachys> #endmeeting