16:00:22 <ihrachys> #startmeeting neutron_ci
16:00:24 <openstack> Meeting started Tue Nov 14 16:00:22 2017 UTC and is due to finish in 60 minutes.  The chair is ihrachys. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:26 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:28 <openstack> The meeting name has been set to 'neutron_ci'
16:00:40 <ihrachys> hi all
16:00:56 <slaweq> hello
16:00:59 <ihrachys> we skipped the last time because of summit but now we are hopefully back to business
16:01:17 <ihrachys> giving people some time to join
16:01:28 <haleyb> hi
16:01:33 <jlibosva> o/
16:01:40 <jlibosva> sorry for joining late
16:02:43 <mlavalle> o/
16:03:01 <ihrachys> ok let's get it started
16:03:06 <ihrachys> #topic Actions from prev meeting
16:03:11 <ihrachys> we had a single one
16:03:16 <ihrachys> "ihrachys to follow up with Chandan about tempest split again"
16:03:22 <ihrachys> I guess I can update on that here
16:03:50 <ihrachys> so we prepared the new repo after cookie cutting it: v
16:03:51 <ihrachys> https://review.openstack.org/506553
16:04:03 * mlavalle got confused with the falling back to standard time
16:04:10 <ihrachys> also added -nv jobs in neutron repo: https://review.openstack.org/516934
16:04:55 <ihrachys> there is a follow up on the patch adding jobs here: https://review.openstack.org/519649 removing -api job from gate pipeline (it's just cosmetic, the job is not triggered by zuul)
16:05:42 <ihrachys> also, Chandan has a patch inheriting those new jobs in the new repo: https://review.openstack.org/519646
16:06:13 <ihrachys> once it's all done, next steps are syncing latest test code from neutron repo to the new one: https://review.openstack.org/519310 (still WIP)
16:06:40 <mlavalle> Nice progress!
16:06:42 <ihrachys> then we'll enable voting for new jobs, remove legacy jobs, and finally clean up neutron tree from tempest code
16:07:14 <ihrachys> I want to block all patches for tempest code after we land the sync patch
16:07:26 <mlavalle> ++
16:07:37 <ihrachys> basically go through all patches touching neutron/tests/tempest/... and -2 them until the code is spun into the new repo
16:08:09 <ihrachys> I *think* we haven't missed anything in the list of tasks, but if you see a gap, feel free to tell
16:08:21 <ihrachys> maybe we'll need to update our docs somewhere to reflect the new location of tests
16:08:22 <mlavalle> will do
16:08:34 <mlavalle> That is a good idea
16:08:58 <slaweq> so new "workflow" will be to add tempest tests after patch to neutron is merged, right?
16:09:15 <slaweq> or will be any way to do it "in parallel"?
16:09:18 <ihrachys> slaweq, kinda. you can post a patch with depends-on
16:09:24 <slaweq> ahh, ok
16:09:26 <ihrachys> and reviewers can enforce the patch present
16:09:44 <ihrachys> also note that the new repo is meant for pike only
16:10:03 <ihrachys> so if you let's say want to get it in an earlier release, you gotta backport it to neutron repo
16:10:19 <slaweq> ok, good to know, thx
16:10:19 <ihrachys> in the future, new releases will all be covered by the new repo
16:10:28 <ihrachys> but we start with current master only
16:11:00 <mlavalle> makes sense
16:11:32 <ihrachys> #topic Grafana
16:11:35 <ihrachys> http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:12:41 <ihrachys> nothing of special interest there it seems
16:12:56 <ihrachys> ryu periodic has failed twice though
16:14:44 <ihrachys> which is weird, I can't see any failures in http://logs.openstack.org/periodic/legacy-periodic-tempest-dsvm-neutron-with-ryu-master/
16:15:43 <ihrachys> I guess some fluke in data
16:16:06 <ihrachys> #topic Fullstack
16:16:24 <ihrachys> so this landed: https://review.openstack.org/#/c/517598/
16:16:36 <ihrachys> which should hopefully make trunk test fixed
16:17:00 <mlavalle> thanks slaweq :-)
16:17:11 <slaweq> I hope it helps
16:17:26 <ihrachys> (taking random failure) http://logs.openstack.org/13/518113/1/check/legacy-neutron-dsvm-fullstack/08c6be8/logs/testr_results.html.gz
16:17:35 <ihrachys> it seems to pass
16:17:45 <ihrachys> though there are still failures there
16:18:00 <slaweq> good to hear that it passes
16:18:51 <ihrachys> it seems that those connectivity/sg failures are now the new constant. here is another log with the same: http://logs.openstack.org/51/396351/43/check/legacy-neutron-dsvm-fullstack/6f49eec/logs/testr_results.html.gz
16:19:14 <ihrachys> in the latter log, see how test_north_south_traffic failed
16:19:23 <ihrachys> it failed waiting for port active
16:20:12 <ihrachys> and here is what I see in dhcp agent: http://logs.openstack.org/51/396351/43/check/legacy-neutron-dsvm-fullstack/6f49eec/logs/dsvm-fullstack-logs/TestLegacyL3Agent.test_north_south_traffic/neutron-dhcp-agent--2017-11-14--10-37-54-629926.txt.gz?level=TRACE#_2017-11-14_10_38_37_370
16:20:17 <ihrachys> RuntimeError: Second simultaneous read on fileno 9 detected
16:20:23 <ihrachys> afair it's eventlet issue
16:20:40 <ihrachys> like when something is not monkey patched
16:20:59 <ihrachys> and it happens when calling to rootwrap
16:21:32 <ihrachys> also, this error does seem like a logic bug in neutron: http://logs.openstack.org/51/396351/43/check/legacy-neutron-dsvm-fullstack/6f49eec/logs/dsvm-fullstack-logs/TestLegacyL3Agent.test_north_south_traffic/neutron-dhcp-agent--2017-11-14--10-37-54-629926.txt.gz?level=TRACE#_2017-11-14_10_39_57_401
16:21:47 <ihrachys> seems like we pass mac address of None there into netaddr
16:22:40 <jlibosva> the first one should be fixed by https://review.openstack.org/#/c/514547/, is that right?
16:22:44 <ihrachys> I believe for the eventlet issue, the following patch may help: https://review.openstack.org/#/c/514547/
16:22:48 <ihrachys> jlibosva, yeah :)
16:23:30 <ihrachys> and afaiu it also helps the issue with obsolete data left in buffer when we raise Timeout
16:23:51 <ihrachys> toshii explained there why it's so
16:24:31 <ihrachys> if we don't unset need_restart flag, it means we raised an error, so we force restart which makes it clean up the socket
16:25:02 <ihrachys> thoughts?
16:25:36 <jlibosva> we should get some oslo cores, that's what I think :)
16:25:45 <ihrachys> yeah
16:26:38 <ihrachys> #action ihrachys to pull oslo folks into reviewing rootwrap patch
16:27:19 <ihrachys> as for mac address being None, I think we still should fix it.
16:27:32 <ihrachys> even though maybe it wouldn't show up without eventlet issue
16:27:40 <mlavalle> the L3 subteam can take a look
16:28:08 <ihrachys> mlavalle, ok cool
16:28:13 <mlavalle> I can bring it up on Thursday
16:28:30 <ihrachys> #action mlavalle to track down "TypeError: None is not str() or unicode()!" error in dhcp agent fullstack tests
16:28:38 <jlibosva> isn't the second only a consequence? it failed to create a namespace so then later it didn't add a device
16:28:41 <ihrachys> mlavalle, I put you personally on the hook ;)
16:28:47 <mlavalle> that's ok
16:28:53 <mlavalle> LOL
16:29:00 <ihrachys> jlibosva, probably, but passing bad data into a lib is bad taste
16:29:12 <mlavalle> yeah, it usually is :-)
16:30:01 <ihrachys> ok, I think it's enough for fullstack. as usual, we will revisit next time when we make progress on rootwrap.
16:30:52 <ihrachys> #topic Scenarios
16:31:25 <ihrachys> I take a random recent failure
16:31:37 <ihrachys> http://logs.openstack.org/95/518395/2/check/neutron-tempest-plugin-dvr-multinode-scenario/da8637d/logs/testr_results.html.gz
16:31:50 <ihrachys> timeout in test_east_west
16:31:59 <ihrachys> afair there was a bug for that
16:32:23 <mlavalle> maybe https://bugs.launchpad.net/neutron/+bug/1717302
16:32:23 <openstack> Launchpad bug 1717302 in neutron "Tempest floatingip scenario tests failing on DVR Multinode setup with HA" [High,Confirmed]
16:33:32 <ihrachys> logs service just dumped on me. is it working for everyone?
16:34:01 <slaweq> not for me :/
16:34:14 <ihrachys> yeah just hanging here
16:34:25 <ihrachys> mlavalle, hard to say since I don't have logs. :)
16:34:35 <mlavalle> not responding to me either
16:34:37 <ihrachys> mlavalle, but speaking of that, do we have progress on the bug?
16:34:50 <slaweq> it works again
16:35:03 <mlavalle> I don't know, Swami was working on that one and he was in Sydney last week
16:35:32 <mlavalle> I will get update in the L3 meeting on Thursday
16:35:37 <ihrachys> ok. I also saw some weird security groups failure that seemed api layer related once but now I don't have it
16:35:44 <ihrachys> could be it was fixed or smth
16:35:59 <ihrachys> it was smth like api layer spilling an error on request validation
16:36:15 <ihrachys> slaweq, not for me
16:37:24 <ihrachys> ok there is little we can do without log server here. I will report the SG api bug if I see it again.
16:38:22 <ihrachys> there was also this bug that affected the job: https://bugs.launchpad.net/neutron/+bug/1719711
16:38:22 <openstack> Launchpad bug 1719711 in neutron "iptables failed to apply when binding a port with AGENT.debug_iptables_rules enabled" [High,Confirmed] - Assigned to Brian Haley (brian-haley)
16:38:23 <mlavalle> yeah, logs don't work yet for me either
16:39:34 <ihrachys> haleyb, was there any progress on that one?
16:39:37 <haleyb> no fix for ^^ yet btw
16:40:01 <ihrachys> do we have an idea what happens there?
16:41:02 <haleyb> not yet, don't know how they got out of sync
16:42:23 <ihrachys> ok. if you feel like you won't have time for it in next weeks, feel free to unassign so that we can find someone to pick it
16:43:23 <ihrachys> btw seems like log server is actually working but very slow
16:43:26 <ihrachys> #topic zuulv3 job migration
16:43:33 <ihrachys> mlavalle, your floor
16:44:00 <mlavalle> I got https://review.openstack.org/#/c/516715/ reviewed by yamamoto
16:44:18 <mlavalle> I will need to rebase, now that the tempest patch merged
16:44:35 <mlavalle> once I do that, I will need another +2
16:45:05 <mlavalle> and I will complete the entire process with the two related patchsets
16:45:27 <mlavalle> https://review.openstack.org/#/c/516724/
16:45:44 <mlavalle> and https://review.openstack.org/#/c/516744/
16:46:06 <mlavalle> once I complete this, I will migrate the other jobs
16:46:09 <mlavalle> left
16:46:19 <mlavalle> that's all I have for today
16:46:34 <ihrachys> mlavalle, ok. how does it fit the narrative of moving the code of tempest into the new repo?
16:46:40 <ihrachys> seems like we already have a job for api no?
16:46:44 <ihrachys> that uses the new repo
16:46:59 <mlavalle> I hadn't thought of that
16:47:02 <ihrachys> so when we switch to it as voting, we won't need the legacy flavor
16:47:28 <mlavalle> then we remove the legacy one I guess
16:47:41 <ihrachys> so it could be that moving it around is not very useful if we get rid of it in next weeks
16:47:53 <ihrachys> we could as well remove it from project-config right away
16:48:01 <ihrachys> that being said... the new jobs are for master only
16:48:09 <ihrachys> we still need legacy jobs for stable
16:48:17 <ihrachys> how does it work in terms of branches?
16:48:30 <ihrachys> do we keep both jobs in master? backport jobs to stable?
16:48:48 <mlavalle> don't know, need to think about it
16:49:56 <ihrachys> yeah. ok that's cool with me. in general, maybe fullstack or functional would be better candidates to start since we don't reshuffle them right now
16:50:19 <mlavalle> good suggestion
16:51:42 <ihrachys> #topic Open discussion
16:51:53 <ihrachys> if you have a topic, bring it on
16:52:00 <ihrachys> also, patches to peek at are welcome
16:52:48 <jlibosva> there is still this patch lacking attention https://review.openstack.org/#/c/514660/
16:52:50 <jlibosva> do we want it?
16:53:25 <ihrachys> oh right. I think it makes sense to have it.
16:53:48 <ihrachys> zuul is nuts there
16:53:51 <ihrachys> prolly to recheck
16:54:03 <ihrachys> rechecked
16:54:16 <mlavalle> yeah, I was about to say
16:54:17 <ihrachys> mlavalle, maybe you can have a look at it, it's small ^
16:54:31 <mlavalle> yeap, will review today
16:54:44 <ihrachys> I will be happy if we don't every use it in tree, it would still be helpful to have it.
16:55:16 <mlavalle> LOL, we both rechecked it
16:55:20 <ihrachys> ok. I also gonna advertise a new scenario for availability zones: https://review.openstack.org/#/c/509156/
16:56:01 <ihrachys> mlavalle, thankfully zuul is smart enough not to run it twice
16:56:04 <ihrachys> anything else?
16:56:14 <ihrachys> or you have your full 3 minutes of free time!
16:56:21 <mlavalle> \o/
16:56:24 <ihrachys> thrilled?
16:56:31 <slaweq> :)
16:56:33 <ihrachys> #endmeeting