15:01:22 <slaweq> #startmeeting neutron_ci
15:01:23 <openstack> Meeting started Wed Jun 17 15:01:22 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:01:24 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:01:25 <slaweq> hi
15:01:27 <openstack> The meeting name has been set to 'neutron_ci'
15:01:28 <lajoskatona> o/
15:01:34 <ralonsoh> hi
15:01:54 <bcafarel> yeah o/
15:02:53 <slaweq> ok, let's start
15:03:10 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
15:03:22 <slaweq> please open link and we can move on
15:03:24 <slaweq> #topic Actions from previous meetings
15:03:36 <njohnston> 0o/
15:03:40 <maciejjozefczyk> \o
15:03:57 <slaweq> first action from last meeting
15:04:01 <slaweq> slaweq to check failure in test_ha_router_failover: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6d0/726168/2/check/neutron-functional/6d0b174/testr_results.html
15:04:09 <slaweq> I didn't saw this issue any time later so I didn't check it. I will if it will start failing more often
15:04:34 <slaweq> or maybe You saw it more often and it's just me who missed that
15:04:40 <slaweq> so please tell me :)
15:04:47 <slaweq> 641590
15:04:57 <slaweq> ^^ sorry :)
15:05:34 <bcafarel> nice score, I will not try to compete :)
15:06:07 <slaweq> :)
15:07:24 <slaweq> ok, so next one
15:07:26 <slaweq> slaweq to add additional logging for fullstack's firewall tests
15:07:31 <slaweq> For now I just did DNM patch https://review.opendev.org/#/c/736034/1 to see exactly when this test is failing and then I will add some additional logs to know more what's going on there.
15:07:43 <slaweq> so I will assign it to myself for next week too
15:07:46 <slaweq> #action slaweq to add additional logging for fullstack's firewall tests
15:07:57 <slaweq> and the last one
15:07:59 <slaweq> maciejjozefczyk to check failing test_port_security_macspoofing_port test
15:09:55 * maciejjozefczyk looking for the link :)
15:10:35 <maciejjozefczyk> #link https://bugs.launchpad.net/tempest/+bug/1728886
15:10:35 <openstack> Launchpad bug 1728886 in tripleo "test_port_security_macspoofing_port doesn't work when l2pop and arpresponding are enabled" [Critical,Fix released]
15:11:30 <slaweq> maciejjozefczyk: so it's fixed and test is passing now, right?
15:12:10 <maciejjozefczyk> slaweq, the problem was in core-ovn and it was fixed there
15:12:24 <maciejjozefczyk> in some stable releases I needed to blacklist it, because those are not using the newest ovn
15:12:46 <slaweq> ok
15:12:49 <slaweq> thx maciejjozefczyk
15:13:54 <slaweq> ok, lets move on
15:13:56 <slaweq> #topic Stadium projects
15:14:01 <slaweq> standardize on zuul v3
15:14:06 <slaweq> Etherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop
15:14:12 <slaweq> there are only 2 missing things there
15:14:15 <slaweq> networking-midonet
15:14:21 <slaweq> and ovn grenade job
15:14:32 <slaweq> for ovn grenade job I have it in my backlog
15:15:12 <slaweq> and for networking-midonet, I asked project core reviewers this week, and they sadly agreed that we should probably deprecate this project as we did with fwaas
15:15:28 <slaweq> so IMO we can cross it out from this list here
15:15:48 <njohnston> +1
15:16:08 <bcafarel> time has come then
15:16:23 <ralonsoh> +1
15:16:29 <slaweq> ok
15:16:34 <bcafarel> for ovn grenade job is it still working? neutron grenade has issues in stable branches (where it is still legacy atm)
15:16:54 <bcafarel> with virtualenv/pip removal from base image (and also our friend uwsgi)
15:17:03 <slaweq> bcafarel: currently we are not running this ovn grenade job so I don't know if it's really working or not
15:17:10 <bcafarel> :)
15:17:17 <slaweq> I will need to make it working and migrate to zuulv3
15:18:16 <slaweq> and we have new topic which also needs to be done in stadium projects
15:18:22 <slaweq> "Switch to Ubuntu Focal"
15:19:13 <slaweq> is there any volunteer to drive this?
15:19:56 <njohnston> I started it but I think with my other focuses someone else might be a good candidate to push forward
15:20:19 <njohnston> with the zuul v3 job almost done I think our job inheritance will do almost all of this
15:20:30 <njohnston> as the base jobs are modified
15:20:45 <njohnston> but the process of debugging any changes is unknown
15:20:52 <njohnston> s/changes/issues/
15:21:02 <slaweq> njohnston: yes, I think it shouldn't be a lot of work really
15:21:27 <bcafarel> I can help it will be a change from fixing xenial jobs :)
15:21:40 <slaweq> bcafarel: :D
15:21:42 <slaweq> thx a lot
15:22:00 <njohnston> looks like the change I did for fullstack/functional is breaking on the step of compiling OVN
15:22:00 <njohnston> https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_12b/734304/3/check/neutron-functional/12b22c0/job-output.txt
15:22:14 <njohnston> that change is https://review.opendev.org/#/c/734304/
15:22:48 <maciejjozefczyk> njohnston, with moving to Focal we can stop compiling the modules
15:23:04 <njohnston> maciejjozefczyk: Perfect, that was my hope
15:23:06 <maciejjozefczyk> njohnston, cause the kernel in Focal has OVS metric actions that we need :)
15:23:11 <slaweq> maciejjozefczyk: unless we will have another feature which we will need there :P
15:23:13 <maciejjozefczyk> meter*
15:23:20 <maciejjozefczyk> slaweq, unless :D
15:23:31 <slaweq> but for now it should be good way to go probably
15:23:46 <bcafarel> one step at a time yep
15:24:05 <njohnston> So probably the first step is to disable the module compilation if focal is detected?
15:24:08 <maciejjozefczyk> wait, the error in njohnston link is about "ipv6_dst_lookup"
15:24:24 <maciejjozefczyk> so
15:24:25 <maciejjozefczyk> #link https://review.opendev.org/#/c/735799/
15:24:27 <maciejjozefczyk> should fix it
15:24:45 <slaweq> maciejjozefczyk: I see other error in https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_12b/734304/3/check/neutron-functional/12b22c0/job-output.txt
15:24:57 <slaweq> RuntimeError: dictionary keys changed during iteration
15:24:59 <bcafarel> RuntimeError: dictionary keys changed during iteration
15:25:12 <slaweq> first! \o/ :P
15:25:19 <njohnston> right, that is in the compile_ovs action
15:25:20 <njohnston> http://paste.openstack.org/show/794904/
15:25:31 <bcafarel> slaweq: that's why you are PTL :)
15:25:32 <njohnston> the dictionary keys error, I mean
15:25:43 <maciejjozefczyk> yhmmm, allright
15:25:48 <slaweq> bcafarel: I hope it's not the only reason ;P
15:26:01 <ralonsoh> and njohnston's question about disabling module compilation?
15:26:10 <ralonsoh> is there a patch for this?
15:26:18 <bcafarel> not yet I think
15:26:23 <njohnston> no, I think that would be a new action item
15:26:28 <ralonsoh> ok, I'll propose one
15:26:29 <maciejjozefczyk> we need to start discussion again if we want to comiple ovs/ovn at all, that seems to be outside module (error is related to ovsdb)
15:26:30 <bcafarel> first focus was on https://review.opendev.org/#/c/735799/
15:26:53 <ralonsoh> we can also add a knob for this, apart from checking the version
15:27:18 <njohnston> +1
15:28:01 <ralonsoh> (ok we have OVN_BUILD_MODULES)
15:28:36 <slaweq> njohnston: I think You can simply add new variable to https://github.com/openstack/neutron/blob/master/roles/configure_functional_tests/tasks/main.yaml#L13
15:29:00 <slaweq> and then set it in the job's definition to not compile ovs from source
15:29:28 <slaweq> the variable in script is here https://github.com/openstack/neutron/blob/master/tools/configure_for_func_testing.sh#L65
15:29:42 <lajoskatona> +1 if possible get rid of depenency to out of tree error sources
15:29:49 <maciejjozefczyk> slaweq, I can work on that part... cause I started doing something similar in: https://review.opendev.org/#/c/727193/
15:29:52 <njohnston> slaweq: sounds good, I will iterate on that after the meeting
15:30:01 * njohnston yields to maciejjozefczyk
15:30:02 <slaweq> thx maciejjozefczyk and njohnston
15:30:57 <maciejjozefczyk> actually we need to make a cleanup around compile_ovs functions, because we have this one both in devstack ovn lib and devstack ovs lib ...
15:31:23 <slaweq> maciejjozefczyk: yes, so we need cleanup
15:31:32 <slaweq> will You take care of this?
15:31:34 <maciejjozefczyk> yes
15:32:00 <slaweq> thx a lot
15:32:33 <slaweq> ok, and bcafarel will You prepare some list of the other jobs which we need to switch to Focal?
15:33:07 <bcafarel> slaweq: that's the plan yep!
15:33:12 <slaweq> thx a lot
15:33:21 <slaweq> so we should be good with that topic :)
15:33:31 <slaweq> so next one
15:33:33 <slaweq> #topic Stable branches
15:33:39 <slaweq> Ussuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1
15:33:41 <slaweq> Train dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1
15:33:53 <slaweq> we are still broken with this uwsgi issue in stable branches, right?
15:34:29 * bcafarel looks for links
15:34:42 <bcafarel> uwsgi should be good in recent branches now though there are other issues
15:34:53 <slaweq> ouch
15:35:31 <bcafarel> #link https://review.opendev.org/#/c/735523/
15:35:46 <bcafarel> uwsgi should be fixed up to stein
15:36:09 <slaweq> good :)
15:36:25 <maciejjozefczyk> ++
15:36:57 <bcafarel> then we need https://review.opendev.org/#/c/735948/1 (grenade job) + ovs_version fix to backport
15:39:09 <slaweq> by ovs_version You mean the patch from jlibosva https://review.opendev.org/#/c/735799/ right?
15:39:38 <bcafarel> sorry, laptop issue
15:39:43 <bcafarel> slaweq: that's the one yes
15:40:16 <slaweq> hmm, but will we be able to merge them? as one requires another
15:40:27 <slaweq> so we are in deadlock in ussuri, aren't we?
15:40:36 <ralonsoh> I think so
15:40:38 <ralonsoh> we need both
15:40:57 <bcafarel> good point, yes they will most probably need to be squashed
15:41:16 <slaweq> so we can combine them too in ussuri, or (better IMO) propose patch to set all those jobs non-voting in ussuri
15:41:25 <slaweq> then merge both patches and revert first one
15:42:43 <slaweq> what do You think?
15:43:29 <ralonsoh> +1 if we only merge the ovs patch with those jobs stopped
15:44:00 <ralonsoh> (well, and the zuul migration one)
15:44:28 <bcafarel> sounds good, each fix patch can (and should) re-enable relevant jobs
15:44:53 <slaweq> ok, bcafarel will You propose patch to make those jobs non-voting temporary or do You want me to do that?
15:46:04 <bcafarel> slaweq: I have the relevant tab already open I can send it (neutron-grenade-multinode* and ovn jobs)
15:46:12 <slaweq> thx bcafarel
15:46:43 <slaweq> I think that should be all related to the stable branches, right?
15:46:51 <slaweq> or do You have anything else?
15:47:40 <bcafarel> older branches will need some fixes too, but one branch at a time!
15:47:47 <slaweq> bcafarel: ok :)
15:48:10 <slaweq> lets move on to the next topic as we are late today :)
15:48:12 <slaweq> #topic Grafana
15:48:49 <slaweq> as we already discussed, there are 2 main issues which are causing many jobs failing: uwsgi problem and ovs compilation problem
15:49:13 <slaweq> uwsgi should be fixed in master and fix for  ovs issue is in the gate already
15:49:51 <slaweq> there was also problem with designate job but it is also going down with failure rate todayu
15:50:02 <slaweq> so seems like it's fixed already
15:50:27 <bcafarel> yes frickler was commenting about it earlier in #openstack-neutron it should be good
15:51:03 <slaweq> do You have anything else regarding grafana and our dashboard?
15:52:22 <slaweq> ok, so lets move on
15:52:25 <slaweq> next topic
15:52:32 <slaweq> #topic fullstack/functional
15:52:44 <slaweq> I found only one new issue in fullstack job for today
15:52:49 <slaweq> Ovs compilation error: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_410/735799/1/check/neutron-fullstack/410977b/job-output.txt
15:53:04 <slaweq> but this one seems different than those which we discussed earlier already
15:53:11 <slaweq> maciejjozefczyk: can You take a look at it?
15:53:26 <slaweq> it's not happening all the time for sure
15:53:58 <maciejjozefczyk> slaweq, yes
15:54:22 <slaweq> ahh, wait
15:54:29 <slaweq> it was on https://review.opendev.org/#/c/735799/
15:54:29 <maciejjozefczyk> bcafarel, found the same in https://review.opendev.org/#/c/735799/
15:54:50 <maciejjozefczyk> yeah... I'll take a look tomorrow morning on it
15:54:54 <slaweq> thx a lot
15:55:08 <bcafarel> sigh so this is reproducible then
15:55:15 <maciejjozefczyk> ;/
15:55:18 <slaweq> bcafarel: no, it's on the same patch
15:55:25 <slaweq> :)
15:55:36 <bcafarel> oh true
15:55:37 <slaweq> I found it also in same job run
15:55:57 <slaweq> ok, lets move on quickly to the last topic for today
15:56:02 <slaweq> #topic Tempest/Scenario
15:56:11 <slaweq> first of all, I sent today 2 patches:
15:56:16 <slaweq> https://review.opendev.org/736177
15:56:18 <slaweq> https://review.opendev.org/736186
15:56:26 <slaweq> both are follow-ups from the PTG discussion
15:56:46 <slaweq> first one is move of not stable dvr multinode job to experimental queue
15:56:58 <maciejjozefczyk> the problem should be solved by https://github.com/openvswitch/ovs/commit/0c4d144a989a444d038d58272d8571e97e00e86f
15:57:08 <slaweq> and the second one is to remove tempest-integrated-networking job with multinode job
15:57:08 <maciejjozefczyk> so maybe we can use this tag instead the one proposed by Kuba
15:57:16 <slaweq> maciejjozefczyk++
15:57:16 <maciejjozefczyk> (oh sorry, too late) :D
15:57:38 <maciejjozefczyk> slaweq, yeah, sorry for disturbing, I'll update the patch
15:57:46 <slaweq> maciejjozefczyk: no problem at all
15:57:52 <slaweq> thx for looking into that
15:58:22 <ralonsoh> +1 to those two patches
15:58:27 <slaweq> ok, and that's all what I have for this week
15:58:29 <slaweq> thx ralonsoh
15:58:47 <slaweq> anything else You want to raise here quickly in last minute?
15:59:36 <slaweq> ok, if not then thx for attending the meeting
15:59:41 <slaweq> have a great evening
15:59:43 <slaweq> o/
15:59:44 <ralonsoh> bye
15:59:46 <slaweq> #endmeeting