15:00:35 <slaweq> #startmeeting neutron_ci
15:00:35 <opendevmeet> Meeting started Tue Mar  8 15:00:35 2022 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
15:00:35 <opendevmeet> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
15:00:35 <opendevmeet> The meeting name has been set to 'neutron_ci'
15:00:41 <slaweq> Grafana dashboard: https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1
15:00:41 <lajoskatona> o/
15:00:53 <slaweq> Please open and we can then start :)
15:00:53 <mlavalle1> o/
15:01:10 <mlavalle1> open
15:02:07 <slaweq> ralonsoh: bcafarel ykarel ping: CI meeting is starting
15:02:09 <ykarel> o/
15:02:15 <ralonsoh> hi
15:02:25 <bcafarel> o/ (thanks for ping)
15:02:26 <slaweq> hi all
15:02:33 <slaweq> lets start
15:02:36 <slaweq> #topic Actions from previous meetings
15:02:41 <slaweq> mlavalle will continue work on https://bugs.launchpad.net/neutron/+bug/1945283
15:03:34 <mlavalle1> didn't had much time last week
15:03:39 <mlavalle1> unfortunately
15:03:46 <mlavalle1> I'll continue this week
15:04:02 <slaweq> #action mlavalle will continue work on https://bugs.launchpad.net/neutron/+bug/1945283
15:04:05 <slaweq> ok, thx
15:04:11 <slaweq> next one
15:04:13 <slaweq> slaweq to update grafana to add fips jobs to the periodic graph
15:04:24 <slaweq> actually ykarel did it already https://review.opendev.org/c/openstack/project-config/+/831378
15:04:28 <slaweq> thx a lot ykarel
15:05:01 <slaweq> next one:
15:05:03 <slaweq> ralonsoh to investigate timeout while waiting for router to be "primary" in functional tests
15:05:20 <ralonsoh> still checking this, I don't find a good way to solve this
15:06:02 <slaweq> ralonsoh: but do You know root cause there?
15:06:28 <ralonsoh> not yet, sorry, I think this is a problem in the greenthread that sets the primary status
15:06:32 <ralonsoh> but I can't confirm that
15:07:05 <slaweq> ok, if You want I can take a look at it too
15:07:10 <ralonsoh> thanks a lot
15:07:35 <slaweq> ok, I will ping You about it tomorrow to talk about it
15:07:39 <ralonsoh> sure
15:07:43 <slaweq> #action slaweq to investigate timeout while waiting for router to be "primary" in functional tests
15:07:54 <slaweq> next one
15:07:55 <slaweq> ralonsoh to investigate mysql failure https://f7e1d46dbc2e3f05ff14-bdc442ba3e90f0570dc90e3be6cfc7bb.ssl.cf5.rackcdn.com/830622/3/check/neutron-functional-with-uwsgi/c6a67e0/testr_results.html
15:08:06 <ralonsoh> yeah, the patch is merged
15:08:19 <ralonsoh> https://review.opendev.org/c/openstack/neutron/+/831409
15:08:37 <ralonsoh> but this patch doesn't fully guarantee a test case execution isolation
15:08:40 <slaweq> thx, one issue down :)
15:10:00 <slaweq> next one
15:10:02 <slaweq> slaweq to check fullstack failure https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/830623/1/check/neutron-fullstack-with-uwsgi/366febd/testr_results.html
15:10:15 <slaweq> I started checking this but I didn't found anything really wrong there
15:10:50 <slaweq> so I will need to add some extra logs to the test itself to be able to know more about what was/wasn't configured in the fake vms and why it's not working
15:11:17 <slaweq> #action slaweq to add extra logs to fullstack tests to investigate https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_366/830623/1/check/neutron-fullstack-with-uwsgi/366febd/testr_results.html
15:11:25 <slaweq> and last one
15:11:27 <slaweq> slaweq to check notifications from neutron to nova in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_275/819147/3/check/neutron-ovs-tempest-dvr-ha-multinode-full/275f51f/testr_results.html
15:11:35 <lajoskatona> this one is this bug: https://bugs.launchpad.net/neutron/+bug/1962854 am I right?
15:11:38 <slaweq> I also took an initial look here
15:12:08 <slaweq> lajoskatona: yes
15:12:46 <slaweq> ok, getting back to the issue with notifications
15:12:55 <slaweq> I reported bug https://bugs.launchpad.net/neutron/+bug/1963899 with my initial findings
15:13:27 <slaweq> it seems for me that for some reason port during rebuild of the instance wasn't set to be DOWN and then UP but was all the time UP
15:13:51 <slaweq> and because of that notification to nova wasn't send at all and instance failed to be spawned after rebuilld
15:13:59 <ralonsoh> didn't you see the OVS agent port status updates?
15:14:09 <slaweq> but I wasn't able to reproduce that issue in any way locally, even with some extra sleeps added in some places
15:14:28 <slaweq> ralonsoh: ovs agent updated port properly
15:14:56 <slaweq> but in the neutron-server's logs I saw that "current port status ACTIVE, new status ACTIVE" thus notification wasn't send at all
15:15:03 <ralonsoh> ahhh
15:15:33 <slaweq> in case when all works fine, port is for short time set to DOWN and then to ACTIVE again
15:15:46 <slaweq> I will continue investigation on it
15:16:14 <slaweq> #action slaweq to continue investigation on nova-neutron notifications issue https://bugs.launchpad.net/neutron/+bug/1963899
15:16:53 <slaweq> ok, that's all about actions from last week
15:16:55 <slaweq> lets move on
15:16:57 <slaweq> #topic Stable branches
15:17:25 <opendevreview> yatin proposed openstack/neutron master: [DNM] Test neutron scenario tests in fips job  https://review.opendev.org/c/openstack/neutron/+/832654
15:17:56 <bcafarel> I managed to unstack most of my backports queue yesterday :) overall we are in good shape
15:18:36 <slaweq> that's good :)
15:18:39 <slaweq> thx bcafarel
15:18:44 <slaweq> #topic Stadium projects
15:18:51 <slaweq> any updates about stadium?
15:18:53 <lajoskatona> no news
15:19:09 <lajoskatona> I can add here the RC1 patches link to have it here: also:
15:19:24 <opendevreview> yatin proposed openstack/neutron master: [DNM] Test neutron scenario tests in fips job fix  https://review.opendev.org/c/openstack/neutron/+/832654
15:19:27 <lajoskatona> https://review.opendev.org/q/topic:yoga-rc1-deadline+is:open+reviewer:katonalala%2540gmail.com
15:19:36 <lajoskatona> I have to check these also
15:19:52 <lajoskatona> that's all for stadium from me
15:20:49 <slaweq> thx lajoskatona
15:20:59 <slaweq> #topic Grafana
15:21:04 <slaweq> https://grafana.opendev.org/d/f913631585/neutron-failure-rate
15:21:13 <opendevreview> yatin proposed openstack/neutron master: [DNM] Test neutron scenario tests in fips job  https://review.opendev.org/c/openstack/neutron/+/832656
15:21:28 <slaweq> I'm not sure why but we have a lot of gaps there recently
15:21:36 <slaweq> but this is probably some infra issue
15:22:21 <slaweq> because e.g. Octavia has got similar gaps in their dashboard
15:22:26 <slaweq> https://grafana.opendev.org/d/584dacaabb/octavia-failure-rate?orgId=1
15:22:58 <slaweq> But, looking at number of rechecks which we need to do to merge patch:
15:23:07 <slaweq> +---------+----------+... (full message at https://matrix.org/_matrix/media/r0/download/matrix.org/pvFbIpnoRbffcyasJFDqJDIz)
15:23:19 <slaweq> it looks still ok as it's below 1 in average
15:23:23 <ralonsoh> cool
15:23:35 <lajoskatona> looks quite good really
15:23:54 <ykarel> nice
15:24:21 <mlavalle1> really nice
15:24:43 <slaweq> looking both: at grafana and doing reviews and checking patches, it seems that we have one, pretty unstable job recently
15:24:47 <slaweq> it's functional job
15:25:02 <slaweq> https://grafana.opendev.org/d/f913631585/neutron-failure-rate?orgId=1&viewPanel=20
15:25:03 <slaweq> other than that it seems really ok IMO
15:25:39 <opendevreview> Elvira García Ruiz proposed openstack/neutron-specs master: Add spec for DNS subdomain support in OVN  https://review.opendev.org/c/openstack/neutron-specs/+/832658
15:26:12 <slaweq> so lets talk about functional job's issues now
15:26:17 <slaweq> #topic fullstack/functional
15:26:30 <slaweq> first one: neutron.tests.functional.agent.test_dhcp_agent.DHCPAgentOVSTestCase Timedout out in "common_utils.wait_until_true(predicate, 10)"
15:26:35 <slaweq> https://d2906f583f75630318f3-beac4b5f1c13ffc5e76ddbf8e8dde130.ssl.cf1.rackcdn.com/830666/5/gate/neutron-functional-with-uwsgi/f92faca/testr_results.html
15:28:36 <ralonsoh> I can take a look at this one this week
15:28:43 <slaweq> in this case I don't see anything obviously wrong
15:29:10 <ralonsoh> how many ocurrences if this test case did you see?
15:29:34 <slaweq> for now it was only once
15:30:05 <opendevreview> liuyulong proposed openstack/neutron-specs master: Spec for distributed datapath for metadata  https://review.opendev.org/c/openstack/neutron-specs/+/832660
15:30:25 <ykarel> also logstash no longer working to check no. of similar failures, right?
15:30:59 <opendevreview> liuyulong proposed openstack/neutron-specs master: Spec for Router Advertisement on Openvswitch Agent  https://review.opendev.org/c/openstack/neutron-specs/+/832660
15:31:05 <slaweq> ykarel: I think so
15:31:24 <slaweq> I know that dpawlik was/is working on logstash or some replacement for it
15:31:31 <slaweq> but I don't know about current state of it
15:31:32 <slaweq> I can ask him
15:31:43 <ykarel> Thanks slaweq for confirming
15:31:48 <slaweq> #action slaweq to ask dpawlik about logstash state
15:31:58 <ykarel> hmm good to know about it as it help to get some matrix
15:32:55 <slaweq> next one
15:33:00 <slaweq> Again timeouts while waiting for router to be switched to primary:
15:33:04 <slaweq> https://zuul.openstack.org/build/b2b1e2c0027c4405abdda540ade0879e
15:33:04 <slaweq> https://zuul.openstack.org/build/e5ce944520584f00815505554522fdc0
15:33:45 <ralonsoh> I'll keep working on this
15:33:56 <slaweq> I will also look at those this week
15:34:05 <slaweq> and last one:
15:34:07 <slaweq> Error with missing namespace
15:34:30 <slaweq> https://zuul.openstack.org/build/31bd6bca81f34c6c9af385daf6f64ee8
15:34:30 <slaweq> https://zuul.opendev.org/t/openstack/build/f8510ec1f1ca4329a5bb28fb1a38614c
15:34:34 <slaweq> I saw it at least twice
15:34:39 <slaweq> so I will open bug for it
15:34:45 <slaweq> #action slaweq to report bug about missing namespace in functional tests
15:35:50 <slaweq> and if anyone have some cycles, please take a look into it
15:36:06 <lajoskatona> +1
15:36:18 <slaweq> and that's basically all issues which I had for today for functional/fullstack jobs
15:36:22 <slaweq> any questions/comments?
15:37:31 <bcafarel> sounds like a no :)
15:37:43 <mlavalle1> not from me
15:37:58 <slaweq> ok
15:38:06 <lajoskatona> nothing from me
15:38:10 <slaweq> so last, quick topic for today
15:38:12 <slaweq> #topic Periodic
15:38:36 <slaweq> here I noticed that our functional and fullstack jobs with FIPS enabled are failing
15:38:36 <slaweq> every day
15:38:48 <opendevreview> Elvira García Ruiz proposed openstack/neutron-specs master: Add spec for DNS subdomain support in OVN  https://review.opendev.org/c/openstack/neutron-specs/+/832658
15:38:51 <slaweq> I'm pretty sure they were green when we had them in experimental queue
15:39:00 <slaweq> so I will check what is wrong there now
15:39:16 <slaweq> #action slaweq to check functional and fullstack FIPS jobs from periodic queue
15:39:18 <bcafarel> pretty sure they were passing, and I think also recently in xena backports no,
15:39:20 <bcafarel> ?
15:39:28 <slaweq> bcafarel: yes
15:39:59 <slaweq> ok, that's basically all what I had for today
15:40:08 <slaweq> anything else You want to discuss today?
15:40:17 <slaweq> if not, I will give You some time back today
15:40:18 <ralonsoh> yes
15:40:20 <ralonsoh> https://bugs.launchpad.net/oslo.db/+bug/1963679
15:40:29 <ralonsoh> and https://review.opendev.org/c/openstack/neutron-lib/+/828738
15:40:36 <ralonsoh> I'm stuck on this
15:40:53 <ralonsoh> I don't know what should we do to migrate to sqlalchemy 2.0
15:41:06 <ralonsoh> I still need to process https://bugs.launchpad.net/oslo.db/+bug/1963679/comments/1
15:41:12 <ralonsoh> just a heads-up
15:41:39 <ralonsoh> (maybe I should have raised this topic in the previous meeting)
15:42:00 <slaweq> ralonsoh: yeah, previous meeting was better for this IMO
15:42:06 <slaweq> or drivers meeting on Friday maybe
15:42:06 <ralonsoh> sorry
15:42:12 <ralonsoh> ok then
15:42:12 <slaweq> no problem :)
15:42:31 <lajoskatona> ralonsoh: thanks for checking this, we can discuss on Friday
15:42:34 <mlavalle1> I'll take a look at that patch anyways
15:43:34 <slaweq> ok, so with that I think we can finish for today
15:43:51 <slaweq> thx a lot for attending the meeting
15:43:55 <slaweq> and have a great week
15:43:56 <bcafarel> o/
15:43:58 <ralonsoh> bye
15:43:59 <slaweq> #endmeeting