16:00:12 <slaweq> #startmeeting neutron_ci
16:00:13 <openstack> Meeting started Tue Nov 19 16:00:12 2019 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:14 <openstack> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:16 <openstack> The meeting name has been set to 'neutron_ci'
16:00:16 <slaweq> hi (again)
16:00:17 <njohnston> o/
16:00:25 <ralonsoh> hi
16:01:21 <slaweq> ok, lets start
16:01:30 <slaweq> first of all
16:01:31 <slaweq> Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:01:46 <slaweq> please open it and we will move on
16:01:48 <slaweq> #topic Actions from previous meetings
16:01:55 <slaweq> slaweq to investigate failed neutron.tests.fullstack.test_qos.TestDscpMarkingQoSOvs
16:02:06 <slaweq> Bug reported https://bugs.launchpad.net/neutron/+bug/1852724
16:02:06 <openstack> Launchpad bug 1852724 in neutron "Fullstack test for dscp_marking_packets fails if first icmp is not send properly" [Medium,Fix released] - Assigned to Slawek Kaplonski (slaweq)
16:02:07 <slaweq> Patch proposed: https://review.opendev.org/694505
16:02:35 <slaweq> it's even merged already
16:02:46 <njohnston> nice
16:03:02 <slaweq> so next one
16:03:04 <slaweq> njohnston prepare etherpad to track stadium progress for zuul v3 job definition and py2 support drop
16:03:44 <njohnston> yeah, I did not get to that, I'll work on it today
16:03:50 <slaweq> ok
16:03:55 <njohnston> somehow I missed it in my todo list
16:03:59 <slaweq> so, just as a reminder
16:04:05 <slaweq> #action njohnston prepare etherpad to track stadium progress for zuul v3 job definition and py2 support drop
16:04:08 <slaweq> :)
16:04:19 <slaweq> and we will get back to this in next meeting
16:04:29 <slaweq> next one
16:04:31 <slaweq> slaweq to take a look at connectivity issues after resize/migration
16:04:41 <slaweq> I did some analysis and described it in comment https://bugs.launchpad.net/neutron/+bug/1850557
16:04:41 <openstack> Launchpad bug 1850557 in neutron "DHCP connectivity after migration/resize not working" [Medium,Confirmed] - Assigned to Slawek Kaplonski (slaweq)
16:04:48 <slaweq> but I don't know exactly why it happens like that
16:05:21 <slaweq> as it happend at least few times in tempest-slow job, which runs tests in serial, I will try to reproduce this issue locally
16:05:59 <slaweq> but tbh I didn't saw it this week in ci results
16:07:12 <slaweq> ok, next one
16:07:18 <slaweq> njohnston delete old tests from neutron-dynamic-routing repo
16:07:51 <njohnston> https://review.opendev.org/695014
16:08:10 <njohnston> up for review
16:08:21 <slaweq> thx njohnston
16:08:32 <slaweq> I will review it as soon as zuul will be happy :)
16:09:06 <slaweq> ok, next one
16:09:08 <slaweq> slaweq to move neutron-tempest-with-os-ken-master to zuulv3 syntax and switch to run neutron related tests only
16:09:15 <slaweq> Patch https://review.opendev.org/694770
16:09:21 <slaweq> ready for review :)
16:09:26 <bcafarel> late o/
16:09:35 <slaweq> hi bcafarel :)
16:09:53 <slaweq> and this is in fact last "non-grenade" legacy job in neutron repo IIRC
16:10:08 <njohnston> nice
16:10:31 <slaweq> next one was
16:10:33 <slaweq> slaweq to fix python version in tempest-slow-py3 job
16:10:42 <slaweq> Bug reported: https://bugs.launchpad.net/tempest/+bug/1853004
16:10:42 <openstack> Launchpad bug 1853004 in tempest "tempest-slow-py3 job uses python 2.7 on subnodes" [Undecided,In progress] - Assigned to Slawek Kaplonski (slaweq)
16:10:48 <slaweq> Patch https://review.opendev.org/694768
16:11:03 <slaweq> but I found out today that this is not only related to tempest-slow-py3
16:11:14 <slaweq> also tempest-multinode-full-py3 has this issue
16:11:24 <slaweq> and it is also addressed by patch 694768
16:11:42 <slaweq> but also neutron-tempest-plugin-multinode-dvr job has got same issue
16:11:48 <slaweq> and patch to fix it is here:
16:12:09 <slaweq> https://review.opendev.org/#/c/695013/
16:12:19 <slaweq> so please review :)
16:12:56 <slaweq> ok, and the last one from last week is
16:12:57 <slaweq> njohnston to check failing NetworkMigrationFromHA in multinode dvr job
16:13:01 * bcafarel tries to keep up with the flurry of review links
16:13:08 <njohnston> I did not have a chance to look at that
16:13:52 <slaweq> will You have time to look at it this week maybe?
16:14:33 <njohnston> I will do my best
16:14:39 <slaweq> thx njohnston :)
16:14:48 <slaweq> I know You will
16:14:54 <slaweq> #action njohnston to check failing NetworkMigrationFromHA in multinode dvr job
16:15:14 <slaweq> that was all actions from last week
16:15:26 <slaweq> anything else You want to add/ask maybe?
16:16:19 <njohnston> nope
16:16:27 <slaweq> ok, so lets move on
16:16:30 <slaweq> #topic Stadium projects
16:16:50 <slaweq> we already talked about neutron-dynamic-routing patch
16:17:02 <slaweq> and I don't think we have anything else to talk about here today
16:17:25 <njohnston> agreed
16:17:27 <slaweq> in next weeks we can track here progress of migrating jobs to zuulv3 and dropping py2 jobs
16:17:35 <slaweq> but for now I don't have anything else
16:18:06 <slaweq> but maybe You have something to talk about here?
16:18:23 <lajoskatona> Hi, I just realized that CI meeting is in progress
16:18:29 <slaweq> hi lajoskatona :)
16:18:32 <njohnston> hello lajoskatona
16:18:39 <lajoskatona> I have one networkingüodl small bug: https://review.opendev.org/668904
16:19:04 <lajoskatona> It is a unit test that fails sporadically
16:19:36 <lajoskatona> If you have some time to check it that would be helpful, sorry for adding new things to the piles
16:20:16 <ralonsoh> sorry, in this patch I don't see the UTs failing
16:20:41 <slaweq> ralonsoh: I think that this patch fixes failing UT
16:20:42 <slaweq> :)
16:20:54 <ralonsoh> I need to read better.....
16:21:31 <lajoskatona> ralonsoh: yes, soryy for bad wording, the ut that fails is networking-odl: test_feature_configs_does_not_mutate_default_features
16:21:44 <lajoskatona> ralonsoh: thanks anyway
16:21:44 <ralonsoh> thanks!
16:21:53 <slaweq> lajoskatona: I'm not odl expert but patch seems reasonable for me
16:22:19 <bcafarel> one-liner to add init/reset is reasonable yeah
16:22:36 <lajoskatona> slaweq: yeah the good thing is that this is in python, not in Java at least :-)
16:23:02 <slaweq> lajoskatona: true :)
16:23:32 <slaweq> lajoskatona: I just +W your patch
16:23:59 <lajoskatona> slaweq: thanks, a lot less recheck from this day
16:24:10 <slaweq> :)
16:24:15 <slaweq> ok, lets move on
16:24:17 <slaweq> #topic Grafana
16:24:25 <slaweq> #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate
16:24:47 <slaweq> as You may see there, empest-slow-py3 and tempest-multinode-full-py3 high failure rate today
16:24:49 <slaweq> same for neutron-grenade
16:25:10 <slaweq> and all that issues are caused by nova's patch https://review.opendev.org/#/c/687954/
16:25:39 <slaweq> nova went too far with dropping py2 support
16:26:01 <slaweq> I know they were talking today how to fix this for now
16:26:07 <slaweq> so hopefully it will be fine soon
16:26:51 <njohnston> hopefully
16:27:01 <slaweq> yep
16:27:02 <lajoskatona> slaweq: so let's wait with rechecks?
16:27:10 <slaweq> lajoskatona: for now, yes
16:27:14 <lajoskatona> slaweq: ok
16:27:36 <slaweq> for sure neutron-grenade job, tempest-slow-py3 and tempest-multinode-full-py3 will fail
16:27:58 <slaweq> tempest-slow-py3 and tempest-multinode-full-py3 should be fixed when my patch to switch it "fully" to py3 will be merged
16:28:13 <slaweq> and for neutron-grenade we have already patch to remove it from neutron queue
16:28:35 <slaweq> so in fact if all those patches will be merged, we should be good even without fix on nova's side
16:29:06 <njohnston> if they need any more reviews, can you post the URLs?
16:29:07 <ralonsoh> btw, with one of those patches (devstack) USE_PYTHON3 will be True by default
16:29:08 <ralonsoh> https://review.opendev.org/#/c/649097/10/stackrc
16:29:49 <njohnston> excellent
16:30:07 <slaweq> njohnston: thx, I already sent links to those patches before
16:30:16 <njohnston> ah ok
16:30:18 <njohnston> nm
16:30:22 <slaweq> :)
16:31:18 <slaweq> other than that I think we are fine
16:31:33 <njohnston> do you think we will need to squash the changes in order to get them through the gate?  Or should we wait for Nova to clean up?
16:31:46 <slaweq> njohnston: no, we don't need to squash anything
16:32:08 <njohnston> ok
16:32:09 <slaweq> as fix for tempest-slow-py3 and tempest-multinode-full-py3 is in tempest repo: https://review.opendev.org/694768
16:32:15 <njohnston> ah
16:33:09 <slaweq> and remove of neutron-grenade is in neutron repo: https://review.opendev.org/#/c/694039/
16:33:34 <slaweq> so first patch in tempest should land and than our patch in neutron will be able to land
16:33:42 <slaweq> and than we should be good probably
16:34:02 <slaweq> but I'm not sure which version will be faster, that one or fix on nova's side
16:34:50 <slaweq> ok, I think we can move on
16:35:10 <slaweq> basically this issue is biggest issue for now, so I don't have prepared any other issues for today
16:35:22 <slaweq> but I want to talk a bit about scenario jobs
16:35:28 <slaweq> #topic Tempest/Scenario
16:36:00 <slaweq> few days ago I sent email http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010838.html
16:36:13 <slaweq> with some comparison of single/multinode jobs
16:36:29 <slaweq> and also with comparison of tempest- and neutron-tempest-plugin- jobs
16:36:59 <slaweq> and I would like to discuss here a bit about what do You think should we do with those jobs?
16:37:36 <ralonsoh> first, what are we going to do with LB?
16:37:57 <ralonsoh> because the future (or not) of LB could decide about those jobs
16:38:02 <slaweq> some job with LB should be still there
16:38:23 <ralonsoh> if we remove Lb too?
16:38:34 <slaweq> ralonsoh: for now we will not remove it for sure
16:38:50 <slaweq> we are starting some discussion about deprecating it
16:38:55 <ralonsoh> I know
16:38:56 <slaweq> but nothing else for now
16:39:08 <slaweq> and for now we have 2 jobs: neutron-tempest-plugin-scenario-linuxbridge and neutron-tempest-linuxbridge
16:39:23 <slaweq> maybe we could merge them into one job?
16:39:23 <ralonsoh> then we should keep it
16:39:42 <ralonsoh> we are doing different things there, but could work
16:40:14 <ralonsoh> I'll check this
16:40:46 <slaweq> ralonsoh: what do You mean by "different things"?
16:41:08 <ralonsoh> the tests are different
16:41:34 <slaweq> yes, tests are different
16:41:56 <slaweq> that's why I think that maybe we can run one job which will run tests from both tempest and than neutron-tempest-plugin repo
16:42:20 <ralonsoh> that could speed up the CI
16:42:35 <slaweq> yes
16:42:45 <slaweq> and would be one less job which can fail maybe :)
16:43:26 <slaweq> but the second question is: do we really need to run all tempest tests for it?
16:43:42 <slaweq> as it is single node job, it don't test things like live-migration for example
16:44:34 <slaweq> so basically it runs a lot of nova related tests and neutron related tests
16:44:47 <slaweq> neutron related things we can cover in neutron-tempest-plugin job as well
16:45:27 <slaweq> and from nova point of view - what we really need to test, if it's only if vm can be spawned and ssh than we have this covered in neutron-tempest-plugin tests also
16:45:43 <slaweq> so maybe we could stay only with neutron-tempest-plugin-scenario-linuxbridge job?
16:46:05 <slaweq> and exactly same situation is for neutron-tempest-iptables_hybrid
16:46:18 <ralonsoh> (I would need to check both jobs and the tests executed)
16:46:42 <slaweq> ralonsoh: sure, please check them and maybe reply to my email on ML
16:46:47 <ralonsoh> slaweq,
16:46:49 <ralonsoh> ok
16:46:50 <slaweq> we can continue this discussion there
16:46:52 <slaweq> thx a lot
16:47:12 <slaweq> next thing are grenade jobs
16:47:29 <slaweq> we have grenade-py3 and neutron-grenade-multinode in check and gate queue
16:47:38 <slaweq> those jobs are the same
16:47:50 <slaweq> only difference between them is that grenade-py3 is single node job
16:48:07 <slaweq> can we maybe drop this one? and use only neutron-grenade-multinode job?
16:48:20 <ralonsoh> single node is not used in production
16:48:21 <ralonsoh> so yes
16:48:29 <bcafarel> and I suppose we don't have any tests that *requires* single node :)
16:48:37 <bcafarel> (so also yes)
16:48:40 <slaweq> bcafarel: exactly :)
16:48:58 <slaweq> ok, so as I see agreement here, I will propose patch to drop single node job
16:49:00 <slaweq> thx
16:49:39 <slaweq> that's all from me about this topic
16:50:05 <slaweq> if You would have any other opinions/ideas about it, please reply to the email and we can continue this discussion there
16:50:13 <slaweq> from other things
16:50:30 <njohnston> ok
16:50:50 <slaweq> I sent patch https://review.opendev.org/#/c/694049/ which will start running queens jobs using neutron-tempest-plugin with some specific tag
16:51:10 <slaweq> so we can drop those jobs from master branch queues
16:51:15 <slaweq> please review it also :)
16:51:45 <bcafarel> ooh nice
16:52:05 <slaweq> and one last thing which I want to mention
16:52:21 <slaweq> I have patch to switch default deployment to uwsgi https://review.opendev.org/#/c/694042/
16:52:30 <slaweq> but it has some problems with grenade jobs
16:52:40 <slaweq> so I will investigate that and I hope it will be ready soon
16:52:46 <njohnston> cool
16:53:36 <ralonsoh> slaweq, https://review.opendev.org/#/c/694266/
16:53:46 <ralonsoh> which one is failing?
16:54:07 <ralonsoh> or you mean in https://review.opendev.org/#/c/694042/
16:54:21 <njohnston> the second one
16:54:33 <ralonsoh> perfect
16:54:46 <slaweq> ralonsoh: yes, it's failing in https://review.opendev.org/#/c/694042/
16:55:25 <slaweq> but I think I know why it is failing
16:55:34 <slaweq> I will just need to test that in gate :)
16:55:49 <slaweq> ok, that's all from me for today
16:55:58 <slaweq> anything else You want to talk about today?
16:56:33 <bcafarel> sounds like all action points from ptg have patches in progress then?
16:56:41 <bcafarel> nice work :)
16:56:41 <slaweq> bcafarel: I hope so
16:56:51 <slaweq> but if I missed anything, forgive me :)
16:57:04 <ralonsoh> no mercy
16:57:08 <slaweq> lol
16:57:26 <slaweq> ok, we are almost on time
16:57:33 <slaweq> thx for attending the meeting guys
16:57:40 <slaweq> and see You online o/
16:57:42 <njohnston> thanks!
16:57:43 <bcafarel> o/
16:57:43 <slaweq> #endmeeting