Wednesday, 2020-03-11

*** yamamoto has joined #openstack-meeting-300:01
*** yamamoto has quit IRC00:06
*** ianychoi has quit IRC00:07
*** ianychoi has joined #openstack-meeting-300:08
*** igordc has joined #openstack-meeting-300:12
*** yamamoto has joined #openstack-meeting-300:57
*** ianychoi has quit IRC01:30
*** ianychoi has joined #openstack-meeting-301:32
*** ianychoi has quit IRC02:10
*** ianychoi has joined #openstack-meeting-302:18
*** hongbin has joined #openstack-meeting-302:20
*** diablo_rojo has quit IRC02:34
*** yamamoto has quit IRC02:35
*** hongbin has quit IRC02:39
*** yamamoto has joined #openstack-meeting-302:43
*** ricolin_ has joined #openstack-meeting-303:01
*** psachin has joined #openstack-meeting-303:26
*** igordc has quit IRC03:30
*** ianychoi has quit IRC03:35
*** ianychoi has joined #openstack-meeting-303:37
*** ricolin_ has quit IRC03:38
*** ricolin has quit IRC04:44
*** ianychoi has quit IRC05:00
*** ianychoi has joined #openstack-meeting-305:03
*** yamamoto has quit IRC05:34
*** links has joined #openstack-meeting-305:36
*** yamamoto has joined #openstack-meeting-305:39
*** njohnston has quit IRC05:39
*** yamamoto has quit IRC05:43
*** ricolin has joined #openstack-meeting-305:51
*** ianychoi has quit IRC06:01
*** ianychoi has joined #openstack-meeting-306:03
*** ianychoi has quit IRC06:11
*** ianychoi has joined #openstack-meeting-306:13
*** e0ne has joined #openstack-meeting-306:16
*** yamamoto has joined #openstack-meeting-306:25
*** e0ne has quit IRC06:26
*** e0ne has joined #openstack-meeting-306:27
*** e0ne has quit IRC06:28
*** e0ne has joined #openstack-meeting-306:28
*** ianychoi has quit IRC06:29
*** ianychoi has joined #openstack-meeting-306:31
*** e0ne has quit IRC06:37
*** ianychoi has quit IRC06:39
*** ianychoi has joined #openstack-meeting-306:41
*** ianychoi has quit IRC06:58
*** ianychoi has joined #openstack-meeting-307:05
*** maciejjozefczyk has joined #openstack-meeting-307:49
*** yamamoto has quit IRC07:59
*** yamamoto has joined #openstack-meeting-308:00
*** slaweq has joined #openstack-meeting-308:03
*** yamamoto has quit IRC08:04
*** yamamoto has joined #openstack-meeting-308:05
*** psachin has quit IRC08:15
*** yamamoto_ has joined #openstack-meeting-308:18
*** yamamoto has quit IRC08:22
*** yamamoto_ has quit IRC08:35
*** yamamoto has joined #openstack-meeting-308:37
*** ralonsoh has joined #openstack-meeting-308:47
*** e0ne has joined #openstack-meeting-308:56
*** belmoreira has joined #openstack-meeting-309:13
*** yamamoto has quit IRC09:56
*** yamamoto has joined #openstack-meeting-309:57
*** yamamoto has quit IRC09:59
*** yamamoto has joined #openstack-meeting-310:20
*** yamamoto has quit IRC10:27
*** rubasov has joined #openstack-meeting-310:38
*** yamamoto has joined #openstack-meeting-311:03
*** yamamoto has quit IRC11:09
*** ttx has quit IRC11:17
*** maciejjozefczyk_ has joined #openstack-meeting-311:17
*** maciejjozefczyk has quit IRC11:18
*** ttx has joined #openstack-meeting-311:19
*** ttx has quit IRC11:19
*** ianychoi has quit IRC11:24
*** ttx has joined #openstack-meeting-311:28
*** njohnston has joined #openstack-meeting-311:33
*** belmoreira has quit IRC12:31
*** jraju__ has joined #openstack-meeting-313:17
*** links has quit IRC13:18
*** irclogbot_3 has quit IRC13:22
*** irclogbot_0 has joined #openstack-meeting-313:23
*** jamesmcarthur has joined #openstack-meeting-313:27
*** yamamoto has joined #openstack-meeting-313:36
*** maciejjozefczyk_ is now known as maciejjozefczyk13:41
*** yamamoto has quit IRC13:49
*** zigo has quit IRC13:49
*** liuyulong has joined #openstack-meeting-313:58
*** yamamoto has joined #openstack-meeting-314:25
*** yamamoto has quit IRC14:31
slaweq#startmeeting neutron_ci15:01
openstackMeeting started Wed Mar 11 15:01:54 2020 UTC and is due to finish in 60 minutes.  The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot.15:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:01
*** openstack changes topic to " (Meeting topic: neutron_ci)"15:01
openstackThe meeting name has been set to 'neutron_ci'15:01
njohnstono/15:02
slaweqnjohnston: ralonsoh bcafarel: here it should be :)15:02
ralonsohhello again15:02
bcafarelo/15:02
slaweqGrafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:02
bcafarelah you were in another chan? that was why I felt lonely here :)15:02
slaweqbcafarel: sorry :)15:02
slaweqit was my mistake15:02
slaweqok, lets start15:02
slaweq#topic Actions from previous meetings15:02
*** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)"15:02
slaweqfirst one15:03
slaweqslaweq to remove neutron-tempest-dvr job from grafana15:03
slaweqpatch: https://review.opendev.org/71204815:03
slaweqI don't think there is much to say about it so lets move on to the next one15:03
slaweqmaciejjozefczyk to take a look at https://bugs.launchpad.net/neutron/+bug/186545315:03
openstackLaunchpad bug 1865453 in neutron "neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestVirtualPorts.test_virtual_port_created_before fails randomly" [High,Confirmed] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)15:03
maciejjozefczykslaweq, yes, im going to do it asap, I was on pto15:04
slaweqok maciejjozefczyk, I will assign it to You for next week, ok?15:04
maciejjozefczykslaweq, im already assigned to this one15:05
slaweqmaciejjozefczyk: ok15:05
slaweq#action maciejjozefczyk to take a look at https://bugs.launchpad.net/neutron/+bug/186545315:05
openstackLaunchpad bug 1865453 in neutron "neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.test_mech_driver.TestVirtualPorts.test_virtual_port_created_before fails randomly" [High,Confirmed] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)15:05
slaweqmaciejjozefczyk: do You think we should mark those tests as unstable temporary?15:05
maciejjozefczykslaweq, let me try to take a look and spend an hour on it, If I'll not find anything at first shot I'll send a patch to mark to unstable, aight?15:06
slaweqmaciejjozefczyk: sure, sounds good15:06
slaweqok, next one15:07
slaweqralonsoh to check "neutron_tempest_plugin.scenario.test_multicast.MulticastTestIPv4 failing often on deletion of server"15:07
ralonsohslaweq, sorry, I didn't have time for this one15:07
slaweqok15:07
slaweqwill You try to check it this week?15:08
ralonsohyes, sure, I think I'll have time15:08
slaweq#action ralonsoh to check "neutron_tempest_plugin.scenario.test_multicast.MulticastTestIPv4 failing often on deletion of server"15:08
slaweqralonsoh: thx15:08
slaweqand the last one15:08
slaweqslaweq to check problem with console output in scenario test15:08
slaweqPatch https://review.opendev.org/71205415:09
slaweqI think this should solve this issue15:09
slaweqplease review when You will have some time15:09
njohnstonwill do15:10
slaweqand that's all on the list of actions from last week15:10
slaweqthx njohnston15:10
bcafarel"for server in servers: server = server.get("server") or server" that makes a funny line15:10
slaweqbcafarel: yes, but it works :)15:10
maciejjozefczykslaweq, ;)15:10
bcafareltrue :)15:10
slaweq#topic Stadium projects15:11
*** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)"15:11
slaweqstandardize on zuul v315:11
slaweqEtherpad: https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop15:11
slaweqaccording to zuul v3 migration, we have only 3 stadium projects left on the list15:11
slaweq    networking-bagpipe15:12
slaweq    networking-midonet15:12
slaweq    networking-odl15:12
slaweqfor bagpipe there is some patch proposed to convert fullstack job15:12
slaweqbut it's very red currently15:12
slaweqfor midonet and odl there is nothing proposed yet15:12
slaweqin overall we are doing good progress on this15:13
slaweqanything else related to stadium projects for today?15:13
bcafarelshould we add some ipv6-only goal section here?15:14
slaweqbcafarel: good point15:14
bcafarelhttps://review.opendev.org/#/q/status:open+topic:ipv6-only-deployment-and-testing+(status:open+OR+status:merged)+(project:%255Eopenstack/neutron.*+OR+project:%255Eopenstack/networking-.*) you mentioned on neutron meeting mostly shows networking-* remaining patches15:14
slaweqI will prepare etherpad with summary of what is wrong there and will add it to the agenda15:15
slaweq#action slaweq to prepare etherpad to track progress with ipv6-only testing goal15:15
bcafarelthanks! I tried to rebase networking-generic-switch patch but apparently there are other failures around (and did not check the others yet)15:15
slaweqthx bcafarel15:16
*** yamamoto has joined #openstack-meeting-315:17
slaweqok, lets move on15:17
slaweq#topic Grafana15:17
*** openstack changes topic to "Grafana (Meeting topic: neutron_ci)"15:17
slaweq#link http://grafana.openstack.org/dashboard/db/neutron-failure-rate15:17
slaweqin overall we still have the same problems as last week(s)15:19
slaweqhigh failure rate of dvr scenario jobs (but those are non-voting)15:19
slaweqand pretty high failure rate of functional job and grenade jobs15:19
ralonsohdo you have a list of grenade and FTs tests failing?15:20
slaweqralonsoh: for grenade jobs nope15:21
slaweqand for functional tests, it's mostly those ovn mech driver tests15:21
slaweqwhich maciejjozefczyk will check15:21
ralonsohyou are right15:21
bcafarelI sent quite a few "recheck grenade" on stable/train backports recently too, older branches are not affected15:22
slaweqfor grenade job, last time when I was checking it, it was mostly issue with some timeout on placement and instance in error state due to that15:22
slaweqbut maybe there is some different issue now15:22
slaweqhttps://zuul.opendev.org/t/openstack/build/e820bdd66eb040fa9db010de1fe821b315:22
slaweqhere is the grenade job failure from today :)15:23
slaweqfailures with "No valid host was found"15:24
slaweqhttps://5e801b59e3f992e71df8-ab7619989a2ab5c37e2a2a5061e93a1d.ssl.cf5.rackcdn.com/711887/1/check/neutron-grenade-multinode/e820bdd/logs/grenade.sh.txt15:24
*** yamamoto has quit IRC15:25
ralonsohNo valid host was found. There are not enough hosts available15:25
ralonsohsame error always15:26
slaweqyep15:26
slaweqand I believe it would be some issue in placement15:26
bcafarelralonsoh: some failures in http://paste.openstack.org/show/790544/ (I did not check them, maybe "no valid host" everywhere)15:27
ralonsohI'll check it15:27
slaweqthere is such error there: https://zuul.opendev.org/t/openstack/build/e820bdd66eb040fa9db010de1fe821b3/log/logs/screen-n-sch.txt#103115:28
slaweqI think it may be related15:28
slaweqit's not from placement but nova15:28
slaweqif anyone has got any cycles and can take a look, that would be great :)15:29
slaweqok, anything else related to grafana for today?15:29
slaweqif not, I think we can move on to the next topic15:30
slaweqI don't have anything about functional/fullstack jobs for today15:31
slaweqso lets talk about scenario15:31
slaweq#topic Tempest/Scenario jobs15:31
*** openstack changes topic to "Tempest/Scenario jobs (Meeting topic: neutron_ci)"15:31
slaweqas I said before, our biggest issues are with dvr related multinode jobs15:31
slaweqso I spent some time today to prepare some summary of what tests are failing there15:32
slaweqand I have: https://etherpad.openstack.org/p/neutron-dvr-jobs-issues15:32
slaweqmost of the failures are in neutron-tempest-plugin-dvr-multinode-scenario15:32
bcafarelthat is a long list :(15:32
slaweqbcafarel: some links are there couple of times as in one job more than one test failed15:33
slaweqand I put it in each section as I wanted to check how much each of those tests is failing15:33
slaweqso our biggest issues are:15:34
slaweqrouter migration from something (HA or Legacy) to dvr15:34
slaweqand I believe that this is the same issue in both cases15:34
slaweqand     neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers15:34
slaweqand also security_groups tests15:34
slaweqbut those may be hopefully fixed when we will use new cirros 0.5.0 in our gate15:35
slaweqas maciejjozefczyk did some improvements in cirros image to address issues with metadata timeouts15:35
ralonsohI think there were some problems with cirros 5 and OVN15:35
ralonsohbut maciejjozefczyk knows this better15:36
maciejjozefczykralonsoh, I remember issue with ip link, but it should be fixed in 0.5.115:36
ralonsohahhh ok, perfect15:36
slaweqmaciejjozefczyk: is 0.5.1 already released?15:36
maciejjozefczykslaweq, yes: http://download.cirros-cloud.net/0.5.1/15:37
bcafarelmaciejjozefczyk: is this new new release for the failures you saw trying 0.5.0?15:38
slaweqmaciejjozefczyk: great, You have already patch to use it in neutron, right?15:38
maciejjozefczykim gonna verify it this week if all is fine, I saw some comments from Radoslaw here: https://review.opendev.org/#/c/711492/15:38
slaweqmaciejjozefczyk: can You also send patch to neutron-tempest-plugin to use it in our jobs?15:38
slaweqwe can then check how many (if any) tests more will be green15:38
slaweq:)15:38
maciejjozefczykslaweq, yes sure, but I think that setting it in global devstack is the right way? I didnt find any setting for this in our tempest configuration, afair15:39
maciejjozefczykor I missed something, anyways, im gonna take a look on it this week15:39
ralonsohwe retrieve that from devstack15:40
slaweqmaciejjozefczyk: setting it in devstack is the best place but we can do it for our jobs even just like DNM patch as I'm curius if that will really help15:40
maciejjozefczykslaweq, I already did it similiar way in https://review.opendev.org/#/c/711425/15:40
maciejjozefczykwith depends-on15:40
maciejjozefczykanyways, lemme check this and I'll be back with results :)15:41
slaweqok, thx15:41
slaweqso my plan for that is:15:41
slaweq1. we will check with new cirros if some of tests will be more stable,15:41
slaweq2. for router migrations tests and connectivity tests I will open new LP15:42
slaweqand we will need to check them in next weeks15:42
slaweqis it ok for You?15:42
maciejjozefczykslaweq, you have my sword15:42
ralonsohhahahaha15:43
slaweq:)15:43
slaweqanything else You have related to scenario/tempest jobs?15:44
maciejjozefczykI'm working on some improvements for QoS tests15:44
maciejjozefczykWe spotted some issues while using QoS tests with OVN as a backend15:44
maciejjozefczyk#link https://bugs.launchpad.net/neutron/+bug/186603915:45
openstackLaunchpad bug 1866039 in neutron "[OVN] QoS gives different bandwidth limit measures than ml2/ovs" [High,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk)15:45
maciejjozefczykIt should be addressed by:15:45
maciejjozefczyk#link https://review.opendev.org/#/c/711048/15:45
maciejjozefczykIf you'll have a minute please take a look:) thanks!15:45
slaweqthx maciejjozefczyk, I will review15:45
maciejjozefczykall from me, thanks!15:46
slaweqanything else? or should we move on?15:46
slaweqok, so lets move on15:47
slaweqI see that njohnston added something related to fullstack tests15:47
slaweqso lets get back to this topic now15:47
slaweq#topic fullstack/functional15:48
*** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)"15:48
slaweqnjohnston: You're up :)15:48
njohnstonYes - I have a change to mark the fullstack security group tests as stable15:48
njohnston https://review.opendev.org/71078215:48
njohnstonIt has not failed in a bit15:48
njohnstonSo I wanted to let you all know, and see how many passing rechecks you think are needed before calling it stable15:49
bcafarelso disabling concurrency on security group tests seems to be enough?15:49
njohnstonit seems to have done the trick15:49
slaweqI don't know for sure but IMO it may be worth to try15:50
slaweqworst case we will mark it as unstable again15:50
ralonsoh+115:50
slaweqthx njohnston for bringing this up :)15:51
njohnstonthat's it for me15:51
slaweqI almost forgot about it15:51
njohnston:-)15:51
bcafarelI'd feel safer with 2 or 3 additional rounds of test rechecks - will see how it runs locally15:51
njohnstonbcafarel: I can definitely do that15:52
slaweqbcafarel: sure :)15:52
njohnstonso far it has passed the security group tests 5 times, I will do 3 more15:52
bcafarelapart from that nice to see it was "only" this secgroup issue :)15:52
*** zigo has joined #openstack-meeting-315:52
slaweqgreat, so I will also keep an eye on it15:53
slaweqdo You have anything else to talk about today?15:53
njohnstonnope15:53
slaweqif not, I think I can give You few minutes back15:53
bcafarelyay15:54
njohnston\o/15:54
ralonsohbye15:54
maciejjozefczyk;)15:54
slaweqok, so thx for attending15:54
slaweqo/15:54
slaweq#endmeeting15:54
*** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/"15:54
openstackMeeting ended Wed Mar 11 15:54:23 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)15:54
openstackMinutes:        http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-11-15.01.html15:54
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-11-15.01.txt15:54
openstackLog:            http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-03-11-15.01.log.html15:54
*** e0ne has quit IRC15:57
*** yamamoto has joined #openstack-meeting-315:57
*** diablo_rojo has joined #openstack-meeting-315:59
*** yamamoto has quit IRC16:03
*** maciejjozefczyk has quit IRC16:07
*** belmoreira has joined #openstack-meeting-316:21
*** ricolin has quit IRC16:25
*** belmoreira has quit IRC16:39
*** jamesmcarthur has quit IRC16:49
*** jamesmcarthur has joined #openstack-meeting-316:50
*** e0ne has joined #openstack-meeting-316:59
*** ricolin has joined #openstack-meeting-317:01
*** jamesmcarthur has quit IRC17:01
*** rubasov has quit IRC17:13
*** jamesmcarthur has joined #openstack-meeting-317:15
*** jamesmcarthur has quit IRC17:23
*** jamesmcarthur has joined #openstack-meeting-317:28
*** jamesmcarthur has quit IRC17:37
*** jamesmcarthur has joined #openstack-meeting-317:37
*** bnemec has quit IRC17:38
*** bnemec has joined #openstack-meeting-317:39
*** jamesmcarthur has quit IRC17:40
*** jraju__ has quit IRC17:48
*** jamesmcarthur has joined #openstack-meeting-317:54
*** e0ne has quit IRC18:16
*** diablo_rojo has quit IRC18:23
*** diablo_rojo has joined #openstack-meeting-318:24
*** ralonsoh has quit IRC19:44
*** jamesmcarthur has quit IRC19:48
*** jamesmcarthur has joined #openstack-meeting-319:49
*** jamesmcarthur has quit IRC19:49
*** slaweq has quit IRC19:53
*** jamesmcarthur has joined #openstack-meeting-319:59
*** yamamoto has joined #openstack-meeting-320:33
*** yamamoto has quit IRC20:38
*** raildo has quit IRC21:12
*** dustinc has quit IRC22:31
*** jamesmcarthur has quit IRC22:34
*** gmann has quit IRC22:36
*** beisner has quit IRC22:36
*** dustinc has joined #openstack-meeting-322:40
*** beisner has joined #openstack-meeting-322:40
*** gmann has joined #openstack-meeting-322:40
*** rajinir has quit IRC22:46
*** irclogbot_0 has quit IRC22:47
*** irclogbot_2 has joined #openstack-meeting-322:48
*** rajinir has joined #openstack-meeting-322:50
*** ildikov has quit IRC22:56
*** srwilkers has quit IRC22:56
*** knikolla has quit IRC22:56
*** csatari has quit IRC22:56
*** ildikov has joined #openstack-meeting-322:59
*** csatari has joined #openstack-meeting-323:00
*** srwilkers has joined #openstack-meeting-323:01
*** knikolla has joined #openstack-meeting-323:01
*** ianychoi has joined #openstack-meeting-323:06
*** yamamoto has joined #openstack-meeting-323:47
*** yamamoto has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!