| *** dustinc has quit IRC | 03:34 | |
| *** psachin has joined #openstack-meeting-3 | 03:37 | |
| *** diablo_rojo has quit IRC | 04:44 | |
| *** psachin has quit IRC | 04:56 | |
| *** links has joined #openstack-meeting-3 | 05:34 | |
| *** njohnston has quit IRC | 06:05 | |
| *** belmoreira has joined #openstack-meeting-3 | 06:25 | |
| *** links has quit IRC | 06:37 | |
| *** links has joined #openstack-meeting-3 | 06:40 | |
| *** lpetrut has joined #openstack-meeting-3 | 06:42 | |
| *** slaweq has joined #openstack-meeting-3 | 06:51 | |
| *** apetrich has quit IRC | 07:11 | |
| *** ralonsoh has joined #openstack-meeting-3 | 07:42 | |
| *** apetrich has joined #openstack-meeting-3 | 07:45 | |
| *** apetrich has quit IRC | 07:45 | |
| *** apetrich has joined #openstack-meeting-3 | 07:46 | |
| *** e0ne has joined #openstack-meeting-3 | 08:05 | |
| *** jraju__ has joined #openstack-meeting-3 | 09:27 | |
| *** links has quit IRC | 09:28 | |
| *** links has joined #openstack-meeting-3 | 09:47 | |
| *** jraju__ has quit IRC | 09:47 | |
| *** njohnston has joined #openstack-meeting-3 | 11:01 | |
| *** yamamoto has joined #openstack-meeting-3 | 11:43 | |
| *** yamamoto has quit IRC | 11:43 | |
| *** raildo has joined #openstack-meeting-3 | 12:05 | |
| *** lpetrut has quit IRC | 12:46 | |
| *** ianychoi has quit IRC | 12:55 | |
| *** liuyulong has joined #openstack-meeting-3 | 13:49 | |
| *** lpetrut has joined #openstack-meeting-3 | 14:29 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
|---|---|---|
| openstack | Meeting started Wed May 13 15:00:48 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| slaweq | hi | 15:00 |
| ralonsoh | hi | 15:01 |
| *** lajoskatona has joined #openstack-meeting-3 | 15:01 | |
| lajoskatona | o/ | 15:01 |
| bcafarel | o/ | 15:01 |
| *** links has quit IRC | 15:02 | |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:02 |
| slaweq | Please open now :) | 15:02 |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 15:02 | |
| slaweq | and now, lets start :) | 15:02 |
| slaweq | #topic Actions from previous meetings | 15:02 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:02 | |
| slaweq | maciejjozefczyk will increase timeout in neutron.tests.functional.plugins.ml2.drivers.ovn.mech_driver.ovsdb.test_ovn_db_sync.TestOvnNbSyncOverTcp.test_ovn_nb_sync_log | 15:02 |
| maciejjozefczyk | slaweq, done | 15:03 |
| slaweq | maciejjozefczyk: thx :) | 15:03 |
| slaweq | that's fast | 15:03 |
| maciejjozefczyk | but I haven't time to verify if this helped | 15:03 |
| slaweq | I didn't saw such failure this week | 15:03 |
| slaweq | but maybe others did | 15:03 |
| ralonsoh | no sorry | 15:03 |
| bcafarel | I don't think I saw one either (so that sounds good) | 15:04 |
| slaweq | ralonsoh: no need to sorry, maybe it's just fixed now :) | 15:04 |
| slaweq | ok, next one | 15:05 |
| slaweq | bcafarel to check neutron-dynamic-routing failurs in stable/train | 15:05 |
| bcafarel | it was indeed caused by recent tempest install change, there is a LP and patch up for review | 15:05 |
| bcafarel | https://review.opendev.org/#/c/725920/ | 15:06 |
| slaweq | thx bcafarel | 15:07 |
| slaweq | +2 | 15:07 |
| slaweq | :) | 15:07 |
| bcafarel | almost there :) | 15:07 |
| slaweq | ok, so next one | 15:07 |
| slaweq | ralonsoh to check ovn jobs timeouts | 15:07 |
| ralonsoh | sorry I dind't spend too much time on this | 15:08 |
| *** lpetrut has quit IRC | 15:08 | |
| ralonsoh | and I didn't find anything yet | 15:08 |
| slaweq | but are ovn jobs still timeouting so often? | 15:08 |
| ralonsoh | not so often | 15:08 |
| slaweq | I think it was like 100% of runs last week | 15:08 |
| slaweq | or almost | 15:08 |
| ralonsoh | but I still need to find why we could have this | 15:08 |
| slaweq | ok | 15:08 |
| ralonsoh | and maybe push a solution for the CI | 15:09 |
| njohnston | o/ sorry I am late | 15:09 |
| slaweq | so can I assign it to You for next week? | 15:09 |
| ralonsoh | sure | 15:09 |
| slaweq | thx | 15:09 |
| slaweq | #action ralonsoh to continue checking ovn jobs timeouts | 15:09 |
| slaweq | hi njohnston :) | 15:09 |
| slaweq | and the last one | 15:09 |
| slaweq | maciejjozefczyk to check failing compile of ovs in the fedora periodic job | 15:09 |
| maciejjozefczyk | #link https://review.opendev.org/#/c/726759/ | 15:09 |
| maciejjozefczyk | Fedora has new kernel and OVS doesn't support compilation for it | 15:09 |
| slaweq | ok, I saw this patch | 15:10 |
| maciejjozefczyk | (newer than 5.5) | 15:10 |
| slaweq | and I have one question - did You test that it will solve problem with fedora job? | 15:10 |
| slaweq | can You maybe send dnm patch on top of this one, where You will add this fedora job to check queue, then it will be run as other jobs and we will see if that helped | 15:10 |
| maciejjozefczyk | actually not on the gates | 15:11 |
| maciejjozefczyk | I can verify now sending another patch that relies on this one | 15:11 |
| * maciejjozefczyk will do after meeting | 15:11 | |
| slaweq | thx | 15:11 |
| maciejjozefczyk | but the failure in the fedora job was related to that compilation, so it should fix this | 15:11 |
| slaweq | just to make sure it solves our issue with periodic job :) | 15:12 |
| maciejjozefczyk | yep ++ | 15:12 |
| slaweq | and that's all actions from last wek | 15:12 |
| slaweq | *week | 15:12 |
| slaweq | anything else You want to add here? or can we move on to the next topic? | 15:12 |
| slaweq | ok, so lets move on | 15:14 |
| slaweq | #topic Stadium projects | 15:14 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:14 | |
| slaweq | standardize on zuul v3 | 15:14 |
| slaweq | as we talked on Monday, I will send this week patch to switch ovn grenade job to zuulv3 syntax | 15:14 |
| slaweq | for networking-odl we have patches ready for review, right? | 15:15 |
| lajoskatona | yes, I ahve to check | 15:16 |
| lajoskatona | have -^ | 15:16 |
| slaweq | ok, and then only midonet will left | 15:16 |
| lajoskatona | I have a comment so I ahve to check, but it is ongoing | 15:16 |
| slaweq | thx lajoskatona | 15:16 |
| slaweq | from other things related to stadium, we fixed midonet gate in master and ussuri | 15:17 |
| slaweq | thx ralonsoh for patch for that | 15:17 |
| ralonsoh | yw! | 15:17 |
| slaweq | and I proposed serie of patches https://review.opendev.org/#/q/status:open+branch:stable/ussuri+topic:ussuri-release to use ussuri jobs from neutron-tempest-plugin in ussuri branch | 15:18 |
| slaweq | please review them :) | 15:18 |
| slaweq | anything else regading stadium projects' CI? | 15:18 |
| bcafarel | I think there are a few patches sent recently for hacking 3, to add to review pile :) | 15:20 |
| bcafarel | "Bump hacking min version to 3.0.1" topic for most of them | 15:21 |
| slaweq | thx bcafarel I will check them tonight | 15:22 |
| *** belmoreira has quit IRC | 15:24 | |
| slaweq | bcafarel: I see such patch only for networking-baremetal, which isn't neutron project | 15:24 |
| slaweq | ok, I see | 15:24 |
| slaweq | for other projects too | 15:24 |
| slaweq | I will review them later today | 15:24 |
| bcafarel | yes search does not work well there | 15:25 |
| bcafarel | I mostly saw them passing on #openstack-neutron bot | 15:26 |
| slaweq | lets move on to the next topic then | 15:27 |
| slaweq | #topic Stable branches | 15:27 |
| *** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:27 | |
| slaweq | Train dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 | 15:27 |
| slaweq | Stein dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 | 15:27 |
| slaweq | I think we should update those dashboards now, as ussuri is officially released | 15:28 |
| bcafarel | true, add it to my pile | 15:28 |
| slaweq | thx bcafarel :) | 15:29 |
| slaweq | #action bcafarel to update stable branches grafana dashboards | 15:29 |
| bcafarel | also overall most backports are waiting atm on pycodestyle capping | 15:29 |
| slaweq | yes, pep8 failing 100% times | 15:29 |
| bcafarel | https://review.opendev.org/#/c/727274/3 just merged on ussuri, we will need it up to rovky included | 15:30 |
| bcafarel | *rocky | 15:30 |
| slaweq | I just +W on train patch | 15:32 |
| slaweq | so we will need still backports for older ones | 15:33 |
| slaweq | bcafarel: other jobs in stable branches are fine, right? | 15:33 |
| slaweq | or do we have there any other problems which I missed? | 15:33 |
| bcafarel | yes the rest of the jobs are fine we actually got a few backports merged without too many rechecks :) | 15:34 |
| slaweq | ok, so lets move on to the next topics | 15:35 |
| slaweq | #topic Grafana | 15:35 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:35 | |
| slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:35 |
| slaweq | and some stats: | 15:35 |
| slaweq | Average number of rechecks in last weeks: | 15:36 |
| slaweq | week 18 of 2020: 1.25 | 15:36 |
| slaweq | week 19 of 2020: 2.0 | 15:36 |
| slaweq | week 20 of 2020: 0.67 | 15:36 |
| slaweq | so in overall it seems that it's not very bad | 15:36 |
| slaweq | I don't see anything alarming there | 15:37 |
| bcafarel | a quiet week for release time :) | 15:38 |
| slaweq | seems so :) | 15:38 |
| slaweq | anything else about grafana and our dashboards? | 15:39 |
| slaweq | ok, so lets move on | 15:41 |
| slaweq | #topic fullstack/functional | 15:41 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 15:41 | |
| slaweq | first functional tests | 15:41 |
| slaweq | I wanted to ask You about what do You think if we would switch functional uwsgi job to be voting? | 15:41 |
| maciejjozefczyk | +1 seems to be quite stable | 15:42 |
| slaweq | IMO it's more stable now, and we are at the beginning of the cycle | 15:42 |
| njohnston | +1 | 15:42 |
| slaweq | so worst case we can revert it :) | 15:42 |
| slaweq | ok, so I will propose such patch this week | 15:42 |
| slaweq | #action slaweq to switch functional uwsgi job to be voting | 15:43 |
| slaweq | and I didn't found any new issues in functional job this week | 15:43 |
| maciejjozefczyk | I found one bug in the way we execute functional tests: https://bugs.launchpad.net/neutron/+bug/1878160 | 15:43 |
| openstack | Launchpad bug 1878160 in neutron "[OVN] Functional tests environment is using old OVN" [High,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk) | 15:43 |
| maciejjozefczyk | The OVS and OVN are compiled from hardcoded version 2.12 | 15:43 |
| maciejjozefczyk | So we actually test master code with old OVN as a backend (at least NBDB) | 15:44 |
| maciejjozefczyk | I started working on a solution for that, in order to have possibility to specify tags from which we can compile OVN/OVS for functional tests | 15:44 |
| maciejjozefczyk | we're going to reuse the same for OVN octavia provider | 15:44 |
| maciejjozefczyk | #link https://review.opendev.org/#/c/727193/ | 15:44 |
| maciejjozefczyk | For now I'm checking how to properly perform it | 15:45 |
| slaweq | but are You going to propose such 2 new functional tests jobs? or is that only for Your testing now? | 15:46 |
| maciejjozefczyk | for OVN octavia provider we can try to propose job for OVN master and OVN latest release job | 15:46 |
| maciejjozefczyk | and I need to discuss withing OVN team if we need such -master and -release job in Neutron | 15:47 |
| slaweq | TBH I'm not sure if we want yet another functional tests job | 15:47 |
| slaweq | we are testing I think master and released ovn in some scenario job | 15:47 |
| slaweq | so maybe it would be enough :) | 15:48 |
| maciejjozefczyk | In scenario jobs we're testing only release OVN branch | 15:48 |
| maciejjozefczyk | master is 'experimental' for now | 15:48 |
| maciejjozefczyk | but the problem with it is that we write new code that needs changes in core OVN, mostly those are send in master and not yet released | 15:48 |
| maciejjozefczyk | some of those changes relies on OVN schema | 15:48 |
| maciejjozefczyk | and we have logic like: if required change is not in schema, fallback to the old behavior | 15:49 |
| slaweq | so for such changes there is experimental queue :) | 15:49 |
| maciejjozefczyk | So when we for now have master or release job only, we test only 1 'code path' | 15:49 |
| maciejjozefczyk | So we merge code that its not tested? :) | 15:50 |
| slaweq | You can run experimental jobs on every patch You want | 15:50 |
| maciejjozefczyk | experimental queue is not voting | 15:50 |
| slaweq | just put comment "check experimental" in gerrit | 15:50 |
| slaweq | yes, but we need some "balance" | 15:51 |
| slaweq | IMHO we already have too many jobs in check queue | 15:51 |
| maciejjozefczyk | yes that is why Lucas find some idea | 15:51 |
| slaweq | and if we will start testing also with various versions of ovs/ovn/other soft - we will be lost | 15:51 |
| maciejjozefczyk | #link https://review.opendev.org/#/c/717270/ | 15:51 |
| maciejjozefczyk | Maybe we can define -master job that will be voting only if change is related to OVN? if not: -release job will be run *only* | 15:52 |
| maciejjozefczyk | For now we're producing a lot of code that fills the gaps between ovs/ovn, and I feel sometimes we don't have everything properly covered and then we find issues in d/s gates | 15:52 |
| slaweq | I | 15:54 |
| slaweq | I'm not sure about it :/ | 15:54 |
| maciejjozefczyk | But those issues could be fine u/s if we would have master jobs voting, which will including the OVN code part, that is triggered by OVN driver. | 15:54 |
| maciejjozefczyk | find* | 15:54 |
| maciejjozefczyk | yes, I don't like to much jobs also... | 15:55 |
| slaweq | if change will touch ovn files and also something else, like e.g. some constants it will run all jobs then, right? | 15:55 |
| maciejjozefczyk | maybe we can add -master to gate? I dunno... I need to talk about it during OVN team meeting, maybe we'll find some solution for that, or create general rule to run exeperimental job manually before merging, then +w | 15:55 |
| slaweq | so in most cases we will probably run all jobs still | 15:55 |
| maciejjozefczyk | for constants yes | 15:56 |
| maciejjozefczyk | for other cases should run only -release job | 15:56 |
| maciejjozefczyk | but yeah, probability of changing constants is high | 15:56 |
| slaweq | ok, we will have to get back to it - maybe it will be good topic for ptg :) | 15:58 |
| bcafarel | :) | 15:58 |
| slaweq | we are almost running out of time now, | 15:58 |
| maciejjozefczyk | yes :) | 15:58 |
| slaweq | so is there anything else anyone wants discuss quickly? | 15:58 |
| ralonsoh | no thanks | 15:58 |
| slaweq | last minute call :) | 15:58 |
| bcafarel | nope | 15:59 |
| slaweq | btw ralonsoh I just replied to Your comment in https://review.opendev.org/#/c/726777/2 | 15:59 |
| ralonsoh | thanks! | 15:59 |
| slaweq | please let me know if You still have any concerns with this | 15:59 |
| slaweq | and thx for attending the meeting | 15:59 |
| slaweq | o/ | 15:59 |
| slaweq | #endmeeting | 15:59 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:59 | |
| openstack | Meeting ended Wed May 13 15:59:40 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-13-15.00.html | 15:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-13-15.00.txt | 15:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-05-13-15.00.log.html | 15:59 |
| bcafarel | o/ | 15:59 |
| ralonsoh | bye | 15:59 |
| maciejjozefczyk | bb | 15:59 |
| lajoskatona | bye | 16:01 |
| *** lajoskatona has left #openstack-meeting-3 | 16:01 | |
| *** maciejjozefczyk has quit IRC | 16:04 | |
| *** liuyulong has quit IRC | 16:17 | |
| *** ralonsoh has quit IRC | 17:44 | |
| *** e0ne has quit IRC | 17:55 | |
| *** belmoreira has joined #openstack-meeting-3 | 19:05 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 19:08 | |
| *** maciejjozefczyk has quit IRC | 19:25 | |
| *** belmoreira has quit IRC | 20:07 | |
| *** raildo has quit IRC | 21:46 | |
| *** slaweq has quit IRC | 21:57 | |
| *** slaweq has joined #openstack-meeting-3 | 22:08 | |
| *** slaweq has quit IRC | 22:13 | |
| *** slaweq has joined #openstack-meeting-3 | 22:23 | |
| *** slaweq has quit IRC | 22:28 | |
| *** hongbin has joined #openstack-meeting-3 | 23:05 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!