| *** jamesmcarthur has joined #openstack-meeting-3 | 00:45 | |
| *** jamesmcarthur has quit IRC | 01:00 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 01:10 | |
| *** slaweq_ has joined #openstack-meeting-3 | 01:11 | |
| *** slaweq_ has quit IRC | 01:16 | |
| *** jamesmcarthur has quit IRC | 01:22 | |
| *** vesper has joined #openstack-meeting-3 | 01:38 | |
| *** vesper11 has quit IRC | 01:39 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 01:45 | |
| *** igordc has joined #openstack-meeting-3 | 01:52 | |
| *** jamesmcarthur has quit IRC | 02:22 | |
| *** slaweq_ has joined #openstack-meeting-3 | 03:11 | |
| *** apetrich has quit IRC | 03:12 | |
| *** slaweq_ has quit IRC | 03:16 | |
| *** links has joined #openstack-meeting-3 | 04:43 | |
| *** bnemec has joined #openstack-meeting-3 | 04:53 | |
| *** hongbin has joined #openstack-meeting-3 | 04:55 | |
| *** igordc has quit IRC | 04:59 | |
| *** slaweq_ has joined #openstack-meeting-3 | 05:11 | |
| *** slaweq_ has quit IRC | 05:15 | |
| *** hongbin has quit IRC | 05:28 | |
| *** slaweq_ has joined #openstack-meeting-3 | 07:11 | |
| *** slaweq_ has quit IRC | 07:16 | |
| *** slaweq_ has joined #openstack-meeting-3 | 08:00 | |
| *** slaweq__ has joined #openstack-meeting-3 | 08:09 | |
| *** slaweq_ has quit IRC | 08:10 | |
| *** slaweq has joined #openstack-meeting-3 | 08:14 | |
| *** slaweq__ has quit IRC | 08:16 | |
| *** ralonsoh has joined #openstack-meeting-3 | 08:42 | |
| *** apetrich has joined #openstack-meeting-3 | 08:48 | |
| *** slaweq_ has joined #openstack-meeting-3 | 08:48 | |
| *** slaweq has quit IRC | 08:48 | |
| *** slaweq__ has joined #openstack-meeting-3 | 09:06 | |
| *** slaweq_ has quit IRC | 09:07 | |
| *** slaweq__ is now known as slaweq | 09:25 | |
| *** slaweq_ has joined #openstack-meeting-3 | 10:40 | |
| *** slaweq has quit IRC | 10:42 | |
| *** e0ne has joined #openstack-meeting-3 | 10:57 | |
| *** slaweq__ has joined #openstack-meeting-3 | 11:01 | |
| *** slaweq_ has quit IRC | 11:03 | |
| *** lkoranda has joined #openstack-meeting-3 | 11:05 | |
| *** bobmel has joined #openstack-meeting-3 | 11:17 | |
| *** psachin has joined #openstack-meeting-3 | 11:30 | |
| *** pcaruana has quit IRC | 11:37 | |
| *** e0ne has quit IRC | 11:42 | |
| *** pcaruana has joined #openstack-meeting-3 | 11:50 | |
| *** slaweq__ has quit IRC | 12:19 | |
| *** slaweq__ has joined #openstack-meeting-3 | 12:23 | |
| *** e0ne has joined #openstack-meeting-3 | 12:27 | |
| *** raildo has joined #openstack-meeting-3 | 12:43 | |
| *** e0ne has quit IRC | 12:53 | |
| *** e0ne has joined #openstack-meeting-3 | 13:05 | |
| *** slaweq has joined #openstack-meeting-3 | 13:11 | |
| *** slaweq__ has quit IRC | 13:13 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 13:20 | |
| *** lpetrut has joined #openstack-meeting-3 | 13:24 | |
| *** jamesmcarthur has quit IRC | 13:37 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 13:47 | |
| *** liuyulong has joined #openstack-meeting-3 | 13:58 | |
| *** lkoranda has quit IRC | 13:58 | |
| *** bobmel has quit IRC | 13:58 | |
| *** lkoranda has joined #openstack-meeting-3 | 14:11 | |
| *** pcaruana has quit IRC | 14:17 | |
| *** psachin has quit IRC | 14:25 | |
| *** bobmel has joined #openstack-meeting-3 | 14:37 | |
| *** links has quit IRC | 14:50 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
|---|---|---|
| openstack | Meeting started Wed Feb 5 15:00:13 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| slaweq | hi | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| njohnston | o/ | 15:00 |
| slaweq | ralonsoh: bcafarel haleyb: CI meeting, are You around? | 15:01 |
| *** lkoranda has quit IRC | 15:01 | |
| ralonsoh | hi | 15:01 |
| bcafarel | o/ | 15:01 |
| ralonsoh | I was waiting in the wrong channel | 15:01 |
| bcafarel | slaweq: thanks for the ping I was looking for correct window :) | 15:01 |
| haleyb | slaweq: i'm in another meeting too, have one eye here :) | 15:01 |
| slaweq | :) | 15:01 |
| slaweq | ok, lets start | 15:01 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:01 |
| slaweq | please open it and we can move on | 15:02 |
| slaweq | #topic Actions from previous meetings | 15:02 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:02 | |
| slaweq | slaweq to talk with gmann about vpnaas jobs on rocky | 15:02 |
| slaweq | tbh I forgot about it :/ | 15:02 |
| gmann | slaweq: i sent summary of what amotoki and we discussed on ML | 15:02 |
| slaweq | maybe gmann is around now so we can ask him how to fix this issue with vpnaas rocky tempest jobs | 15:02 |
| slaweq | gmann: ok, so I need to find this email than :) | 15:03 |
| gmann | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012241.html | 15:03 |
| gmann | 4th point | 15:03 |
| gmann | slaweq: ping me once you are done then we can discuss further | 15:04 |
| slaweq | gmann: so basically we should backport to rocky https://review.opendev.org/#/c/695834/ | 15:07 |
| slaweq | or at least "partially" backport it | 15:07 |
| slaweq | ? | 15:07 |
| gmann | slaweq: yeah that will be good for long term maintenance and how py2 EOL things happening | 15:07 |
| slaweq | ok, and this will not be a problem if we will have those tests in-tree but will run them from neutron-tempest-plugin repo actually? | 15:08 |
| gmann | i am fixing stable branches with stable u-c t use in tempest tox run which solve issue but backport that is what i will suggest in case of another issue | 15:08 |
| gmann | true | 15:09 |
| slaweq | ok, thx for explanation | 15:09 |
| gmann | fixing current stable branches summary - http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012371.html | 15:09 |
| slaweq | #action slaweq to backport https://review.opendev.org/#/c/695834/ to stable branches in neutron-vpnaas | 15:09 |
| gmann | FYI, all stable branch till rocky are broken now | 15:09 |
| bcafarel | sigh | 15:10 |
| slaweq | :/ | 15:10 |
| slaweq | ok, lets move on | 15:11 |
| slaweq | next action was | 15:11 |
| slaweq | slaweq to update grafana dashboard with missing jobs | 15:11 |
| slaweq | and I also forgot about it :/ | 15:11 |
| slaweq | #action slaweq to update grafana dashboard with missing jobs | 15:11 |
| slaweq | I will do it this week | 15:11 |
| slaweq | any questions/comments to this topic? | 15:12 |
| njohnston | nope | 15:12 |
| ralonsoh | no | 15:13 |
| slaweq | so we can move on to the next topic | 15:13 |
| slaweq | #topic Stadium projects | 15:13 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:13 | |
| slaweq | migration to zuulv3 | 15:13 |
| slaweq | https://etherpad.openstack.org/p/neutron-train-zuulv3-py27drop | 15:13 |
| slaweq | I was checking this etherpad few days ago, and I even send some small patches related to it | 15:13 |
| slaweq | (but still needs some work) | 15:13 |
| slaweq | and generally we are pretty good there | 15:14 |
| slaweq | most of the legacy jobs have got patches already in review | 15:14 |
| njohnston | +1 | 15:14 |
| *** e0ne has quit IRC | 15:15 | |
| slaweq | huge thx for bcafarel for sending many related patches :) | 15:15 |
| bcafarel | np, some of them are still not working properly | 15:15 |
| bcafarel | slaweq: as you know neutron-functional well if you have some time to take a look at https://review.opendev.org/#/c/703601/ | 15:16 |
| bcafarel | I can't seem to convince it to install/find neutron :( | 15:16 |
| bcafarel | (nothing urgent of course) | 15:16 |
| slaweq | bcafarel: ok, I will take a look | 15:16 |
| bcafarel | thanks :) | 15:17 |
| slaweq | np | 15:17 |
| slaweq | anything else related to the stadium projects' ci? | 15:18 |
| njohnston | nope | 15:18 |
| ralonsoh | no | 15:18 |
| slaweq | ok, lets move on than | 15:19 |
| slaweq | #topic Grafana | 15:19 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:19 | |
| slaweq | http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:19 |
| slaweq | first of all our gate jobs are not running since few days | 15:21 |
| slaweq | and it's mostly due to broken neutron-ovn-tempest-ovs-release job | 15:21 |
| slaweq | second issue is that we have some gap in data in during last weekend and begin of this week | 15:22 |
| slaweq | but that's probably infra issue | 15:22 |
| njohnston | Is neutron-ovn-tempest-ovs-release one of the missing jobs that needs to be updated in grafana? I can't find it. | 15:23 |
| slaweq | njohnston: yes | 15:23 |
| slaweq | sorry for that :/ | 15:23 |
| njohnston | ok | 15:23 |
| njohnston | np :-) | 15:23 |
| slaweq | also it seems we had some problem yesterday as many jobs has got high numbers there | 15:24 |
| slaweq | but I don't know about any specific issue from yesterday | 15:25 |
| slaweq | but this spike can be also due to low number of running jobs (or data stored) in the day before | 15:26 |
| slaweq | other than that I don't see anything really wrong | 15:28 |
| *** lpetrut has quit IRC | 15:28 | |
| slaweq | ok, lets move on than | 15:29 |
| slaweq | #topic fullstack/functional | 15:29 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 15:29 | |
| slaweq | I have couple of issues in fullstack tests for today | 15:29 |
| slaweq | Error when connecting to the placement service (same as in last week too): | 15:29 |
| slaweq | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f66/703143/3/check/neutron-fullstack/f667c93/controller/logs/dsvm-fullstack-logs/TestPlacementBandwidthReport.test_configurations_are_synced_towards_placement_NIC-Switch-agent_/neutron-server--2020-02-04--12-09-12-759753_log.txt | 15:29 |
| slaweq | maybe lajoskatona or rubasov could take a look at it | 15:30 |
| slaweq | I pinged rubasov to join this meeting | 15:30 |
| slaweq | maybe he will join soon | 15:31 |
| *** rubasov has joined #openstack-meeting-3 | 15:31 | |
| rubasov | hi | 15:31 |
| slaweq | hi rubasov | 15:31 |
| slaweq | recently we spotted few times issue like in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f66/703143/3/check/neutron-fullstack/f667c93/controller/logs/dsvm-fullstack-logs/TestPlacementBandwidthReport.test_configurations_are_synced_towards_placement_NIC-Switch-agent_/neutron-server--2020-02-04--12-09-12-759753_log.txt | 15:32 |
| slaweq | neutron-server can't connect to (fake) placement service | 15:32 |
| slaweq | did You maybe saw something like that before? | 15:32 |
| slaweq | or do You know why it could happen? | 15:33 |
| rubasov | did not see this before | 15:33 |
| slaweq | rubasov: can You try to take a look into that? | 15:35 |
| rubasov | I don't really have ideas right now | 15:35 |
| slaweq | not today ofcourse :) but if You will have some time | 15:35 |
| rubasov | sure we'll look into it with lajoskatona | 15:35 |
| rubasov | he wrote that fake placement service originally IIRC | 15:35 |
| rubasov | how frequent this is? | 15:36 |
| slaweq | it's not very frequent, I saw it once per week or something like that | 15:36 |
| slaweq | rubasov: I will open LP to track it | 15:37 |
| rubasov | okay, I'll put it on my todo list | 15:37 |
| slaweq | and I will send it to You and Lajos - maybe You will have some time to take a look | 15:37 |
| rubasov | that's even better, thank you | 15:38 |
| slaweq | #action slaweq to open LP related to fullstack placement issue | 15:38 |
| slaweq | thx rubasov | 15:38 |
| slaweq | ok, another issue | 15:38 |
| slaweq | in https://c355270b22583c2d2af0-42801c5a43c64ea303a559bec7f7cdd7.ssl.cf5.rackcdn.com/705903/2/check/neutron-fullstack/79cf197/controller/logs/dsvm-fullstack-logs/TestBwLimitQoSOvs.test_bw_limit_qos_policy_rule_lifecycle_egress_/neutron-server--2020-02-05--11-00-19-067345_log.txt | 15:38 |
| rubasov | thanks | 15:38 |
| slaweq | it seems for me like neutron-server just hanged and that caused test timeout | 15:38 |
| slaweq | actually not, it was also connection issue: https://c355270b22583c2d2af0-42801c5a43c64ea303a559bec7f7cdd7.ssl.cf5.rackcdn.com/705903/2/check/neutron-fullstack/79cf197/controller/logs/dsvm-fullstack-logs/TestBwLimitQoSOvs.test_bw_limit_qos_policy_rule_lifecycle_egress_.txt | 15:39 |
| slaweq | this time to neutron-server | 15:39 |
| *** ianychoi_ is now known as ianychoi | 15:40 | |
| njohnston | well if the neutron server crashed hard then both the log would end precipitously like that and ECONNREFUSED is what clients would see | 15:40 |
| njohnston | if it was a hang then the client would have timeouts | 15:41 |
| slaweq | njohnston: but in the end of the logs You can see | 15:41 |
| slaweq | 2020-02-05 11:01:19.086 22341 DEBUG neutron.agent.linux.utils [-] Running command: ['kill', '-15', '3180'] create_process /home/zuul/src/opendev.org/openstack/neutron/neutron/agent/linux/utils.py:87 | 15:41 |
| slaweq | 2020-02-05 11:01:19.280 22341 DEBUG neutron.tests.fullstack.resources.process [-] Process stopped: neutron-server stop /home/zuul/src/opendev.org/openstack/neutron/neutron/tests/fullstack/resources/process.py:85 | 15:41 |
| slaweq | so it seems that neutron-server was properly stopped at the end | 15:42 |
| slaweq | if it would crash earlier wouldn't this be an error? | 15:42 |
| ralonsoh | I think so... | 15:42 |
| njohnston | which log is that in? I don't see it in https://c355270b22583c2d2af0-42801c5a43c64ea303a559bec7f7cdd7.ssl.cf5.rackcdn.com/705903/2/check/neutron-fullstack/79cf197/controller/logs/dsvm-fullstack-logs/TestBwLimitQoSOvs.test_bw_limit_qos_policy_rule_lifecycle_egress_/neutron-server--2020-02-05--11-00-19-067345_log.txt | 15:43 |
| slaweq | njohnston: it's in https://c355270b22583c2d2af0-42801c5a43c64ea303a559bec7f7cdd7.ssl.cf5.rackcdn.com/705903/2/check/neutron-fullstack/79cf197/controller/logs/dsvm-fullstack-logs/TestBwLimitQoSOvs.test_bw_limit_qos_policy_rule_lifecycle_egress_.txt | 15:43 |
| slaweq | this is "test log" | 15:43 |
| njohnston | ok | 15:44 |
| ralonsoh | slaweq, do you have a bug for this one? | 15:46 |
| ralonsoh | I can review it later | 15:46 |
| slaweq | ralonsoh: nope, I saw it only once so far so and I didn't open bug for it | 15:46 |
| slaweq | but I can | 15:46 |
| ralonsoh | (at least this is not a QoS error) | 15:46 |
| slaweq | I will ping You when I will open LP for that | 15:47 |
| ralonsoh | thanks for the "present" | 15:47 |
| slaweq | #action slaweq to open LP related to "hang" neutron-server | 15:47 |
| slaweq | ralonsoh: yw :D | 15:47 |
| slaweq | and that's all what I have for today for functional/fullstack jobs | 15:47 |
| slaweq | anything else You have maybe? | 15:48 |
| ralonsoh | slaweq, https://review.opendev.org/#/c/705760/ is almost merged | 15:48 |
| ralonsoh | I'll abandon https://review.opendev.org/#/c/705903/ | 15:48 |
| slaweq | ralonsoh: great | 15:49 |
| bcafarel | so, recheck time once 705760 is in? | 15:49 |
| slaweq | that should unblock our gate hopefully | 15:49 |
| *** pcaruana has joined #openstack-meeting-3 | 15:49 | |
| njohnston | excellent | 15:50 |
| slaweq | ok, so we moved to scenario/tempest test now | 15:51 |
| slaweq | #topic Tempest/Scenario | 15:51 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:51 | |
| slaweq | we already mentioned broken neutron-ovn-tempest-ovs-release job which should be fixed with 705760 | 15:51 |
| slaweq | from other issues I have one with test_show_network_segment_range https://9f9aee74b45263b2a9d8-795792c1f104e79962e44448ab55e3f1.ssl.cf1.rackcdn.com/681466/2/check/neutron-tempest-plugin-api/b340c8e/testr_results.html | 15:53 |
| *** e0ne has joined #openstack-meeting-3 | 15:53 | |
| slaweq | and I think I saw something similar couple of times already | 15:53 |
| slaweq | I'm not sure if that was always the same test but similar issue for sure | 15:54 |
| ralonsoh | again? | 15:54 |
| ralonsoh | it is the same one | 15:54 |
| bcafarel | KeyError on project_id?? | 15:54 |
| ralonsoh | I really don't understand why this specific key is not present | 15:54 |
| ralonsoh | and this is not a trivial one, but project_id | 15:54 |
| slaweq | yes, I also don't understant it | 15:55 |
| slaweq | wait, actually this one was 21.01 | 15:55 |
| slaweq | so maybe it's an old issue | 15:56 |
| slaweq | sorry for the noise than :) | 15:56 |
| ralonsoh | no, but this test error is recurrent | 15:57 |
| slaweq | yes, and actually I don't understand it excatly | 15:57 |
| slaweq | if You look at code: https://github.com/openstack/neutron-tempest-plugin/blob/master/neutron_tempest_plugin/api/admin/test_network_segment_range.py#L201 | 15:58 |
| slaweq | it failed after checking "id", "name" and other attributes | 15:58 |
| slaweq | so it's not like dict is empty | 15:59 |
| ralonsoh | one question | 15:59 |
| slaweq | there is "only" project_id missing from it | 15:59 |
| ralonsoh | this test is using neutron-client | 15:59 |
| ralonsoh | not os-client | 15:59 |
| ralonsoh | is that correct? | 15:59 |
| slaweq | idk | 15:59 |
| slaweq | but IMO those tests are using tempest clients, no? | 16:00 |
| ralonsoh | Ok, I'll check it | 16:01 |
| slaweq | thx ralonsoh | 16:01 |
| slaweq | #action ralonsoh to check missing project_id issue | 16:01 |
| ralonsoh | we run out of time | 16:01 |
| slaweq | ok we are out of time today | 16:01 |
| slaweq | thx for attending | 16:01 |
| slaweq | o/ | 16:01 |
| ralonsoh | bye | 16:01 |
| njohnston | \o | 16:01 |
| slaweq | #endmeeting | 16:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:01 | |
| openstack | Meeting ended Wed Feb 5 16:01:35 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-02-05-15.00.html | 16:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-02-05-15.00.txt | 16:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-02-05-15.00.log.html | 16:01 |
| bcafarel | o/ | 16:01 |
| *** bobmel has quit IRC | 16:04 | |
| *** ralonsoh has left #openstack-meeting-3 | 16:04 | |
| *** bobmel has joined #openstack-meeting-3 | 16:05 | |
| *** bobmel has quit IRC | 16:10 | |
| *** e0ne has quit IRC | 16:17 | |
| *** slaweq_ has joined #openstack-meeting-3 | 16:23 | |
| *** jamesmcarthur has quit IRC | 16:24 | |
| *** slaweq has quit IRC | 16:25 | |
| *** psachin has joined #openstack-meeting-3 | 16:38 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 16:38 | |
| *** jamesmcarthur_ has joined #openstack-meeting-3 | 16:41 | |
| *** jamesmcarthur has quit IRC | 16:44 | |
| *** jamesmcarthur_ has quit IRC | 16:55 | |
| *** e0ne has joined #openstack-meeting-3 | 16:56 | |
| *** psachin has quit IRC | 16:58 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 16:58 | |
| *** e0ne has quit IRC | 17:05 | |
| *** jamesmcarthur has quit IRC | 17:05 | |
| *** e0ne has joined #openstack-meeting-3 | 17:09 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 17:27 | |
| *** igordc has joined #openstack-meeting-3 | 17:34 | |
| *** jamesmcarthur has quit IRC | 17:56 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 18:03 | |
| *** e0ne has quit IRC | 18:27 | |
| *** jamesmcarthur has quit IRC | 18:54 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 19:15 | |
| *** bobmel has joined #openstack-meeting-3 | 19:45 | |
| *** jamesmcarthur has quit IRC | 20:21 | |
| *** raildo has quit IRC | 20:30 | |
| *** raildo has joined #openstack-meeting-3 | 20:33 | |
| *** e0ne has joined #openstack-meeting-3 | 20:36 | |
| *** e0ne has quit IRC | 20:44 | |
| *** slaweq_ has quit IRC | 20:44 | |
| *** slaweq_ has joined #openstack-meeting-3 | 20:47 | |
| *** slaweq_ has quit IRC | 21:15 | |
| *** liuyulong has quit IRC | 21:41 | |
| *** bobmel has quit IRC | 21:47 | |
| *** raildo has quit IRC | 22:37 | |
| *** bobmel has joined #openstack-meeting-3 | 22:55 | |
| *** bobmel has quit IRC | 22:56 | |
| *** jamesmcarthur has joined #openstack-meeting-3 | 23:03 | |
| *** jamesmcarthur has quit IRC | 23:06 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!