| *** yamamoto has joined #openstack-meeting-3 | 00:39 | |
| *** ricolin has joined #openstack-meeting-3 | 01:40 | |
| *** apetrich has quit IRC | 02:14 | |
| *** markvoelker has joined #openstack-meeting-3 | 04:03 | |
| *** markvoelker has quit IRC | 04:07 | |
| *** links has joined #openstack-meeting-3 | 05:32 | |
| *** trident has quit IRC | 06:39 | |
| *** trident has joined #openstack-meeting-3 | 06:42 | |
| *** markvoelker has joined #openstack-meeting-3 | 06:43 | |
| *** markvoelker has quit IRC | 06:48 | |
| *** ralonsoh has joined #openstack-meeting-3 | 07:47 | |
| *** mdelavergne has joined #openstack-meeting-3 | 07:56 | |
| ttx | #startmeeting large_scale_sig | 08:00 |
|---|---|---|
| openstack | Meeting started Wed Jul 8 08:00:46 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 08:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 08:00 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 08:00 | |
| openstack | The meeting name has been set to 'large_scale_sig' | 08:00 |
| ttx | #topic Rollcall | 08:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 08:00 | |
| ttx | Who is here for the Large Scale SIG meeting ? | 08:00 |
| mdelavergne | Hi! | 08:00 |
| amorin | Hello | 08:01 |
| ttx | France is up and awake | 08:01 |
| mdelavergne | Yep! | 08:01 |
| amorin | :p | 08:02 |
| ttx | I was hoping we'd have a few recruits from opendev week | 08:02 |
| ttx | Let's wait a few minutes and see if anyone else joins, otherwise it will be a short meeting | 08:03 |
| amorin | did you meet some new people during the summit? | 08:03 |
| ttx | masahito is on IRC, just pinged him | 08:04 |
| *** masahito has joined #openstack-meeting-3 | 08:04 | |
| masahito | hi | 08:05 |
| ttx | o/ | 08:05 |
| ttx | amorin: there were a couple of people who said in the chat they would join | 08:05 |
| ttx | I wonder how much our US-unfriendly time has been harming that will to join | 08:06 |
| ttx | I'll try to identify and contact those who expressed interest in the chat | 08:06 |
| amorin | ack | 08:06 |
| ttx | see if we need to rotate meeting time or anything | 08:06 |
| ttx | #action ttx to identify from the chat interested candidates from Opendev event and invite them to next meeting | 08:07 |
| ttx | masahito: did you have a good time with Opendev last week? | 08:07 |
| masahito | yeah. | 08:07 |
| ttx | Did anyone else have any specific feedback about it? | 08:07 |
| ttx | things to change before teh next one(s)? | 08:08 |
| mdelavergne | I couldn't attend personnally | 08:08 |
| ttx | ok, moving on then | 08:09 |
| ttx | #topic Progress on "Documenting large scale operations" goal | 08:09 |
| *** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)" | 08:09 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-documentation | 08:09 |
| amorin | my action is still in todo, wasnt able to move that forward | 08:10 |
| ttx | ok, will add it again for the sake of meeting notes | 08:10 |
| ttx | #action amorin to add some meat to the wiki page before we push the Nova doc patch further | 08:10 |
| ttx | amorin: you did, however, post your metrics/billing story on https://etherpad.openstack.org/p/large-scale-sig-documentation | 08:11 |
| ttx | thanks for that! | 08:11 |
| *** huyupeng has joined #openstack-meeting-3 | 08:11 | |
| ttx | Good example of fine-tuning ceilometer down to reasonable amounts of data | 08:12 |
| amorin | ah yes :p | 08:12 |
| ttx | Can you tell what you are using for metrics storage? | 08:12 |
| amorin | yes, we are using a mongodb | 08:12 |
| amorin | I can add some details on the etherpad | 08:12 |
| ttx | ok, and are those metrics directly use for billing, or do you have separate data collection for that? | 08:13 |
| amorin | so, the metrics, after beeing sent to the rabbit cluster | 08:13 |
| amorin | are consumed by a ceileter service (I dont remember the name of this one) | 08:14 |
| amorin | but we patched this service a little bit | 08:14 |
| amorin | to do some computation and push the results in a mongodb | 08:14 |
| amorin | then the date stored in mongo are used by some OVH internal robots to do the billing | 08:14 |
| amorin | I dont know if it's answering your question? | 08:15 |
| ttx | it does! thanks | 08:15 |
| ttx | A reminder for others to push their story so that we can compare approaches! | 08:16 |
| ttx | #action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation | 08:16 |
| ttx | Anything else on the "Documenting large scale operations" topic ? | 08:16 |
| ttx | #topic Progress on "Scaling within one cluster" goal | 08:17 |
| *** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)" | 08:17 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling | 08:17 |
| ttx | masahito: last week we discussed review teams for oslo.metrics. | 08:17 |
| ttx | last meeting I mean | 08:17 |
| ttx | I think it would make sense to add one of the original authors to the oslo.metrics core review team | 08:17 |
| ttx | should we add you? Or someone else from LINE? | 08:17 |
| masahito | Me is fine. | 08:18 |
| ttx | OK, adding you now | 08:18 |
| masahito | If needed, I'll add other folks. | 08:18 |
| ttx | ah! Actually you're already in | 08:18 |
| ttx | https://review.opendev.org/#/admin/groups/2096,members | 08:19 |
| ttx | I must have had that discussion with myself already | 08:19 |
| ttx | So for the moment that's us two + the other oslo-core | 08:19 |
| ttx | Next step is to add basic tests, so that we are reasonably confident we do not introduce regressions | 08:20 |
| *** mdelavergne31 has joined #openstack-meeting-3 | 08:20 | |
| ttx | Let us know if someone is interested in looking into that. I have it on my TODO list but it can take me a while before i get there | 08:20 |
| ttx | Anything else to report on the "Scaling within one cluster" topic? | 08:21 |
| *** mdelavergne has quit IRC | 08:21 | |
| ttx | #topic Large Scale SIG slide for OpenStack 10th birthday campaign | 08:21 |
| *** openstack changes topic to "Large Scale SIG slide for OpenStack 10th birthday campaign (Meeting topic: large_scale_sig)" | 08:21 | |
| ttx | We were offered a slide in the 10th birthday slide deck to promote the Large Scale SIG | 08:22 |
| ttx | I proposed the following content: | 08:22 |
| ttx | #link https://etherpad.opendev.org/p/JcyxFIi-F9I_cIi2QOyX | 08:22 |
| ttx | and it was accepted for the 10th anniversary deck | 08:22 |
| ttx | Let me know if you have comments, I can still get them to update it | 08:22 |
| amorin | sounds good to me | 08:23 |
| ttx | #topic Next meeting | 08:23 |
| *** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)" | 08:23 | |
| ttx | Let's plan to have it in two weeks... If as I contact potential newcomers the time is not working out for them, I'll start an email thread to discuss a better time | 08:24 |
| ttx | Does that work for everyone? | 08:24 |
| amorin | yes | 08:24 |
| ttx | We've been struggling to reach critical mass lately, so let's not lose the existing members :) | 08:24 |
| ttx | (+ I think Belmiro is in holidays right now) | 08:25 |
| ttx | #info next meeting: Jul 22, 8:00UTC | 08:25 |
| amorin | I will be off in 2 weeks btw | 08:25 |
| ttx | amorin: ok... I was considering skipping it to be honest... but I was also hoping the new people would be here today. In two weeks I can still get some of teh event momentum to get them to join | 08:26 |
| ttx | in 4 weeks they will probably have forgotten about it | 08:26 |
| ttx | So let's still plan in two weeks, even if we make it a very short meeting | 08:27 |
| amorin | yes, ok | 08:27 |
| masahito | I'm okay. | 08:27 |
| ttx | amorin: have a good time off! | 08:27 |
| ttx | Thanks everyone | 08:27 |
| amorin | thanks :p | 08:27 |
| amorin | before closing up | 08:27 |
| amorin | I'd like to share some stuff we are working on at OVH | 08:28 |
| ttx | please do! | 08:28 |
| amorin | or maybe I should do it on mailing list? | 08:28 |
| ttx | probably larger reach on the ML | 08:28 |
| ttx | but now the suspense is killing me | 08:28 |
| amorin | that will be quick :p | 08:28 |
| amorin | I am pretty sure other large scale groups already hit some DB size issue | 08:29 |
| amorin | so we built a tool, that we call osarchiver | 08:29 |
| amorin | https://github.com/ovh/osarchiver/ | 08:29 |
| ttx | I remember it being around for a while already | 08:29 |
| amorin | basically it's removing (or moving to other DB) some data from a running mysql cluster | 08:29 |
| amorin | yes | 08:29 |
| amorin | I dont know if it make sense, but maybe we can move it under opendev gerrit? | 08:30 |
| ttx | I think it makes sense, if only to make it more discoverable | 08:30 |
| amorin | ok, | 08:30 |
| ttx | There were talks about restarting osops during the opendev event last week | 08:30 |
| ttx | I'll give it some thought on where we could land it for maximum impact | 08:31 |
| amorin | great :p | 08:31 |
| ttx | amorin: could you post a ML thread about the tool and ask if it would make sense to land it somewhere in openstack space | 08:31 |
| mdelavergne31 | this seems nice | 08:31 |
| ttx | I'll take it from there | 08:31 |
| amorin | yes, will do | 08:31 |
| amorin | the second topic I'd like to talk about is a small patch we did on oslo.messaging: | 08:32 |
| amorin | https://review.opendev.org/#/c/735385/ | 08:32 |
| amorin | to add a ping endpoint on RPC | 08:32 |
| ttx | #action amorin to start a thread on osarchiver proposing to land it somewhere in openstack | 08:32 |
| amorin | behind this patch, we want to monitor our neutron agent (and nova compute) a little bit closer | 08:33 |
| ttx | OK, I'll make sure Sean sees it. I know he was off lately | 08:34 |
| amorin | I dont know if people from LINE or other companies are using something similar | 08:34 |
| amorin | anyway, I just wanted to let the group know about that :p | 08:35 |
| amorin | and that's all for me :p | 08:35 |
| ttx | i think Sean is trying to avoid having two features to do ping/healthcheck | 08:35 |
| masahito | btw, we directly monitor rabbit cluster for the healthcheck | 08:36 |
| amorin | masahito: what do you mean by that | 08:36 |
| amorin | ? | 08:36 |
| ttx | So it could be a good idea to discuss that in a ML thread too, and get operators to +1 your approach | 08:36 |
| amorin | ok, so I will do a mail about that also | 08:36 |
| amorin | masahito: I am really interested in your approach on monitoring this :p | 08:37 |
| masahito | amorin: oops, sorry I misundersood the patch. It's about endpoint for dispatcher. | 08:37 |
| ttx | you can add [largescale-sig] to the subject to encourage large deployers to chime in | 08:37 |
| amorin | yes ok | 08:38 |
| ttx | I'll add those two work items (osarchiver and the ping) to the etherpads so that we remember to track them | 08:38 |
| amorin | masahito: ok, please, check the patch, let me know, maybe I am completely wrong in my approach | 08:38 |
| amorin | thanks | 08:38 |
| ttx | #action amorin to start a [largescale-sig] thread about his middleware ping approach, SIG members can comment if that makes sense for them | 08:39 |
| ttx | #link https://review.opendev.org/#/c/735385/ | 08:40 |
| ttx | Ok I think I captured everything | 08:40 |
| ttx | Anything else before we close? | 08:40 |
| ttx | I'll take that as a no | 08:41 |
| ttx | Have a great week! | 08:41 |
| amorin | that's good for me | 08:41 |
| amorin | thanks, great week also | 08:41 |
| ttx | #endmeeting | 08:41 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 08:41 | |
| openstack | Meeting ended Wed Jul 8 08:41:34 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 08:41 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-07-08-08.00.html | 08:41 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-07-08-08.00.txt | 08:41 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-07-08-08.00.log.html | 08:41 |
| mdelavergne31 | thanks you too! | 08:41 |
| masahito | thanks | 08:41 |
| *** mdelavergne31 has quit IRC | 08:41 | |
| *** huyupeng has quit IRC | 08:45 | |
| *** yamamoto has quit IRC | 09:39 | |
| *** yamamoto has joined #openstack-meeting-3 | 09:42 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** yamamoto has joined #openstack-meeting-3 | 10:02 | |
| *** e0ne has joined #openstack-meeting-3 | 10:12 | |
| *** yamamoto has quit IRC | 10:18 | |
| *** apetrich has joined #openstack-meeting-3 | 10:44 | |
| *** yamamoto has joined #openstack-meeting-3 | 10:51 | |
| *** yamamoto has quit IRC | 10:56 | |
| *** masahito has quit IRC | 11:11 | |
| *** markvoelker has joined #openstack-meeting-3 | 11:34 | |
| *** raildo has joined #openstack-meeting-3 | 11:37 | |
| *** markvoelker has quit IRC | 11:39 | |
| *** lpetrut_ has joined #openstack-meeting-3 | 13:21 | |
| *** liuyulong has joined #openstack-meeting-3 | 13:51 | |
| *** liuyulong has quit IRC | 13:57 | |
| *** lpetrut_ has quit IRC | 14:28 | |
| *** maciejjozefczyk has joined #openstack-meeting-3 | 14:57 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
| openstack | Meeting started Wed Jul 8 15:00:34 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| slaweq | hi | 15:00 |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| ralonsoh | hi | 15:00 |
| njohnston | o/ | 15:01 |
| *** bcafarel has joined #openstack-meeting-3 | 15:01 | |
| bcafarel | o/ sorry somehow my irc client now had #openstack-meeting3 (without "-") | 15:01 |
| maciejjozefczyk | \o | 15:02 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:02 |
| slaweq | please open link and we can start :) | 15:02 |
| slaweq | #topic Actions from previous meetings | 15:03 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:03 | |
| slaweq | ralonsoh will check get_datapath_id issues in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html | 15:03 |
| ralonsoh | I didn't find the root cause of the timeout | 15:03 |
| ralonsoh | but I've pushed a patch for ovsdbapp | 15:03 |
| ralonsoh | to add a bit more information when the txn fails | 15:03 |
| ralonsoh | if that depends on the RXN or the TXN queue | 15:04 |
| ralonsoh | sorry for that... it's not easy to track those errors | 15:04 |
| slaweq | ok, so it's WIP | 15:05 |
| slaweq | thx ralonsoh | 15:05 |
| ralonsoh | yes | 15:05 |
| slaweq | please keep us updated if You will find anything | 15:05 |
| ralonsoh | sure | 15:05 |
| slaweq | btw. do we have LP for that? | 15:05 |
| ralonsoh | no | 15:05 |
| ralonsoh | just a patch in gerrit | 15:05 |
| ralonsoh | If I find another error like this one, I'll open a LP bug | 15:06 |
| slaweq | ok, thx | 15:06 |
| slaweq | ok, next one | 15:06 |
| slaweq | slaweq will check errors with non existing interface in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_63a/711425/11/check/neutron-functional/63ac4ca/testr_results.html | 15:06 |
| slaweq | I did not have time for that one | 15:06 |
| slaweq | and logs from the link above are already gone | 15:06 |
| slaweq | as I didn't saw it later, I will not bother with it anymore for now | 15:07 |
| slaweq | if it will happen again, I will try to check it | 15:07 |
| slaweq | next one | 15:07 |
| slaweq | slaweq to move non-voting jobs to experimental queue in EM branches | 15:07 |
| slaweq | Rocky: https://review.opendev.org/#/c/739668/ | 15:07 |
| slaweq | Queens: https://review.opendev.org/#/c/739669/ | 15:07 |
| slaweq | Pike: https://review.opendev.org/#/c/739910/ (thx bcafarell) | 15:07 |
| slaweq | and one more question - what about Ocata? | 15:07 |
| slaweq | do we still support this branch? | 15:08 |
| slaweq | I think so but I couldn't find any open patch for it | 15:08 |
| ralonsoh | that was set as EOL | 15:08 |
| ralonsoh | if I'm not wrong | 15:08 |
| bcafarel | I am looking for link I think it is EOL now | 15:08 |
| njohnston | EOL IIRC | 15:09 |
| bcafarel | ah "Moving the stable/ocata to 'Unmaintained' phase and then EOL" | 15:09 |
| slaweq | I don't see ocata-eol tag in the repo | 15:09 |
| bcafarel | maybe the paperwork is not fully completed for https://releases.openstack.org/ (and tags) | 15:09 |
| bcafarel | but at least on neutron side we did not see recent backport requests | 15:10 |
| njohnston | as I recall there was something about how individual projects can EOL it at their individual discretion | 15:10 |
| slaweq | ok, I will check its state after I will come back from the PTO | 15:10 |
| slaweq | ok, next one | 15:11 |
| slaweq | slaweq to investigate failures in test_create_router_set_gateway_with_fixed_ip test | 15:11 |
| slaweq | I didn't have time for that one too | 15:11 |
| slaweq | it's not very urgent as it happens in non-voting job | 15:11 |
| slaweq | but if someone wants to check, feel free to take it ;) | 15:12 |
| ralonsoh | I'll ping you tomorrow about this one | 15:12 |
| slaweq | ralonsoh: sure, thx | 15:12 |
| slaweq | ok, next one | 15:12 |
| slaweq | maciejjozefczyk to check neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers failure in ovn jobs | 15:12 |
| maciejjozefczyk | I havent time to check that one ;/ I'm going to take a look tomorrow | 15:13 |
| slaweq | ok | 15:14 |
| slaweq | I will assign it to You for next week | 15:14 |
| slaweq | #action maciejjozefczyk to check neutron_tempest_plugin.scenario.test_connectivity.NetworkConnectivityTest.test_connectivity_through_2_routers failure in ovn jobs | 15:14 |
| ralonsoh | (busy week for everyone) | 15:14 |
| slaweq | yeah | 15:15 |
| slaweq | as usual | 15:15 |
| slaweq | lets move to the next topic | 15:15 |
| slaweq | #topic Stadium projects | 15:15 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:15 | |
| slaweq | zuul v3 migration | 15:15 |
| slaweq | I finally pushed patch for last missing neutron job: https://review.opendev.org/729591 - lets see how it will work | 15:15 |
| ralonsoh | cool! | 15:16 |
| slaweq | it still didn't even started | 15:16 |
| slaweq | but I will keep an eye on it | 15:16 |
| bcafarel | it will before your PTO :) | 15:16 |
| ralonsoh | hehehe | 15:16 |
| slaweq | bcafarel: I hope so | 15:16 |
| slaweq | :) | 15:16 |
| slaweq | ok, anything else regardig stadium for today? | 15:17 |
| slaweq | ok, I guess it means "no" | 15:19 |
| slaweq | so lets move on | 15:19 |
| slaweq | #topic Switch to Ubuntu Focal | 15:19 |
| *** openstack changes topic to "Switch to Ubuntu Focal (Meeting topic: neutron_ci)" | 15:19 | |
| slaweq | any updates on that? | 15:20 |
| ralonsoh | devstack patch was merged | 15:20 |
| ralonsoh | https://review.opendev.org/#/c/704831/ | 15:20 |
| ralonsoh | but https://review.opendev.org/#/c/734304 is failing | 15:20 |
| ralonsoh | I need to review the logs for FT/fullstack | 15:20 |
| maciejjozefczyk | we need this one first to land https://review.opendev.org/#/c/737984/ | 15:21 |
| bcafarel | maciejjozefczyk: will it fix errors like those seen in https://review.opendev.org/#/c/738163/ ? | 15:22 |
| bcafarel | Exception: Invalid directories: /usr/local/share/ovn, /usr/share/ovn, /usr/local/share/openvswitch, /usr/share/openvswitch, None | 15:22 |
| bcafarel | (some other jobs failing too, but I did not have time to dig further) | 15:22 |
| ralonsoh | maciejjozefczyk, but apart from your patch, FTs/fullstack are failing because in Focal python-openvswitch does not exist | 15:23 |
| ralonsoh | E: Unable to locate package python-openvswitch | 15:23 |
| slaweq | :/ | 15:23 |
| slaweq | should we ask someone from Ubuntu team for help with that? | 15:23 |
| ralonsoh | we should, yes | 15:24 |
| maciejjozefczyk | ahh | 15:24 |
| maciejjozefczyk | python3-? | 15:24 |
| ralonsoh | maciejjozefczyk, I'll check that today | 15:24 |
| bcafarel | ralonsoh: I have a quick hack to pull python3- for test in 738163 (hardcoded just for testing) | 15:24 |
| slaweq | ok, so package name was renamed to python3-openvswitch in Focal? Is that correct? | 15:26 |
| ralonsoh | I need to check that | 15:26 |
| bcafarel | yup | 15:26 |
| slaweq | ok, thx ralonsoh and bcafarel for working on this | 15:26 |
| slaweq | I think we can move on to the next topic | 15:27 |
| slaweq | #topic Stable branches | 15:27 |
| *** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:27 | |
| slaweq | Ussuri dashboard: http://grafana.openstack.org/d/pM54U-Kiz/neutron-failure-rate-previous-stable-release?orgId=1 | 15:27 |
| slaweq | Train dashboard: http://grafana.openstack.org/d/dCFVU-Kik/neutron-failure-rate-older-stable-release?orgId=1 | 15:27 |
| bcafarel | stable branches maybe, stable ci not these days | 15:27 |
| slaweq | LOL | 15:28 |
| maciejjozefczyk | ;/ | 15:28 |
| slaweq | bcafarel: so we can say (not so)stable branches | 15:28 |
| slaweq | ok? | 15:28 |
| slaweq | :D | 15:28 |
| bcafarel | :) approved! | 15:28 |
| bcafarel | for the previous issues we are almost good now finally (pike https://review.opendev.org/#/c/739456/ waiting on second +2) | 15:28 |
| bcafarel | but new pep8 error (isort) needs a requirements backport which failed CI https://review.opendev.org/#/c/739912 | 15:29 |
| slaweq | bcafarel: +W | 15:29 |
| slaweq | and we need same backport all the way to stein, right? | 15:30 |
| bcafarel | apparently yes | 15:30 |
| slaweq | :/ | 15:30 |
| slaweq | at least we are good with pep8 issue in master now :) | 15:31 |
| ralonsoh | what are those errors in py36 tests? | 15:31 |
| ralonsoh | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_513/739912/1/check/openstack-tox-py36/51376c8/testr_results.html | 15:31 |
| bcafarel | yes I am not sure where they come from (but appear in both py36/py37 and all other backports) | 15:32 |
| ralonsoh | please, keep an eye on https://github.com/PyCQA/pylint/issues/3722 | 15:33 |
| slaweq | maybe yet another ci blocker issue (because we still have not enough) ;) | 15:34 |
| slaweq | ok, lets move on | 15:35 |
| slaweq | #topic Grafana | 15:35 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:35 | |
| slaweq | #link http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:35 |
| slaweq | I see one thing which worry me a bit | 15:36 |
| slaweq | neutron-ovn-tempest-full-multinode-ovs-master is failing 100% of times since some time | 15:36 |
| slaweq | I opened LP for that https://bugs.launchpad.net/neutron/+bug/1886807 | 15:37 |
| openstack | Launchpad bug 1886807 in neutron "neutron-ovn-tempest-full-multinode-ovs-master job is failing 100% times" [High,Confirmed] | 15:37 |
| slaweq | idk if that is job configuration or something like that but seems that in many tests it's failing due to ssh issues | 15:37 |
| slaweq | other than that, I think that things are more or less ok in master branch now | 15:38 |
| maciejjozefczyk | yes thats strange because I don't remember anything that could break this... We also didn't changed the ovn branch | 15:38 |
| maciejjozefczyk | yes | 15:38 |
| maciejjozefczyk | I'm gonna take a look tomorrow what was pushed to ovn master recently | 15:39 |
| slaweq | maciejjozefczyk: at first glance I suspect some issue with infra, it's multinode job and maybe ssh to vms which lands on the second node don't work | 15:39 |
| slaweq | that's something what I would try to check first | 15:39 |
| maciejjozefczyk | yep | 15:39 |
| slaweq | btw https://review.opendev.org/737984 just merged :) | 15:40 |
| slaweq | #action maciejjozefczyk to check failing neutron-ovn-tempest-full-multinode-ovs-master job | 15:40 |
| slaweq | thx maciejjozefczyk for looking into that | 15:40 |
| maciejjozefczyk | ack | 15:40 |
| maciejjozefczyk | ;) | 15:40 |
| slaweq | ok, lets move on | 15:41 |
| slaweq | #topic fullstack/functional | 15:41 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 15:41 | |
| slaweq | I found one new (for me at least) issue | 15:41 |
| slaweq | neutron.tests.functional.test_server.TestWsgiServer.test_restart_wsgi_on_sighup_multiple_workers | 15:41 |
| slaweq | https://f7a63aeb9edd557a2176-4740624f0848c8c3257f704064a4516f.ssl.cf2.rackcdn.com/736026/4/gate/neutron-functional/d7d5c47/testr_results.html | 15:41 |
| slaweq | did You saw something like that recently? | 15:41 |
| bcafarel | not that I remember (and that code was not touched recently) | 15:43 |
| ralonsoh | the worker didn't restart, isn't it? | 15:43 |
| *** links has quit IRC | 15:43 | |
| slaweq | bcafarel: yes, but I saw it I think twice this week | 15:43 |
| slaweq | hmm | 15:44 |
| slaweq | I see such error in the test logs: https://f7a63aeb9edd557a2176-4740624f0848c8c3257f704064a4516f.ssl.cf2.rackcdn.com/736026/4/gate/neutron-functional/d7d5c47/controller/logs/dsvm-functional-logs/neutron.tests.functional.test_server.TestWsgiServer.test_restart_wsgi_on_sighup_multiple_workers.txt | 15:44 |
| slaweq | do You think it may be related? | 15:44 |
| slaweq | actually I think I saw something similar in the past | 15:46 |
| slaweq | when You look at the test's code: https://github.com/openstack/neutron/blob/master/neutron/tests/functional/test_server.py#L163 | 15:46 |
| slaweq | it failed on condition which checkes if some specific file was created and had got expected size | 15:47 |
| slaweq | so it could be that size was wrong or there was no this file at all | 15:47 |
| slaweq | I will change that condition to add some logging to it | 15:47 |
| slaweq | so will be easier to check what happend there | 15:47 |
| njohnston | +1 | 15:49 |
| slaweq | #action slaweq to change condition in the TestNeutronServer to have better logging | 15:49 |
| slaweq | ok, we can get back to this issue when we will know more what happen there | 15:50 |
| bcafarel | +1 | 15:50 |
| slaweq | lets move on | 15:50 |
| slaweq | #topic Tempest/Scenario | 15:50 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 15:50 | |
| slaweq | I had only this issue related to ovn job but we talked about it already | 15:50 |
| slaweq | so just 2 short info | 15:50 |
| slaweq | and ask for reviews :) | 15:51 |
| slaweq | Please review https://review.opendev.org/#/c/736186/ | 15:51 |
| slaweq | I had to rebase it to resolve some conflict there | 15:51 |
| slaweq | and also I increased timeouts in singlenode tempest jobs: https://review.opendev.org/739955 | 15:52 |
| slaweq | please take a look at this one too | 15:52 |
| slaweq | and that's all from me about scenario jobs | 15:52 |
| slaweq | #topic On demand agenda | 15:52 |
| *** openstack changes topic to "On demand agenda (Meeting topic: neutron_ci)" | 15:52 | |
| slaweq | do You have anything else You want to talk about today? | 15:53 |
| ralonsoh | no thanks | 15:53 |
| slaweq | I have one quick item | 15:53 |
| slaweq | as You know next 2 weeks I will be on PTO | 15:53 |
| maciejjozefczyk | slaweq, enjoy :) | 15:54 |
| slaweq | do You want me to cancel this ci meeting or ralonsoh can You chair it? | 15:54 |
| ralonsoh | I can (probably) | 15:54 |
| slaweq | I'm asking You because You are listed as co-chair of this meeting already :) | 15:54 |
| slaweq | ralonsoh: thx a lot | 15:54 |
| slaweq | so I will not cancel it, please run it if You will have time or cancel if You want | 15:55 |
| slaweq | that's all from my side | 15:55 |
| slaweq | thx for attending today | 15:55 |
| slaweq | and have a great rest of the week | 15:55 |
| ralonsoh | bye | 15:55 |
| slaweq | #endmeeting | 15:55 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:55 | |
| openstack | Meeting ended Wed Jul 8 15:55:34 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:55 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-08-15.00.html | 15:55 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-08-15.00.txt | 15:55 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-07-08-15.00.log.html | 15:55 |
| bcafarel | o/ | 15:56 |
| maciejjozefczyk | bb | 15:56 |
| njohnston | o/ | 15:57 |
| *** markvoelker has joined #openstack-meeting-3 | 16:53 | |
| *** markvoelker has quit IRC | 16:53 | |
| *** ricolin has quit IRC | 18:00 | |
| *** e0ne has quit IRC | 18:03 | |
| *** ralonsoh has quit IRC | 18:07 | |
| *** maciejjozefczyk has quit IRC | 18:56 | |
| *** gmann_ has joined #openstack-meeting-3 | 19:17 | |
| *** gmann_ is now known as gmann | 19:18 | |
| *** apetrich has quit IRC | 19:31 | |
| *** apetrich has joined #openstack-meeting-3 | 19:31 | |
| *** beisner has joined #openstack-meeting-3 | 19:42 | |
| *** csatari has joined #openstack-meeting-3 | 19:49 | |
| *** srwilkers has joined #openstack-meeting-3 | 19:59 | |
| *** knikolla has joined #openstack-meeting-3 | 20:05 | |
| *** raildo has quit IRC | 21:01 | |
| *** slaweq has quit IRC | 21:54 | |
| *** slaweq has joined #openstack-meeting-3 | 21:56 | |
| *** slaweq has quit IRC | 22:00 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!