| *** zigo has quit IRC | 00:53 | |
| *** zigo has joined #openstack-meeting-3 | 02:27 | |
| *** baojg has quit IRC | 02:27 | |
| *** haleyb has quit IRC | 02:27 | |
| *** baojg has joined #openstack-meeting-3 | 02:33 | |
| *** baojg has left #openstack-meeting-3 | 02:33 | |
| *** baojg has joined #openstack-meeting-3 | 02:33 | |
| *** yamamoto has joined #openstack-meeting-3 | 03:11 | |
| *** ianychoi_ is now known as ianychoi | 06:16 | |
| *** apetrich has joined #openstack-meeting-3 | 06:26 | |
| *** ralonsoh has joined #openstack-meeting-3 | 06:29 | |
| *** e0ne has joined #openstack-meeting-3 | 06:41 | |
| *** e0ne has quit IRC | 06:45 | |
| *** apetrich has quit IRC | 06:56 | |
| *** apetrich has joined #openstack-meeting-3 | 07:14 | |
| *** tosky has joined #openstack-meeting-3 | 07:19 | |
| *** slaweq has joined #openstack-meeting-3 | 07:27 | |
| *** priteau has joined #openstack-meeting-3 | 07:50 | |
| *** mdelavergne has joined #openstack-meeting-3 | 07:51 | |
| ttx | #startmeeting large_scale_sig | 08:00 |
|---|---|---|
| openstack | Meeting started Wed Sep 23 08:00:19 2020 UTC and is due to finish in 60 minutes. The chair is ttx. Information about MeetBot at http://wiki.debian.org/MeetBot. | 08:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 08:00 |
| *** openstack changes topic to " (Meeting topic: large_scale_sig)" | 08:00 | |
| openstack | The meeting name has been set to 'large_scale_sig' | 08:00 |
| ttx | #topic Rollcall | 08:00 |
| *** openstack changes topic to "Rollcall (Meeting topic: large_scale_sig)" | 08:00 | |
| ttx | Who is here for the Large Scale SIG meeting ? | 08:00 |
| mdelavergne | Hi | 08:00 |
| ttx | mdelavergne: hi! | 08:00 |
| ttx | pinging amorin | 08:01 |
| amorin | hello! | 08:01 |
| amorin | thank for the ping, completely forgot :( | 08:01 |
| ttx | belmoreira seems offline | 08:02 |
| ttx | masahito might be around | 08:03 |
| ttx | While we wait and are between French people -- I had a nice chat with the ops at Societe generale yesterday. They are in the first stages of scaling (think 100->1000 and are a bit in the unknown, which reinforces the need for documenting what to expect when scaling up/out | 08:04 |
| ttx | I told then as they transition to 100->1000 they will likely hit the need to scale out at some point | 08:05 |
| ttx | them* | 08:05 |
| mdelavergne | Nice! | 08:05 |
| ttx | But it's unclear what to watch for, and what will fail first | 08:05 |
| mdelavergne | They might have some surprises :( | 08:06 |
| *** masahito has joined #openstack-meeting-3 | 08:06 | |
| ttx | (they are already sharding RabbitMQ a bit so I expect that won't be the first thing) | 08:06 |
| ttx | masahito: hi! | 08:06 |
| ttx | alright, let's get started | 08:06 |
| ttx | Our agenda for today is at: | 08:06 |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-meeting | 08:06 |
| ttx | #topic Meaningful monitoring | 08:07 |
| *** openstack changes topic to "Meaningful monitoring (Meeting topic: large_scale_sig)" | 08:07 | |
| masahito | hi! sorry at late. We're hitting a rabbit mq down right now :-( | 08:07 |
| ttx | Two weeks ago we had our second EU-US meeting, and we actually had new participants joining! | 08:07 |
| ttx | masahito: we can catch up later, you should focus on the urgent things :) | 08:07 |
| ttx | One of the things the newcomers were interested in pushing was the idea of "meaningful monitoring" | 08:08 |
| ttx | which I think could be a third workstream for the group | 08:08 |
| ttx | as it's a pretty common challenge across large scale deployments | 08:08 |
| ttx | and also matches the "how do you do metrics/billing in your deployment" question | 08:08 |
| ttx | Thoughts on that? | 08:09 |
| ttx | Is that something interesting to work on, or is it basically a solved problem? | 08:10 |
| mdelavergne | It's definitely a good point, but where is it possible to begin on tackling this problem? | 08:11 |
| *** belmoreira has joined #openstack-meeting-3 | 08:12 | |
| ttx | I'd start by defining the need, and discuss whether the current tooling available is covering it or not | 08:12 |
| belmoreira | sorry for joining late | 08:12 |
| mdelavergne | hi! | 08:12 |
| ttx | Also collecting stories | 08:12 |
| ttx | belmoreira: hi! | 08:12 |
| ttx | We were discussing "meaningful monitoring" as a potential new workstream for the group | 08:13 |
| ttx | The new US participants (James Penick and Erik from Blizzard) were interested in that | 08:13 |
| ttx | belmoreira: does that sound like a valuable thing for the group to explore? | 08:14 |
| belmoreira | +1 from me. With all the work with oslo metrics looks a good next step | 08:14 |
| ttx | OK I'll start documenting that and come up with a plan we can bootstrap from the US+EU side of the meeting | 08:14 |
| belmoreira | there was any discussion regarding notifications? | 08:15 |
| ttx | #action ttx to draft a plan to tackle "meaningful monitoring" as a new SIG workstream | 08:15 |
| ttx | belmoreira: no, but I'd say that's part of "meaningful" | 08:15 |
| ttx | like... "actionable monitoring" | 08:15 |
| belmoreira | there are projects and produce a lot of notifications but no easy/interesting way to get "meaningful" from them | 08:16 |
| belmoreira | s/and/that | 08:16 |
| ttx | ack | 08:16 |
| ttx | I think that's what they meant. There is data being produced, but it's difficult to derive operational status from it | 08:17 |
| ttx | moving on to our other workstreams... | 08:18 |
| ttx | #topic Progress on "Documenting large scale operations" goal | 08:18 |
| *** openstack changes topic to "Progress on "Documenting large scale operations" goal (Meeting topic: large_scale_sig)" | 08:18 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-documentation | 08:18 |
| ttx | On the OSops side, the repository was just created: | 08:18 |
| ttx | #link https://opendev.org/openstack/osops | 08:18 |
| ttx | It's managed under the "Ops Docs and Tooling" SIG, which is obviously closely related to ours | 08:18 |
| ttx | but probably a better host for tools that are not specific for large scale | 08:19 |
| ttx | amorin: you should reach out to smcginnis (or the SIG) to see where you could propose OSarchiver | 08:19 |
| ttx | Personally I'd say it should have its own directory under tools/ as it is something you intend to maintain... | 08:19 |
| belmoreira | I think this is a great initiative and for me this should be the "transition" path for the projects adopt/integrate this tools in the projects | 08:20 |
| belmoreira | this shows that ops still need something that the projects don't offer | 08:21 |
| ttx | ok, I think we lost amorin, but he will pick up the message later I expect | 08:21 |
| ttx | The other open task on that workstream is to collect more metrics/billing stories. | 08:21 |
| ttx | A task we could just move to the new "Meaningful monitoring" workstream, as part of the "current status" collection | 08:21 |
| ttx | Nothing new has been contributed there, so I will push back the action there | 08:22 |
| ttx | #action all to describe briefly how you solved metrics/billing in your deployment in https://etherpad.openstack.org/p/large-scale-sig-documentation | 08:22 |
| ttx | Anything else on that topic? | 08:22 |
| ttx | #topic Progress on "Scaling within one cluster" goal | 08:23 |
| *** openstack changes topic to "Progress on "Scaling within one cluster" goal (Meeting topic: large_scale_sig)" | 08:23 | |
| ttx | #link https://etherpad.openstack.org/p/large-scale-sig-cluster-scaling | 08:23 |
| ttx | The work on middleware ping has landed, so we are left with oslo.metrics work... | 08:23 |
| ttx | I did push a basic test framework for oslo.metrics at | 08:23 |
| ttx | #link https://review.opendev.org/#/c/752262/ | 08:23 |
| ttx | I'll continue by trying to add meaningful tests | 08:24 |
| ttx | (but need to dive deeper into the code first) | 08:24 |
| ttx | masahito: any progress on posting your latest changes ? | 08:24 |
| ttx | amorin: did you discuss testing it internally at OVH? | 08:24 |
| masahito | sorry.... nothing now. | 08:24 |
| ttx | That's ok, I'll just push back the action item | 08:25 |
| amorin | ttx yes | 08:25 |
| ttx | #action masahito to push latest patches to oslo.metrics | 08:25 |
| amorin | I talked about it | 08:25 |
| amorin | we did not yet tested it | 08:25 |
| amorin | but we are considering doing it | 08:25 |
| ttx | ok, great! Keep us posted | 08:25 |
| amorin | yup | 08:25 |
| ttx | Our second subtopic is around scaling stories on: | 08:25 |
| ttx | #link https://etherpad.openstack.org/p/scaling-stories | 08:25 |
| ttx | Nothing new posted there... our next action is the forum session at the summit in a month. | 08:26 |
| ttx | Anything else on this topic? | 08:26 |
| belmoreira | I can contribute for the forum session | 08:26 |
| belmoreira | if there is space | 08:26 |
| ttx | belmoreira: theer is! The goal will be to create a lively conversation space to encourage others to share their own journey scaling up/out | 08:27 |
| ttx | So it's good to have 2-3 people signed up to bootstrap the discussion | 08:27 |
| belmoreira | great, a good topic would be moving from cells to regions because scalability issues | 08:27 |
| ttx | We have James Penick (or someone from his team) + Arnaud that have volunteered to help too | 08:27 |
| amorin | belmoreira: nice topic indeed! | 08:28 |
| amorin | very curious about that | 08:28 |
| amorin | is it because of neutron? | 08:28 |
| belmoreira | amorin of course :) | 08:28 |
| ttx | Maybe we can organize the discussion along different stages. Like "first thing that fails as you add more nodes" (scaling up woes), then moving from cells to regions (scaling out woes) | 08:29 |
| belmoreira | ttx yes, that makes sense | 08:30 |
| ttx | I'm personally very interested in gettign an order of magnitude of number of nodes/activity for a single cluster, as it's the top question I'm being asked by new companies starting to scale up their deployments | 08:30 |
| ttx | like "what should we expect? What should we watch?" | 08:31 |
| amorin | we have input at OVH about that | 08:31 |
| ttx | And I'm always like... "well, somewhere between 100 and 1000 physical servers, something will fail and force you to scale out" | 08:31 |
| ttx | which is not super-helpful :) | 08:31 |
| ttx | also I'm not even sure that 100-1000 range is current in Ussuri | 08:32 |
| amorin | there are plenty of differents things that could break | 08:32 |
| amorin | and it depends also on what kind of usage you are doing with openstack | 08:32 |
| amorin | that's why it's hard to answer I think | 08:32 |
| ttx | amorin: right, type of activity will change the numbers completely | 08:33 |
| belmoreira | and the risk that you are willing to take | 08:33 |
| ttx | Also it's normal that "plenty of things can fail", since if there was only one thing, we would have handled that low hanging fruit already | 08:33 |
| amorin | yup | 08:34 |
| ttx | but still having guidance would be super helpful for the ops at that stage of their openstack life | 08:34 |
| ttx | it can be fuzzy | 08:34 |
| ttx | but just knowing that others have goine through that and survived... is good | 08:34 |
| ttx | #topic PTG/Summit plans update | 08:35 |
| *** openstack changes topic to "PTG/Summit plans update (Meeting topic: large_scale_sig)" | 08:35 | |
| ttx | So I did file a Forum session on scaling stories, which was just accepted/scheduled | 08:35 |
| * ttx checks when | 08:36 | |
| ttx | https://www.openstack.org/summit/2020/summit-schedule/events/24746/share-your-openstack-scaling-story | 08:36 |
| ttx | Tuesday, October 20, 7:30am-8:15am CT (2:30pm - 3:15pm UTC) | 08:36 |
| amorin | great | 08:36 |
| ttx | hmm not CT, PT | 08:37 |
| belmoreira | friendly time slot | 08:37 |
| ttx | #info Forum session Tuesday, October 20, 7:30am-8:15am PT (2:30pm - 3:15pm UTC) | 08:37 |
| ttx | yes, it's good for us | 08:37 |
| ttx | Let me start an etherpad about it | 08:38 |
| ttx | #link https://etherpad.opendev.org/p/w-forum-scaling-stories | 08:38 |
| ttx | I dumped the base themes, feel free to add to that | 08:41 |
| ttx | we only have 45 minutes so I would keep it to 4-5 questions max | 08:41 |
| amorin | ack | 08:42 |
| belmoreira | +1 | 08:43 |
| ttx | Alright let's refine that doc between now and the event | 08:43 |
| ttx | Otherwise for PTG week we will have two one-hour sessions: | 08:43 |
| ttx | #info PTG meeting Wednesday Oct 28 7UTC-8UTC and 16UTC-17UTC | 08:44 |
| ttx | I suspect those will be approved but will let you know when I know more | 08:44 |
| ttx | The idea will be to encourage people from the Forum session to follow up at the PTG hours | 08:44 |
| ttx | (and then to sign them up to join the SIG meeting) | 08:44 |
| ttx | We'll likely use both to introduce the various SIG workstreams and ask for input/help | 08:45 |
| ttx | (both PTG hours) | 08:45 |
| ttx | Any question on that | 08:45 |
| ttx | ? | 08:45 |
| ttx | ok then | 08:46 |
| ttx | #topic Next meeting | 08:46 |
| *** openstack changes topic to "Next meeting (Meeting topic: large_scale_sig)" | 08:46 | |
| ttx | Next meeting will be US-EU-friendly on Oct 7, 16utc. | 08:46 |
| ttx | Then we'll have summit and PTG weeks, and then go back to our regular cadence in November. | 08:46 |
| ttx | #info next meeting: Oct 7, 16:00UTC | 08:47 |
| ttx | Anything else before we close? | 08:47 |
| ttx | Alright! Thanks everyone! | 08:48 |
| ttx | #endmeeting | 08:48 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 08:48 | |
| openstack | Meeting ended Wed Sep 23 08:48:16 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 08:48 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-09-23-08.00.html | 08:48 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-09-23-08.00.txt | 08:48 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/large_scale_sig/2020/large_scale_sig.2020-09-23-08.00.log.html | 08:48 |
| mdelavergne | Thanks, see you | 08:48 |
| amorin | thanks | 08:48 |
| *** mdelavergne has quit IRC | 08:48 | |
| *** masahito has quit IRC | 08:55 | |
| *** priteau has quit IRC | 08:56 | |
| *** lpetrut has joined #openstack-meeting-3 | 09:16 | |
| *** yamamoto has quit IRC | 09:59 | |
| *** lkoranda has joined #openstack-meeting-3 | 10:11 | |
| *** slaweq has quit IRC | 10:17 | |
| *** yamamoto has joined #openstack-meeting-3 | 10:19 | |
| *** yamamoto has quit IRC | 10:26 | |
| *** slaweq has joined #openstack-meeting-3 | 10:27 | |
| *** yamamoto has joined #openstack-meeting-3 | 10:34 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** e0ne has joined #openstack-meeting-3 | 11:12 | |
| *** yamamoto has joined #openstack-meeting-3 | 11:15 | |
| *** raildo has joined #openstack-meeting-3 | 11:25 | |
| *** yamamoto has quit IRC | 11:25 | |
| *** tosky_ has joined #openstack-meeting-3 | 12:02 | |
| *** tosky is now known as Guest73219 | 12:03 | |
| *** tosky_ is now known as tosky | 12:03 | |
| *** yamamoto has joined #openstack-meeting-3 | 12:03 | |
| *** Guest73219 has quit IRC | 12:05 | |
| *** yamamoto has quit IRC | 12:16 | |
| *** priteau has joined #openstack-meeting-3 | 12:35 | |
| *** baojg has quit IRC | 12:41 | |
| *** baojg has joined #openstack-meeting-3 | 12:41 | |
| *** yamamoto has joined #openstack-meeting-3 | 12:54 | |
| *** yamamoto has quit IRC | 12:55 | |
| *** yamamoto has joined #openstack-meeting-3 | 12:55 | |
| *** yamamoto has quit IRC | 13:10 | |
| *** tosky_ has joined #openstack-meeting-3 | 13:26 | |
| *** tosky is now known as Guest13336 | 13:26 | |
| *** tosky_ is now known as tosky | 13:26 | |
| *** Guest13336 has quit IRC | 13:28 | |
| *** tosky_ has joined #openstack-meeting-3 | 13:29 | |
| *** tosky is now known as Guest61020 | 13:31 | |
| *** tosky_ is now known as tosky | 13:31 | |
| *** Guest61020 has quit IRC | 13:32 | |
| *** elod is now known as elod_afk | 13:41 | |
| *** yamamoto has joined #openstack-meeting-3 | 13:50 | |
| *** yamamoto has quit IRC | 14:02 | |
| *** lkoranda has quit IRC | 14:04 | |
| *** artom has joined #openstack-meeting-3 | 14:10 | |
| *** apetrich has quit IRC | 14:19 | |
| *** elod_afk is now known as elod | 14:53 | |
| slaweq | #startmeeting neutron_ci | 15:00 |
| openstack | Meeting started Wed Sep 23 15:00:14 2020 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 15:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 15:00 |
| ralonsoh | hi | 15:00 |
| *** lajoskatona has joined #openstack-meeting-3 | 15:01 | |
| lajoskatona | Hi | 15:01 |
| bcafarel | o/ | 15:01 |
| slaweq | ok, I think we can start | 15:01 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 15:01 |
| slaweq | Please open now :) | 15:01 |
| slaweq | #topic Actions from previous meetings | 15:02 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 15:02 | |
| slaweq | ralonsoh to check timing out neutron-ovn-tempest-full-multinode-ovs-master jobs - bug https://bugs.launchpad.net/neutron/+bug/1886807 | 15:02 |
| openstack | slaweq: Error: Could not gather data from Launchpad for bug #1886807 (https://launchpad.net/bugs/1886807). The error has been logged | 15:02 |
| ralonsoh | slaweq, sorry, I didn't start this task | 15:03 |
| slaweq | ralonsoh: np | 15:03 |
| slaweq | should I assign this one to You for next week? | 15:03 |
| ralonsoh | yes | 15:03 |
| bcafarel | probably the bot complaining launchpad is slooow | 15:03 |
| slaweq | #action ralonsoh to check timing out neutron-ovn-tempest-full-multinode-ovs-master jobs - bug https://bugs.launchpad.net/neutron/+bug/1886807 | 15:04 |
| openstack | slaweq: Error: Could not gather data from Launchpad for bug #1886807 (https://launchpad.net/bugs/1886807). The error has been logged | 15:04 |
| slaweq | thx ralonsoh | 15:04 |
| slaweq | ok, next one | 15:04 |
| slaweq | ralonsoh to check issue with pep8 failures like https://zuul.opendev.org/t/openstack/build/6c8fbf9b97b44139bf1d70b9c85455bb | 15:04 |
| ralonsoh | yes, I pushed a patch for this one | 15:04 |
| ralonsoh | and Oleg another one on top of mine | 15:05 |
| ralonsoh | let me check | 15:05 |
| ralonsoh | https://review.opendev.org/#/c/748594/ | 15:05 |
| ralonsoh | https://review.opendev.org/#/c/749320/ | 15:05 |
| ralonsoh | No more issues reported | 15:05 |
| slaweq | thx ralonsoh :) | 15:06 |
| slaweq | ok, and the last one from last week | 15:06 |
| slaweq | bcafarel to check failing periodic fedora job | 15:06 |
| * bcafarel looks for links | 15:06 | |
| bcafarel | short version fedora jobs are broken on f31, infra aims to switch to 32 | 15:07 |
| slaweq | bcafarel: and on 32 it works, right? | 15:07 |
| slaweq | if yes, I'm ok with that | 15:07 |
| bcafarel | with https://review.opendev.org/#/c/750292 as depends-on that job works | 15:08 |
| tosky | f31 is out of support in ~2 months anyway | 15:08 |
| tosky | one month after the release of f33 | 15:08 |
| slaweq | bcafarel++ ok, thx for checking | 15:08 |
| bcafarel | but the devstack fedora job fails in linked review on neutron-dhcp-agent issues (quite a few "no such file or directory" on pid and conf files...) | 15:08 |
| slaweq | ouch | 15:10 |
| slaweq | https://zuul.opendev.org/t/openstack/build/d7d455eadaac4f5081fca5dd7197b6ea/log/controller/logs/screen-q-dhcp.txt?severity=4 | 15:10 |
| slaweq | it can be something with python 3.8 maybe | 15:10 |
| slaweq | as f32 is using that version of python already | 15:10 |
| slaweq | hmm, ubuntu focal also uses python 3.8 | 15:11 |
| bcafarel | focal is on 3.8 too (just checked) | 15:12 |
| slaweq | and we have there errors like https://zuul.opendev.org/t/openstack/build/af52abc771a541548d8e528af79a524d/log/controller/logs/screen-q-dhcp.txt?severity=4 | 15:12 |
| slaweq | but job passed | 15:12 |
| slaweq | anyone wants to report bug and investigate that? | 15:12 |
| ralonsoh | maybe next week | 15:12 |
| slaweq | ok, so I will report a bug and we will see next week | 15:13 |
| slaweq | it's not critical for us I think | 15:13 |
| slaweq | but worth to have it tracked | 15:13 |
| slaweq | #action slaweq to report bug with neutron-dhcp-agent and Fedora 32 | 15:13 |
| ralonsoh | TBH, this should not be logged as an error | 15:13 |
| slaweq | ralonsoh: so maybe that's redherring | 15:14 |
| ralonsoh | we are just checking if the interface exists and has MAC | 15:14 |
| ralonsoh | could be, yes | 15:14 |
| lajoskatona | Than perhaps warning should be enough? | 15:15 |
| slaweq | or even debug | 15:15 |
| ralonsoh | info | 15:15 |
| ralonsoh | we return false | 15:15 |
| ralonsoh | not an exception | 15:15 |
| ralonsoh | we should inform about this | 15:15 |
| ralonsoh | that could be used in an active wait, waiting for the interface to be ready | 15:15 |
| lajoskatona | I run a quick search on logstash and as I see it happens quite often, but the jobs are passing | 15:16 |
| slaweq | ok, but still there are failed tests on that job | 15:16 |
| slaweq | and we have to check why so many tests are failing due to ssh issue | 15:16 |
| slaweq | I will report this bug and we will see | 15:17 |
| bcafarel | could it be similar to failing tests we see in focal? | 15:17 |
| slaweq | ralonsoh: if You have time please report a bug for this logging thing | 15:17 |
| ralonsoh | sure | 15:17 |
| slaweq | bcafarel: nah, that's other tests | 15:17 |
| slaweq | so I don't think it's the same thing really | 15:17 |
| bcafarel | it would have been too easy to have one single root cause :( | 15:18 |
| slaweq | bcafarel: exactly | 15:18 |
| slaweq | ok, I think we can move on now | 15:18 |
| slaweq | #topic Switch to Ubuntu Focal | 15:18 |
| *** openstack changes topic to "Switch to Ubuntu Focal (Meeting topic: neutron_ci)" | 15:18 | |
| slaweq | Etherpad: https://etherpad.opendev.org/p/neutron-victoria-switch_to_focal | 15:19 |
| slaweq | I updated it a bit today | 15:19 |
| slaweq | I know that gmann is going to pull the trigger and switch all devstack based jobs to focal today | 15:19 |
| slaweq | so we may have problems with some jobs and in such case we should propose patch(es) to switch unstable jobs to bionic temporary | 15:20 |
| slaweq | and then focus on debugging those issues | 15:20 |
| gmann | yeah these are on gate now https://review.opendev.org/#/c/731207/ https://review.opendev.org/#/c/734700/ | 15:20 |
| slaweq | for unstable neutron-tempest-plugin scenario jobs I proposed today https://review.opendev.org/#/c/753552/ | 15:21 |
| slaweq | hopefully it will help | 15:21 |
| slaweq | but lets see | 15:21 |
| bcafarel | if https://review.opendev.org/#/c/738163/ passes fine on top of it it will be nice | 15:22 |
| bcafarel | sorry https://review.opendev.org/#/c/748367 | 15:22 |
| bcafarel | (too many tabs with "Focal" in it...) | 15:23 |
| slaweq | thx bcafarel for taking care of it | 15:23 |
| slaweq | any other updates regarding this topic? | 15:23 |
| bcafarel | did the review-priority patches get merged? after the jobs are switched it may be a bit longer to merge stuff | 15:24 |
| slaweq | bcafarel: which review-priority patches? | 15:25 |
| bcafarel | https://review.opendev.org/746347 / https://review.opendev.org/747774 | 15:25 |
| bcafarel | hopefully we can get them in with all the focal switch + ovs hash issue | 15:26 |
| lajoskatona | no they are failing with the ovs build issue :-( | 15:26 |
| slaweq | yep, both failed on ovn jobs :/ | 15:27 |
| slaweq | ok, I think we can move on | 15:27 |
| slaweq | #topic Stadium projects | 15:27 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 15:27 | |
| slaweq | anything new regarding ci of stadium projects? | 15:27 |
| lajoskatona | nothing from my side | 15:28 |
| slaweq | ok, thx lajoskatona | 15:29 |
| slaweq | so lets move on | 15:29 |
| slaweq | #topic Stable branches | 15:29 |
| *** openstack changes topic to "Stable branches (Meeting topic: neutron_ci)" | 15:29 | |
| slaweq | anything here? bcafarel? | 15:29 |
| bcafarel | we may see some activity on not spotted focal issues there, but so far so good | 15:29 |
| bcafarel | (late answer for stadium projects) | 15:29 |
| bcafarel | for stable, a few rechecks here and there needed, but backports got in nicely in last week (up to queens) | 15:30 |
| ralonsoh | bcafarel, hmmmmm | 15:31 |
| ralonsoh | https://review.opendev.org/#/c/752387/ | 15:31 |
| ralonsoh | still waiting!! | 15:31 |
| ralonsoh | ok, sorry, the OVN CI jobs | 15:31 |
| slaweq | :) | 15:32 |
| slaweq | ok, so lets move on | 15:32 |
| bcafarel | ah ussuri yep it probably is broken now | 15:32 |
| slaweq | #topic Grafana | 15:32 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 15:32 | |
| slaweq | #link http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1 | 15:33 |
| slaweq | what worries me is neutron-functional-with-uwsgi job which is failing pretty often recently | 15:34 |
| slaweq | but when I was checking some jobs' results today, it was mostly some ovs or db timeouts | 15:34 |
| ralonsoh | agree with this | 15:34 |
| slaweq | so looked for me that this may be just overloaded infra maybe | 15:35 |
| *** baojg has quit IRC | 15:35 | |
| slaweq | so unless You have some specific example of issues to investigate there, I would say to wait few more days and see how it will be | 15:35 |
| *** baojg has joined #openstack-meeting-3 | 15:36 | |
| *** baojg has quit IRC | 15:37 | |
| slaweq | and that is basically all from my side for today | 15:37 |
| *** baojg has joined #openstack-meeting-3 | 15:37 | |
| slaweq | do You have anything else You want to discuss today? | 15:38 |
| ralonsoh | no thanks | 15:38 |
| slaweq | if not I will give You 20 minutes back :) | 15:38 |
| slaweq | ok, so thx for attending the meeting | 15:39 |
| slaweq | and see You online :) | 15:39 |
| slaweq | o/ | 15:39 |
| bcafarel | o/ and let's see how focal goes | 15:39 |
| lajoskatona | Bye | 15:39 |
| slaweq | #endmeeting | 15:39 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:39 | |
| openstack | Meeting ended Wed Sep 23 15:39:19 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:39 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-09-23-15.00.html | 15:39 |
| ralonsoh | bye | 15:39 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-09-23-15.00.txt | 15:39 |
| *** lajoskatona has left #openstack-meeting-3 | 15:39 | |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2020/neutron_ci.2020-09-23-15.00.log.html | 15:39 |
| *** lpetrut has quit IRC | 15:45 | |
| *** belmoreira has quit IRC | 15:57 | |
| *** belmoreira has joined #openstack-meeting-3 | 15:59 | |
| *** tosky has quit IRC | 15:59 | |
| *** belmoreira has quit IRC | 16:00 | |
| *** baojg has quit IRC | 16:12 | |
| *** baojg has joined #openstack-meeting-3 | 16:13 | |
| *** apetrich has joined #openstack-meeting-3 | 16:21 | |
| *** ralonsoh has quit IRC | 16:28 | |
| *** haleyb has joined #openstack-meeting-3 | 16:37 | |
| *** etp has quit IRC | 16:49 | |
| *** etp has joined #openstack-meeting-3 | 16:52 | |
| *** etp has quit IRC | 17:00 | |
| *** priteau has quit IRC | 17:02 | |
| *** etp has joined #openstack-meeting-3 | 17:02 | |
| *** e0ne has quit IRC | 17:56 | |
| *** yamamoto has joined #openstack-meeting-3 | 18:00 | |
| *** yamamoto has quit IRC | 18:04 | |
| *** baojg has quit IRC | 18:36 | |
| *** baojg has joined #openstack-meeting-3 | 18:37 | |
| *** baojg has quit IRC | 18:57 | |
| *** baojg has joined #openstack-meeting-3 | 18:58 | |
| *** tosky has joined #openstack-meeting-3 | 19:15 | |
| *** slaweq has quit IRC | 19:25 | |
| *** yamamoto has joined #openstack-meeting-3 | 20:01 | |
| *** yamamoto has quit IRC | 20:05 | |
| *** slaweq has joined #openstack-meeting-3 | 20:14 | |
| *** slaweq has quit IRC | 20:16 | |
| *** slaweq has joined #openstack-meeting-3 | 20:17 | |
| *** apetrich has quit IRC | 20:32 | |
| *** baojg has quit IRC | 20:44 | |
| *** gmann has quit IRC | 20:50 | |
| *** hemanth_n has quit IRC | 20:50 | |
| *** knikolla has quit IRC | 20:50 | |
| *** srwilkers has quit IRC | 20:51 | |
| *** knikolla has joined #openstack-meeting-3 | 20:52 | |
| *** srwilkers has joined #openstack-meeting-3 | 20:52 | |
| *** hemanth_n has joined #openstack-meeting-3 | 20:53 | |
| *** gmann has joined #openstack-meeting-3 | 20:53 | |
| *** mlavalle has joined #openstack-meeting-3 | 21:20 | |
| *** baojg has joined #openstack-meeting-3 | 21:37 | |
| *** yamamoto has joined #openstack-meeting-3 | 22:01 | |
| *** yamamoto has quit IRC | 22:06 | |
| *** yamamoto has joined #openstack-meeting-3 | 22:12 | |
| *** slaweq has quit IRC | 22:13 | |
| *** knikolla has quit IRC | 22:50 | |
| *** hemanth_n has quit IRC | 22:50 | |
| *** jungleboyj has quit IRC | 22:50 | |
| *** srwilkers has quit IRC | 22:50 | |
| *** hemanth_n has joined #openstack-meeting-3 | 22:51 | |
| *** knikolla has joined #openstack-meeting-3 | 22:51 | |
| *** jungleboyj has joined #openstack-meeting-3 | 22:51 | |
| *** srwilkers has joined #openstack-meeting-3 | 22:54 | |
| *** hemanth_n has quit IRC | 22:56 | |
| *** hemanth_n has joined #openstack-meeting-3 | 22:59 | |
| *** elod has quit IRC | 23:32 | |
| *** elod has joined #openstack-meeting-3 | 23:34 | |
| *** etp has quit IRC | 23:35 | |
| *** etp has joined #openstack-meeting-3 | 23:36 | |
| *** tosky has quit IRC | 23:49 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!