| opendevreview | Merged openstack/nova master: add hypervisor version weigher https://review.opendev.org/c/openstack/nova/+/880231 | 01:29 |
|---|---|---|
| dvo-plv_ | gibi, sean-k-monney: Hello, could you pelase verify os-trait patch: https://review.opendev.org/c/openstack/os-traits/+/876069 to unblock zuul verification for nova patch, cause it fails according to the deendencies | 08:06 |
| dvo-plv_ | sean-k-mooney: sorry, I have made a mistake in your nick | 08:07 |
| gibi | dvo-plv_: the os-traits patch looks good to me. After that merges, we need to propose a new os-traits release, so that you can depend on that release of os-traits in your nova patch | 08:23 |
| bauzas | dang, it took me a while to figure out we were mocking the default policies by a fixture... | 08:23 |
| bauzas | context : https://39770df410ace4902fc0-3eec3a080da8356877afe7c3a8a6cc53.ssl.cf5.rackcdn.com/881562/1/check/openstack-tox-py39/c658228/testr_results.html | 08:23 |
| bauzas | now, I know, we also have a fake policy data : https://github.com/openstack/nova/blob/master/nova/tests/unit/fake_policy.py#L16 | 08:25 |
| bauzas | TIL. | 08:25 |
| dvo-plv_ | gibi: I have only update requirements.txt for nova, or there is some additional doc files what should be updated too | 08:34 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add a new policy for cold-migrate with host https://review.opendev.org/c/openstack/nova/+/881562 | 08:48 |
| sahid | o/ | 09:00 |
| gibi | dvo-plv_: you only need to updat the requirements.txt on the nova side. | 09:09 |
| bauzas | gibi: but before doing it, we need to have an upper-constraints change, right ? :) | 09:11 |
| gibi | bauzas: right, I above mentioned to dvo-plv_ that we need an os-traits release | 09:11 |
| gibi | but then there will be either an automatic upper constraints bump | 09:11 |
| gibi | or do a manual one in the requirements repo | 09:11 |
| gibi | then the requirements.txt on the nova side can be bumped | 09:11 |
| gibi | in the meantime we should not forget to bump the min version in placement too | 09:12 |
| bauzas | dvo-plv_: fwiw, I +Wd https://review.opendev.org/c/openstack/os-traits/+/876069 with a comment | 09:13 |
| opendevreview | Merged openstack/os-traits master: Add 'COMPUTE_NET_VIRTIO_PACKED' https://review.opendev.org/c/openstack/os-traits/+/876069 | 09:22 |
| dvo-plv_ | bauzas: Could I update patch with comment, ot it should be done in some another way. because it alredy merged? | 09:32 |
| bauzas | dvo-plv_: just add another patch as a follow-up :) | 09:32 |
| dvo-plv_ | okay, sure | 09:34 |
| kashyap | Can anyone remind me: for picking the min libvirt/QEMU versions what OSes do we care? I'm guessing the "usual": Debian, Ubuntu, Fedora, and RHEL | 09:35 |
| kashyap | In the past we've also considered openSUSE / SLES. Should they be considered too? | 09:36 |
| * kashyap looks | 09:36 | |
| dvo-plv_ | gibi: Does os-trait release has some estimates? I would like to plan our next activities according to them | 09:36 |
| gibi | dvo-plv_: if you ping me after the os-trait patch lands I can propose the release quickly and it should not take more than couple day to get that release proposal landed | 09:37 |
| kashyap | bauzas: Have we got a name after Bobcat? | 09:49 |
| bauzas | kashyap: nope, afaik not yet | 09:54 |
| opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | 09:54 |
| dvo-plv_ | bauzas: I added comment ot the trait: https://review.opendev.org/c/openstack/os-traits/+/882249 | 10:10 |
| opendevreview | Merged openstack/nova master: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/862964 | 10:27 |
| opendevreview | Merged openstack/nova master: Have host look for CPU controller of cgroupsv2 location. https://review.opendev.org/c/openstack/nova/+/873127 | 10:27 |
| opendevreview | Merged openstack/nova master: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882160 | 10:31 |
| sean-k-mooney | bauzas: i wont get to it for a week or two but im going to work on a backlog spec for AZ enhancements. i feel liek ther eare enough rough edges and pain point that collecting them in one docs would make sense but while i thik i know how to adrss some of them i dont know how to adress all fo them so im hoping that if we aggreated the pain point some cominalities will emerge | 11:15 |
| sean-k-mooney | that can then be adress in the C cycle after some tought | 11:15 |
| opendevreview | Danylo Vodopianov proposed openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | 11:20 |
| opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Selete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | 12:45 |
| opendevreview | Amit Uniyal proposed openstack/nova master: WIP: Delete dangling volumes https://review.opendev.org/c/openstack/nova/+/882284 | 13:00 |
| bauzas | tell me I'm wrong but os-port-interfaces and os-security-groups API calls proxy to Neutron, right? | 13:17 |
| sean-k-mooney | yes | 13:17 |
| bauzas | if so, looks like we have a performance regression in the gate https://review.opendev.org/c/openstack/nova/+/882052 | 13:17 |
| bauzas | I got two timeouts | 13:17 |
| sean-k-mooney | tiem outs where | 13:18 |
| sean-k-mooney | i dont see them in the zuul jobs | 13:18 |
| sean-k-mooney | you mean in a test | 13:18 |
| bauzas | yes, see my comments | 13:19 |
| bauzas | you'll see the stacktrace | 13:19 |
| bauzas | I haven't looked yet at Neutron APIs | 13:19 |
| sean-k-mooney | urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='173.231.255.102', port=80): Read timed out. (read timeout=60) | 13:20 |
| sean-k-mooney | that could be related to memory pressure or a slow node | 13:20 |
| bauzas | yes, or no workers available | 13:21 |
| sean-k-mooney | looking at the memory tracker the low point was 1342048 | 13:21 |
| sean-k-mooney | so i dont think that is the issue 130MB is not much but its not terible for these jobs | 13:21 |
| bauzas | read timeouts can happen on a synchronous call if the call takes more than 60 secs to return | 13:22 |
| bauzas | so this could be something else but just a memory issue | 13:22 |
| sean-k-mooney | ya im wondering if the neutron server was deadlocked on something | 13:22 |
| bauzas | like a DB lock | 13:22 |
| sean-k-mooney | ya i know they had issue with db deadlocks last year | 13:23 |
| sean-k-mooney | im seeing quite a few trace backs | 13:24 |
| sean-k-mooney | realed to security gorups that apprently dont exist | 13:25 |
| sean-k-mooney | https://paste.opendev.org/show/bxCPO7Jo1jSzEoRNwHKS/ | 13:27 |
| sean-k-mooney | this seams to be the relevent logs https://paste.opendev.org/show/bMFDSLaKg5TZVpRa3OXD/ | 13:30 |
| sean-k-mooney | there are tracbacks before and after that | 13:31 |
| sean-k-mooney | bauzas: if you care its roughly here https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d/log/controller/logs/screen-q-svc.txt#53526 | 13:32 |
| bauzas | I'll check whether there is an upstream gate report on it | 13:33 |
| bauzas | bug report* | 13:33 |
| sean-k-mooney | neutron does seam to be respondign to the get request May 04 12:02:50.886890 np0033942429 neutron-server[162430]: INFO neutron.wsgi [req-49c6f58c-b29d-4c83-8ad1-5a588383533f req-bc14944a-4c80-4107-9ce5-de32f17088f8 tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] 173.231.255.102 "GET | 13:34 |
| sean-k-mooney | /v2.0/security-groups/b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76 HTTP/1.1" status: 200 len: 1820 time: 0.1247852 | 13:34 |
| sean-k-mooney | but i dont knwo when nova actully made it | 13:35 |
| sean-k-mooney | it could have been sitting in apache waiting for other requests for a while | 13:36 |
| opendevreview | Merged openstack/os-traits master: Comment to the trait was added https://review.opendev.org/c/openstack/os-traits/+/882249 | 13:37 |
| opendevreview | Sylvain Bauza proposed openstack/nova stable/2023.1: Fix get_segments_id with subnets without segment_id https://review.opendev.org/c/openstack/nova/+/882293 | 13:38 |
| bauzas | lajoskatona: slaweq: ralonsoh: not sure you already know but it seems we have some Neutron API deadlock in the gate ^ | 13:41 |
| bauzas | and I wasn't able to find an exiting neutron bug report that relates to it | 13:42 |
| ralonsoh | bauzas, but I'm not sure about this method, I don't see in the "list_subnets" method where the neutron client is returning the segment_id | 13:44 |
| ralonsoh | I see that in "show_subnet" | 13:45 |
| bauzas | ralonsoh: ah sorry, you looked at the wrong thing due to my lack of explanation :) | 13:45 |
| bauzas | ralonsoh: see rather the discussion we had between sean-k-mooney and I, the context is https://zuul.opendev.org/t/openstack/build/6e0e0879058f439d8e494cfb68e5211d | 13:46 |
| ralonsoh | yes, we have the same problem | 13:46 |
| ralonsoh | you discussed that with lajoskatona | 13:46 |
| bauzas | and https://zuul.opendev.org/t/openstack/build/de4b1d075b0745aab2bdec9bc8319877 | 13:46 |
| bauzas | then I forgot about it | 13:47 |
| ralonsoh | bauzas, if recall correctly (maybe I'm wrong), the issue with create_security_group_rule was that we where calling the nova client | 13:48 |
| ralonsoh | sean-k-mooney, I'm checking https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e0/882052/1/gate/grenade-skip-level-always/6e0e087/testr_results.html | 13:52 |
| ralonsoh | and I don't see where Neutron is failing | 13:53 |
| bauzas | ralonsoh: I can be wrong but in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e0/882052/1/gate/grenade-skip-level-always/6e0e087/testr_results.html we have test_security_group_rules_create[id-850795d7-d4d3-4e55-b527-a774c0123d3a,smoke] failing on File "/opt/stack/new/tempest/tempest/lib/services/compute/security_group_rules_client.py", line 35, in create_security_group_rule | 13:55 |
| bauzas | this is an API call to the os-security-groups API which belongs to Nova | 13:55 |
| bauzas | but this actually is a proxy call to Neutron | 13:56 |
| ralonsoh | yes and I'm reviewing the logs and I can't find any error | 13:56 |
| ralonsoh | I'm checking the same call in other tempest runs | 13:56 |
| bauzas | that's unfortunate that given the timeout, we don't have the request-id | 14:00 |
| ralonsoh | bauzas, but what call is timing out? because is not the SG creation | 14:03 |
| bauzas | ralonsoh: this is failing on https://github.com/openstack/tempest/blob/master/tempest/api/compute/security_groups/test_security_group_rules.py#L67 | 14:06 |
| bauzas | I think I can trace the nova-api call with May 04 12:02:50.750575 np0033942429 devstack@n-api.service[165740]: DEBUG nova.api.openstack.wsgi [None req-49c6f58c-b29d-4c83-8ad1-5a588383533f tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] Action: 'create', calling method: <function Controller.__getattribute__.<locals>.version_select at 0x7f7d37c793f0>, body: {"security_group_ | 14:06 |
| bauzas | rule": {"parent_group_id": "b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76", "ip_protocol": "tcp", "from_port": 22, "to_port": 22}} {{(pid=165740) _process_stack /opt/stack/new/nova/nova/api/openstack/wsgi.py:511}} | 14:06 |
| bauzas | so now I'm trying to find logs from req-49c6f58c-b29d-4c83-8ad1-5a588383533f | 14:08 |
| bauzas | look, I can see the req-id in neutron May 04 12:02:50.886890 np0033942429 neutron-server[162430]: INFO neutron.wsgi [req-49c6f58c-b29d-4c83-8ad1-5a588383533f req-bc14944a-4c80-4107-9ce5-de32f17088f8 tempest-SecurityGroupRulesTestJSON-878638434 tempest-SecurityGroupRulesTestJSON-878638434-project-member] 173.231.255.102 "GET /v2.0/security-groups/b90c2750-9bbd-44fc-9ad2-2eba1fcc4c76 HTTP/1.1" status: 200 len: 1820 time: 0.1247852 | 14:10 |
| ralonsoh | but this is a get call, not the SG rule creation | 14:12 |
| ralonsoh | well, actually this is the req-id of the SG rule creation | 14:12 |
| lajoskatona | bauzas, ralonsoh: Hi, this is related to https://bugs.launchpad.net/neutron/+bug/2015065 as I see (sorry perhaps you already linked the lp link) | 14:12 |
| bauzas | lajoskatona: excellent, will mention it then | 14:13 |
| bauzas | lajoskatona: fwiw, this is not only on neutron-ovs-grenade-dvr-multinode but also on our jobs too :) | 14:13 |
| lajoskatona | bauzas: yeah, with rodolfo we added later that as comment with opensearch links and examples also from tempest | 14:15 |
| bauzas | cool ++ | 14:15 |
| ralonsoh | bauzas, this is using neutronclient, right? | 14:16 |
| bauzas | good question | 14:17 |
| bauzas | afaik yes, but recently we wanted to use the sdk | 14:17 |
| bauzas | I don't think we merged any change yet, but lemme doublecheck | 14:17 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882313 | 14:17 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/2023.1: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882314 | 14:17 |
| ralonsoh | I don't see any change in neutronclient nor nova | 14:18 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/zed: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882315 | 14:18 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/zed: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882316 | 14:18 |
| *** blarnath is now known as d34dh0r53 | 14:18 | |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/yoga: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882317 | 14:18 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/yoga: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882318 | 14:18 |
| bauzas | ralonsoh: you're correct, still neutronclient https://github.com/openstack/nova/blob/master/nova/network/security_group_api.py#L379 | 14:19 |
| ralonsoh | yes | 14:19 |
| bauzas | anyway, I need to go get my daughter from school, bbiab (~15 mins) | 14:19 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882319 | 14:20 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882320 | 14:20 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882321 | 14:21 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/wallaby: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882322 | 14:21 |
| dvo-plv_ | gibi: we have finished with os-traits patch. Could you please propose the release | 14:42 |
| bauzas | dvo-plv_: I can do it | 14:56 |
| dvo-plv_ | great, thank you | 14:58 |
| gibi | bauzas: if you have cycles right now then thanks for proposing it | 15:01 |
| bauzas | just doing it now | 15:01 |
| gibi | bauzas++ | 15:02 |
| bauzas | Uggla: your approval is nice for https://review.opendev.org/c/openstack/releases/+/882325 | 15:09 |
| bauzas | gibi: dvo-plv_: ^ | 15:09 |
| Uggla | bauzas, I will have a look | 15:10 |
| sean-k-mooney | dvo-plv_: when that is released plese ensure you bump the min requirement for os-traits in the nova patch that uses it | 15:11 |
| bauzas | sean-k-mooney: he'll need to wait for the upper-constraints bot patch to be generated first if I'm not wrong | 15:17 |
| sean-k-mooney | for it to pass ci | 15:17 |
| sean-k-mooney | but they can do the bump | 15:17 |
| bauzas | correct, but dvo-plv_'s main concern was that zuul wasn't happy with its patch, hence his request to super-fast-approve traits | 15:22 |
| bauzas | so I'm just explaining that the release is only half of the definition of done | 15:22 |
| sean-k-mooney | dvo-plv_: for what its worth this should all be resolved early next week | 15:25 |
| sean-k-mooney | i would expect the release to hapeen todya or tomorrow and the uper constraits bump should land shortly after | 15:25 |
| bauzas | well, there are humans behind the releases approval and the upper-constraints patch approval too, so I'd give them a few more days | 15:27 |
| bauzas | but yeah, early next week seems a reasonable ETA | 15:27 |
| gibi | I can cosy up to elodilles to get attention on https://review.opendev.org/c/openstack/releases/+/882325 :) | 15:32 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/yoga: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882317 | 15:52 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/yoga: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882318 | 15:52 |
| Uggla | bauzas, in case you haven't seen https://review.opendev.org/c/openstack/manila-specs/+/881934 and https://review.opendev.org/c/openstack/manila-specs/+/881894 | 16:01 |
| bauzas | Uggla: ack, nice to see | 16:03 |
| Uggla | bauzas, yep pretty cool. | 16:04 |
| bauzas | I'll to review both specs one day | 16:04 |
| Uggla | bauzas, despite I have not fully read them yet. | 16:04 |
| bauzas | (saying one day because I didn't had time this week yet to do what I promised) | 16:04 |
| elodilles | bauzas gibi : there is a problem with the os-traits release version bump, see my comment on the patch | 16:12 |
| bauzas | elodilles: hmmm, I'm a bit torn but I trust you | 16:13 |
| bauzas | elodilles: we haven't removed any support afaik | 16:13 |
| elodilles | bauzas: py36 & py37 classifiers were removed from setup.cfg, which means pypi would not offer the released package to py36 and py37 systems | 16:15 |
| bauzas | yup yup I understand the reasoning | 16:15 |
| elodilles | note that we always bump MAJOR version in these cases | 16:15 |
| elodilles | so while the code might still support py36 and/or py37, still we are stating it isn't :/ i'm not fond of dropping these classifiers either, but that went through most of the deliverables already | 16:17 |
| bauzas | ok, thanks for the catch anyway | 16:18 |
| bauzas | elodilles++ | 16:18 |
| bauzas | elodilles: https://review.opendev.org/c/openstack/releases/+/882325 is updated | 16:18 |
| elodilles | bauzas: thanks, +2'd | 16:19 |
| bauzas | gracias | 16:20 |
| elodilles | de nada | 16:21 |
| gibi | elodilles: good catch | 16:24 |
| gibi | thanks for looking at it | 16:24 |
| sean-k-mooney | elodilles: python 3.7 wupport was remvoed form placment and nova quite a while ago | 16:42 |
| sean-k-mooney | elodilles: i guess we missed those in os-traits | 16:42 |
| sean-k-mooney | elodilles: i dont think python 36 has been supproted since wallaby | 16:43 |
| sean-k-mooney | that was the last release as we kept it for centos 8 support | 16:43 |
| sean-k-mooney | we can do the major bump just point out that this is long overdue | 16:43 |
| elodilles | sean-k-mooney: yes the last time py3.7 listed amongst supported runtimes was victoria: https://governance.openstack.org/tc/reference/runtimes/victoria.html | 16:57 |
| elodilles | sean-k-mooney: what i meant is that there is difference between we don't list as supported and not allowing it to being installed on non-supported python | 16:59 |
| elodilles | if we know that there is some clear incompatibility, then it is OK to remove the classifier for that python version. (though i understand we might not realise when we introduce such changes and that might be a worse situation) | 17:01 |
| dvo-plv_ | sean-k-mooney: yes, sure, guys already mentioned that I have to udpate requirments for nova. I will do it, when os-trait packag will be ready | 17:59 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Reproduce bug 1995153 https://review.opendev.org/c/openstack/nova/+/882319 | 18:32 |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/xena: Save cell socket correctly when updating host NUMA topology https://review.opendev.org/c/openstack/nova/+/882320 | 18:32 |
| opendevreview | Merged openstack/nova stable/yoga: Fix LM rollback w/o multi port bindings extension https://review.opendev.org/c/openstack/nova/+/839922 | 20:48 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!