Tuesday, 2023-06-20

*** auniyal0 is now known as auniyal07:45
noonedeadpunkHey folks! Completely forgot to ask that during PTG last week :( But wdyt about this bug report? For me it sounds more like a nova issue, that service fails and do not try to re-auth? https://bugs.launchpad.net/openstack-ansible/+bug/202337010:56
noonedeadpunkSo what's hapenning in OSA here, is that we reset a password (in 99.9% to exactly same one). While I kinda agree we should not do that at the first place, but this should not break service either?10:57
*** Continuity__ is now known as Continuity11:24
stephenfinmelwitt: Sorry for the delay, reviewed that DB archive series. I addressed your feedback on https://review.opendev.org/c/openstack/nova/+/880669/ also. Thanks for that11:26
opendevreviewMerged openstack/nova master: testing: Fix and robustify archive_deleted_rows test  https://review.opendev.org/c/openstack/nova/+/87705511:49
sean-k-mooneynoonedeadpunk: reset the password where12:52
sean-k-mooneynoonedeadpunk: you update the db password? rabbit?12:52
noonedeadpunksean-k-mooney: in keystone. 12:52
noonedeadpunkso issued previously tokens are invalidated12:52
noonedeadpunkdespite password is kinda the same12:53
sean-k-mooneyoh that sound like a keystone issue then12:53
sean-k-mooneypassword or what by the way12:53
noonedeadpunkbut it happens kinda only when asking neutron api by nova service, and only first request fails, and then it re-asks for the token I believe12:53
sean-k-mooneythe password of the nova user? or the enduser creating the vm12:53
noonedeadpunkof the nova user12:53
noonedeadpunkand system service user per say12:54
sean-k-mooneyright so if you udpate it in keystoen you need to update it in the nova.conf and then restart the nova service12:54
sean-k-mooneyunless this is marked as mutable a sighup wont be enough12:54
noonedeadpunkyeah, but it's reseted to the same value12:54
noonedeadpunkSo you kinda don't need to update conf12:55
noonedeadpunkAnd thus - don't restart service12:55
sean-k-mooneyregardless of if its updateed to the same or diffent value i woudl expect keystone to acppet existing non expired tokens12:55
noonedeadpunkAnd it wasn't an issue in X and even in Y I believe12:55
sean-k-mooneyso that why i said it sounded like a keystone issue12:56
sean-k-mooneyit sounds like they have made updating the password invalidation all current tokens12:56
sean-k-mooneywhich would be a major api brake12:56
sean-k-mooneyas in it should requrie a major api bump as that woudl completely change the meaning of the token experation12:57
noonedeadpunkI'm not sure that's exactly the case, was more guessing what hapens12:58
noonedeadpunkAs I can hardly imagine any other reason for such behaviour12:58
noonedeadpunkThe interesting part also, that according to the report, only neutron connection breaks12:59
sean-k-mooneywell if the user and password is the same when we create the admin client it shoudl get a new token12:59
sean-k-mooneythis is presumable port binding?12:59
noonedeadpunkYeah, I believe so. 13:00
sean-k-mooneywe generally dont use nova for anything else we use the end useers token13:00
sean-k-mooneyya so portbidning is admin only so we use the admin clinet withthe user/password form teh config13:00
noonedeadpunkAnd it fails with `nova.exception.NeutronAdminCredentialConfigurationInvalid`13:00
noonedeadpunk(which is weird exception kinda)13:01
noonedeadpunkOne thing ofc, is that we will just stop resetting password to the same value. This was added quite some time ago to ensure that password is correct13:02
noonedeadpunkbut even though I'd say it shouldn't fail that way. So maybe just having some retry or try/except would be good to get new token when needed13:08
anskiyI'm the reporter of the https://bugs.launchpad.net/openstack-ansible/+bug/2023370. I see this behaviour in Y too, except that in Z it does recover, whilst in Y one needs to restart Nova API service. I haven't tested this in X, but I can do it if that would help. 13:08
noonedeadpunkanskiy: I think latest Y should be quite the same?13:08
noonedeadpunkAs OSSA-2023-003 was backported to Y13:09
anskiynoonedeadpunk: oh, yes, true.13:09
noonedeadpunkno idea though13:09
* bauzas reading back the channel13:25
bauzasnoonedeadpunk: sean-k-mooney: I don't know how Neutron ensures Keystone authd, but at least for Nova, all our meat is provided by keystonemiddleware13:27
bauzasI assume the same for Neutron13:27
bauzasso yeah, sounds a Keystone issue, probably in middleware13:27
bauzasI don't see why we would need to recycle the nova-api services, but if middleware does some sort of caching, this may explain13:28
noonedeadpunkyeah, ok, will try to bother keystone folks then13:29
bauzasnoonedeadpunk: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html13:30
bauzasparticularly https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time13:30
noonedeadpunkyeah, I'm pretty sure memcached is being configured13:31
bauzashttps://blog.benhall.me.uk/wp-content/uploads/2015/01/cache-invalidation.png13:32
bauzaspretty well known stance, I assume13:32
noonedeadpunkthe werid thing though, is that on subsequent request it jsut passes13:32
noonedeadpunkwhich made me thinking if it's jsut some simple retry that can be added somewhere13:33
bauzaswell, sounds a design problem to me, which is never simple to solve :)13:34
bauzaslike, memcached has to be flushed against a user if that user password changed13:34
bauzasunrelated, was the f... is going on with nova-live-migration jobs ? I got two CI failures on the same patch with the same error https://review.opendev.org/c/openstack/nova/+/88623213:36
stephenfinbauzas: melwitt commented about that last week13:40
stephenfin<melwitt> fyi just got two CI failures on the same patch in the nova-live-migration and nova-grenade-multinode jobs https://bugs.launchpad.net/neutron/+bug/1940425 and it reminded me I have been seeing this failure often lately13:40
stephenfin<melwitt> we likely need to look into this to improve gate fail rate13:40
bauzasstephenfin: ack and the story is ?13:40
bauzasok13:40
* bauzas grabs gibi's opensearch13:40
bauzashttps://zuul.openstack.org/builds?job_name=nova-live-migration&skip=0 sounds pretty bad if you want MHO13:41
bauzasralonsoh: lajoskatona: you'll be happy to hear we have some kind of neutron hard failure on nova-live-migration jobs13:42
ralonsohbauzas, please!13:42
bauzaswhat a nice day, isn't it ? :)13:43
lajoskatonabauzas: this one perhaps: https://bugs.launchpad.net/neutron/+bug/2024160 ?13:43
ralonsohdo you know what is happening?13:43
bauzasralonsoh: one of the failing logs https://ffd910c1fba2c4e789e5-da0bf4b2b0817e3979adc64c27f459a5.ssl.cf5.rackcdn.com/880719/5/check/nova-live-migration/08a7455/testr_results.html13:43
ralonsohso this is related only with the trunk ports, right?13:43
bauzasyeah13:43
ralonsohok, we have the triage meeting in 20 mins13:43
bauzastempest.api.compute.admin.test_live_migration.LiveMigrationTest is failing13:43
bauzasdue to a failed assert13:43
ralonsohI'll raise it as urgent13:43
ralonsohcan you disable this test temporarily?13:44
ralonsohhold on13:44
bauzaslajoskatona: sounds related13:44
ralonsohis this CI in a gate/check queue?13:44
ralonsohI mean, is this blocking you from merging patches?13:44
lajoskatonaralonsoh, bauzas: yeah by the logs seems bug/2024160 , and to the OVN version bump13:45
ralonsohbauzas, is this in a gate/check queue? is this blocking you from merging patches?13:46
ralonsohif so, I'll disable this test temporarily13:46
bauzasralonsoh: (sorry, was elsewhere) yeah this is a voting job13:48
bauzason both check and gate13:48
bauzasTempest seems impacted too13:48
ralonsohok, I'll disable make this test unstable temporarily13:48
ralonsohbut we'll start working on this right now13:49
bauzasI can add a skip test in zuul13:50
bauzasideally, we should have a tempest bug report against it13:50
bauzaswant me to declare it ?13:50
ralonsohbauzas, isn't enough with the Neutron LP bug?13:51
ralonsohI'm sending a patch right now for tempest13:51
bauzasralonsoh: sorry I misread your last comment, I thought you said the opposite13:52
ralonsohbroken phone, I think13:52
ralonsohsorry if I wasn't clear13:52
ralonsoh--> https://review.opendev.org/c/openstack/tempest/+/88649613:56
bauzasralonsoh++13:56
lajoskatona+113:59
bauzasreminder : nova meeting in 1 hour-ish here15:02
* gibi lurk16:01
bauzassorry16:01
bauzas#startmeeting nova16:01
opendevmeetMeeting started Tue Jun 20 16:01:36 2023 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
opendevmeetThe meeting name has been set to 'nova'16:01
bauzas#topic Bugs (stuck/critical) 16:01
bauzas#info No Critical bug16:01
bauzaswhoops, forgot to tell howdy :)16:02
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:02
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 27 new untriaged bugs (+9 since the last meeting)16:03
bauzasbut last week was hectic :)16:03
bauzasso, let's try to look again this week 16:03
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:05
bauzasI'll try to look at them this week16:05
bauzasso,16:05
bauzas#info bug baton is being passed to bauzas16:05
bauzasany bug to raise ?16:05
bauzasok, moving on16:06
bauzas#topic Gate status 16:06
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:06
bauzaswe have a new winner this week16:06
bauzasin case you haven't seen it, 16:06
bauzas#link https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034176.html Hold your rechecks !16:06
bauzasI'd appreciate if tempest cores could quickly approve https://review.opendev.org/c/openstack/tempest/+/88649616:07
bauzas#link https://etherpad.opendev.org/p/nova-ci-failures16:07
bauzaspeople wanting to discuss it now ?16:08
* gibi is wondering if it is only bauzas and gibi is in the room16:09
bauzasmeh16:09
bauzasat least I'll write this for people looking at our meeting logs: p16:09
bauzas:p16:09
bauzasbut yeah, quickly moving then16:10
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:10
* gmann here but in another meeting in parallel 16:10
bauzasall greens, nothing to relate16:10
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:10
bauzas#info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures16:10
bauzasthere we go16:10
bauzas#topic Release Planning 16:10
bauzas#link https://releases.openstack.org/bobcat/schedule.html16:10
bauzas#info Nova deadlines are set in the above schedule16:10
bauzas#info Nova spec review day next Tuesday (June 27th)16:10
bauzas#action bauzas to notify the ML about the spec review day16:11
bauzasand as a reminder :16:11
bauzas#link https://releases.openstack.org/bobcat/schedule.html#b-nova-spec-freeze Nova Spec Freeze on July 6th !16:11
bauzasso, basically folks, we have 3 weeks for approving specs16:11
bauzasthat said, iirc the cinder spec freeze is this week :)16:11
bauzasnothing to tell about the above ?16:12
gibiI see some movement on the SRIOV spec but not ready yet16:13
bauzasindeed16:14
bauzasanyway, moving on 16:14
bauzas#topic Forum/PTG feedback 16:15
bauzas#link https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034152.html16:15
bauzasin case you haven't read it yet :)16:15
bauzaswe won't do a retrospective, we don't have a quorum here :)16:15
bauzasand actually, all the folks around were on the Summit too :p16:15
bauzasso, next topic16:16
gibi:)16:17
bauzas#topic Review priorities 16:18
bauzas#link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2)16:18
bauzas#info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review16:18
bauzas#topic Stable Branches 16:18
bauzasI haven't seen elodilles_pto since a while and his nick tells me something16:18
bauzasmaybe "pto" means something like "people to obligate" :)16:19
bauzasbut meh16:19
bauzaslet's say he's not here16:20
bauzasso I won't tell anything about the stable branches16:20
bauzasmaybe just16:20
bauzas#info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci16:20
bauzasbut I also have 16:20
bauzas#info train-eol patch proposed https://review.opendev.org/c/openstack/releases/+/88536516:20
bauzasso, there was a question in ^16:21
bauzasshould we just abandon the existing changes in this branch ?16:21
bauzasgibi: would be nice if you could vote about it16:21
bauzasok, last topic 16:23
bauzas#topic Open discussion 16:23
bauzas(bauzas) Specless blueprint approval request for https://blueprints.launchpad.net/nova/+spec/num-instances-weigher16:23
bauzasgiven we don't have quorum today, I'll ask him again next week16:23
bauzasI'm done for me, anything else?16:24
* gibi voten on the EOL patch now16:24
gibiI have nothing extra for today16:25
bauzasok, so,16:26
bauzasthanks folks16:26
bauzas#endmeeting16:26
opendevmeetMeeting ended Tue Jun 20 16:26:15 2023 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:26
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-20-16.01.html16:26
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-20-16.01.txt16:26
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-20-16.01.log.html16:26
bauzaswas quick and silent :)16:27
Uggla_yep but nothing special this week I guess16:28
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (db)  https://review.opendev.org/c/openstack/nova/+/83119316:34
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (objects)  https://review.opendev.org/c/openstack/nova/+/83940116:34
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction)  https://review.opendev.org/c/openstack/nova/+/83119416:34
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part)  https://review.opendev.org/c/openstack/nova/+/83309016:34
opendevreviewribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process  https://review.opendev.org/c/openstack/nova/+/88007516:34
opendevreviewribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion  https://review.opendev.org/c/openstack/nova/+/88147216:34
opendevreviewribaudr proposed openstack/nova master: Add metadata for shares  https://review.opendev.org/c/openstack/nova/+/85050016:34
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/85482316:34
opendevreviewribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/85482416:34
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/86028416:34
opendevreviewribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/86028516:34
opendevreviewribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares  https://review.opendev.org/c/openstack/nova/+/86028616:34
opendevreviewribaudr proposed openstack/nova master: Support rescuing an instance with shares (driver part)  https://review.opendev.org/c/openstack/nova/+/86028716:34
opendevreviewribaudr proposed openstack/nova master: Support rescuing an instance with shares (compute manager part)  https://review.opendev.org/c/openstack/nova/+/86028816:34
opendevreviewribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol  https://review.opendev.org/c/openstack/nova/+/88386216:34
opendevreviewribaudr proposed openstack/nova master: Check shares support (compute manager)  https://review.opendev.org/c/openstack/nova/+/88575116:34
opendevreviewribaudr proposed openstack/nova master: Check shares support (only API exception)  https://review.opendev.org/c/openstack/nova/+/88575216:34
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API)  https://review.opendev.org/c/openstack/nova/+/83683016:34
opendevreviewribaudr proposed openstack/nova master: Check shares support (API)  https://review.opendev.org/c/openstack/nova/+/85049916:34
opendevreviewribaudr proposed openstack/nova master: Add helper methods to attach/detach shares  https://review.opendev.org/c/openstack/nova/+/88575316:34
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach notification  https://review.opendev.org/c/openstack/nova/+/85050116:34
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach notification  https://review.opendev.org/c/openstack/nova/+/85102816:34
opendevreviewribaudr proposed openstack/nova master: Add shares to InstancePayload  https://review.opendev.org/c/openstack/nova/+/85102916:34
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach_error notification  https://review.opendev.org/c/openstack/nova/+/86028216:34
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach_error notification  https://review.opendev.org/c/openstack/nova/+/86028316:34
opendevreviewribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working.  https://review.opendev.org/c/openstack/nova/+/85208616:34
opendevreviewribaudr proposed openstack/nova master: Add virt/libvirt error test cases  https://review.opendev.org/c/openstack/nova/+/85208716:34
opendevreviewribaudr proposed openstack/nova master: Docs about Manila shares API usage  https://review.opendev.org/c/openstack/nova/+/87164216:34
opendevreviewribaudr proposed openstack/nova master: Allow config to support virtiofs (driver)  https://review.opendev.org/c/openstack/nova/+/88652216:34
gibiit was almost like the PTG :)16:35
sean-k-mooneyouch16:36
sean-k-mooneyi can feel the heat form that burn form here16:36
gibi:)16:43
dansmithheh16:54
opendevreviewmelanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time  https://review.opendev.org/c/openstack/nova/+/87705619:34
opendevreviewmelanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time  https://review.opendev.org/c/openstack/nova/+/87705620:04

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!