Tuesday, 2024-01-16

opendevreviewsean mooney proposed openstack/nova master: [DNM] debug kernel paincs  https://review.opendev.org/c/openstack/nova/+/90562801:26
*** efried1 is now known as efried06:07
auniyalHi sean-k-mooney, gibi, bauzas, can you please llok at this change its a simple and important change which can help many tests. 06:11
auniyalhttps://review.opendev.org/c/openstack/nova/+/89358406:11
gibielodilles: we can land the zed backport of https://review.opendev.org/q/topic:%22bug/2025480%22 now08:03
gibiauniyal: done08:06
auniyalgibi, thanks there are 2 more patches with same topic - can you please review them as well https://review.opendev.org/q/topic:%22refactor-volumeAttachment-calls%2208:07
elodillesgibi: ACK, thanks! +2+W'd \o/08:31
opendevreviewMerged openstack/nova stable/zed: Reproduce bug #2025480 in a functional test  https://review.opendev.org/c/openstack/nova/+/90437409:13
gibisean-k-mooney bauzas: re: powermgmt; it seems that during compute startup when power management offlines unused pcpus the privsep daemon is not spawned automatically. During VM boot the privsep daemon spawned automatically09:23
gibiit seems the privsep decorator fails to apply09:43
gibihttps://paste.opendev.org/show/b3m4lKm2z7c9yP2psvSJ/09:43
UgglaHello, can someone clarify this: This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo). I mean in which case it is a FQDN or not ?09:45
UgglaAnd do you know if the behavior of libvirt changed recently ?09:46
gibibauzas: sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/885293 it is not backported to 2023.1 :/09:54
gibibauzas: do you happen to have a list of bugfixes you implemented for the powermgmt feature. I want to check all to see what is not backported to 2023.109:55
bauzasgibi: I don't have a lot of bugfixes for this09:56
bauzasI think you found the only one missing :(09:56
bauzas(sorry, was in meeting)09:56
gibiOK09:57
gibiI will backport it09:57
bauzasthanks09:58
bauzasUggla: are you talking of hypervisor_hostname field ?10:02
bauzasif so, this is virt-dependent10:03
Ugglayep, in fact it seems we have users blocked by the patched introduced by Dan to check rename compute. They registered their nodes with the short name and libvirt seems to provide the FQDN and thus making the comparison to fail and block the compute startup.10:04
bauzasUggla: libvirt populates this field by https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L961110:05
bauzaswhich itself calls libvirt with https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L106510:06
Uggla@bauzas, yep I looked at the methods doc --> This returns a system hostname on which the hypervisor is running (based on the result of the gethostname system call, but possibly expanded to a fully-qualified domain name via getaddrinfo)10:06
Ugglabut I guess that's not libvirt which changed it's behavior.10:07
bauzasyeah, just checked the libvirt release notes10:08
bauzasthis is possibly something that the operator did which broke10:08
UgglaThe question was more personal to understand what libvirt is doing.10:08
bauzasand what dansmith writes actually prevents that, which is definitely what  :)10:08
bauzaswe want :)10:08
bauzasaccidental hostname rewrites break placement and many other things10:09
UgglaIt seams the operator upgraded to 2023.1 and he is blocked by the check.10:09
Ugglabecause short name != fqdn10:10
bauzasthat's good that they're blocked then10:10
UgglaI guess it should not be blocked if the compute were registered wit the fqdn previously.10:11
Ugglabut I'm not sure we have guideline that enforce this ?10:12
UgglaAnd how the operator ccan get out this issue without db surgery ?10:12
bauzasnova only registers the nodename 10:13
bauzasthat means that *something* changed it10:13
gibibauzas: there is another non backported fix for power mgmt: https://review.opendev.org/c/openstack/nova/+/885352 you implemented this on master but not backported. Therefore I see the bug in 2023.1 but did not noticed the master fix and implemented another fix that is now backported https://review.opendev.org/c/openstack/nova/+/903169 . Unfortunately 10:13
bauzasand now nova refuses to start10:13
gibihttps://review.opendev.org/c/openstack/nova/+/903169 has a bug in it as it expecting a wrong exception type10:13
gibibauzas: I will do some reverts and backports to clean this up10:14
bauzasdoh10:14
bauzasgibi: apologies for the mess10:14
bauzasI should create bugfixes with the same gerrit topic than the feature10:15
gibino worries I made my one fair share of mess in it10:15
bauzasinstead of using the bug number topic 10:15
gibiI prefer bug number topics10:15
bauzasyeah I understand10:15
bauzasthis is a tradeoff10:16
bauzasthen I should mention some changeid in the commit msg, so gerrit could possibly link it10:16
bauzastbc, I don't trust my memory and that one is another evidence10:17
bauzasthe series was small and there were only two bugfixes, despite that I failed to remember about that 10:17
bauzasmy brain is so fcked10:17
opendevreviewMerged openstack/nova stable/zed: Do not untrack resources of a server being unshelved  https://review.opendev.org/c/openstack/nova/+/90437510:19
gibibauzas: I don't know how to avoid this via automation10:24
gibielodilles: do I need to backport a revert, or I can do a revert on the stable branch independently of a revert on master?10:25
gibielodilles: i.e can I do git revert on stable, or do I need to do git cherry-pick -x on the revert commit from master to stable?10:26
gibielodilles: I will go with the cherry-pick as I affraid of the stable backport script will not allow an independent revert :/10:32
Uggla@bauzas you said we store only the shortname for compute name in the db right ?10:33
bauzasnope10:33
bauzasI said we store what libvirt gives us 10:33
bauzasif after an upgrade libvirt is giving us something else, then now with 2023.1 we fail10:34
bauzaspreviously, we were blindly recreating duplicate resource providers leading to inconsistencies10:34
Ugglayep it means that when the compute was registered he got the shortname and after upgrade or something else libvirt supply the fqdn.10:35
Uggla@bauzas, note I understand the patch goal.10:36
elodillesgibi: we usually simply backport reverts if it is needed all the way from master branch to multiple stable branches. if only needed for some older branches, then [stable-only] is fine. and of course if the bug is superimportant highly critical ultimate fix is needed, then also mark them as [stable-only] and merge them parallel in all stable branches :)10:47
elodillesthese are my memories from the past ^^^ :)10:48
gibinot super critical 10:49
gibigoing with the cherry-picks10:50
opendevreviewBalazs Gibizer proposed openstack/nova master: Revert "[pwmgmt]ignore missin governor when cpu_state used"  https://review.opendev.org/c/openstack/nova/+/90567110:50
opendevreviewBalazs Gibizer proposed openstack/nova stable/2023.2: Revert "[pwmgmt]ignore missin governor when cpu_state used"  https://review.opendev.org/c/openstack/nova/+/90567210:51
opendevreviewBalazs Gibizer proposed openstack/nova stable/2023.2: cpu: make governors to be optional  https://review.opendev.org/c/openstack/nova/+/90567310:51
opendevreviewBalazs Gibizer proposed openstack/nova stable/2023.1: Revert "[pwmgmt]ignore missin governor when cpu_state used"  https://review.opendev.org/c/openstack/nova/+/90567410:51
opendevreviewBalazs Gibizer proposed openstack/nova stable/2023.1: cpu: fix the privsep issue when offlining the cpu  https://review.opendev.org/c/openstack/nova/+/90567510:51
opendevreviewBalazs Gibizer proposed openstack/nova stable/2023.1: cpu: make governors to be optional  https://review.opendev.org/c/openstack/nova/+/90567610:51
gibielodilles, bauzas, sean-k-mooney ^^10:51
gibiit is complicated as on master we only need to revert, on 2023.2 we revert and backport the proper fix from master, on 2023.1 we revert, backport a related fix from 2023.2 and then backport the original good fix from master->2023.210:52
* bauzas goes off for gym but I'll look later10:52
bauzasgibi: ++10:52
Ugglado we agree that there is no "proper" way to rename a compute node. And that we need to stop the compute service, remove the compute from the configuration, and restart the compute to have it registered with the new name ?11:06
zigokashyap: For Ubuntu stuff, please ask jamespage or coreycb_ please, I have no idea about #ubuntu-kernel on Libera at all, and I honestly don't know anything about the Ubuntu kernel in general.11:07
kashyapzigo: Thans for the contacts :) In the past I had a couple of chats on #ubuntu-kernel, while it used to be still on FN :)11:07
jamespagekashyap: lacking some context - just reading the etherpad linked above somewhere11:12
kashyapjamespage: Hi, I didn't ping you with full context yet. :)11:12
kashyapjamespage: It's about a certain class of kernel panics we're seeing in the upstream CI.  An example: https://bugs.launchpad.net/nova/+bug/201861211:13
kashyapAdmittedly  (a) these are difficult to reproduce locally; and (b) distro kernel maintainers (understandably) might not have time to debug "unsupported distros" (as evidenced above).11:16
kashyapI need to step out to get some lunch; need to drop here.  (I don't have an IRC bouncer anymore)11:16
jamespageack11:16
auniyalin cinderfixture for attachment_update https://review.opendev.org/c/openstack/nova/+/658904/4/nova/tests/fixtures.py#199711:19
auniyalwhy this volume_id must be self.MULTIATTACH_VOL (the sent volume is a dynamically created new volume, )  MULTIATTACH_VOL is a contant 11:19
auniyalgibi ^11:19
auniyalI am trying to to create a server from snapshot-image and while updating attachment in cinder its failing at this step - can someone  please tell why this is wrtten and what I can I do to use MULTIATTACH_VOL.11:23
auniyalso attachment can get updated11:24
opendevreviewMerged openstack/nova master: Adds server show in helpers  https://review.opendev.org/c/openstack/nova/+/89358411:33
opendevreviewAmit Uniyal proposed openstack/nova master: Fixes bug 2048184  https://review.opendev.org/c/openstack/nova/+/90481712:00
opendevreviewAmit Uniyal proposed openstack/nova master: Updates glance fixture for create image  https://review.opendev.org/c/openstack/nova/+/90568412:00
opendevreviewAmit Uniyal proposed openstack/nova master: WIP: snapshot tests  https://review.opendev.org/c/openstack/nova/+/90568512:00
bauzasUggla: in our OSP product, we have some solution for renaming a compute by scaling in and the out the computes13:36
SvenKieskeHey, has anyone ever thought about enabling "core scheduling" support by default? It's in libvirt since v8.9.0 but I can't find any information regarding openstack implementation status: https://www.libvirt.org/news.html#v8-9-0-2022-11-0114:42
SvenKieskewhen searching for this in nova I also found this gem, which seems not to be tracked anymore (last activity 2019, but bug was never closed): https://bugs.launchpad.net/nova/+bug/141797514:46
opendevreviewAmit Uniyal proposed openstack/nova master: WIP: snapshot tests  https://review.opendev.org/c/openstack/nova/+/90568514:49
opendevreviewAmit Uniyal proposed openstack/nova master: Updates glance fixture for create image  https://review.opendev.org/c/openstack/nova/+/90568414:56
opendevreviewAmit Uniyal proposed openstack/nova master: Fixes bug 2048184  https://review.opendev.org/c/openstack/nova/+/90481714:56
opendevreviewAmit Uniyal proposed openstack/nova master: WIP: snapshot tests  https://review.opendev.org/c/openstack/nova/+/90568514:56
bauzasSvenKieske: that sounds a new feature request, if you want to use it with nova15:12
gibidansmith melwitt bauzas : a fresh kernel panic on master https://review.opendev.org/c/openstack/nova/+/90567115:13
dansmithgibi: that's similar to the one that we opened against the ubuntu team and they asked us to repro on a newer kernel, so that's ... "good"15:23
* gibi feels useful15:23
SvenKieskebauzas: yeah sure, I'm just a little baffled that I seem to be the first person to mention this feature request, as this is a rather useful feature, from a security perspective.15:31
kashyapgibi: Hi, dansmith tells me you've ran into another kernel panic.  Got a log link?15:47
dansmithkashyap: https://review.opendev.org/c/openstack/nova/+/90567115:47
dansmithkashyap: also for future reference: https://meetings.opendev.org/irclogs/15:48
kashyapdansmith: Hi; thanks!15:48
*** Continuity__ is now known as Continuity15:48
kashyapdansmith: Oh, yeah; I know we log upstream stuff; forgot for a min :)15:48
dansmithoh, okay15:48
kashyapNow the challenge is find the elusive local reproducer :-( 15:49
kashyapInteresting, this was discovered by a power-management related patch15:49
dansmithI doubt it's related, and I also doubt we're going to find a local reproducer15:50
gibikashyap: power management is disable by default in our upstream CI15:52
kashyapHm, I see.15:52
gibiso the patch itself is not related15:52
kashyapdansmith: Hm, if we can't figure out a way get a local reproducer, it throws a wrench in our communication with kernel maintainers (whether it be distro or upstream).  I'm sure you know this15:53
kashyapI'm wondering how we can unjam this "deadlock" situation.15:54
dansmiththere's still information in the stack traces to provide hints on where to go from here15:54
bauzasnova meeting in 5 mins15:55
bauzas*here15:55
dansmithand also, kernel developers are (IME) used to working on issues they can't reproduce because they're related to hardware and/or deadlocks where we can't get forensics out15:55
dansmitha kernel developer might also have a better idea about how to force a repro from the details of what was happening when it faulted15:56
kashyapdansmith: Yeah; that's a fair point; at least _some_ of the traces should lead us somewhere15:57
kashyapAnd to your previous point, from a recent experience one kernel trace, I did see at least one kernel maintainer immediately say "it's a known issue due to a hardware bug"15:59
dansmithit's also in very early boot of a kernel and userspace they have good access to15:59
bauzas#startmeeting nova16:00
opendevmeetMeeting started Tue Jan 16 16:00:02 2024 UTC and is due to finish in 60 minutes.  The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.16:00
opendevmeetUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:00
opendevmeetThe meeting name has been set to 'nova'16:00
elodilleso/16:00
grandchildo/16:00
bauzasheya16:00
dansmitho/ (kinda)16:00
bauzas#link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting16:00
bauzaslet's try to have a short meeting16:00
bauzassome folks have another meeting in 15 mins16:01
bauzas#topic Bugs (stuck/critical) 16:01
bauzas#info No Critical bug16:01
bauzas#link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 46 new untriaged bugs (+3 since the last meeting)16:01
bauzas#info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster16:01
bauzasUggla worked a lot on the bugs16:01
bauzasUggla: have you modified the status for all of the ones you looked at ?16:01
fwieselo/16:02
bauzasanyway, moving on16:03
bauzaselodilles: are you okay if you could look at the bugs next week ?16:03
elodillessure o/16:03
gibio/16:04
bauzascool16:04
bauzas#info bug baton is elodilles16:04
bauzas#info bug baton is elodilles16:04
bauzasshit16:04
bauzas#topic Gate status 16:04
bauzas#link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs 16:04
bauzas#link https://etherpad.opendev.org/p/nova-ci-failures-minimal16:04
bauzas#link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status16:05
bauzas#info Please look at the gate failures and file a bug report with the gate-failure tag.16:05
bauzasdo guys want to discuss about the guest kernel issues ?16:05
bauzas(periodic runs are all ok)16:05
bauzaslooks not, moving on then16:06
bauzas#topic Release Planning 16:06
bauzas#link https://releases.openstack.org/caracal/schedule.html#nova16:06
bauzas#info Caracal-3 (and feature freeze) milestone in 6 weeks16:06
bauzas#topic Review priorities 16:06
bauzasas a reminder, please look at this etherpad if you want to review our implementations :)=16:07
bauzas#link https://etherpad.opendev.org/p/nova-caracal-status16:07
bauzas#topic Stable Branches 16:07
bauzaselodilles: your time16:07
elodilleso/16:07
elodillesnothing to report actually16:07
elodillesstate is the same as last week16:07
elodilles(i've pinged release cores to approve Nova's Zed release)16:07
bauzas++16:08
elodillesthat's all16:08
bauzasI think I did +1 for the zed release16:08
elodillesyepp, you did16:08
bauzascool16:08
bauzaselodilles: thanks16:08
elodillesthanks too :)16:08
bauzas#topic vmwareapi 3rd-party CI efforts Highlights 16:08
bauzasgrandchild: fwiesel: your time :)16:08
fwiesel#Info ETA on exemption for public access by end of week. Hopefully under DNS name: openstack-ci-logs.global.cloud.sap16:09
bauzas<316:09
fwieselSo, not much happened. Still wrapped in red tape. But at least I have a dns name.16:09
bauzasso we'll wait :)16:09
fwieselThat's it from my side. Any questions?16:09
bauzasthanks a lot16:09
bauzasnope from me16:09
fwieselYou're welcome.16:09
bauzasI think we said we should wait until milestone-2 for seeing whether we would remove vmwareapi, but given what you did and what you continue, of course we won't16:10
bauzasso let's continue to see what things happen16:10
fwieselThanks. Much appreciated.16:11
bauzasthanks folks for the work, definitely nice16:11
bauzas#topic Open discussion 16:11
bauzasnothing in the agenda, anything anyone ?16:11
bauzaslooks not,16:12
bauzasthanks all, was a quick one16:12
bauzas#endmeeting16:12
opendevmeetMeeting ended Tue Jan 16 16:12:27 2024 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:12
opendevmeetMinutes:        https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.html16:12
opendevmeetMinutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.txt16:12
opendevmeetLog:            https://meetings.opendev.org/meetings/nova/2024/nova.2024-01-16-16.00.log.html16:12
fwieselThanks chat with you next week, hopefully with more news16:13
bauzas++16:14
Uggla@bauzas, sorry I was discussing with Artom, and I did not managed to answer.16:26
UgglaYep I updated the status of the bugs. I read a lot of them, but did not manage to triage a lot of them.16:27
opendevreviewTakashi Kajinami proposed openstack/placement master: Bump hacking  https://review.opendev.org/c/openstack/placement/+/90570616:29
UgglaI think https://bugs.launchpad.net/nova/+bug/2048154 is important to be fixed. I hope @Amit could have a look.16:29
opendevreviewTakashi Kajinami proposed openstack/python-novaclient master: Bump hacking  https://review.opendev.org/c/openstack/python-novaclient/+/90570716:30
opendevreviewTakashi Kajinami proposed openstack/osc-placement master: Bump hacking  https://review.opendev.org/c/openstack/osc-placement/+/90571516:41
fricklerkashyap: dansmith: added some comments on the etherpad. given that the issue (afaict) only happens for volume resize tests, I'm wondering how we could verify that no actual data corruption is happening on the volume in that scenario16:44
fricklerbecause if it does happen, chasing possible kernel bugs is kind of moot16:44
kashyapfrickler: Hi, yeah, I've just seen your comment on the Etherpad.  Good question. (I'm in a meeting, and will be slow here)16:45
dansmithfrickler: these test instances haven't even made it out of initial kernel boot, so no real opportunity to corrupt anything yet16:45
frickleris this happening on the initial boot, not after resize?16:46
dansmithfrickler: and we're not booting from the volumes either, since we're using the preloaded kernel boot method, which means corruption of the on-disk data causing the crash wouldn't be a thing16:46
dansmithbefore the recent switch to kernel preload that could have been a possible option, but with kernel preload, disk corruption due to the resize should be ruled out AFAIK16:47
dansmithI'm also not sure these are only happening on resize tests16:49
fricklerwell the errors seem to be happening around the time when the switch to the actual root-fs would be happening. and in the traceback I was looking at, it was happening after the resize. do you happen to have a pointer to the preloaded kernel change?16:57
dansmithfrickler: I think it's still loading modules from the initramfs when it crashes16:59
dansmithfrickler: https://review.opendev.org/c/openstack/nova/+/90221717:00
dansmith(happened while I was out)17:00
dansmithfrickler: I think what you're seeing there is that it tried to load all the modules from the initramfs that it needed to have access to storage, which all crashed, and then it kept going trying to do the switch_root and failed to do so17:01
dansmithit did get vda from the virtio driver so I would think it would have worked even, but with that level of broken having happened who knows17:02
fricklerfrom a quick scan, that patch only changes some specific nova jobs, not tempest-integrated-compute?17:03
dansmithfrickler: tbf, I hadn't even looked at the patch, I was going based on sean-k-mooney, melwitt saying nova-next was the only one still doing image boot17:04
dansmiththe original bug we opened for this GPF was before we even had the resize-volume-backed test, FWIW :)17:05
tkajinamo/ I'd appreciate it if https://review.opendev.org/c/openstack/nova/+/905314 can get some attention because it's now blocking requirement bump.17:20
dansmithfrickler: yeah, that instance in question is doing preloaded kernel boot:17:22
dansmithJan 16 11:45:59.955345 np0036421623 nova-compute[74855]:     <kernel>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/kernel</kernel>17:22
dansmithJan 16 11:45:59.955345 np0036421623 nova-compute[74855]:     <initrd>/opt/stack/data/nova/instances/76256ac2-5db8-4dab-b3b8-0297a2ad2b71/ramdisk</initrd>17:22
dansmithso, shouldn't have loaded the ramdisk from disk17:23
sean-k-mooneydansmith: sorry was drinking coffee reading back17:23
sean-k-mooneyfrickler: dansmith https://review.opendev.org/c/openstack/nova/+/902809/117:24
sean-k-mooneywe had a folow up patch to overried the images for jobs defiend out of repo17:24
sean-k-mooneyso tempest-integrated-compute was updated here https://review.opendev.org/c/openstack/nova/+/902809/1/.zuul.yaml#94417:25
sean-k-mooneydansmith: i quickly tried enabling the larger vms here by the way17:26
sean-k-mooneygoing to 32 gb feels excessinve but based on the low point (and concurnace 8) we were using about 10G on the contoler17:27
sean-k-mooneyso if we had a 16G flavor or even a 11G flavor we woudl entirly elimiate our usage of swap form the job17:27
clarkbfwiw we do have larger flavors they are just limited in quantity as only a few (maybe one?) cloud currently provide them17:28
sean-k-mooneyya i was looking at that17:28
sean-k-mooneywe have 32G instance form one vexhost cloud and 16G instnace form another17:29
sean-k-mooneyand then 8G instnace in general17:29
clarkbbut also I think any "make nodes bigger" effort should coincide with a "understand where the memory use is coming from and whether or not it represents bugs/leaks" effort17:29
sean-k-mooneyya so it looks like if we were able to move to 16G as standard we would signifcatly increase the performance of the job (tempest full was about an hour)17:30
clarkbfor example privsep uses tremendous amounts of memory for what it is17:30
sean-k-mooneybut if we were to move to that we would likely expand to fill it17:30
clarkbwith the little info I have I would classify that as a bug17:30
sean-k-mooneyclarkb: frickler dansmith  while ye are aroudn could ye look at my zram change when ye have time https://review.opendev.org/c/openstack/devstack/+/89069317:32
sean-k-mooneyclarkb: on the privsep topic, i have not looked at it in detail before but do you knwo roughly how much ram its using per process17:33
sean-k-mooneyi assume it partly depend on the service? i.e. nova vs neutron17:33
dansmithsean-k-mooney: yeah I'm for that for sure, but not sure the qa people will want it on by default17:33
fricklersean-k-mooney: tbh I've stayed away from that patch for now since I'm wary of making the CI setup even more complex17:33
dansmithgmann: kopecmartin ^17:33
sean-k-mooneyack i can change the defaults os if people prefer it to be off by default ill refactor for that17:34
JayFsean-k-mooney: /me points an ironic change at 890693 for science17:34
sean-k-mooneyits off by default locally but on by default in ci currently17:34
dansmithsean-k-mooney: if you disable it by default I'll +2 now, but if not, I want a read from gmann at least17:34
sean-k-mooneyack ill do that i need to deploy a devstack vm for my healthcheck stuff anyway so ill upated it then17:35
clarkbsean-k-mooney: the memory usage varies by the service talking to the privsep instance (which implies to me that workload probably impacts buffer sizes). In total I think it was about half a gig for privsep17:35
sean-k-mooneyclarkb: i think tis more related to the size of our oslo.config global + the code that is imported17:35
clarkbsean-k-mooney: oh ya I guess everything is a regex? Those shouldn't be terribly large but if you have enough of them... might also be worth checking if libre2 is more efficient maybe17:36
sean-k-mooneywe have some chaced module level varables for example that might be related to this17:37
clarkbsean-k-mooney: for zswap I notice you set swappiness to 100. I think devstack sets swappiness elsewhere you might need to reconcile the two to ensure they don't fight each other17:37
clarkbsean-k-mooney: https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/roles/configure-swap/tasks/main.yaml#L46-L61 this is the code that does it and I think devstack jobs call that first so you would override which I guess is fine17:38
sean-k-mooneyclarkb: i was thinking fo https://github.com/openstack/nova/blob/a72f7eaac78927892b937d451cbacb24a83c05ac/nova/api/validation/parameter_types.py#L10617:38
sean-k-mooneyalthough privsep shoudl not be using the api stuff17:38
sean-k-mooneyso hopefully that wont matter17:39
clarkbI don't have any other comments on the zswap change. Seems like it may be worth experimenting with17:39
dansmithyep17:39
sean-k-mooney i know people have seen a lot of success with desktop responsiveness on rassbery pi with it17:40
sean-k-mooneyso i was hoping it woudl help us in a simialr way17:40
sean-k-mooneyok ill go remove the default change in teh patch17:40
sean-k-mooneyand we can then enabel it on a per job basises in nova17:40
sean-k-mooneyJayF: did you see https://review.opendev.org/c/openstack/nova/+/905406 by the way17:44
sean-k-mooneyJayF: that extract the common instnace metadata code up to the driver level and refactors libvirt to use it17:44
JayFhttps://review.opendev.org/c/openstack/nova/+/900831 I really want this to land before I push up more patches17:44
JayFI've been burned in the past by stacking a bunch of patches and being in rebase hell17:45
sean-k-mooneyah yes well want that to land to make backport simpler17:45
JayFSo I have that in my back pocket as a "next thing" but waiting for 900831 to land first17:45
clarkbJayF: this may or may not be useful but the way I try to deal with those is to "squash back" I make all of my edits on the tip of my dev branch and then commit will be something like "squash fix for commit foo" then I git rebase -i HEAD~X where X is the number of commits I need to go back to and squash the new commit back into the old one. Then git review the whole thing. I17:48
clarkbfind it simplifies things because I can focus on the necessary fixes and then focus on the accounting and usually I only need a single rebase to update a whole stack with any number of fixes17:48
JayFclarkb: it's more that I have enough things on my list that I can action and will move more quickly so I just prioritize those over things which will languish for longer17:49
bauzasgit reflog FTW17:49
JayFI can't remember the last time I was just like "there's nothing to do today" :D 17:49
jrosserclarkb: a colleague of mine demoed this recently https://github.com/tummychow/git-absorb 17:55
jrosserwhich is some kind of magic17:56
clarkbthat looks similar to what I do with the pairing of changes to squash done automatically based on file heuristics17:56
clarkbThere is also git restack which corvus wrote so that you don't have to manually determine the value of X when rebasing17:57
sean-k-mooneyjohnthetubaguy: are you able to be the second core to review https://review.opendev.org/c/openstack/nova/+/900831 its JayF fix for https://launchpad.net/bugs/204303619:05
sean-k-mooneyotherwizse melwitt dansmith ^ perhasp ye can review19:05
opendevreviewMerged openstack/osc-placement master: Bump hacking  https://review.opendev.org/c/openstack/osc-placement/+/90571519:33
opendevreviewSylvain Bauza proposed openstack/nova master: Check if destination can support the src mdev types  https://review.opendev.org/c/openstack/nova/+/90417720:19
opendevreviewSylvain Bauza proposed openstack/nova master: Reserve mdevs to return to the source  https://review.opendev.org/c/openstack/nova/+/90420920:19
opendevreviewSylvain Bauza proposed openstack/nova master: WIP(docs): Modify the mdevs in the migrate XML  https://review.opendev.org/c/openstack/nova/+/90425820:19
opendevreviewsean mooney proposed openstack/nova master: enable zswap in nova ci jobs.  https://review.opendev.org/c/openstack/nova/+/90579120:38
opendevreviewsean mooney proposed openstack/nova master: enable zswap in nova ci jobs.  https://review.opendev.org/c/openstack/nova/+/90579120:44
opendevreviewMerged openstack/python-novaclient master: Bump hacking  https://review.opendev.org/c/openstack/python-novaclient/+/90570720:51
opendevreviewMerged openstack/placement master: Bump hacking  https://review.opendev.org/c/openstack/placement/+/90570621:06

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!