| opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | 00:09 |
|---|---|---|
| seyeongkim | Do you think there’s a chance this patch could be merged into master, or does it need more work? https://review.opendev.org/c/openstack/nova/+/963665 Thanks! | 04:09 |
| opendevreview | Nicolai Ruckel proposed openstack/nova stable/2025.1: Preserve vTPM state between power off and power on https://review.opendev.org/c/openstack/nova/+/979866 | 05:22 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | 07:10 |
| Uggla | gmann, gibi, 978494: Fix wrong strict assertion of quota class set id | https://review.opendev.org/c/openstack/nova/+/9784gibi finally merged. What was just a matter of crossing fingers. :) | 07:49 |
| *** ralonsoh_ is now known as ralonsoh | 07:50 | |
| Uggla | gibi, bauzas if you can review / +w 979008: Update compute rpc alias for Gazpacho | https://review.opendev.org/c/openstack/nova/+/979008, 979016: Add service version for Gazpacho | https://review.opendev.org/c/openstack/nova/+/979016 and 979023: Add Gazpacho prelude section | https://review.opendev.org/c/openstack/nova/+/979023 | 07:55 |
| Uggla | gibi, bauzas, and this one too: 980165: Amend vTPM live migration spec | https://review.opendev.org/c/openstack/nova-specs/+/980165 | 07:56 |
| bauzas | sure, I'll review those patches | 08:11 |
| bauzas | (I'm used to :) ) | 08:11 |
| bauzas | sean-k-mooney: fwiw, I started to peek at numa aware vswitches numa affinity problems and I created functests here for reproducing : https://review.opendev.org/c/openstack/nova/+/979999/ | 09:48 |
| bauzas | at the moment, I'm not seeing any issue by the fixtures, as the scheduler provides limits to the compute claim | 09:48 |
| bauzas | that's only when the filter doesn't pass the limits or if the reqspec is modified that we no longer get the limits for the claim and then the compute doesn't look at the numa nodes affinities | 09:49 |
| opendevreview | Dominik proposed openstack/nova master: NUMA Topology with Resource Providers: Libvirt NUMA Migrate https://review.opendev.org/c/openstack/nova/+/971177 | 10:10 |
| sean-k-mooney | bauzas: the probelm is not in the schduler | 11:51 |
| sean-k-mooney | bauzas: so in the schduler it works its on the compute node | 11:51 |
| sean-k-mooney | bauzas: when we reproduced it in a devstack enve we saw the schduelr do the right thing but when the compute agent called the same code it did not | 11:52 |
| sean-k-mooney | bauzas: by the way the limits form the schdluer are not provided to the compute agenet | 11:53 |
| sean-k-mooney | we reconsudct the instance numa toplogy blob in the compute and recompute the pining ectra we dont pass it over rpc today | 11:53 |
| sean-k-mooney | bauzas: my guess its this si really dumb and we are just not passing a parmater somewhere | 11:54 |
| sean-k-mooney | bauzas: as an asside if the numa shculder filter is not used numa is not suprpoted at all but but ti ssond liek you may have fount part of the issue network_metadata is missgin? | 11:56 |
| sean-k-mooney | the compute agent shoudl enfoce numa regardelss fothe schduler config obvioulsy but the point im making is we have alwasy said useign numa feature without the schduler filter is not supprot upstream the compute agenet protections are just a saftey net so that you can evenutally fix your scdhluer config without needing ot move your vms | 11:58 |
| opendevreview | Merged openstack/nova master: Add service version for Gazpacho https://review.opendev.org/c/openstack/nova/+/979016 | 12:08 |
| opendevreview | Merged openstack/python-novaclient master: Add version foot protector https://review.opendev.org/c/openstack/python-novaclient/+/978254 | 12:27 |
| bauzas | sean-k-mooney: I'd appreciate if you could look at my func reproducer patch https://review.opendev.org/c/openstack/nova/+/979999 | 13:21 |
| bauzas | I agree with you on the fact the compute claim should do the same than the filter and doublecheck the network metadata | 13:22 |
| bauzas | but I'm still wondering why those limits are not passed, because functests say "yes this is passed unless reschedules without alternativesé" | 13:22 |
| sean-k-mooney | bauzas: the are not passed because we can race with tow diffent request | 13:33 |
| sean-k-mooney | i.e. we cant safely use what the schduler has calulated without moving the instnace calim to the schduler | 13:33 |
| sean-k-mooney | bauzas: the instnace claim buidl the numa constraits form teh falvor/image locally on teh comptue agent | 13:41 |
| sean-k-mooney | my guess based on what you discsoced is we are not incuding the network metadtaa in that | 13:42 |
| opendevreview | Takashi Natsume proposed openstack/nova master: Update contributor guide for 2026.1 Gazpacho https://review.opendev.org/c/openstack/nova/+/961896 | 14:09 |
| opendevreview | Merged openstack/nova stable/2025.2: [nova-tox-py312-threading]Ignore failing tests https://review.opendev.org/c/openstack/nova/+/980013 | 14:15 |
| nicolairuckel | Is the CI having a bad day today? There are random jobs failing for my changes. | 14:18 |
| opendevreview | ribaudr proposed openstack/nova master: Add Gazpacho prelude section https://review.opendev.org/c/openstack/nova/+/979023 | 14:22 |
| nicolairuckel | I had to rerun them because one of the gates failed (which was correct at that point) but one job was failing that ran before. So I rerun them again and now that one passed but another one fails. | 14:22 |
| nicolairuckel | https://review.opendev.org/c/openstack/nova/+/978908 | 14:22 |
| dansmith | sean-k-mooney: IIRC, the only way the whole admin/instance password thing works is either by injection in libvirt (disabled/discouraged) or via metadata.. cloud-init seems to not even support this, but cloudbase-init (for windows) does.. that line up with your understanding? | 14:24 |
| dansmith | asking you because you're usually one of the more vocal anti-injection voices :) | 14:24 |
| sean-k-mooney | dansmith: liek alwasy there is a third option | 14:24 |
| sean-k-mooney | it can be set via the qemu guest agent without injection | 14:24 |
| sean-k-mooney | but yes cloudbase init is the only thing that generates it on the compute startup and post teh value back via metadta | 14:25 |
| dansmith | that's the set-password thing, I know that.. that's sort of different than what I'm asking | 14:25 |
| sean-k-mooney | ya so your summery basicly matches my understanding | 14:25 |
| dansmith | sean-k-mooney: libvirt will generate/store it in sysmeta I think, cloudbase-init can rewrite it IIRC | 14:25 |
| dansmith | hmm, maybe it only does that during set.. so maybe I'm thinking of older rax/xen behavior.. you'd create an instance in horizon and immediately get back a "here's your admin password" dialog | 14:29 |
| sean-k-mooney | dansmith: by the way i didnt think we saved the admin-password | 14:29 |
| sean-k-mooney | i tought we only returned it on create/rebuild (encyprted via your keypair) | 14:30 |
| sean-k-mooney | but that was only in memory | 14:30 |
| sean-k-mooney | perhasp i imagined that | 14:30 |
| dansmith | we store it in sysmeta in password_N keys | 14:30 |
| sean-k-mooney | the reason i ask is i dont think its in server show | 14:30 |
| dansmith | I thought it was in the create response but I'm not seeing where that would be now | 14:31 |
| dansmith | it's not in server show | 14:31 |
| sean-k-mooney | it is in the create respocne and in the rebuild responce | 14:31 |
| sean-k-mooney | i assumed we did not have it in show because we did nto have it rather then not including it intentionally hence the question | 14:32 |
| dansmith | I just don't see where it could be actually used during create I mean | 14:32 |
| sean-k-mooney | i guess we must | 14:33 |
| sean-k-mooney | https://docs.openstack.org/api-ref/compute/#show-server-password | 14:33 |
| dansmith | I'll keep looking, not asking you for that, just if you knew of any other edges | 14:33 |
| dansmith | yep, that's the way to get it on a running instance | 14:33 |
| sean-k-mooney | we likely kept it in its onw api to just not send it iin general when not needed | 14:33 |
| sean-k-mooney | dansmith: no none that come to mind at the moment | 14:34 |
| dansmith | ack, I'll try to chase down what we're doing (or not) with the password on create (I wonder if that was only ever honored by xen), | 14:34 |
| sean-k-mooney | honestly it woudl be better if we sotred this in barbiacna but this is hella old | 14:34 |
| dansmith | yeah, that's an idea I guess if we were going to modernize it | 14:35 |
| sean-k-mooney | same for keyparis in keyston or barbican | 14:35 |
| dansmith | the cloudbase-init post back to metadata sort of can't be private, so I guess it still needs to encrypt it manually or something | 14:35 |
| dansmith | it's all pretty crusty hacks for 2026 | 14:36 |
| sean-k-mooney | ya so that realying on just ssl at the infra level | 14:36 |
| sean-k-mooney | but also cloudbase dont really do openstack stuff anymore | 14:36 |
| sean-k-mooney | so we coudl perhaps deprecate/remove that | 14:36 |
| sean-k-mooney | i think that was partly put in because we dont have a way for such an agent to call barbican or similar | 14:37 |
| dansmith | what I mean is, no verifiable SSL cert for the metadata service | 14:37 |
| sean-k-mooney | i.e. we dont have the k8s thing of providing a bootstrap/service token | 14:37 |
| dansmith | yeah, deprecate would be best here I think.. but I also want to know if windows users are still relying on the nova->cloudbase-init for the initial password setting part | 14:38 |
| sean-k-mooney | dansmith: by the way we alwasy encyrpt the password with your keypair at least when storign it/returning it | 14:39 |
| sean-k-mooney | i dont knwo if cloud-base was doing that when posting it | 14:40 |
| sean-k-mooney | or if that was on the metadta api side | 14:40 |
| sean-k-mooney | im hoping it was in cloudbase | 14:40 |
| sean-k-mooney | so that nova was never handelign the plaintext password | 14:40 |
| dansmith | yep | 14:40 |
| sean-k-mooney | windows user do often use this but they can use winrm (an i think ssh in newer windwos) or userdata to set the inital password | 14:41 |
| dansmith | also, it looks like the password on create is only used by libvirt if we do injection or if we're creating config_drive.. I haven't yet seen that we store it if not doing either of htose | 14:41 |
| sean-k-mooney | they likely wont | 14:41 |
| sean-k-mooney | but that does at least work | 14:41 |
| dansmith | oh also, I'm not sure if the admin password on create is encrypted ... it seems like not.. once you do set-admin-password its only encrypted, you're right about that | 14:42 |
| sean-k-mooney | ... i dont alwasy like to be right | 14:42 |
| dansmith | yeah, ssh in newer windows for sure | 14:42 |
| dansmith | sean-k-mooney: FWIW, the adminPass generated on boot is stored in plaintext in the config_drive in meta_data.json | 15:31 |
| dansmith | not stored in the database, so not visible via usual http metadata, but.. that would be easy to mount world-readable somewhere and expose your root password _if_ you decided to use it | 15:32 |
| sean-k-mooney | not ideal. i hope we seting it to 440 or 400 today but ya with out nova providsion storage encycrption storing it in the configdriver vs root disk does nto really chagne the postture | 15:33 |
| dansmith | it's either fat or iso9660, so no permissions.. depends on how you mount it | 15:35 |
| dansmith | or if you have an OS that automounts CDs, it probably mounts it 555 | 15:35 |
| sean-k-mooney | oh you mean in the guest rather then on the host | 15:35 |
| dansmith | yes | 15:35 |
| sean-k-mooney | ack | 15:36 |
| dansmith | I meant expose your root password to any random untrusted thing that is running that can read world-readable things on the filesystem | 15:36 |
| sean-k-mooney | i tought you ment an operator or soemoen with acess to read the file on the host | 15:36 |
| sean-k-mooney | but yes so it wont automoutn by default however cloud init will do that | 15:36 |
| sean-k-mooney | so if you have a clodu iamge i think its bindmounted in o /etc/cloud/config.... | 15:37 |
| sean-k-mooney | i have never looked at hwo that is doen in the guest really so im not sure what permissison it will use | 15:37 |
| dansmith | Windows will "mount" it automatically and a general-purpose OS would automount it | 15:37 |
| dansmith | like ubuntu-desktop would end up with it mounted | 15:38 |
| sean-k-mooney | oh right the DE(gnome) ectra woudl | 15:38 |
| sean-k-mooney | not really sure now to avoid that other then contineut to advise agians enabling the fature | 15:39 |
| dansmith | since cloud-init won't set the root password from that (AFAICT) then it's not a big deal there, but on Windows where cloudbase-init would/did it'd be readable | 15:39 |
| dansmith | advise against enabling what, configdrive? | 15:39 |
| sean-k-mooney | the admin password injection | 15:39 |
| sean-k-mooney | not config drive | 15:39 |
| dansmith | there's no way to disable that part AFAICT | 15:40 |
| dansmith | the admin password is always generated in the API and passed to driver.spawn().. if libvirt is creating a config_drive, it will put it in there.. no injection enabling required | 15:40 |
| sean-k-mooney | oh ok was not aware of that. i assuem we only do that on windows images base on the os_type value in glance | 15:40 |
| dansmith | if cloudbase-init was running, sees it and honors it then... | 15:41 |
| dansmith | no, I'm testing this with a linux guest | 15:41 |
| sean-k-mooney | oh ok. perhasp we shoudl add a flag for that somewher then | 15:41 |
| dansmith | $ grep -Po admin_pass.*?, /mnt/openstack/latest/meta_data.json | 15:43 |
| dansmith | admin_pass": "CExk4PUfQLgA", | 15:43 |
| dansmith | well, it doesn't matter for linux guests regardless because cloud-init won't use it | 15:43 |
| dansmith | and for windows guests, if they rely on that, it has to work | 15:43 |
| sean-k-mooney | dansmith: do you know what other palthforms do. i asume aws and kubvirt have a similar issue | 15:43 |
| dansmith | it's not really related to what I'm doing so it doesn't matter, I just thought I'd point it out based on the conversation above | 15:44 |
| sean-k-mooney | ack | 15:44 |
| dansmith | FWIW, I don't see anywhere that we allow a POST/PUT from cloudbase-init via metadata to locally generate and set it | 15:46 |
| dansmith | do you know that's a thing for sure? | 15:46 |
| sean-k-mooney | it is but its hard to find ill grab a link | 15:53 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/api/metadata/password.py#L58 | 15:54 |
| sean-k-mooney | dansmith: it can only be set once via that flow | 15:54 |
| sean-k-mooney | https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 | 15:55 |
| dansmith | oh thanks.. how does that get linked from the actual handler that services the GET? I didn't see it ... (don't answer, I can dig) | 15:55 |
| dansmith | so it already has to be encrypted in the way we're expecting in that case I guess.. all we do is split it into pieces | 15:56 |
| sean-k-mooney | my guess is that handeler is jsut alwasy used for both but not sure | 15:56 |
| dansmith | this stuff is so gross | 15:57 |
| sean-k-mooney | that was my reaction when i rediscoved it exested a few years ago i dont know if i have the linke to the cloudbase init code but i thikn its useing the keypair to encyrpt it | 15:58 |
| dansmith | it has to if it wants horizon to be able to use it I think | 15:58 |
| dansmith | well, horizon lets the user choose a private key to decrypt with so.. I guess that could be something else, IDK | 15:58 |
| dansmith | the cloudbase-init code is available, I haven't gone looking through it yet but I will | 15:59 |
| sean-k-mooney | it has to be the private key that releated to the publick key for the server | 16:01 |
| sean-k-mooney | https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/metadata/services/httpservice.py#L71-L81 | 16:02 |
| sean-k-mooney | https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/plugins/common/setuserpassword.py#L50 | 16:03 |
| sean-k-mooney | https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/metadata/services/base.py#L115-L126 | 16:03 |
| dansmith | yep, so same as nova basically | 16:04 |
| sean-k-mooney | so it alwasy the first public key in the list which for opentack is where we put the ssh/x509 pub key form the servers keypair | 16:04 |
| sean-k-mooney | yep so it shoudl be end to end encypted | 16:05 |
| sean-k-mooney | althogh i hate the idea that horizon has a view where you can send it your private key ever | 16:05 |
| sean-k-mooney | hopefully they do that clint side with javascript of somehtign liek that | 16:06 |
| dansmith | yep they do | 16:07 |
| dansmith | and yep, showing the password in cleartext ever seems like an anti-pattern anyway | 16:07 |
| dansmith | I assume there has to be some windows-specific workflow that requires it be generated on the instance at first boot or something | 16:07 |
| dansmith | like maybe post-sysprep or some such | 16:07 |
| dansmith | post-un-sysprep I mean | 16:07 |
| opendevreview | Florian proposed openstack/nova master: Add check for PCIe devices attach limit for volume and ports https://review.opendev.org/c/openstack/nova/+/955584 | 16:09 |
| opendevreview | Merged openstack/nova master: Update compute rpc alias for Gazpacho https://review.opendev.org/c/openstack/nova/+/979008 | 16:24 |
| opendevreview | Merged openstack/nova master: Add Gazpacho prelude section https://review.opendev.org/c/openstack/nova/+/979023 | 16:24 |
| opendevreview | Merged openstack/nova-specs master: Amend vTPM live migration spec https://review.opendev.org/c/openstack/nova-specs/+/980165 | 17:59 |
| Matko[m] | Would anybody know if the ironic driver can or ever consumed in any shape or form the memcache_servers and memcache_secret_key configs from the ironic section in nova.conf? | 21:25 |
| Matko[m] | I'm doing some cleanup in openstack-helm and found those configs are configured in the ironic section: https://opendev.org/openstack/openstack-helm/src/branch/master/nova/templates/configmap-etc.yaml#L166-L172 | 21:25 |
| Matko[m] | I want to remove them as I couldn't find anything consumed them, even through libraries. I want to cover all bases and confirm with the Nova team this is actually not used. | 21:25 |
| sean-k-mooney | Matko[m]: the ironic driver used to use tooz to provide a distibute hash ring | 21:26 |
| sean-k-mooney | i think tha tmight have eben able to use memcache to store it | 21:26 |
| sean-k-mooney | https://opendev.org/openstack/tooz/src/branch/master/tooz/drivers/memcached.py | 21:27 |
| sean-k-mooney | so maybe if it was integrated via that | 21:27 |
| sean-k-mooney | Matko[m]: it woudl proably be better to also check with the ironic folks | 21:27 |
| sean-k-mooney | this i beilve is what it sued form that lib https://opendev.org/openstack/tooz/src/branch/master/tooz/hashring.py#L30 | 21:28 |
| sean-k-mooney | Matko[m]: thos ecould also be commign form the keystoneAuth optiosn we add | 21:29 |
| Matko[m] | yes, I found the usage of tooz. However there is nothing feeding the config to tooz | 21:29 |
| Matko[m] | I did look into keystoneauth too | 21:30 |
| Matko[m] | There is no trace of memcache in keystoneauth project: https://github.com/search?q=repo%3Aopenstack%2Fkeystoneauth+memcache&type=code | 21:30 |
| Matko[m] | No dependency on oslo.cache or dogpile that might consume the config: https://github.com/openstack/keystoneauth/blob/master/requirements.txt | 21:30 |
| Matko[m] | I'll ask the Ironic team then. | 21:30 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conf/ironic.py | 21:30 |
| sean-k-mooney | that is where that config section is regeistered and its not an option we are addign explcitly | 21:31 |
| sean-k-mooney | so that measn it coming from either keystoen auth or the sdk i think | 21:31 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/conf/ironic.py#L128-L131 | 21:31 |
| sean-k-mooney | so if those have any effect i suspect it via the sdk/keystoneAuth common logic not anythin ironic specific | 21:32 |
| Matko[m] | I looked into those as well and couldn't find anything. The openstack-helm code I linked to was added 9 years ago. And I wonder/suspect if it was a bad copy/paste from similar code in the chart for keystone_authtoken which has memcache configs. | 21:33 |
| sean-k-mooney | did you check our current config ref | 21:33 |
| Matko[m] | yes | 21:33 |
| Matko[m] | I would also assume list_opts would list and include all the supported configs in the docs. | 21:33 |
| sean-k-mooney | ya we generate the config ref form that | 21:34 |
| Matko[m] | https://docs.openstack.org/nova/latest/configuration/config.html#ironic | 21:34 |
| Matko[m] | if it doesn't ring a bell to anybody, then I'll conclude it was a bad copy/paste. | 21:34 |
| sean-k-mooney | i dont see them there | 21:34 |
| sean-k-mooney | so i think they are not used any more | 21:34 |
| Matko[m] | alright, thanks for your help and time | 21:35 |
| sean-k-mooney | i suspect they may have been used when we use the python-ironic client | 21:35 |
| sean-k-mooney | we swapped to the sdk about 3-4 releases ago | 21:35 |
| opendevreview | Ashish Gupta proposed openstack/nova master: Add native threading support to test case ID propagation fixtures https://review.opendev.org/c/openstack/nova/+/980179 | 22:33 |
Generated by irclog2html.py 4.1.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!