Thursday, 2024-05-16

opendevreviewmelanie witt proposed openstack/nova-specs master: Re-propose specs for ephemeral encryption  https://review.opendev.org/c/openstack/nova-specs/+/90765400:52
gmannthanks01:46
opendevreviewDmitriy Rabotyagov proposed openstack/nova master: Handle MAC dash-addresses during migration  https://review.opendev.org/c/openstack/nova/+/91976007:31
opendevreviewPierre Riteau proposed openstack/nova master: Fix formatting issues in extra-specs docs  https://review.opendev.org/c/openstack/nova/+/91746209:08
stephenfinbauzas: Friendly reminder about that OpenAPI spec needing your attention, please and thank you10:03
bauzasstephenfin: I'm currently working hard on creating my presentation for the next OpenInfra Day in Paris next week, but I'll try10:03
*** sfinucan is now known as stephenfin11:35
opendevreviewMerged openstack/osc-placement master: Update testing of python versions  https://review.opendev.org/c/openstack/osc-placement/+/90461212:18
opendevreviewMerged openstack/osc-placement master: Make python 3.12 functional job voting  https://review.opendev.org/c/openstack/osc-placement/+/91910112:29
opendevreviewMerged openstack/nova master: Fix formatting issues in extra-specs docs  https://review.opendev.org/c/openstack/nova/+/91746213:16
opendevreviewMerged openstack/nova stable/2023.2: Adding server actions tests to grenade-multinode  https://review.opendev.org/c/openstack/nova/+/90033913:36
dansmithbauzas: this stack fixes python 3.12 functional/unit tests if you could help merge that before we regress :) https://review.opendev.org/c/openstack/nova/+/91976313:42
bauzasdansmith: okay, I don't really have time but I trust you, +Wd13:43
dansmithbauzas: okay, it's a stack of three.. maybe gibi could look at the other two, or maybe melwitt when she's around13:44
sean-k-mooneyonly the second one https://review.opendev.org/c/openstack/nova/+/919764/213:44
sean-k-mooneyis missing a +w at this point13:44
bauzasthis is done13:44
sean-k-mooneycool13:44
bauzasthe fixes are simple but I don't really have time to think about some concerns13:45
dansmiththanks13:45
dansmithbauzas: yep, safe I think13:45
sean-k-mooneyform the nova side yes13:45
* bauzas tries to understand why Glance returns some gateway exception when I want to backup my instance13:45
bauzasdansmith: glance-api is absolutely blind on the req-id, so how could I see any logs from glance apart from devstack@g-api13:46
dansmithis it/13:46
dansmithbut gateway exception usually means apache not glance13:47
dansmithunless glance is 500'ing or something13:51
bauzasokay, I can look at httpd's logs 13:53
bauzasdansmith: can you help me for a sec ? I want to backup my instance but then it's queued on the glance side for a while and eventually I no longer see the image13:58
dansmithabout to start a meeting, but can try in parallel13:58
dansmithusing backup or snapshot?13:58
bauzasnothing is told on the glance side, only the server event record in nova gives me the exception13:58
bauzasserver backup13:58
dansmithbecause you want to test our backup stuff, or because you want a snapshot of your image?13:59
dansmithI don't know what the plumbing through our server backup stuff looks like in detail, which is why I'm asking13:59
dansmithso you see an image created on the glance side (POST /images) .. do you see PUT/POST /images/$uuid/data ?14:00
*** sfinucan is now known as stephenfin14:00
bauzasdansmith: yep14:02
bauzasPOST /images14:02
bauzasI have an instance, I just want a new image from it14:02
dansmithokay, I'd be using snapshot not backup.. backup is a bit of a different thing.. still results in an image, but snapshot is more straightfortward14:03
sean-k-mooneyif using osc i think you shoudl b eusing "openstack server create image"14:04
sean-k-mooneyto take the snapshot14:04
sean-k-mooneybut ya backup is for snapshot roation really14:04
dansmithsean-k-mooney: exactly14:04
sean-k-mooneyalso backup does not work for BFV as an fyi14:04
sean-k-mooneynot sure why but it does not14:05
sean-k-mooneywe call it out in the api ref in anycase https://docs.openstack.org/api-ref/compute/#create-server-back-up-createbackup-action14:05
sean-k-mooneyso its a know limitaiton not a bug14:05
bauzasokay, I'll try snapshot then14:06
bauzasdoh14:20
bauzaseq-67d7b42c-c6a6-46cc-a86b-7eb66afabb79 demo admin] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 5257 and delta 0] {{(pid=2376125) enforce_limits /usr/local/lib/python3.9/site-packages/oslo>14:20
sean-k-mooneyis that a 1TB limit ?14:25
sean-k-mooneywith a 5.2TB image14:25
dansmithno I think it's KiB14:25
sean-k-mooneyimage size is in GB right14:25
dansmithor maybe MiB14:25
dansmithI don't remember :P14:25
sean-k-mooneywe have local_MB for our storage class14:26
opendevreviewMerged openstack/nova master: Fix notification object hashes for python 3.12  https://review.opendev.org/c/openstack/nova/+/91976314:26
sean-k-mooneybut the flavor and glance i tought was in  GBs14:26
opendevreviewMerged openstack/nova master: Fix hacking test with syntax error  https://review.opendev.org/c/openstack/nova/+/91976414:26
dansmithhttps://docs.openstack.org/glance/latest/admin/quotas.html14:26
dansmithMiB14:27
sean-k-mooneyok so the quota is also in MiB14:27
dansmithI wrote that feature, but they already had some limits in MiB so that's why it uses the same units they chose14:28
dansmithmost of the image stuff is MiB, volumes in GiB14:28
bauzasyep, saw it14:28
bauzas1000 for me14:28
bauzasso, only images for less than 1GB14:28
bauzashence why14:28
sean-k-mooneydansmith: ok the galnce api is kind of incositent. the size field on an image is in bytes, the min_disk is in GB and virtual_size is undefined but presumable also bytes14:29
dansmithbytes yeah14:30
dansmithsean-k-mooney: what min_disk are you talking about?14:33
sean-k-mooneyon the image14:34
sean-k-mooneyyou can spcify the min disk for an instance or voluem14:34
sean-k-mooneythat is defiend in the api ref as in GB14:34
sean-k-mooneywhich is why i was confused the quota woudl be in MiB when the api either uese bytes or GB elsehwere14:34
sean-k-mooneyanyway not imporant14:34
dansmithisn't that a nova thing?14:36
dansmithor nova/cinder anyway14:36
sean-k-mooneyhttps://docs.openstack.org/api-ref/image/v2/index.html#create-image14:38
sean-k-mooneyits a filed in the respocne 14:38
sean-k-mooneyoh and the request14:39
dansmithyeah those aren't core glance properties, I think those were added for nova/cinder and probably inherited the units14:39
sean-k-mooneythey have been there since i worked on openstack and have never been called out as any diffent then name14:40
dansmithyeah I'm just sayin' they're not glance-specific :)14:40
zigoHi there! One quick question: how can I decrypt the output of "nova get-password <server>" knowing that I don't have access to my private key (it's stored in my yubikey...) ?14:47
sean-k-mooneydansmith: but it considerd a base property by glance "https://github.com/openstack/glance/blob/705b145ab83defb3cb2f8d3243e1ce900fe337b9/glance/api/v2/images.py#L1121" proably becasue of how old and wid spread it is14:48
sean-k-mooneyzigo: its encypted with the serverers keypair14:49
zigosean-k-mooney: Yeah, I know, but with what command do I decrypt it?14:49
sean-k-mooneyzigo: so you can use your ssh private key14:49
zigoIt's in my yubikey ...14:49
zigoI don't have it as a file.14:49
dansmithsean-k-mooney: in how it's stored, because of age yeah14:49
sean-k-mooneyzigo: i guess you cant unless you can use ssh to do the decyption somehow14:51
zigosean-k-mooney: novaclient uses in novaclient/crypto.py: cmd = ['openssl', 'rsautl', '-decrypt', '-inkey', private_key]14:52
zigoThough rsautl is deprecated ... :/14:52
sean-k-mooneyyep and im not sure we ever supproted it in osc14:53
sean-k-mooneybecause we dont want to main that14:53
bauzashuzzah it worked14:53
bauzasupping the registered limit, I mean14:53
bauzassize             | 2557188505614:54
bauzasvoila the reason14:54
bauzas25GB14:54
clarkbzigo: https://docs.yubico.com/hardware/yubihsm-2/hsm-2-user-guide/hsm2-openssl-libp11.html this is for their hsm product but maybe similar for regular yubikeys15:26
clarkbalternatively ssh using the key then change the password and don't bother involving nova at all?15:27
zigoclarkb: There's no such thing as sshd in our windows instances, just cloud-base-init. I'm trying to figure out what format "nova get-password" is giving out, and if I can unencrypt it with my yubikey...15:36
clarkbzigo: got it (I think you can run an sshd on windows now fwiw, but that makes sense). Maybe those yubikey hsm docs are the clue you need for using the hardware to decrypt15:37
zigoThanks. I'm reading it.15:38
dansmithyep sshd on windows is trivial now15:38
dansmithbuilt-in almost15:38
opendevreviewMerged openstack/nova master: Make python 3.12 unit and functional voting  https://review.opendev.org/c/openstack/nova/+/91976715:42
bauzasmelwitt: I want to demonstrate VGPU quota usage on my demo next week16:54
bauzaswhen setting unifiedlimits driver in nova.conf, which services are using it ? n-sch and n-api right?16:55
bauzasnone from the conductor or the compute ?16:55
melwittbauzas: I think only n-api. I'll double check16:56
bauzasI quickly looked at the code, I only found the scheduler and the api were importing nova.quota16:56
melwittbauzas: there's a "recheck" in nova-conductor that enforces "strict quota" like races16:58
bauzasthen I need to also update nova_cell1.conf16:59
bauzasUnable to initialize OpenStackSDK session: An auth plugin is required to determine endpoint URL: keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin17:00
bauzashumpf, I probably need to set something else17:00
bauzasmelwitt: do you know what I missed ? ^17:01
bauzas(that's the API exception I get when wanting to create a new instance)17:01
melwittlooking..17:02
* bauzas checking the keystone section in conf17:02
bauzasoh, I have nothing17:02
bauzasdevstack doesn't automatically set it17:02
melwittyeah.. maybe something switched to using the sdk since then? I don't remember seeing anything like this17:04
melwittI need to lookup what's needed for sdk. though it seems surprising it wouldn't be done automatically in devstack like you said17:04
melwittoh huh that's actually coming from oslo.limit. so likely something needed in an [oslo_limit] conf section. still looking17:08
bauzasyeah, still getting the api exception17:09
bauzasafter restarting n-api (and setting [keystone] section)à17:09
* bauzas tries to look at anything from oslo.limit17:09
melwitthttps://docs.openstack.org/oslo.limit/latest/user/usage.html#configuration17:10
bauzashttps://docs.openstack.org/oslo.limit/latest/user/usage.html#configuration17:10
bauzasheh17:10
melwittI'm just confused how/where this got missed. and I'll will add stuff to the quota docs 17:11
melwittok, here it is in devstack, it should have configured it for you https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L81717:12
dansmithdo we not have a job with unified limits enabled?17:12
melwitt(I'll still add to docs)17:12
melwittwe do, it's on in nova-next17:12
dansmithack, so it must be in devstack unless we're hacking it all on in our job config right?17:13
melwittyeah I just linked the devstack stuff above17:14
melwittbauzas maybe didn't do the config through devstack or something (?)17:14
dansmithI know, I'm just talking about the assertion of it being missed or not supported 17:14
dansmithright17:14
melwittNOVA_USE_UNIFIED_LIMITS=True should do the stuff17:14
melwittoh, sure17:14
bauzasno, I just modified nova.conf directly17:15
bauzasI don't want to set quotas for all resources but only VGPU17:16
bauzasI know it will mean that all the quotas limits will be unlimitied :)17:16
melwittok cool. I'll propose a patch to add the oslo_limit config info to the docs, that's what I missed17:17
melwittbauzas: make sure you set a limit for DISK_GB otherwise it will treat it as 0 limit17:18
melwittthat's the only new "core" quota resource17:18
bauzasmelwitt: oh ok17:18
melwittoh wait, actually you have to set all of them bc you didn't do in devstack. but you can just copy what's in devstack17:19
melwitthttps://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L789-L81217:20
melwittand then this to make the nova service user able to read the quotas https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L827-L82917:20
bauzasyup, I'm done now, I just need to tweak the registered limits17:24
bauzasProject 1471d08833d141c583b5f04344476ebd is over a limit for [Resource class:DISK_GB is over limit of 0 due to current usage 80 and delta 40, Resource class:MEMORY_MB is over limit of 0 due to current usage 16384 and delta 8192, Resource class:VCPU is over limit of 0 due to current usage 2 and delta 1, Resource class:VGPU is over limit of 0 due to current usage 2 and delta 1, Resource servers is over limit of 0 due to current usa17:24
bauzasge 2 and delta 1] (HTTP 403) (Request-ID: req-980c41e8-6f18-4e12-984b-7eb5daa7d9e8)17:24
bauzasso, I not only need to add a limit for DISK_GB but also the other classes17:24
melwittyes. any class that's going to be requested has to have a registered limit else oslo.limit treats it as zero17:29
bauzasyup I know it :)17:30
melwitt(which is why we were discussing at the PTG about how to make an unlimited option in nova)17:30
bauzasyeah my brain fried17:30
bauzassometimes my brain can't reconcile two things :)17:31
melwitt:)17:31
bauzasyay that works17:35
bauzasI have everything now for my demo, I just need to record it :)17:35
bauzasmy next nights => bye bye17:35
melwitt\o/17:36
bauzasthanks melwitt for the quick resolution and the help17:36
bauzasCERN will be more happy to test unified limits for VGPU if they haven't done it yet :)17:36
melwittnp. you can ping me if you have questions or run into any other issues17:36
melwittthat would be super :)17:37
bauzasno, unified limits is waaaay simplier that f**** nvidia kernel usage :)17:37
melwitthaha! yeah, I guess if you can nvidia then you can do anything 😆 17:38
bauzasat least now I know more about cuda and numba :)17:39
bauzaslike the fact we need to use a specific cuda release related to the mdev type17:39
bauzas(and the kernel nvidia version)17:40
sean-k-mooneyi mean its not surpsing the have spit there mdev by vdi vs comptue usage in there skus forever18:11

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!