| opendevreview | melanie witt proposed openstack/nova-specs master: Re-propose specs for ephemeral encryption https://review.opendev.org/c/openstack/nova-specs/+/907654 | 00:52 |
|---|---|---|
| gmann | thanks | 01:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/nova master: Handle MAC dash-addresses during migration https://review.opendev.org/c/openstack/nova/+/919760 | 07:31 |
| opendevreview | Pierre Riteau proposed openstack/nova master: Fix formatting issues in extra-specs docs https://review.opendev.org/c/openstack/nova/+/917462 | 09:08 |
| stephenfin | bauzas: Friendly reminder about that OpenAPI spec needing your attention, please and thank you | 10:03 |
| bauzas | stephenfin: I'm currently working hard on creating my presentation for the next OpenInfra Day in Paris next week, but I'll try | 10:03 |
| *** sfinucan is now known as stephenfin | 11:35 | |
| opendevreview | Merged openstack/osc-placement master: Update testing of python versions https://review.opendev.org/c/openstack/osc-placement/+/904612 | 12:18 |
| opendevreview | Merged openstack/osc-placement master: Make python 3.12 functional job voting https://review.opendev.org/c/openstack/osc-placement/+/919101 | 12:29 |
| opendevreview | Merged openstack/nova master: Fix formatting issues in extra-specs docs https://review.opendev.org/c/openstack/nova/+/917462 | 13:16 |
| opendevreview | Merged openstack/nova stable/2023.2: Adding server actions tests to grenade-multinode https://review.opendev.org/c/openstack/nova/+/900339 | 13:36 |
| dansmith | bauzas: this stack fixes python 3.12 functional/unit tests if you could help merge that before we regress :) https://review.opendev.org/c/openstack/nova/+/919763 | 13:42 |
| bauzas | dansmith: okay, I don't really have time but I trust you, +Wd | 13:43 |
| dansmith | bauzas: okay, it's a stack of three.. maybe gibi could look at the other two, or maybe melwitt when she's around | 13:44 |
| sean-k-mooney | only the second one https://review.opendev.org/c/openstack/nova/+/919764/2 | 13:44 |
| sean-k-mooney | is missing a +w at this point | 13:44 |
| bauzas | this is done | 13:44 |
| sean-k-mooney | cool | 13:44 |
| bauzas | the fixes are simple but I don't really have time to think about some concerns | 13:45 |
| dansmith | thanks | 13:45 |
| dansmith | bauzas: yep, safe I think | 13:45 |
| sean-k-mooney | form the nova side yes | 13:45 |
| * bauzas tries to understand why Glance returns some gateway exception when I want to backup my instance | 13:45 | |
| bauzas | dansmith: glance-api is absolutely blind on the req-id, so how could I see any logs from glance apart from devstack@g-api | 13:46 |
| dansmith | is it/ | 13:46 |
| dansmith | but gateway exception usually means apache not glance | 13:47 |
| dansmith | unless glance is 500'ing or something | 13:51 |
| bauzas | okay, I can look at httpd's logs | 13:53 |
| bauzas | dansmith: can you help me for a sec ? I want to backup my instance but then it's queued on the glance side for a while and eventually I no longer see the image | 13:58 |
| dansmith | about to start a meeting, but can try in parallel | 13:58 |
| dansmith | using backup or snapshot? | 13:58 |
| bauzas | nothing is told on the glance side, only the server event record in nova gives me the exception | 13:58 |
| bauzas | server backup | 13:58 |
| dansmith | because you want to test our backup stuff, or because you want a snapshot of your image? | 13:59 |
| dansmith | I don't know what the plumbing through our server backup stuff looks like in detail, which is why I'm asking | 13:59 |
| dansmith | so you see an image created on the glance side (POST /images) .. do you see PUT/POST /images/$uuid/data ? | 14:00 |
| *** sfinucan is now known as stephenfin | 14:00 | |
| bauzas | dansmith: yep | 14:02 |
| bauzas | POST /images | 14:02 |
| bauzas | I have an instance, I just want a new image from it | 14:02 |
| dansmith | okay, I'd be using snapshot not backup.. backup is a bit of a different thing.. still results in an image, but snapshot is more straightfortward | 14:03 |
| sean-k-mooney | if using osc i think you shoudl b eusing "openstack server create image" | 14:04 |
| sean-k-mooney | to take the snapshot | 14:04 |
| sean-k-mooney | but ya backup is for snapshot roation really | 14:04 |
| dansmith | sean-k-mooney: exactly | 14:04 |
| sean-k-mooney | also backup does not work for BFV as an fyi | 14:04 |
| sean-k-mooney | not sure why but it does not | 14:05 |
| sean-k-mooney | we call it out in the api ref in anycase https://docs.openstack.org/api-ref/compute/#create-server-back-up-createbackup-action | 14:05 |
| sean-k-mooney | so its a know limitaiton not a bug | 14:05 |
| bauzas | okay, I'll try snapshot then | 14:06 |
| bauzas | doh | 14:20 |
| bauzas | eq-67d7b42c-c6a6-46cc-a86b-7eb66afabb79 demo admin] hit limit for project: [Resource image_size_total is over limit of 1000 due to current usage 5257 and delta 0] {{(pid=2376125) enforce_limits /usr/local/lib/python3.9/site-packages/oslo> | 14:20 |
| sean-k-mooney | is that a 1TB limit ? | 14:25 |
| sean-k-mooney | with a 5.2TB image | 14:25 |
| dansmith | no I think it's KiB | 14:25 |
| sean-k-mooney | image size is in GB right | 14:25 |
| dansmith | or maybe MiB | 14:25 |
| dansmith | I don't remember :P | 14:25 |
| sean-k-mooney | we have local_MB for our storage class | 14:26 |
| opendevreview | Merged openstack/nova master: Fix notification object hashes for python 3.12 https://review.opendev.org/c/openstack/nova/+/919763 | 14:26 |
| sean-k-mooney | but the flavor and glance i tought was in GBs | 14:26 |
| opendevreview | Merged openstack/nova master: Fix hacking test with syntax error https://review.opendev.org/c/openstack/nova/+/919764 | 14:26 |
| dansmith | https://docs.openstack.org/glance/latest/admin/quotas.html | 14:26 |
| dansmith | MiB | 14:27 |
| sean-k-mooney | ok so the quota is also in MiB | 14:27 |
| dansmith | I wrote that feature, but they already had some limits in MiB so that's why it uses the same units they chose | 14:28 |
| dansmith | most of the image stuff is MiB, volumes in GiB | 14:28 |
| bauzas | yep, saw it | 14:28 |
| bauzas | 1000 for me | 14:28 |
| bauzas | so, only images for less than 1GB | 14:28 |
| bauzas | hence why | 14:28 |
| sean-k-mooney | dansmith: ok the galnce api is kind of incositent. the size field on an image is in bytes, the min_disk is in GB and virtual_size is undefined but presumable also bytes | 14:29 |
| dansmith | bytes yeah | 14:30 |
| dansmith | sean-k-mooney: what min_disk are you talking about? | 14:33 |
| sean-k-mooney | on the image | 14:34 |
| sean-k-mooney | you can spcify the min disk for an instance or voluem | 14:34 |
| sean-k-mooney | that is defiend in the api ref as in GB | 14:34 |
| sean-k-mooney | which is why i was confused the quota woudl be in MiB when the api either uese bytes or GB elsehwere | 14:34 |
| sean-k-mooney | anyway not imporant | 14:34 |
| dansmith | isn't that a nova thing? | 14:36 |
| dansmith | or nova/cinder anyway | 14:36 |
| sean-k-mooney | https://docs.openstack.org/api-ref/image/v2/index.html#create-image | 14:38 |
| sean-k-mooney | its a filed in the respocne | 14:38 |
| sean-k-mooney | oh and the request | 14:39 |
| dansmith | yeah those aren't core glance properties, I think those were added for nova/cinder and probably inherited the units | 14:39 |
| sean-k-mooney | they have been there since i worked on openstack and have never been called out as any diffent then name | 14:40 |
| dansmith | yeah I'm just sayin' they're not glance-specific :) | 14:40 |
| zigo | Hi there! One quick question: how can I decrypt the output of "nova get-password <server>" knowing that I don't have access to my private key (it's stored in my yubikey...) ? | 14:47 |
| sean-k-mooney | dansmith: but it considerd a base property by glance "https://github.com/openstack/glance/blob/705b145ab83defb3cb2f8d3243e1ce900fe337b9/glance/api/v2/images.py#L1121" proably becasue of how old and wid spread it is | 14:48 |
| sean-k-mooney | zigo: its encypted with the serverers keypair | 14:49 |
| zigo | sean-k-mooney: Yeah, I know, but with what command do I decrypt it? | 14:49 |
| sean-k-mooney | zigo: so you can use your ssh private key | 14:49 |
| zigo | It's in my yubikey ... | 14:49 |
| zigo | I don't have it as a file. | 14:49 |
| dansmith | sean-k-mooney: in how it's stored, because of age yeah | 14:49 |
| sean-k-mooney | zigo: i guess you cant unless you can use ssh to do the decyption somehow | 14:51 |
| zigo | sean-k-mooney: novaclient uses in novaclient/crypto.py: cmd = ['openssl', 'rsautl', '-decrypt', '-inkey', private_key] | 14:52 |
| zigo | Though rsautl is deprecated ... :/ | 14:52 |
| sean-k-mooney | yep and im not sure we ever supproted it in osc | 14:53 |
| sean-k-mooney | because we dont want to main that | 14:53 |
| bauzas | huzzah it worked | 14:53 |
| bauzas | upping the registered limit, I mean | 14:53 |
| bauzas | size | 25571885056 | 14:54 |
| bauzas | voila the reason | 14:54 |
| bauzas | 25GB | 14:54 |
| clarkb | zigo: https://docs.yubico.com/hardware/yubihsm-2/hsm-2-user-guide/hsm2-openssl-libp11.html this is for their hsm product but maybe similar for regular yubikeys | 15:26 |
| clarkb | alternatively ssh using the key then change the password and don't bother involving nova at all? | 15:27 |
| zigo | clarkb: There's no such thing as sshd in our windows instances, just cloud-base-init. I'm trying to figure out what format "nova get-password" is giving out, and if I can unencrypt it with my yubikey... | 15:36 |
| clarkb | zigo: got it (I think you can run an sshd on windows now fwiw, but that makes sense). Maybe those yubikey hsm docs are the clue you need for using the hardware to decrypt | 15:37 |
| zigo | Thanks. I'm reading it. | 15:38 |
| dansmith | yep sshd on windows is trivial now | 15:38 |
| dansmith | built-in almost | 15:38 |
| opendevreview | Merged openstack/nova master: Make python 3.12 unit and functional voting https://review.opendev.org/c/openstack/nova/+/919767 | 15:42 |
| bauzas | melwitt: I want to demonstrate VGPU quota usage on my demo next week | 16:54 |
| bauzas | when setting unifiedlimits driver in nova.conf, which services are using it ? n-sch and n-api right? | 16:55 |
| bauzas | none from the conductor or the compute ? | 16:55 |
| melwitt | bauzas: I think only n-api. I'll double check | 16:56 |
| bauzas | I quickly looked at the code, I only found the scheduler and the api were importing nova.quota | 16:56 |
| melwitt | bauzas: there's a "recheck" in nova-conductor that enforces "strict quota" like races | 16:58 |
| bauzas | then I need to also update nova_cell1.conf | 16:59 |
| bauzas | Unable to initialize OpenStackSDK session: An auth plugin is required to determine endpoint URL: keystoneauth1.exceptions.auth_plugins.MissingAuthPlugin | 17:00 |
| bauzas | humpf, I probably need to set something else | 17:00 |
| bauzas | melwitt: do you know what I missed ? ^ | 17:01 |
| bauzas | (that's the API exception I get when wanting to create a new instance) | 17:01 |
| melwitt | looking.. | 17:02 |
| * bauzas checking the keystone section in conf | 17:02 | |
| bauzas | oh, I have nothing | 17:02 |
| bauzas | devstack doesn't automatically set it | 17:02 |
| melwitt | yeah.. maybe something switched to using the sdk since then? I don't remember seeing anything like this | 17:04 |
| melwitt | I need to lookup what's needed for sdk. though it seems surprising it wouldn't be done automatically in devstack like you said | 17:04 |
| melwitt | oh huh that's actually coming from oslo.limit. so likely something needed in an [oslo_limit] conf section. still looking | 17:08 |
| bauzas | yeah, still getting the api exception | 17:09 |
| bauzas | after restarting n-api (and setting [keystone] section)à | 17:09 |
| * bauzas tries to look at anything from oslo.limit | 17:09 | |
| melwitt | https://docs.openstack.org/oslo.limit/latest/user/usage.html#configuration | 17:10 |
| bauzas | https://docs.openstack.org/oslo.limit/latest/user/usage.html#configuration | 17:10 |
| bauzas | heh | 17:10 |
| melwitt | I'm just confused how/where this got missed. and I'll will add stuff to the quota docs | 17:11 |
| melwitt | ok, here it is in devstack, it should have configured it for you https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L817 | 17:12 |
| dansmith | do we not have a job with unified limits enabled? | 17:12 |
| melwitt | (I'll still add to docs) | 17:12 |
| melwitt | we do, it's on in nova-next | 17:12 |
| dansmith | ack, so it must be in devstack unless we're hacking it all on in our job config right? | 17:13 |
| melwitt | yeah I just linked the devstack stuff above | 17:14 |
| melwitt | bauzas maybe didn't do the config through devstack or something (?) | 17:14 |
| dansmith | I know, I'm just talking about the assertion of it being missed or not supported | 17:14 |
| dansmith | right | 17:14 |
| melwitt | NOVA_USE_UNIFIED_LIMITS=True should do the stuff | 17:14 |
| melwitt | oh, sure | 17:14 |
| bauzas | no, I just modified nova.conf directly | 17:15 |
| bauzas | I don't want to set quotas for all resources but only VGPU | 17:16 |
| bauzas | I know it will mean that all the quotas limits will be unlimitied :) | 17:16 |
| melwitt | ok cool. I'll propose a patch to add the oslo_limit config info to the docs, that's what I missed | 17:17 |
| melwitt | bauzas: make sure you set a limit for DISK_GB otherwise it will treat it as 0 limit | 17:18 |
| melwitt | that's the only new "core" quota resource | 17:18 |
| bauzas | melwitt: oh ok | 17:18 |
| melwitt | oh wait, actually you have to set all of them bc you didn't do in devstack. but you can just copy what's in devstack | 17:19 |
| melwitt | https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L789-L812 | 17:20 |
| melwitt | and then this to make the nova service user able to read the quotas https://github.com/openstack/devstack/blob/85b8d2ccab53bb2b38090c8801404072fae2f7ab/lib/nova#L827-L829 | 17:20 |
| bauzas | yup, I'm done now, I just need to tweak the registered limits | 17:24 |
| bauzas | Project 1471d08833d141c583b5f04344476ebd is over a limit for [Resource class:DISK_GB is over limit of 0 due to current usage 80 and delta 40, Resource class:MEMORY_MB is over limit of 0 due to current usage 16384 and delta 8192, Resource class:VCPU is over limit of 0 due to current usage 2 and delta 1, Resource class:VGPU is over limit of 0 due to current usage 2 and delta 1, Resource servers is over limit of 0 due to current usa | 17:24 |
| bauzas | ge 2 and delta 1] (HTTP 403) (Request-ID: req-980c41e8-6f18-4e12-984b-7eb5daa7d9e8) | 17:24 |
| bauzas | so, I not only need to add a limit for DISK_GB but also the other classes | 17:24 |
| melwitt | yes. any class that's going to be requested has to have a registered limit else oslo.limit treats it as zero | 17:29 |
| bauzas | yup I know it :) | 17:30 |
| melwitt | (which is why we were discussing at the PTG about how to make an unlimited option in nova) | 17:30 |
| bauzas | yeah my brain fried | 17:30 |
| bauzas | sometimes my brain can't reconcile two things :) | 17:31 |
| melwitt | :) | 17:31 |
| bauzas | yay that works | 17:35 |
| bauzas | I have everything now for my demo, I just need to record it :) | 17:35 |
| bauzas | my next nights => bye bye | 17:35 |
| melwitt | \o/ | 17:36 |
| bauzas | thanks melwitt for the quick resolution and the help | 17:36 |
| bauzas | CERN will be more happy to test unified limits for VGPU if they haven't done it yet :) | 17:36 |
| melwitt | np. you can ping me if you have questions or run into any other issues | 17:36 |
| melwitt | that would be super :) | 17:37 |
| bauzas | no, unified limits is waaaay simplier that f**** nvidia kernel usage :) | 17:37 |
| melwitt | haha! yeah, I guess if you can nvidia then you can do anything 😆 | 17:38 |
| bauzas | at least now I know more about cuda and numba :) | 17:39 |
| bauzas | like the fact we need to use a specific cuda release related to the mdev type | 17:39 |
| bauzas | (and the kernel nvidia version) | 17:40 |
| sean-k-mooney | i mean its not surpsing the have spit there mdev by vdi vs comptue usage in there skus forever | 18:11 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!