| *** k_mouza has joined #openstack-nova | 00:00 | |
| *** k_mouza has quit IRC | 00:05 | |
| *** tosky has quit IRC | 00:14 | |
| *** LinPeiWen46 has joined #openstack-nova | 00:36 | |
| *** openstackgerrit has joined #openstack-nova | 00:42 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication https://review.opendev.org/759828 | 00:42 |
|---|---|---|
| *** csatari has quit IRC | 00:57 | |
| *** csatari has joined #openstack-nova | 01:00 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Remove tenant_id https://review.opendev.org/737241 | 01:07 |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: [Trivial] update the upgrade release goal https://review.opendev.org/763294 | 01:10 |
| *** sapd1 has quit IRC | 01:15 | |
| *** spatel has joined #openstack-nova | 01:19 | |
| *** spatel has quit IRC | 01:23 | |
| *** martinkennelly has quit IRC | 01:25 | |
| *** gyee has quit IRC | 01:38 | |
| *** macz_ has quit IRC | 02:08 | |
| *** hamalq has quit IRC | 02:09 | |
| *** zzzeek has quit IRC | 02:17 | |
| *** zzzeek has joined #openstack-nova | 02:19 | |
| *** rcernin has quit IRC | 02:22 | |
| *** rcernin has joined #openstack-nova | 02:26 | |
| *** jobewan has joined #openstack-nova | 02:29 | |
| *** zzzeek has quit IRC | 02:35 | |
| *** zzzeek has joined #openstack-nova | 02:36 | |
| *** mkrai has joined #openstack-nova | 02:50 | |
| *** swp20 has joined #openstack-nova | 03:03 | |
| *** xinranwang has joined #openstack-nova | 03:12 | |
| *** mkrai has quit IRC | 03:14 | |
| *** mkrai_ has joined #openstack-nova | 03:14 | |
| *** macz_ has joined #openstack-nova | 03:20 | |
| *** macz_ has quit IRC | 03:25 | |
| *** Yumeng has joined #openstack-nova | 03:54 | |
| *** vishalmanchanda has joined #openstack-nova | 04:30 | |
| *** mkrai_ has quit IRC | 04:49 | |
| *** mkrai__ has joined #openstack-nova | 04:49 | |
| *** dklyle has quit IRC | 04:57 | |
| *** macz_ has joined #openstack-nova | 05:02 | |
| *** macz_ has quit IRC | 05:06 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #openstack-nova | 05:33 | |
| *** yingjisun has joined #openstack-nova | 05:46 | |
| *** k_mouza has joined #openstack-nova | 06:01 | |
| *** JamesBenson has quit IRC | 06:03 | |
| *** JamesBenson has joined #openstack-nova | 06:04 | |
| *** k_mouza has quit IRC | 06:06 | |
| *** ratailor has joined #openstack-nova | 06:08 | |
| *** JamesBenson has quit IRC | 06:09 | |
| *** JamesBenson has joined #openstack-nova | 06:15 | |
| *** ninad has joined #openstack-nova | 06:19 | |
| ninad | Hi | 06:20 |
| ninad | I am using linuxbridge for the neutron services and follow the installation guide as documented but during manila instance creation I am getting [Errno 113] EHOSTUNREACH _test_server_connection | 06:20 |
| ninad | can someone please help me? | 06:21 |
| *** hamalq has joined #openstack-nova | 06:25 | |
| *** macz_ has joined #openstack-nova | 06:29 | |
| *** macz_ has quit IRC | 06:34 | |
| *** ninad has quit IRC | 06:35 | |
| *** mkrai__ has quit IRC | 06:49 | |
| *** mkrai__ has joined #openstack-nova | 06:52 | |
| *** swp20 has quit IRC | 06:55 | |
| *** swp20 has joined #openstack-nova | 06:57 | |
| *** slaweq has joined #openstack-nova | 07:01 | |
| *** yingjisun_ has joined #openstack-nova | 07:12 | |
| *** yingjisun has quit IRC | 07:12 | |
| *** yingjisun_ is now known as yingjisun | 07:12 | |
| *** mkrai__ has quit IRC | 07:34 | |
| *** ralonsoh has joined #openstack-nova | 07:43 | |
| *** Yumeng has quit IRC | 07:46 | |
| *** bhagyashris|off is now known as bhagyashris | 07:55 | |
| openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | 07:58 |
| *** mkrai__ has joined #openstack-nova | 08:01 | |
| *** rpittau|afk is now known as rpittau | 08:03 | |
| openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | 08:06 |
| *** andrewbonney has joined #openstack-nova | 08:10 | |
| *** tesseract has joined #openstack-nova | 08:18 | |
| *** hamalq has quit IRC | 08:28 | |
| *** rcernin has quit IRC | 08:37 | |
| *** ircuser-1 has quit IRC | 08:38 | |
| *** tosky has joined #openstack-nova | 08:44 | |
| *** lpetrut has joined #openstack-nova | 08:47 | |
| *** yingjisun has quit IRC | 08:51 | |
| *** mgoddard has joined #openstack-nova | 08:58 | |
| *** xek has joined #openstack-nova | 08:59 | |
| *** mlavalle has quit IRC | 09:09 | |
| *** mlavalle has joined #openstack-nova | 09:12 | |
| lyarwood | aarents: apologies, had a fun day downstream yesterday, your test changes LGTM, I missed that you had already pulled in the devstack change within the tempest change so assuming it's still passing this should be good to go now. | 09:24 |
| *** hamalq has joined #openstack-nova | 09:29 | |
| aarents | lyarwood: many thanks, I just repush with nit fix sugested by gmann | 09:31 |
| *** hamalq has quit IRC | 09:34 | |
| *** ociuhandu has joined #openstack-nova | 09:41 | |
| *** dtantsur|afk is now known as dtantsur | 09:41 | |
| *** k_mouza has joined #openstack-nova | 09:57 | |
| *** jangutter_ has quit IRC | 10:01 | |
| *** jangutter has joined #openstack-nova | 10:02 | |
| *** hamalq has joined #openstack-nova | 10:07 | |
| *** jangutter_ has joined #openstack-nova | 10:11 | |
| *** xinranwang has quit IRC | 10:11 | |
| *** hamalq has quit IRC | 10:11 | |
| *** jangutter has quit IRC | 10:14 | |
| *** luksky has joined #openstack-nova | 10:18 | |
| stephenfin | sean-k-mooney: I think there's something broken in https://review.opendev.org/#/c/602432/. It appears grenade hasn't passed on that for the last couple of revisions. Haven't gone diving through logs yet | 10:21 |
| *** gibi_away is now known as gibi | 10:25 | |
| *** hamalq has joined #openstack-nova | 10:27 | |
| *** hamalq has quit IRC | 10:32 | |
| *** ociuhandu has quit IRC | 10:36 | |
| *** mkrai__ has quit IRC | 10:40 | |
| *** ociuhandu has joined #openstack-nova | 10:43 | |
| *** xek_ has joined #openstack-nova | 10:45 | |
| *** xek has quit IRC | 10:45 | |
| *** swp20 has quit IRC | 10:47 | |
| *** mkrai__ has joined #openstack-nova | 10:48 | |
| *** bnemec has quit IRC | 11:00 | |
| *** bnemec has joined #openstack-nova | 11:01 | |
| *** ociuhandu has quit IRC | 11:08 | |
| *** hamalq has joined #openstack-nova | 11:09 | |
| openstackgerrit | Jorhson Deng proposed openstack/nova master: To deal instance with soft-deleting in _init_instance https://review.opendev.org/761264 | 11:13 |
| *** hamalq has quit IRC | 11:14 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: zuul: Add devstack-plugin-ceph-compute-local-ephemeral to experimental https://review.opendev.org/743220 | 11:14 |
| *** ratailor_ has joined #openstack-nova | 11:18 | |
| *** ratailor has quit IRC | 11:21 | |
| *** swp20 has joined #openstack-nova | 11:23 | |
| *** ociuhandu has joined #openstack-nova | 11:28 | |
| *** tkajinam has quit IRC | 11:29 | |
| *** tkajinam has joined #openstack-nova | 11:30 | |
| *** hamalq has joined #openstack-nova | 11:30 | |
| *** hamalq has quit IRC | 11:35 | |
| *** ociuhandu has quit IRC | 11:41 | |
| *** ociuhandu has joined #openstack-nova | 11:42 | |
| sean-k-mooney | stephenfin: if we were to go from new to old i guess it would break | 11:45 |
| sean-k-mooney | the xml would be generated on the souce with type ethernet | 11:46 |
| sean-k-mooney | and the os-vif on the dest would be a noop | 11:46 |
| sean-k-mooney | so nothing would plug the interface | 11:46 |
| *** macz_ has joined #openstack-nova | 11:46 | |
| *** ratailor_ has quit IRC | 11:46 | |
| *** mkrai__ has quit IRC | 11:47 | |
| sean-k-mooney | the singel node grenade have been passing but not multinode | 11:48 |
| sean-k-mooney | the only way i think of to fix that is to first backport the neutron change then flip the default for the option in os-vif/depercate it for removal and backport that too before we merge on master | 11:50 |
| sean-k-mooney | and i would then have to keep backporting the neutron and os-vif patches first for every release. | 11:50 |
| *** hamalq has joined #openstack-nova | 11:51 | |
| *** macz_ has quit IRC | 11:51 | |
| sean-k-mooney | stephenfin: lyarwood ^ is there a better way around that | 11:51 |
| * lyarwood reads back | 11:54 | |
| *** hamalq has quit IRC | 11:56 | |
| lyarwood | sean-k-mooney: we shouldn't be going from new to old in the tests right? | 12:06 |
| lyarwood | sean-k-mooney: only old to new, or is that not defined in grenade | 12:07 |
| sean-k-mooney | we do live migration back and fort | 12:07 |
| sean-k-mooney | in the multi node job | 12:07 |
| stephenfin | sean-k-mooney: You could make this dependent on a service version check | 12:08 |
| sean-k-mooney | not really i would have to keep changin it | 12:08 |
| stephenfin | why? Once everything's on Wallaby with a suitably new version of os-vif, we're good, no? | 12:08 |
| sean-k-mooney | i thory this should go to osp 10 so all the way back to queens | 12:08 |
| stephenfin | ah | 12:08 |
| sean-k-mooney | im thinking of stoping at train to be honest | 12:09 |
| sean-k-mooney | but it defiently needs to be backported a few releases | 12:09 |
| stephenfin | (osp 10 is newton, fwiw) | 12:09 |
| sean-k-mooney | ... of course it is | 12:09 |
| sean-k-mooney | since its related to a security issue we are still ment to backport it althou technically i guess 13 would be enough at this point | 12:10 |
| stephenfin | what you've proposed would require os-vif be upgraded before nova, right? I don't think we can count on that | 12:10 |
| stephenfin | *os-vif and neutron | 12:10 |
| sean-k-mooney | without the neturon patch you just get erros in the neutron logs but it fixes its self | 12:11 |
| sean-k-mooney | the os-vif change however would be needed first ya | 12:11 |
| *** hamalq has joined #openstack-nova | 12:12 | |
| sean-k-mooney | lyarwood: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_ff2/602432/26/check/nova-grenade-multinode/ff2bd9a/logs/new/tempest_conf.txt | 12:13 |
| sean-k-mooney | live_migrate_back_and_forth = True | 12:13 |
| sean-k-mooney | that is why its failing | 12:13 |
| gibi | stephenfin: do you have a lauchpad bp filled for this? https://review.opendev.org/#/c/755109 | 12:14 |
| *** raildo has joined #openstack-nova | 12:14 | |
| sean-k-mooney | its off in live migration job and that all the new version so it passes the https://zuul.opendev.org/t/openstack/build/e2fa8ae7802f41309877b537f51ba660/log/controller/logs/tempest_conf.txt#76 | 12:14 |
| stephenfin | gibi: I though I did but I'm not sure | 12:14 |
| * stephenfin checks | 12:14 | |
| gibi | the bp link in the specs does not seem to work for me | 12:14 |
| sean-k-mooney | ya cant find it either | 12:16 |
| *** hamalq has quit IRC | 12:16 | |
| sean-k-mooney | but it should be a simple copy past of the into section | 12:16 |
| stephenfin | gibi: Didn't look like it. Apologies. Have created one now https://blueprints.launchpad.net/nova/+spec/modernize-os-hypervisors-api | 12:17 |
| gibi | thanks | 12:17 |
| * gibi doing the paperwork | 12:17 | |
| sean-k-mooney | weird i cant add the spec via the spec link so i put it in the whiteborad | 12:19 |
| sean-k-mooney | i tought i could do that once i was on the bug team but i guess not | 12:19 |
| *** ociuhandu has quit IRC | 12:22 | |
| gibi | sean-k-mooney: I guess we edited the whiteboard in parallel and my change overwrote yours, sorry. | 12:28 |
| sean-k-mooney | no launchpad gave me a permission error | 12:30 |
| gibi | interesting, I got the following mail from launchpad | 12:31 |
| gibi | Blueprint changed by sean mooney: | 12:31 |
| gibi | Whiteboard set to: | 12:31 |
| gibi | spec: https://review.opendev.org/#/c/755109 | 12:31 |
| gibi | so at least some part of your update went through | 12:31 |
| *** hamalq has joined #openstack-nova | 12:34 | |
| openstackgerrit | Merged openstack/nova-specs master: [Trivial] update the upgrade release goal https://review.opendev.org/763294 | 12:34 |
| *** kevinz has joined #openstack-nova | 12:36 | |
| *** hamalq has quit IRC | 12:39 | |
| *** zzzeek has quit IRC | 12:45 | |
| *** martinkennelly has joined #openstack-nova | 12:46 | |
| *** zzzeek has joined #openstack-nova | 12:47 | |
| *** ociuhandu has joined #openstack-nova | 12:52 | |
| lyarwood | sean-k-mooney: sorry had to go afk | 12:55 |
| lyarwood | sean-k-mooney: so old to new works, it's just new to old that's failing right? | 12:56 |
| sean-k-mooney | lyarwood: yep | 12:57 |
| lyarwood | sean-k-mooney: do we care in that case, I know we don't downstream | 12:57 |
| sean-k-mooney | new to old nova uses type ethernet | 12:57 |
| *** ociuhandu has quit IRC | 12:57 | |
| sean-k-mooney | but does not tell os-vif to plug the interface by passing crate-port=true | 12:57 |
| sean-k-mooney | well once i do the backport it will alos work but we do allow rolling upgrades | 12:58 |
| sean-k-mooney | so we kind of do care yes | 12:58 |
| lyarwood | yeah I was thinking of major upgrades where we don't allow you to return to the older computes | 12:58 |
| lyarwood | minor upgrades I guess do allow this | 12:58 |
| sean-k-mooney | ya i would basiclaly need to know the version of nova on the dest | 13:00 |
| sean-k-mooney | but i cant just do a normal compute service check if im backporting | 13:00 |
| lyarwood | could we use something in migrate_data? | 13:00 |
| lyarwood | like I did with LUKS volumes | 13:00 |
| * lyarwood finds a reference | 13:00 | |
| sean-k-mooney | we coudl if we dont change the object | 13:01 |
| sean-k-mooney | i could stash something in the port profile or something | 13:01 |
| sean-k-mooney | but i then need to supprot both in the code | 13:01 |
| lyarwood | https://github.com/openstack/nova/blob/60071a2c83ad1d7ed6fd50f8af0bb4d92aa84bea/nova/virt/libvirt/driver.py#L9301-L9311 | 13:01 |
| sean-k-mooney | i ripped out the bridge way of doing things so would have to put it back in | 13:02 |
| lyarwood | that's not the best example as it does need an object change | 13:02 |
| sean-k-mooney | ya but i get the point | 13:02 |
| lyarwood | but yeah if the src can look for something stashed in there it might help | 13:02 |
| sean-k-mooney | i could put it into one of the unversioned dict of string fields | 13:02 |
| sean-k-mooney | ya so stash it n pre-live-migrate | 13:03 |
| sean-k-mooney | and read it on the souce when updating the xml | 13:03 |
| sean-k-mooney | ok ill have to think about it | 13:03 |
| sean-k-mooney | i know more or less how to do that but need to find where makes the most sense | 13:04 |
| lyarwood | kk | 13:04 |
| sean-k-mooney | im thing in here https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L30-L59 | 13:04 |
| sean-k-mooney | probaly the profile_json | 13:05 |
| sean-k-mooney | since nova owns that | 13:05 |
| sean-k-mooney | that is what is stored in the vifs filed of the liveMigrateData | 13:05 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L154 | 13:05 |
| lyarwood | kk | 13:06 |
| lyarwood | yeah that could work | 13:06 |
| lyarwood | and you would want to populate that on the dest btw | 13:06 |
| lyarwood | if it isn't set or True then the source would wire things up in the legacy way | 13:07 |
| sean-k-mooney | we are doing it here too https://review.opendev.org/#/c/738432/ | 13:07 |
| sean-k-mooney | lyarwood: yep | 13:07 |
| lyarwood | and then once it's backported you can remove the logic in the next release | 13:08 |
| sean-k-mooney | yes we coudl drop i in X | 13:08 |
| lyarwood | X, not W to be clear | 13:08 |
| lyarwood | yeah | 13:08 |
| lyarwood | migrate_data hackarounds ftw! | 13:09 |
| sean-k-mooney | im just glad we still have some dict of string fields in them or we would have to use system metadata which would be a pian | 13:09 |
| sean-k-mooney | or just not backport i guess | 13:10 |
| sean-k-mooney | in theory master should always be deployable and upgradeable however so we should fix this | 13:10 |
| sean-k-mooney | patchset 27 it is | 13:10 |
| *** rpittau is now known as rpittau|brb | 13:16 | |
| sean-k-mooney | stephenfin: melwitt: lyarwood: summerised this in a toplevel comment on the review | 13:18 |
| lyarwood | sean-k-mooney: ack thanks, LGTM | 13:19 |
| *** ociuhandu has joined #openstack-nova | 13:24 | |
| *** nweinber has joined #openstack-nova | 13:32 | |
| *** hamalq has joined #openstack-nova | 13:40 | |
| sean-k-mooney | johnthetubaguy: if you see this are you working on unified limits this cycle or did you say you wont have time? | 13:41 |
| sean-k-mooney | johnthetubaguy: just basically wondering since i dont think i have seen the spec repoposed althoguh i might have missed it | 13:41 |
| *** hamalq has quit IRC | 13:45 | |
| *** liuyulong has joined #openstack-nova | 14:00 | |
| *** ociuhandu has quit IRC | 14:02 | |
| *** ociuhandu has joined #openstack-nova | 14:02 | |
| *** tbachman has joined #openstack-nova | 14:30 | |
| *** liuyulong has quit IRC | 14:33 | |
| *** ociuhandu has quit IRC | 14:39 | |
| f0o | Hi, I've a question about passwords in nova. On Horizon I keep getting that the password is not set. Which config option in nova.conf do I need to make it create a password? I see `#password_length=12` and `#enable_instance_password=true` in the default config, are these values not the default ones? | 14:40 |
| sean-k-mooney | f0o: they might be but password injection only work if you have the qemu guest agent | 14:42 |
| sean-k-mooney | in general its not recommended to use passwords | 14:42 |
| f0o | I know but windows :/ | 14:43 |
| sean-k-mooney | f0o: yes they are the defaults | 14:43 |
| sean-k-mooney | you can use keypairs with windows | 14:43 |
| sean-k-mooney | you just dont use ssh keys | 14:43 |
| sean-k-mooney | you use x509 certs for winrm | 14:43 |
| f0o | if I read Cloudbase's docs correctly, it does not rely on injection but onyl on the metadata endpoint | 14:43 |
| *** happyhemant has joined #openstack-nova | 14:44 | |
| f0o | I remeber having this working at my previous job (nova+vmware) basically out of the box, now with kvm I cant seem to get a password back | 14:44 |
| f0o | or do I need to enable injection for it to return a password via api? | 14:45 |
| sean-k-mooney | as i said you need the qemu geust agent for ti to work with libvirt/qemu | 14:45 |
| f0o | k | 14:45 |
| f0o | what throws me off is: | 14:45 |
| f0o | The secure and proper way to set passwords in OpenStack Windows instances is by letting Cloudbase-Init generate a random password and post it encrypted on the Nova metadata service. | 14:45 |
| f0o | to me this means that cloud-init in this case is actually not doing it's job because the password is not set. trying to find the issue here since it can be both sides | 14:46 |
| sean-k-mooney | well the proper way to do it is via user-data yes | 14:46 |
| sean-k-mooney | but that is different to what those config options do | 14:46 |
| sean-k-mooney | f0o: https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-set-admin-password.html | 14:47 |
| sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-set-admin-password.html#other-end-user-impact | 14:48 |
| sean-k-mooney | that is what is missing | 14:48 |
| sean-k-mooney | this is unrelated to cloudbase-init | 14:48 |
| openstackgerrit | Merged openstack/nova master: Add missing exception https://review.opendev.org/762898 | 14:48 |
| f0o | I'm not 100% sure tho | 14:48 |
| f0o | but if they claim that the cloudbase-init creates and shoots the password to nova | 14:48 |
| *** mgoddard has quit IRC | 14:48 | |
| f0o | then injection shouldnt matter | 14:48 |
| f0o | or am I off? | 14:48 |
| sean-k-mooney | its not using injection and that is not what cloudbae does | 14:48 |
| sean-k-mooney | cloudbase-init just reimplmente cloud-init for windows | 14:49 |
| sean-k-mooney | it does not interact with nova at all | 14:49 |
| sean-k-mooney | it just consumes the metadta generated by nova | 14:49 |
| dansmith | gibi: looking through our install docs, | 14:49 |
| sean-k-mooney | its a one way comunication | 14:49 |
| f0o | >> Cloudbase-Init generate a random password and post it encrypted on the Nova metadata service. << that part is what makes me think it actually attempts to do something tho | 14:49 |
| sean-k-mooney | we dont allow external things to add metadata that way | 14:50 |
| dansmith | gibi: we already do provide quite a bit of sample config per compute and per "controller", and it's all seemingly duplicated in each doc, and for each distro flavor :/ | 14:50 |
| sean-k-mooney | its not part of the api | 14:50 |
| f0o | well that's a great lie from the vendor then lol | 14:50 |
| sean-k-mooney | the only way to do that woudl be to set a property on server | 14:50 |
| sean-k-mooney | but cloudbase-init is only installed in the vm image | 14:51 |
| *** rpittau|brb is now known as rpittau | 14:51 | |
| sean-k-mooney | it does not know about the openstack its one and has no credentials | 14:51 |
| sean-k-mooney | so it cant set a property | 14:51 |
| f0o | yeah that's exactly my thoughts as well | 14:51 |
| lpetrut | sorry for stepping in, but you can in fact post changes to the metadata server | 14:51 |
| f0o | hence why I got so massively confused when they write "oh dont worry, we post the password to nova" | 14:52 |
| lpetrut | https://docs.openstack.org/api-ref/compute/?expanded=replace-metadata-items-detail#replace-metadata-items | 14:53 |
| f0o | thanks for the clarifications tho :) | 14:54 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/victoria: Add missing exception https://review.opendev.org/763389 | 14:54 |
| f0o | I thought I was being blind or insane | 14:54 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/ussuri: Add missing exception https://review.opendev.org/763390 | 14:54 |
| lpetrut | cloudbase-init will just encrypt the generated password and post it to http://169.254.169.254/openstack, so you can then retrieve it and decrypt it. IIRC the nova client even accepts a key that decrypts that password. | 14:55 |
| sean-k-mooney | lpetrut: you can i was not aware of that. i tought this was readonly | 14:56 |
| sean-k-mooney | unless you used the property api. i guess not but i dont think we have any cli commnds for this | 14:57 |
| openstackgerrit | Elod Illes proposed openstack/nova stable/train: Test for disabling greendns https://review.opendev.org/761763 | 14:59 |
| lpetrut | well, the idea is that the netron metadata agent is also forwarding POST/PUT requests | 14:59 |
| lpetrut | not just GET requests to http://169.254.169.254/openstack | 14:59 |
| bauzas | dansmith: around for a RPC question ? | 14:59 |
| dansmith | bauzas: yeah | 14:59 |
| bauzas | dansmith: ack, just wondering why https://review.opendev.org/#/c/761452/ is getting the 6.0 version as the minor version | 15:00 |
| bauzas | " Nov 13 10:14:02.664855 ubuntu-focal-rax-iad-0021755641 devstack@n-api.service[72892]: ERROR nova.api.openstack.wsgi [None req-dbe8b15b-bd32-46ab-93b4-b0097a6a86a3 None None] Unexpected exception in API method: oslo_messaging.rpc.client.RPCVersionCapError: Requested message version, 5.0 is incompatible. It needs to be equal in major version and less than or equal in minor version as the specified version cap 6.0." | 15:00 |
| lpetrut | sean-k-mooney: so basically each guest can update its own metadata by just sending it to http://169.254.169.254/openstack | 15:01 |
| sean-k-mooney | lpetrut: that might be a bug. i think that was only ment to be updated via the nova api | 15:01 |
| bauzas | dansmith: but I provided an additional endpoint for the v5.0 proxy | 15:01 |
| dansmith | bauzas: well, you haven't implemented 6.0 in the client, but all the servers are reporting that they support 6.0 (via the service version) and everything is new, so it's choosing to use 6.0 but the client doesn't support it, right? | 15:03 |
| lpetrut | sean-k-mooney: fwiw here's where cloudbase-init is sending the password: https://github.com/cloudbase/cloudbase-init/blob/master/cloudbaseinit/metadata/services/httpservice.py#L55-L80 | 15:04 |
| *** mkrai has joined #openstack-nova | 15:04 | |
| dansmith | bauzas: remember, the client has to be able to speak both versions as well, depending on the version pin, but it's choosing to speak the new one based on the service version, and you only speak 5.x on the client side right now | 15:04 |
| bauzas | dansmith: sure, but if the client is only supporting 5.0, the server should still accept it given the additional endpoint, nope ? | 15:04 |
| lpetrut | sean-k-mooney: I wouldn't say it's a bug, it allows the guest to communicate back various metadata items. I'm not sure if it's filtered in any way. | 15:04 |
| dansmith | bauzas: yes, but the auto pin will try to configure the client with 6.0 | 15:04 |
| sean-k-mooney | lpetrut: i see well that is undocumented behavior that they are relying on i think | 15:04 |
| dansmith | bauzas: because everything is new | 15:05 |
| bauzas | dansmith: why this worked then with https://review.opendev.org/#/c/541005/6 ? | 15:05 |
| *** ociuhandu has joined #openstack-nova | 15:05 | |
| sean-k-mooney | lpetrut: unless im mistaken and you can point me to docs or a spec to the controy i dont think that was ever intended to work | 15:06 |
| dansmith | bauzas: were we auto calculating the pin then? we'd have to look at devstack config from back then to know | 15:06 |
| lpetrut | sean-k-mooney: I'm sure I can find some docs on this, checking | 15:06 |
| bauzas | dansmith: maybe, I dunno | 15:07 |
| *** mgoddard has joined #openstack-nova | 15:07 | |
| sean-k-mooney | lpetrut: this is the metadata docs https://docs.openstack.org/nova/latest/user/metadata.html | 15:09 |
| dansmith | bauzas: Nov 13 10:14:02.660873 ubuntu-focal-rax-iad-0021755641 devstack@n-api.service[72892]: INFO nova.compute.rpcapi [None req-dbe8b15b-bd32-46ab-93b4-b0097a6a86a3 None None] Automatically selected compute RPC version 6.0 from minimum service version 54 | 15:10 |
| dansmith | bauzas: that's trying to configure rpcapi.py with version 6.0, but it doesn't support that | 15:10 |
| bauzas | dansmith: yup, I understood it | 15:11 |
| bauzas | dansmith: OK, I can try to support 6.0 for the rpcapi | 15:11 |
| sean-k-mooney | lpetrut: that behavior is not descitbe in either the metadata docs or api docs | 15:12 |
| dansmith | bauzas: well, you have to :) | 15:12 |
| sean-k-mooney | lpetrut: when you do that is the metadata stored back to the db | 15:12 |
| sean-k-mooney | they way it would be if you called the nova api | 15:12 |
| lpetrut | sean-k-mooney: fwiw, here's an explicit handler for "POST": https://github.com/openstack/nova/blob/master/nova/api/metadata/password.py#L62 | 15:12 |
| sean-k-mooney | lpetrut: right that is for the nova api | 15:12 |
| sean-k-mooney | it supproted if you call the nova api with an authenicated token | 15:13 |
| sean-k-mooney | but doing it form the guest vai the 169 adress is not as far as i know | 15:13 |
| lpetrut | nope, that specific request doesn't require a token | 15:14 |
| lpetrut | https://github.com/openstack/nova/blob/master/nova/api/metadata/base.py#L214-L225 | 15:14 |
| sean-k-mooney | so this is something ill have to bring up with the security team | 15:14 |
| sean-k-mooney | we have discussed it publicly at this point but its potentially an issue | 15:15 |
| lpetrut | from what I can tell, the guest can only update specific fields, the password being one of them | 15:15 |
| *** k_mouza has quit IRC | 15:15 | |
| lpetrut | I've just checked, so actually the "password" field is the only one that can be updated | 15:16 |
| f0o | sean-k-mooney: actually it seems that /password accepts POST on the metadata service... The issue was that the sshkey used was ECDSA and cloudbase-init went bananas. I get the password posted correctly if I load it up with an RSA key | 15:16 |
| sean-k-mooney | f0o: ecdsa need a newer version of pycyrptography to work | 15:17 |
| lpetrut | tbh I'm a bit surprised that the nova metadata docs don't mention this, this feature has been there since forever (<2014 I think) | 15:17 |
| sean-k-mooney | as i said im not sure it has been | 15:17 |
| sean-k-mooney | i think this was unitnetional | 15:17 |
| sean-k-mooney | there is no spec for it | 15:18 |
| lpetrut | here it is: https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 | 15:18 |
| sean-k-mooney | i checked | 15:18 |
| lpetrut | 2012, pre nova specs era :) | 15:18 |
| *** tosky has quit IRC | 15:18 | |
| sean-k-mooney | this is only nova v1 api behavior | 15:19 |
| sean-k-mooney | actully no its liberty which is not pre sepecs | 15:19 |
| lpetrut | I guess it seemed simple enough not to mandate a spec, but it's definitely intentional | 15:20 |
| lpetrut | and it's not just nova v1 | 15:20 |
| sean-k-mooney | lpetrut: we require specs for all api cahnge regradless of how trivial it is | 15:20 |
| *** mlavalle has quit IRC | 15:21 | |
| sean-k-mooney | lpetrut: yes the nova v1 comment was because you said it was pre specs | 15:21 |
| sean-k-mooney | we have specs for similar feature in libvirt https://specs.openstack.org/openstack/nova-specs/specs/liberty/implemented/libvirt-set-admin-password.html | 15:21 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/train: Add missing exception https://review.opendev.org/763393 | 15:21 |
| lpetrut | oh, got it | 15:21 |
| *** tosky has joined #openstack-nova | 15:22 | |
| lpetrut | I think that libvirt driver feature is slightly different than https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 | 15:22 |
| sean-k-mooney | we are missing an api microverion bump for this too | 15:23 |
| lpetrut | bump the microversion for what? | 15:23 |
| sean-k-mooney | the metadata api | 15:23 |
| lpetrut | https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 this is there since 2012 :) | 15:24 |
| lpetrut | it's not something that was added now | 15:24 |
| sean-k-mooney | yep and it was not documented and did not follow our spec proceducer and did not do an api microverion bump to the metadata api | 15:24 |
| sean-k-mooney | lpetrut: sure imjust not sure it shoudl continue to be supported unless we at least fix the docs | 15:25 |
| lpetrut | well, just because it wasn't properly documented doesn't mean that it should become unsupported, there are projects relying on it | 15:25 |
| sean-k-mooney | if its just this field it might be ok. if other fiels can be arbitrally updated then it could be a an issue | 15:25 |
| lpetrut | sean-k-mooney: indeed. well, it's ok, it's just this field | 15:26 |
| *** tosky has quit IRC | 15:26 | |
| sean-k-mooney | lpetrut: not that im aware of other the cloudbase-init | 15:26 |
| lpetrut | well, since virtually all Windows Openstack instances use it, I'd say breaking it wouldn't be desired, even though it's just one project | 15:27 |
| *** elod has quit IRC | 15:27 | |
| sean-k-mooney | apparently it was for hyperv https://blueprints.launchpad.net/nova/+spec/hyper-v-metadata-password-post | 15:27 |
| sean-k-mooney | but it was not appoved | 15:27 |
| *** elod has joined #openstack-nova | 15:27 | |
| *** k_mouza has joined #openstack-nova | 15:28 | |
| sean-k-mooney | thre is no https://blueprints.launchpad.net/nova/+spec/get-password blueprit | 15:28 |
| sean-k-mooney | oh there is | 15:28 |
| sean-k-mooney | it did not come up | 15:28 |
| lpetrut | https://blueprints.launchpad.net/nova/+spec/hyper-v-metadata-password-post seems like an extension of this feature | 15:28 |
| sean-k-mooney | this was first added as an api extention https://review.opendev.org/#/c/17273/ | 15:29 |
| sean-k-mooney | before we removed those | 15:29 |
| sean-k-mooney | so this was nota catully part of the metadata api | 15:30 |
| sean-k-mooney | it was a vendor extntion https://review.opendev.org/#/c/17273/15/nova/api/openstack/compute/contrib/server_password.py | 15:30 |
| lpetrut | nice. thanks for checking. this was an interesting lesson of Nova history :) | 15:30 |
| sean-k-mooney | so ya i think we moved it into the metrada service wehn we got rid fo extensions | 15:31 |
| lpetrut | sorry for mentioning the server actions, that's completely unrelated. it took a while since I last had contact with this code so I was a bit confused. | 15:33 |
| sean-k-mooney | so this changed in liberty | 15:36 |
| sean-k-mooney | that is when we remvod the contib folder | 15:37 |
| sean-k-mooney | ah it move to legacy_v2 contrib | 15:38 |
| *** macz_ has joined #openstack-nova | 15:39 | |
| sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/api-no-more-extensions.html | 15:41 |
| *** dklyle has joined #openstack-nova | 15:41 | |
| sean-k-mooney | so this was deprecated in libvirty and removed in newton | 15:42 |
| lpetrut | yep, API extensions were deprecated but then got included in the nova api | 15:43 |
| sean-k-mooney | they were not just all accpeted | 15:44 |
| sean-k-mooney | they needed to be upstreamed | 15:44 |
| lpetrut | makes sense. well, this specific one is part of the nova tree. seems upstream to me :) | 15:44 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: WIP: Reproducer for unpinned to pinned resize bug https://review.opendev.org/763399 | 15:46 |
| sean-k-mooney | it got moved by https://github.com/openstack/nova/commit/003c868da73d84d33fba81ee9b033b8ae321e7ab | 15:46 |
| artom | stephenfin, sean-k-mooney, ^^ sanity check that for me pretty please? I feel like I've missed something obvious | 15:46 |
| artom | And yet... | 15:47 |
| sean-k-mooney | what are yo trying to check? | 15:47 |
| sean-k-mooney | unpinned to pinned works fine | 15:47 |
| stephenfin | artom: You need to disable the workaround option | 15:47 |
| sean-k-mooney | or at least it used too | 15:47 |
| stephenfin | artom: '[workarounds] disable_fallback_pcpu_query' | 15:48 |
| stephenfin | and then it'll fail | 15:48 |
| sean-k-mooney | ah yes | 15:48 |
| sean-k-mooney | so it uses pcpus | 15:48 |
| sean-k-mooney | artom: the current behavior with the fallback enabeld is expected | 15:49 |
| artom | stephenfin, can I be lazy and ask you to remind me what the fallback query actually queries for? | 15:49 |
| artom | Ah, just VCPUs? | 15:49 |
| gibi | dansmith: re: install doc. So those docs could be the place where we define the minimal config. As a first look at leat the compute doc did not ask for a DB config :) | 15:49 |
| sean-k-mooney | artom: yep vcpus | 15:49 |
| dansmith | gibi: yeah | 15:49 |
| artom | Wouldn't it fail on the host though? | 15:49 |
| sean-k-mooney | artom: no | 15:49 |
| dansmith | gibi: I dunno about you, but I'm not sure that having three separate docs for three separate distro flavors is really necessary, | 15:49 |
| sean-k-mooney | artom: it should | 15:50 |
| gibi | dansmith: I did not diff them but they look similar | 15:50 |
| stephenfin | artom: when that's configured, the scheduler will make a second, fallback request for VCPU inventory, yeah | 15:50 |
| dansmith | gibi: especially since the ubuntu one (at least) already has some stale stuff I can see (not using systemctl) | 15:50 |
| *** tosky has joined #openstack-nova | 15:50 | |
| sean-k-mooney | but im not sure stephenfin pach has been merged | 15:50 |
| dansmith | gibi: yeah there are a couple tweaks made to one that aren't in the other, unrelated to distro stuff | 15:50 |
| *** mkrai has quit IRC | 15:50 | |
| dansmith | gibi: having them separate makes that a real likely possibility | 15:50 |
| stephenfin | as for failing on the host, I'm trying to recall... | 15:50 |
| sean-k-mooney | artom: if they dont have cpu_dedicate_set configure then it should boot on the host | 15:50 |
| gibi | dansmith: so we could merge them and just add notes about the destro specific things | 15:51 |
| sean-k-mooney | stephenfin: it should only fail on the host if cpu_dedicated_set is defiend | 15:51 |
| stephenfin | sean-k-mooney: yeah, correct | 15:51 |
| artom | stephenfin, I'd expect virt.hardware to not pin CPUs if neither vcpu_pin_set or cpu_dedicated_set is configured... | 15:51 |
| stephenfin | artom: we can't do that - we'd break upgrades | 15:51 |
| dansmith | gibi: or we could merge them and try to stay out of the distro business | 15:51 |
| sean-k-mooney | artom: your expectiojn is wrong | 15:51 |
| sean-k-mooney | yep what stephenfin said | 15:52 |
| dansmith | gibi: like say "install packages now, probably $ubuntuish or $redhatish" | 15:52 |
| stephenfin | say you have a user that was using pinned instances before Train. Their host would be reporting VCPU and those pinned instances would be booting whether or not vcpu_pin_set was defined | 15:52 |
| artom | stephenfin, hol'up, maybe that's where I got it wrong | 15:53 |
| stephenfin | after the upgrade, those hosts will still report VCPU (because 'cpu_dedicated_set' isn't defined), which is why we need the fallback query | 15:53 |
| artom | I assumed if you don't have vcpu_pin_set you can't boot pinned instances... | 15:53 |
| sean-k-mooney | you can | 15:53 |
| artom | Or will we just chose any old CPUs? | 15:53 |
| sean-k-mooney | vcpu_pin_set is nothign to do with pinning | 15:53 |
| stephenfin | nope. That _should_ have been the case, but it wasn't | 15:53 |
| gibi | dansmith: yeah, that make sense | 15:53 |
| sean-k-mooney | its the set of host you can boot a vm on | 15:53 |
| sean-k-mooney | *cpus | 15:53 |
| sean-k-mooney | if unset its all cpus | 15:53 |
| stephenfin | artom: within the constraints of NUMA, yes | 15:53 |
| stephenfin | pinning means NUMA, which means a guest NUMA node is constrained to a host NUMA node | 15:54 |
| stephenfin | yeah, what sean-k-mooney said | 15:54 |
| artom | Aha, thanks for bringing me up to speed | 15:54 |
| stephenfin | I said we should have required it because pinning to e.g. host core 0 isn't a great idea, but there's no relationship between the option and pinning, as sean-k-mooney says | 15:55 |
| artom | Well, there is if it's set :P | 15:55 |
| artom | Are the semantics the same with cpu_dedicated_set? IOW, if that's None, can we still pin to anything? | 15:55 |
| stephenfin | sure, but unpinned instances booted to the same host will be bound to that set too | 15:55 |
| artom | Ah, it was just the set of CPUs usable by instances | 15:56 |
| gibi | dansmith: having just a controller config enough? or we need a separate scheduler config, conductor config, etc? | 15:56 |
| stephenfin | artom: at the moment, yes, because of upgrade reasons | 15:56 |
| artom | Of any kind | 15:56 |
| artom | I feel like I should know this stuff | 15:56 |
| stephenfin | yes, exactly. Not specific to pinning policy | 15:56 |
| gibi | dansmith: like the scheduler only need the API database I guess | 15:56 |
| dansmith | gibi: well, with cells there needs to be some differences, that's why we're here | 15:56 |
| stephenfin | artom: once we drop support for vcpu_pin_set, you'll have to define 'cpu_dedicated_set' | 15:56 |
| sean-k-mooney | artom: vcpu_pin_set applied to unpinned instance too | 15:57 |
| sean-k-mooney | we set it in the xml | 15:57 |
| sean-k-mooney | artom: which si still a gap in live migration by the way | 15:57 |
| sean-k-mooney | we dont update it for non numa guests | 15:57 |
| stephenfin | I have a draft patch, but I can't submit it because doing so will require removal of the reshape. I'm hoping bauzas gets that offline reshape spec sorted this cycle | 15:58 |
| bauzas | my spec is still open | 15:58 |
| sean-k-mooney | stephenfin: well you mean once we drop the fallback query | 15:58 |
| gibi | nova meeting starts in 1 minute | 15:59 |
| stephenfin | sean-k-mooney: right | 15:59 |
| sean-k-mooney | once that is gon you can only boot pinned vms with pcpus | 15:59 |
| stephenfin | which we'll do at the same time as we drop vcpu_pin_set | 15:59 |
| stephenfin | correct | 15:59 |
| sean-k-mooney | yep | 15:59 |
| sean-k-mooney | but even removing vcpu_pin_set would not be enough | 15:59 |
| sean-k-mooney | since without cpu_dedicated_set but with the fallback enabeld you can boot pinned gusts with vcpus | 16:00 |
| *** ociuhandu_ has joined #openstack-nova | 16:00 | |
| *** vishalmanchanda has quit IRC | 16:00 | |
| sean-k-mooney | so its really the fallback that is imporant | 16:00 |
| artom | sean-k-mooney, oh yeah, I remember the live migration for cpu_shared_set | 16:01 |
| artom | Should get to it, at some point :P | 16:01 |
| *** ociuhandu has quit IRC | 16:03 | |
| sean-k-mooney | artom: is this reated to a bug or something by the way | 16:04 |
| sean-k-mooney | or just better testing | 16:04 |
| artom | sean-k-mooney, the whitebox CI failures | 16:05 |
| artom | I haven't managed to reproduce the delete failure | 16:05 |
| sean-k-mooney | oh well this is not a bug | 16:05 |
| artom | (In functional tests, at any rate) | 16:05 |
| sean-k-mooney | it might be a missconfiguration or a bug in the white box tests | 16:05 |
| artom | But it's happening because it's deleting the pinned instance when cpu_dedicated_set is None | 16:05 |
| artom | So it complains about unpinning CPUs | 16:05 |
| artom | Well... | 16:05 |
| sean-k-mooney | anytime we define cpu_dedicated_set you should also disable the fallback | 16:05 |
| sean-k-mooney | we likely are not doing the latter | 16:06 |
| artom | We don't define it | 16:06 |
| sean-k-mooney | its enabled by default | 16:06 |
| artom | I mean cpu_dedicated_set | 16:06 |
| artom | What I'm seeing is the test_resize_unpinned_server_to_pinned tearDown is failing | 16:06 |
| artom | Because it can't delete the pinned instance | 16:06 |
| artom | Because nova.exception.CPUUnpinningInvalid: CPU set to unpin [2, 6] must be a subset of pinned CPU set [] | 16:07 |
| artom | AFAICT there's no race or anything, it's just booting the resized pinned instance on the host without cpu_dedicated_set or vcpu_pin_set configured | 16:07 |
| artom | Pinning it to 2 random CPUs | 16:07 |
| artom | Then complaining when it can't delete it | 16:07 |
| sean-k-mooney | well we need delete teh vms before we reconfigure | 16:08 |
| artom | Which... does kinda smell like a minor bug... | 16:08 |
| artom | We *don't* reconfigure | 16:08 |
| artom | This is all without any config changes or service restarts | 16:08 |
| sean-k-mooney | no i mean im not sure we are waiting for all vms from the previous test to be deleted | 16:08 |
| artom | AFAICT, yes | 16:09 |
| sean-k-mooney | artom: we should never see nova.exception.CPUUnpinningInvalid: CPU set to unpin [2, 6] must be a subset of pinned CPU set [] | 16:09 |
| sean-k-mooney | if we do it means the config have chagned | 16:10 |
| sean-k-mooney | or we are hitting the race which stephen fixed | 16:10 |
| artom | But... what happens in the case of having booted an instance with pinned CPUs on a host without cpu_dedicated_set configured? | 16:10 |
| artom | And then deleting that | 16:10 |
| artom | Wouldn't we expect that error? | 16:10 |
| sean-k-mooney | if its still not set its fine | 16:10 |
| artom | Or does it use the database and not the config? | 16:11 |
| sean-k-mooney | its not vallide to add cpu_dedicate_set to a host with vms | 16:11 |
| artom | I know | 16:11 |
| artom | That's not what we're doing here | 16:11 |
| sean-k-mooney | can you point me to the test | 16:11 |
| artom | sean-k-mooney, gmeet? :) | 16:11 |
| sean-k-mooney | sure | 16:11 |
| artom | sean-k-mooney, https://meet.google.com/eqm-jjcx-fjw | 16:13 |
| sean-k-mooney | im going to make a cup of coffee but say 15 past | 16:13 |
| sean-k-mooney | ill join when i get back | 16:13 |
| artom | Take your time :) | 16:13 |
| sean-k-mooney | just back :) | 16:16 |
| *** evrardjp has quit IRC | 16:17 | |
| *** hamalq has joined #openstack-nova | 16:20 | |
| dansmith | gibi: the distro-ness also affects our ability to describe what should go in each config file, | 16:24 |
| gibi | dansmith: do you have an example? | 16:24 |
| *** martinkennelly has quit IRC | 16:24 | |
| dansmith | gibi: because the problem on the ML is making all the distro packages work both across multiple nodes, as well as AIO, and currently there's no nova-conductor-cellN vs nova-superconductor service(s) | 16:25 |
| *** hamalq has quit IRC | 16:25 | |
| dansmith | gibi: so I'm not quite sure how to write this to make sense for both | 16:25 |
| dansmith | gibi: I'm going to try, but we'll probably need some input from people like owalsh .. maybe that means we need to use something other than the install doc to describe things, I dunno | 16:26 |
| dansmith | writing docs is hard :( | 16:26 |
| gibi | so the problem is that distros map packages to services but the nova-conductor package needs to be mapped to two type of services with different config? | 16:27 |
| dansmith | yeah | 16:27 |
| dansmith | and also, | 16:27 |
| owalsh | hey | 16:28 |
| dansmith | and this is what he was complaining about, | 16:28 |
| dansmith | normally they'd all point to nova.conf, assuming they'd be on different hosts, | 16:28 |
| dansmith | but if they're all on the same host, I'd have to describe changing/overriding unit files to get the right config to conductor vs. api, etc | 16:28 |
| owalsh | tkajinam made an important point on the ML. It's not just AIO, it's the typical production deployment of ironic nova-compute | 16:29 |
| owalsh | on the same host as nova-api etc... | 16:29 |
| dansmith | owalsh: ack, yeah, that's the same problem, just another reason you might hit it | 16:30 |
| dansmith | owalsh: anyway, what I'm describing above is just that I was going to start working on the config samples in this doc: https://docs.openstack.org/nova/latest/install/controller-install-ubuntu.html | 16:30 |
| dansmith | but realized that I'll not be able to describe it fully without talking about changing the distro unit files, etc | 16:30 |
| gibi | dansmith: so either we start describing deployment scenarios (which feels bad) or start using separate config file name for each service | 16:31 |
| owalsh | dansmith: ack, and that's something puppet-nova could do | 16:31 |
| dansmith | gibi: well, I really think we should avoid getting into prescribing config file names for things, because again, that's very very distro and deployment specific, | 16:32 |
| dansmith | gibi: and I think that the suggestion on the ML to standardize those layers a bit actually got us into a more sticky situation | 16:32 |
| *** mgariepy has quit IRC | 16:34 | |
| gibi | but then we need to iterate on at least AIO deployment scenario, Ironic, and a real cellv2 scenario | 16:34 |
| *** gyee has joined #openstack-nova | 16:34 | |
| gibi | anyhow I agree that docing this properly is hard | 16:35 |
| owalsh | implementing it in code is no picnic either :-) | 16:36 |
| *** mgariepy has joined #openstack-nova | 16:36 | |
| gibi | owalsh: good point :) | 16:36 |
| dansmith | at the very least, I think we should get rid of the 3x distro-based flavored docs for each | 16:36 |
| gibi | +1 ^^ | 16:36 |
| dansmith | maybe I can just sprinkle some notes in around the configuration bits about service differences, and see if that helps clarity at all | 16:37 |
| gibi | yeah, that could be a good starting point | 16:37 |
| *** ociuhandu_ has quit IRC | 16:40 | |
| gibi | dansmith: I don't know if you see this but there are also ~bugs uncovered by these config discussion https://bugs.launchpad.net/nova/+bug/1903908 (thank to owalsh for the report) | 16:41 |
| openstack | Launchpad bug 1903908 in puppet-nova "nova conf [api]/dhcp_domain is required on nova-compute" [Undecided,New] - Assigned to Oliver Walsh (owalsh) | 16:41 |
| dansmith | I saw that, I'm not sure how I feel about it | 16:42 |
| dansmith | anything that the configdrive code uses will be shared with the metadata api | 16:42 |
| *** ociuhandu has joined #openstack-nova | 16:42 | |
| *** lpetrut has quit IRC | 16:42 | |
| gibi | yeah, I don't like that metadata calls deep into the metadata code | 16:43 |
| dansmith | configdrive? | 16:44 |
| dansmith | well, that's pretty much the whole point of it :) | 16:44 |
| gibi | the gneeration of the config drive | 16:44 |
| gibi | but this result in the code use by two different services | 16:44 |
| gibi | I guess it is just the fact that I associate the the metadata code only to the metadata service and always forget that ther is a dependency to it from the nova-compute | 16:45 |
| gibi | I guess that lead to the fact that we moved the domain config to the [api] section | 16:46 |
| gibi | anyhow this is for another day as I have to go offline | 16:46 |
| gibi | o/ | 16:46 |
| *** ociuhandu has quit IRC | 16:48 | |
| dansmith | ack, I'll put this up for discussion in a bit | 16:49 |
| *** gyee has quit IRC | 16:50 | |
| owalsh | dansmith: I was going to respond to tkajinam on the ML but I don't really have a response, just a +1 really | 16:50 |
| dansmith | owalsh: okay I guess it's really the same thing anyway | 16:52 |
| *** rpittau is now known as rpittau|afk | 16:52 | |
| owalsh | yes, the first point IIUC is if we have nova.conf and nova-cpu.conf, both containing the generated sample config, how do I know where to set option foo? | 16:52 |
| dansmith | if the packages want to use nova-compute.conf for the compute package, that'd solve it, AFAIK | 16:52 |
| *** ociuhandu has joined #openstack-nova | 16:52 | |
| owalsh | ack, only realistic solution I can see right now, but it's also a bit unpleasant | 16:55 |
| dansmith | well, nova/ironic has pretty much never fit into nova properly so .. unsurprising that we have this problem | 16:56 |
| dansmith | we could also just special case the check and not complain if we're using ironic, | 16:56 |
| dansmith | since we're not on a compromise-able situation | 16:56 |
| owalsh | I don't think so, the deb/rpm doesn't know what nova-compute will be used for | 16:57 |
| owalsh | ... when it creates the service | 16:57 |
| dansmith | what I meant was, the compute package could keep using nova.conf, and if your tool then configures nova.conf for ironic-and-db-creds-because-single-node, nova-compute wouldn't explode when it starts | 16:59 |
| dansmith | it wouldn't solve the AIO case, but it would solve the ironic one | 16:59 |
| owalsh | rpm/deb would need a new service unit file e.g "nova-ironic" that uses nova.conf vs nova-compute.conf | 17:00 |
| dansmith | going back to the nova-db.conf suggestion.. if we allow the wsgi app to read that, then you can again just configure all the services to read from that, except for nova-compute, and then the only thing that doesn't work is AIO-multicell right/ | 17:01 |
| *** hamalq has joined #openstack-nova | 17:01 | |
| owalsh | did seem like the most elegant solution, but also not had a lot of time to think of any gotchas | 17:02 |
| owalsh | and I think AIO-multicell could be considered not supported outside of devstack | 17:02 |
| *** happyhemant has quit IRC | 17:03 | |
| owalsh | or if you really really want to do this for some of CI job then use containers | 17:03 |
| dansmith | well, I think we should fix wsgi to let you specify config files like everything else, and then let you guys decide how you want to template and split the config files to make the packages work (or not) | 17:03 |
| *** macz_ has quit IRC | 17:05 | |
| *** gyee has joined #openstack-nova | 17:05 | |
| owalsh | ack, might also be worth looking at how we could parameterize the conf file path in the dev/rpm services | 17:06 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: extra logging for unpin_cpus https://review.opendev.org/763409 | 17:08 |
| dansmith | even just having /etc/nova.conf that everything reads, and then each service configured to load /etc/nova/nova-conductor.conf, nova-scheduler.conf, etc would maybe be useful... common config and per-service config.. the latter could be empty in a lot of cases | 17:08 |
| owalsh | ack, I think that was the direction this was going but the wsgi issue was a blocker | 17:13 |
| dansmith | yeah | 17:14 |
| openstackgerrit | Dan Smith proposed openstack/nova master: RFC: Update the install documents for cells and less distro specific https://review.opendev.org/763412 | 17:14 |
| *** ralonsoh_ has joined #openstack-nova | 17:27 | |
| *** ralonsoh has quit IRC | 17:28 | |
| *** ociuhandu_ has joined #openstack-nova | 17:28 | |
| *** ociuhandu has quit IRC | 17:31 | |
| *** ociuhandu_ has quit IRC | 17:33 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | 17:33 |
| *** JamesBenson has quit IRC | 17:33 | |
| *** hamalq has quit IRC | 17:45 | |
| *** hamalq has joined #openstack-nova | 17:45 | |
| *** hemna has quit IRC | 17:46 | |
| *** hemna has joined #openstack-nova | 17:47 | |
| *** macz_ has joined #openstack-nova | 17:53 | |
| *** hamalq has quit IRC | 17:56 | |
| *** hamalq has joined #openstack-nova | 17:56 | |
| *** jangutter has joined #openstack-nova | 17:58 | |
| *** mlavalle has joined #openstack-nova | 18:00 | |
| *** jangutter_ has quit IRC | 18:03 | |
| *** mgoddard has quit IRC | 18:05 | |
| *** macz_ has quit IRC | 18:08 | |
| *** jangutter_ has joined #openstack-nova | 18:21 | |
| *** dtantsur is now known as dtantsur|afk | 18:23 | |
| *** jangutter has quit IRC | 18:24 | |
| *** k_mouza has quit IRC | 18:34 | |
| *** jangutter has joined #openstack-nova | 18:38 | |
| *** jangutter_ has quit IRC | 18:39 | |
| *** jangutte_ has joined #openstack-nova | 18:39 | |
| *** jangutter has quit IRC | 18:42 | |
| *** andrewbonney has quit IRC | 18:45 | |
| sean-k-mooney | stephenfin: i dont know if you saw the discussion about the metadata serivce but we have some undocumented behavior realated to an old nova v2 api extention that got merged into the service when we remove the nova api v3 code and support for cell in newton | 18:51 |
| *** macz_ has joined #openstack-nova | 18:51 | |
| sean-k-mooney | stephenfin: specifically form within the guest without creds you can do a post to the password filed and update that filed | 18:51 |
| sean-k-mooney | stephenfin: so it looks like we need to update the documatnion for that and consider if we actully want to support that longterem i think we need to at least for now since cloudbase use it | 18:53 |
| *** ralonsoh_ has quit IRC | 18:53 | |
| sean-k-mooney | that said it was never actully accpeted into the v2.x api it was incorrectly included when we killed extention and drop the nova v3 api | 18:54 |
| sean-k-mooney | stephenfin: https://github.com/openstack/nova/commit/a2101c4e7017715af0a29675b89e14ee2884bd89 is the change that allowed the password to be updated once via the 169 adress | 18:59 |
| sean-k-mooney | which built on the server_password api extntion | 19:00 |
| sean-k-mooney | anyway none of our docs say ^ is a thing you can do. | 19:00 |
| *** bbowen has quit IRC | 19:07 | |
| *** macz_ has quit IRC | 19:19 | |
| *** mgoddard has joined #openstack-nova | 19:27 | |
| *** macz_ has joined #openstack-nova | 19:37 | |
| *** tesseract has quit IRC | 19:38 | |
| *** mgoddard has quit IRC | 19:39 | |
| *** luksky has quit IRC | 20:26 | |
| *** luksky has joined #openstack-nova | 20:26 | |
| *** rcernin has joined #openstack-nova | 20:33 | |
| *** k_mouza has joined #openstack-nova | 20:35 | |
| *** rcernin has quit IRC | 20:37 | |
| *** k_mouza has quit IRC | 20:39 | |
| *** bbowen has joined #openstack-nova | 20:40 | |
| *** ircuser-1 has joined #openstack-nova | 20:55 | |
| *** rcernin has joined #openstack-nova | 21:02 | |
| *** luksky has quit IRC | 21:41 | |
| *** luksky has joined #openstack-nova | 21:54 | |
| *** nweinber has quit IRC | 22:15 | |
| *** xek_ has quit IRC | 22:40 | |
| NobodyCam | Good Afternoon Nova folks, I've started seeing hypervisor state flapping between up and down, The hypervisors are up, I've read that adjusting server_down_time and report_interval can help with this, are these the correct setting to adjust and is there a way to gauge what these values should be? | 23:10 |
| melwitt | NobodyCam: are you experiencing a problem with getting occasional NoValidHost because of this or seeing the hypervisor state going up and down or both? | 23:31 |
| NobodyCam | really both | 23:31 |
| melwitt | ok, for the scheduling thing, consider moving the ComputeFilter earlier in your list of configured filters if you have a large number of compute nodes. I have seen things where if there are a lot of computes (like 1000) the scheduling process is so slow that some nodes are considered "down" by the ComputeFilter because by the time that filter is reached, 60s have elapsed | 23:33 |
| melwitt | for the other settings, "service_down_time report_interval should be less than service_down_time. If service_down_time is less than report_interval, services will routinely be considered down, because they report in too rarely" from https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.report_interval is the main thing you need to make sure you have right | 23:34 |
| *** rcernin has quit IRC | 23:35 | |
| *** rcernin has joined #openstack-nova | 23:35 | |
| melwitt | here's another setting to make sure is set harmoniously with service_down_time https://docs.openstack.org/nova/latest/configuration/config.html#scheduler.periodic_task_interval | 23:36 |
| melwitt | I think those 3 settings are the only ones you need to adjust and they need to be set as recommended in that doc, that report_interval needs to be less than service_down_time and periodic_task_interval needs to also be less than service_down_time | 23:37 |
| melwitt | AFAIK the driving factor for choosing service_down_time will be how long scheduling is taking for you. if you move ComputeFilter first in the list, for example, and still get sporadic NoValidHost bc of compute node "down" when it's not really down, then you will need to increase service_down_time to accommodate the scheduling time. just make sure you don't set report_interval or periodic_task_interval to longer than service_down_time | 23:44 |
| *** hamalq has quit IRC | 23:53 | |
| *** hamalq has joined #openstack-nova | 23:54 | |
| NobodyCam | Thank you very much melwitt I will review our current configuration an let you know if it improves after any changes | 23:54 |
| melwitt | NobodyCam: np, good luck, will be interested to hear how it goes | 23:54 |
| *** martinkennelly has joined #openstack-nova | 23:56 | |
| *** martinkennelly has quit IRC | 23:59 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!