Monday, 2019-11-11

*** ociuhandu has joined #openstack-nova00:20
*** ociuhandu has quit IRC00:21
*** ociuhandu has joined #openstack-nova00:22
*** ociuhandu has quit IRC00:27
*** eandersson has quit IRC00:44
*** tbachman_ has joined #openstack-nova00:58
*** tbachman has quit IRC01:01
*** tbachman_ is now known as tbachman01:01
*** nanzha has joined #openstack-nova01:17
*** spsurya has joined #openstack-nova01:18
*** ociuhandu has joined #openstack-nova01:18
*** ociuhandu has quit IRC01:23
*** eandersson has joined #openstack-nova01:25
*** eandersson is now known as eandersson201:39
*** eandersson has joined #openstack-nova01:40
*** mlavalle has joined #openstack-nova01:44
*** mlavalle has quit IRC01:44
*** ceryx has joined #openstack-nova01:56
*** ociuhandu has joined #openstack-nova01:57
*** ociuhandu has quit IRC02:02
*** ociuhandu has joined #openstack-nova02:33
*** ociuhandu has quit IRC02:38
*** ociuhandu has joined #openstack-nova02:40
*** ociuhandu has quit IRC02:44
*** nanzha has quit IRC03:21
*** nanzha has joined #openstack-nova03:21
*** mkrai has joined #openstack-nova03:23
*** Liang__ has joined #openstack-nova03:23
*** Liang__ is now known as LiangFang03:25
*** psachin has joined #openstack-nova03:28
*** mkrai has quit IRC03:45
*** boxiang has joined #openstack-nova03:54
*** eandersson has quit IRC03:56
*** ceryx has quit IRC03:56
*** ceryx has joined #openstack-nova03:57
*** eandersson has joined #openstack-nova03:57
*** ceryx has quit IRC03:57
*** eandersson has quit IRC03:57
*** ceryx has joined #openstack-nova03:57
*** eandersson has joined #openstack-nova03:57
*** mkrai has joined #openstack-nova03:58
*** nanzha has quit IRC04:04
*** nanzha has joined #openstack-nova04:04
*** bhagyashris has joined #openstack-nova04:30
*** mkrai has quit IRC04:38
*** ociuhandu has joined #openstack-nova04:44
*** ociuhandu has quit IRC04:50
*** ricolin has joined #openstack-nova04:54
*** bhagyashris has quit IRC04:56
*** psachin has quit IRC05:10
*** zhanglong has joined #openstack-nova05:15
*** nanzha has quit IRC05:23
*** nanzha has joined #openstack-nova05:27
*** ratailor has joined #openstack-nova05:29
openstackgerritTakashi NATSUME proposed openstack/nova master: Update keypairs in saving an instance object  https://review.opendev.org/68304305:30
*** ociuhandu has joined #openstack-nova05:30
*** ociuhandu has quit IRC05:39
*** mkrai has joined #openstack-nova05:41
*** bhagyashris has joined #openstack-nova05:42
*** links has joined #openstack-nova06:17
*** bhagyashris_ has joined #openstack-nova06:24
*** bhagyashris has quit IRC06:27
*** bhagyashris__ has joined #openstack-nova06:28
*** bhagyashris_ has quit IRC06:29
*** takashin has left #openstack-nova06:30
*** tetsuro has joined #openstack-nova06:36
openstackgerritBrin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/68277206:58
*** ociuhandu has joined #openstack-nova07:01
*** ociuhandu has quit IRC07:05
*** rcernin has quit IRC07:08
*** bhagyashris__ is now known as bhagyashris07:09
*** nanzha has quit IRC07:10
*** nanzha has joined #openstack-nova07:17
*** jmlowe has quit IRC07:19
*** ccamacho has joined #openstack-nova07:27
*** nanzha has quit IRC07:32
openstackgerritya.wang proposed openstack/nova-specs master: Add spec for live migration no performance impact.  https://review.opendev.org/69365507:36
*** danil has joined #openstack-nova07:37
danilhello folks. Please help me. Is there a way to get nova client through openstacksdk? I mean that I can not just use novaclient library due to dependencies. Thanks07:38
*** ccamacho has quit IRC07:39
*** jangutter has joined #openstack-nova07:45
*** boxiang has quit IRC07:47
*** zhubx has joined #openstack-nova07:47
*** gibi_ptg is now known as gibi07:58
* gibi is back but jetlagged08:00
*** maciejjozefczyk has joined #openstack-nova08:03
*** awalende has joined #openstack-nova08:09
*** tkajinam has quit IRC08:19
*** zhubx has quit IRC08:27
*** zhubx has joined #openstack-nova08:28
*** ccamacho has joined #openstack-nova08:33
*** ircuser-1 has quit IRC08:34
*** zhanglong has quit IRC08:37
*** zhanglong has joined #openstack-nova08:39
*** trident has quit IRC08:45
*** ociuhandu has joined #openstack-nova08:46
*** zhanglong has quit IRC08:49
*** ralonsoh has joined #openstack-nova08:49
*** zhanglong has joined #openstack-nova08:51
*** ociuhandu has quit IRC08:51
*** gshippey has joined #openstack-nova08:55
*** zhanglong has quit IRC08:56
*** slaweq has joined #openstack-nova08:57
*** zhanglong has joined #openstack-nova08:58
*** trident has joined #openstack-nova08:59
*** nanzha has joined #openstack-nova09:01
*** rcernin has joined #openstack-nova09:03
*** lpetrut has joined #openstack-nova09:03
*** luyao has joined #openstack-nova09:04
*** maciejjozefczyk has quit IRC09:10
*** maciejjozefczyk has joined #openstack-nova09:10
*** brinzhang has joined #openstack-nova09:15
*** trident has quit IRC09:16
*** ivve has joined #openstack-nova09:17
*** maciejjozefczyk has quit IRC09:17
*** zhanglong has quit IRC09:20
*** trident has joined #openstack-nova09:24
*** zhanglong has joined #openstack-nova09:30
*** slaweq has quit IRC09:31
*** dtantsur|afk is now known as dtantsur09:33
*** avolkov has joined #openstack-nova09:39
*** maciejjozefczyk has joined #openstack-nova09:44
danilhello folks. Please help me. Is there a way to get nova client through openstacksdk? I mean that I can not just use novaclient library due to dependencies. Thanks09:46
*** maciejjozefczyk has quit IRC09:49
*** zhanglong has quit IRC09:55
*** rcernin has quit IRC09:56
*** ygk_12345 has joined #openstack-nova09:59
*** brinzhang has quit IRC10:01
*** brinzhang has joined #openstack-nova10:02
*** LiangFang has quit IRC10:03
*** brinzhang_ has joined #openstack-nova10:07
*** luksky has joined #openstack-nova10:09
*** brinzhang has quit IRC10:11
*** rcernin has joined #openstack-nova10:12
*** cdent has joined #openstack-nova10:43
*** mkrai has quit IRC10:46
*** dlbewley has quit IRC10:49
*** rcernin has quit IRC10:49
*** dlbewley has joined #openstack-nova10:50
*** brinzhang has joined #openstack-nova10:55
*** brinzhang_ has quit IRC10:58
*** bhagyashris has quit IRC11:06
*** ociuhandu has joined #openstack-nova11:10
*** ociuhandu has quit IRC11:15
*** ygk_12345 has quit IRC11:17
*** FlorianFa has quit IRC11:26
*** bhagyashris has joined #openstack-nova11:29
*** ociuhandu has joined #openstack-nova11:29
*** ociuhandu has quit IRC11:34
*** ygk_12345 has joined #openstack-nova11:46
*** pcaruana has joined #openstack-nova11:47
*** awalende has quit IRC11:48
*** ygk_12345 has left #openstack-nova11:49
*** awalende has joined #openstack-nova11:49
*** bhagyashris has quit IRC11:53
*** bhagyashris has joined #openstack-nova11:53
*** ociuhandu has joined #openstack-nova12:00
*** dviroel has joined #openstack-nova12:03
*** ociuhandu has quit IRC12:04
*** pcaruana has quit IRC12:09
*** danil has quit IRC12:09
*** CeeMac has joined #openstack-nova12:12
*** bhagyashris has quit IRC12:13
*** rcernin has joined #openstack-nova12:16
*** dave-mccowan has joined #openstack-nova12:26
*** dave-mccowan has quit IRC12:31
*** awalende has quit IRC12:33
*** awalende has joined #openstack-nova12:39
*** ratailor has quit IRC12:41
*** slaweq has joined #openstack-nova12:43
*** brinzhang has joined #openstack-nova12:44
*** rcernin has quit IRC12:44
brinzhangdansmith: efried: In nova PTG in Shanghai we talked about this spec https://review.opendev.org/#/c/663563/, and we reached the agreement on this feature, you can see ML/Etherpad  https://etherpad.openstack.org/p/nova-shanghai-ptg   http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010643.html12:47
*** rouk has joined #openstack-nova12:47
jrollefried: I don't have anything, may be able to get something but not sure it'll be simple :) this might be a reasonable starting point: https://github.com/tpm2-software/tpm2-tools/wiki/Getting-Started12:47
jrollefried: if you don't like the looks of that, I can bug an expert here12:47
*** slaweq has quit IRC12:47
brinzhangdansmith: efried: I was updated (re-write) this spec, please review again (there are -2 on it now)12:48
*** sean-k-mooney has quit IRC12:52
*** sean-k-mooney has joined #openstack-nova12:58
*** mkrai has joined #openstack-nova13:00
*** ociuhandu has joined #openstack-nova13:00
*** slaweq has joined #openstack-nova13:04
*** ociuhandu has quit IRC13:05
*** slaweq has quit IRC13:08
*** bbowen has joined #openstack-nova13:15
*** mriedem has joined #openstack-nova13:15
*** mkrai has quit IRC13:20
*** ociuhandu has joined #openstack-nova13:24
*** jaosorior has joined #openstack-nova13:35
openstackgerritMatt Riedemann proposed openstack/nova master: Remove TODO from ComputeTaskManager._live_migrate  https://review.opendev.org/69369613:38
*** ociuhandu has quit IRC13:56
*** ociuhandu has joined #openstack-nova13:57
*** mkrai has joined #openstack-nova13:59
*** Erdosip has joined #openstack-nova14:01
ErdosipHy!14:01
*** zhubx has quit IRC14:03
*** zhubx has joined #openstack-nova14:03
ErdosipI would like to investigate about CEPH multiple pool support in the same nova process.. AFAIK this is not supported now, since nova only able to use one RBD, but we would like to use multiple... Are there any technical problems with this, to implement?14:03
*** brinzhang_ has joined #openstack-nova14:04
*** tbachman has quit IRC14:04
ErdosipI would like to investigate about CEPH multiple pool support in the same nova process.. AFAIK this is not supported now, since nova only able to use one RBD pool, but we would like to use multiple... Are there any technical problems with this, to implement?14:04
Erdosip(sorry, i thinked, i can edit my previous message...)14:05
*** ociuhandu has quit IRC14:06
*** brinzhang has quit IRC14:07
*** nicholas has joined #openstack-nova14:13
*** slaweq has joined #openstack-nova14:15
*** mmethot has quit IRC14:15
*** priteau has joined #openstack-nova14:17
*** mmethot has joined #openstack-nova14:18
*** slaweq has quit IRC14:19
*** nweinber has joined #openstack-nova14:19
mriedemErdosip: this is likely a better question for the openstack-discuss mailing list, tag the subject line with [nova] so it's filtered properly14:23
Erdosipokay, thanks!14:24
sean-k-mooneyErdosip: the only way to connect to multiple cephs form the same nova host today is via cinder today as far as i know14:29
*** bnemec has joined #openstack-nova14:29
*** bnemec has quit IRC14:29
sean-k-mooneyi dont know what it would take to support it with the rbd image backend but as mriedem says if you ask on the mailing list someone more familar with that may respond14:30
*** maciejjozefczyk has joined #openstack-nova14:32
dansmithbrinzhang_: skimming your changes, it does not look like my core concerns are addressed, so I'll leave my -2 on there14:33
* mriedem drives back to MN, back on in 4 hours or so14:38
*** mriedem has quit IRC14:38
*** luksky has quit IRC14:41
*** ociuhandu has joined #openstack-nova14:43
*** ociuhandu has quit IRC14:48
*** bnemec has joined #openstack-nova14:49
efriedbrinzhang_: I'll look again14:52
efriedjroll: oh, nice, good link, will read.14:53
*** brinzhang has joined #openstack-nova14:53
jroll:)14:54
dansmithErdosip: I've planned to add a tiny bit of multiple-pool support to nova when glance becomes able to replicate images in multiple pools14:55
*** brinzhang has quit IRC14:55
*** brinzhang has joined #openstack-nova14:55
dansmithErdosip: but it would just be the ability to know about multiple pools, there'd still be one "primary" pool that nova tries to get everything into in order to use them14:55
*** brinzhang_ has quit IRC14:56
efriedjroll: I've gotten to the point where I can create an encrypted vTPM. I had found another "hello world" type script out there. It runs seamlessly whether encrypted or not -- that is, afaict the encryption/decryption happens at the libvirt level, and is transparent from the perspective of the VM.14:56
*** brinzhang has quit IRC14:56
jroll\o/14:56
jrollefried: cool, that makes sense to me14:57
*** brinzhang has joined #openstack-nova14:57
efriedThere have been some points of awkwardness thus far, including the need to recycle the libvirtd service before it'll pick up the "secret"14:57
jrollor rather, aligns with how I understood the encrypted vtpm bits14:57
jrolleek14:57
efriedWhich would kind of be ... yeah, eek.14:57
jrollI hope that's an unintentional bug :)14:57
efriedI'm still playing; it may be the case that creating the secret via the API rather than CLI could circumvent that -- not sure.14:57
*** brinzhang has quit IRC14:58
efriedbut even then, I'm still worried about your security requirements being satisfied here.14:58
*** brinzhang has joined #openstack-nova14:58
efriedYou create a "secret" in libvirt and give it a value. And then you create the libvirt XML with the UUID of that secret. And... that's it. There never seems to be a need to retype that secret value.14:59
dansmithefried: libvirt's cli is just an api client, in case you didn't know14:59
dansmithefried: virsh is, I mean14:59
*** brinzhang has quit IRC14:59
efrieddansmith: I figured, point being to do it from the same process as wants to use the secret14:59
*** brinzhang has joined #openstack-nova15:00
efriedsorry, I wasn't clear on that: use the libvirt API from within nova, rather than me typing in the CLI for my PoC.15:00
jrollefried: huh, yeah, I wonder where libvirt keeps that secret15:00
efriedjroll: well, it keeps it in a file in a directory.15:00
jrollsigh.15:00
dansmithefried: yeah and my point is virsh and nova look the same to libvirtd, which is the thing that manipulates the guests, xmls, etc15:00
jrollthe secret itself is in a file?15:00
* jroll needs to play with this as well15:00
*** brinzhang has quit IRC15:01
sean-k-mooneyefried: ideally you would want to store the secret in the hosts tpm rather then on disk15:01
*** brinzhang has joined #openstack-nova15:01
efrieddansmith: Okay, I think I understand what you're saying. There's just one libvirtd "server side" and it hears things the same way whether through API or virsh.15:01
dansmithefried: virsh is just an api client like nova, but yes that's what I'm saying15:02
efriedjroll: Yes, the secret is in a file. Presumably the fact that said file is encrypted is the thing that makes this more "secure".15:02
sean-k-mooneyefried: ya kind of virsh is just like the nova client15:02
sean-k-mooneyall of the logic is implemetne in the libvirt deamon15:02
efriedsean-k-mooney: we're 100% virtual here, there is no TPM on the host.15:02
*** brinzhang has quit IRC15:02
sean-k-mooneyefried: well there could be15:02
efriedfor coding purposes, there isn't.15:02
efriedjroll: but the fact that it's just sitting around there on the disk means that your disk thief could just... boot up the VM?15:03
*** brinzhang has joined #openstack-nova15:03
efriedand would then have to have root access to the VM I guess15:03
jrollefried: the TPM state is in an encrypted file. there is a passphrase to decrypt the file, which we give to libvirt via `virsh secret-set-value` or the API. I'm wondering where libvirt stores that secret, is that also in a file? how does libvirt encrypt that file? (these are rhetorical questions, fwiw, digging now)15:03
sean-k-mooneyefried: shoudl we maybe look at barbican to store the secreat rather then libvirt15:04
sean-k-mooneyi havent really looked at how this worksin in libvirt yet15:04
*** brinzhang has quit IRC15:04
sean-k-mooneybut ideally i think we dont want it sotred on the compute host right15:04
efriedjroll: I'm still not 100% on any of this, but my understanding is that libvirt uses whatever's in the "secret" to encrypt/decrypt the contents of the vTPM.15:05
*** brinzhang has joined #openstack-nova15:05
efriedbut once you've done the secret-set-value... that's it.15:05
jrollefried: right, trying to figure out where "whatever's in the secret" is stored15:05
jrollwhere/how15:05
* jroll upgrades to latest libvirt15:05
sean-k-mooneyit might just be in memory15:06
sean-k-mooneynot everythin you do to a libvirt domain has to be stored to disk15:06
*** jaosorior has quit IRC15:06
efriedjroll:15:06
efriedstack@breakfast:~/nova$ sudo ls -l /etc/libvirt/secrets/15:06
efriedtotal 815:06
efried-rw------- 1 root root   4 Nov  9 12:36 d3e91e0e-20a5-4f60-ae85-9c8bfed6939f.base6415:06
efried-rw------- 1 root root 211 Nov  9 12:36 d3e91e0e-20a5-4f60-ae85-9c8bfed6939f.xml15:06
jrolloh goody15:06
jrollthanks.15:06
sean-k-mooneyefried: is that encrypted or plain text?15:07
jrollI'm gonna go find a table or three. definitely not to flip.15:07
efriedThe .base64 is just the b64-encoded value of the passphrase I set15:07
efriedthe XML is the XML chunk I started with.15:07
sean-k-mooneyso plain texted effectivly15:07
efriedyup15:07
efriedSo I've been thinking of a model where we secret-set-value just long enough to boot the VM, then blow away the secret.15:08
sean-k-mooneyso you might as well not encrypt the tpm at that point15:08
lyarwoodthat isn't passed as plain text to QEMU btw15:08
efried...assuming the tpm stays "unencrypted" while the domain is running.15:08
lyarwoodit's the same as with LUKS, it's encypted using the master key for the domain15:08
dansmithefried: ahh, so... fake security then? :D15:08
efriedbut .... what dansmith said.15:09
dansmithheh15:09
efriedthat doesn't seem like real security.15:09
ErdosipI think it's under /etc/libvirt/security15:09
efriedI mean, I guess your disk thief would have to steal the disk while the VM is booting.15:09
*** ociuhandu has joined #openstack-nova15:10
*** Erdosip has quit IRC15:10
sean-k-mooneyim not sure the vm woudl have to be running15:10
sean-k-mooneythere are cases were we leave the domain intact15:10
sean-k-mooneysuch as a showdown form inside the vm15:10
sean-k-mooneyor suspend/pause15:11
sean-k-mooneymaybe even stop15:11
efriedsean-k-mooney: the secret in the domain xml is just the uuid of the secret. So presumably (hopefully) the decryption happens in-memory when the xml is loaded up to actually start the guest.15:11
sean-k-mooneystart calls power on which calls reboot which recreates the domain15:11
sean-k-mooneysure but the issue is the secret is not encrypted on disk15:12
efriedso regardless of whether the domain xml is there or needs to be recreated, it's when the guest starts up that we would need the secret to be present.15:12
lyarwoodagain, the secret isn't provided to QEMU as a plaintext string15:12
sean-k-mooneyefried: im wondering is there a way to pass it at that point and never save it to disk15:13
efriedlyarwood: but it's on disk in cleartext. (base-64 encoded, but effectively plaintext)15:13
sean-k-mooneylyarwood: sure but efried just said it in /etc/libvirt/secrets/ and is just base64 encoded15:13
lyarwoodright and that's pretty locked down given the listing above15:13
efriedlyarwood: I suspect I'm missing what you're getting at.15:13
*** canori01 has joined #openstack-nova15:13
efriedlyarwood: you mean 600 owned by root locked down?15:14
lyarwoodif someone had access to root it's already game over15:14
sean-k-mooneylyarwood: that not good enough in my view15:14
efriedpretty sure in jroll's view too.15:14
sean-k-mooneythe operator shoudl not be able to see the secreate15:14
lyarwoodthey could already get in and decrypt various things from the running QEMU process15:14
canori01Hello,  would I evacuate an instance that's running into 'AllocationUpdateFailed: Failed to update allocations for consumer' running on Stein15:14
lyarwoodwell that just isn't within libvirt's security model AFAIK15:15
canori01I mean, how would I recover from that15:15
*** ociuhandu has quit IRC15:15
efriedlyarwood: Actually I think we're drawing the line at "compromised root is acceptable risk; stolen disk must be secure".15:15
jrollmore like we realize compromised root on a running hypervisor is game over anyway, but yeah15:16
efriedIOW I think it's the nova admin creds that are the root of trust.15:16
lyarwoodefried: kk, danpb had a blog about secret handling a while ago that might cover a way around that ./me checks15:16
*** munimeha1 has joined #openstack-nova15:16
efriedthanks lyarwood15:16
sean-k-mooneyefried: https://libvirt.org/formatsecret.html#SecretAttributes15:16
jrolllyarwood: thanks, I was just going to ask that15:16
sean-k-mooneyso it look like you can mare a secret as ephemeral15:16
efriedhaving displayed any understanding whatsoever, lyarwood, you're now permanently on call for support.15:16
sean-k-mooneyso its only kept in memory and never on disk15:17
sean-k-mooneyso if we store it in barbican and then do that that would be better15:17
efriedoh, cool, I saw that but hadn't played with it yet, will have to try that out and see if it does the trick.15:17
sean-k-mooneywell ephemeral=yes private=yes15:17
efriedThe secret I created while playing was private but not ephemeral.15:17
jrollcan confirm ephemeral=yes leaves it off of disk15:18
sean-k-mooneyso form a nova point of view can we use barbican to store it15:18
efriedcool, so the nova user is the root of trust because has access to the barbican entry.15:18
efriedyeah, what sean-k-mooney said.15:18
jrolland restarting libvirt nukes an ephemeral secret15:18
jroll\o/15:18
sean-k-mooneyjroll: that is proably ok/good15:19
efriednow we just gotta hope you don't have to restart libvirt to make it pick up the secret :P15:19
lyarwoodthanks sean-k-mooney15:19
jrollefried: fwiw, I didn't need to restart, but I'm on older libvirt playing with tls secrets, not tpm15:19
*** awalende has quit IRC15:19
sean-k-mooneyjroll: i dont know if libvirt makes a difference15:20
efriedjroll: didn't need to restart to do what?15:20
*** awalende has joined #openstack-nova15:20
efriedin my case everything appeared to be moving along fine until I tried booting the guest, then the nova log got an exception complaining it couldn't find my secret.15:20
efriedanyway, ephemeral would be entirely useless in that case, so it MUST work (famous last words)15:21
jrollefried: ah, I was just doing 'virsh secret-list'15:21
jrollheh15:21
jrollsean-k-mooney: if there's a bug in newer libvirt it might :)15:21
sean-k-mooneyoh i ment between secrete usage/types15:22
sean-k-mooneyi think it just has a secret that is use for many things15:22
efriedjroll: btw, there was a question as to whether we needed to bother providing the non-encrypted version of vTPM support at all.15:22
jrollolder libvirt won't let me create a vtpm secret type ¯\_(ツ)_/¯15:22
efriedright, that needs to be >=5.6.0, which afaict you have to compile yourself.15:22
jrollefried: I'd be pro-not-supporting that, I don't like footguns15:22
sean-k-mooneye.g. i dont think a secreate for tls  or tpm is a different thing internally in libvirt15:22
*** jaosorior has joined #openstack-nova15:22
efriedpenick indicated the same. Just that the requirements are going to be way more stringent for it if so.15:23
*** rchurch has joined #openstack-nova15:23
*** awalende_ has joined #openstack-nova15:23
sean-k-mooneyfor what its worth rhel/fedora will be shiping it relitvly quickly15:23
sean-k-mooneyim not sure when exactly15:23
sean-k-mooneybut we have an advance virt stream we use for openstack15:24
efriedI guess "with ussuri" is soon enough <shrug>15:24
sean-k-mooneythat ships a newer version then base rhel15:24
efriedbut also the barbican requirement15:24
*** ociuhandu has joined #openstack-nova15:24
*** awalende has quit IRC15:24
efriedanyway, glad you agree jroll, it makes the code & test surface easier.15:25
efriedlyarwood: since you mentioned the qemu side -- do you have any idea what's the minimum qemu version to support encrypted vTPM? I couldn't find that in the docs anywhere.15:25
efriedcanori01: it would help to know why the allocation update is failing (out of resources? provider conflict? other?), and what action is being performed when that happens.15:26
*** awalende_ has quit IRC15:27
canori01efried: I am replacing old hardware. So I powered off the old hypervisor and then did a nova-evacuate. All the guests evacuated except for two which gave me the provider conflict15:28
lyarwoodefried: I don't sorry15:29
*** ociuhandu has quit IRC15:29
lyarwoodefried: 2.12 is listed in the Stein spec FWIW15:29
efriedlyarwood: yeah, that was for non-encrypted, and idk if there's a higher req for encrypted.15:30
efriedcanori01: If it was a provider conflict, you should just be able to "try again".15:30
lyarwoodefried: no idea sorry, kashyap is out today otherwise I'd ask him if he could work that out for us.15:30
efriedah, good plan. I'll stalk him, thanks.15:30
sean-k-mooneyefried: tpm emulation was added in 2.11 but im not sure if that included encryption15:30
efriedit did not15:31
efriedBut I don't even know whether qemu required any update for that aspect. It might not have.15:31
sean-k-mooneyalso this is an intersting site https://repology.org/project/libvirt/versions15:31
sean-k-mooneylook like apline has the depencyies you need but very little else at teh moment15:33
*** TxGirlGeek has joined #openstack-nova15:34
sean-k-mooneyFedora Rawhide  would "work" too although i suspect nova wont be happy with it in general15:34
efriedsean-k-mooney: oh, that reminds me, I ran into another weird quirk with the home-built libvirt...15:35
efriedsean-k-mooney: When I restarted the libvirt service, it blew up because it was looking in /usr/lib/x86_64-linux-gnu/ which contained the old libvirt 4.xx15:35
efriedI had to go into the service and set LD_LIBRARY_PATH=/usr/lib explicitly to make it work.15:36
efriedwhich seems... weird.15:36
efriedhad to do the same with n-cpu.15:36
sean-k-mooneyya that does seam strange15:36
sean-k-mooneywell clearly i have more work to do15:37
dansmithefried: why is that surprising?15:37
dansmithefried: if you have an old version installed in the standard system library path, and start a new daemon binary that requires the new libs,15:38
dansmithyou have to tell it where they are15:38
dansmith(if I'm understanding what you're saying)15:38
efrieddansmith: Is /usr/lib/x86_64-linux-gnu/ the "standard system library path"?15:38
efriedThat was the surprising part to me. I would have thought /usr/lib was.15:38
dansmithefried: for multi-arch stuff on fedora-like distros it is I think yeah15:39
sean-k-mooneydansmith: well the devstack plugin uninstalles the packages and libvirt was compiles with --system15:39
efriedsean-k-mooney: Doesn't your plugin just use `make install`?15:39
dansmithsean-k-mooney: clearly not if the old version was still there15:39
efriedI would have thought that should have covered symlinking or otherwise replacing versions in any "standard" paths.15:39
dansmithefried: not generally15:40
sean-k-mooneyefried: yes but the autogen sript was passed --system so when it invoked configure it should have set it up to instll in the default system lib path15:40
dansmithefried: make install usually puts them in /usr/local/lib unless you tell it otherwise15:40
dansmithor, --system would yeah15:40
sean-k-mooneydansmith: well i dont try an nuke the old verions15:40
dansmithefried: cat /etc/ld.so.conf or /etc/ld.so.conf.d/*15:41
dansmithefried: will show you where the loader looks for libs15:41
sean-k-mooneydansmith: this shoudl do the right thing yes? https://opendev.org/x/devstack-plugin-libvirt-qemu/src/branch/master/devstack/libs/libvirt#L94-L9615:41
dansmithefried: even on my ubuntu system, /usr/lib/$arch is in there15:41
dansmithsean-k-mooney: should do what? land it in /usr/lib? probably, but he's saying that happened AFAICT15:42
efriedubuntu is what I'm using. And yeah, same. In fact, /usr/lib isn't there at all.15:42
sean-k-mooney/usr/lib is not there at all?15:43
efriedstack@breakfast:~/nova$ cat /etc/ld.so.conf.d/*15:43
efried/usr/lib/x86_64-linux-gnu/libfakeroot15:43
efried# libc default configuration15:43
efried/usr/local/lib15:43
efried# Multiarch support15:43
efried/usr/local/lib/x86_64-linux-gnu15:43
efried/lib/x86_64-linux-gnu15:43
efried/usr/lib/x86_64-linux-gnu15:43
efriedcanori01: I want to say we were working on this, or something very similar, recently, around ignoring *provider* conflicts and just retrying, but it might have been for a different lifecycle operation, I can't remember. Have you looked for an existing launchpad bug?15:45
sean-k-mooneyefried: this is what i have http://paste.openstack.org/show/785944/15:45
canori01efried: I was lookin at https://bugs.launchpad.net/nova/+bug/1798688 but I'm not 100% sure if it's related15:47
openstackLaunchpad bug 1798688 in OpenStack Compute (nova) "AllocationUpdateFailed_Remote: Failed to update allocations for consumer. Error: another process changed the consumer after the report client read the consumer state during the claim" [High,Fix released] - Assigned to Matt Riedemann (mriedem)15:47
efriedsean-k-mooney: yeah, that's what I have in /usr/lib as well, point is that the libvirt-bin and n-cpu services were looking in the x86... dir.15:47
sean-k-mooneyoh ok15:47
*** jaosorior has quit IRC15:47
efriedsean-k-mooney: iow I assume we should either be installing there, or be blowing those away with symlinks.15:47
sean-k-mooneyit might be best to just drop the --system15:48
sean-k-mooneyit clearly not doing it write15:49
sean-k-mooneythen it will install in /usr/local/lib which is in the lib path15:49
efriedI want to say other things blew up when I tried that15:49
*** tbachman has joined #openstack-nova15:50
*** awalende has joined #openstack-nova15:50
efriedanyway, hopefully this is all moot irl. Unless I want to try to run a CI job around this thing. And I'm not sure if there's going to be value in that.15:50
sean-k-mooneywell ill try and spend some more time on this and see if i can make it a bit more robost.15:50
*** links has quit IRC15:52
efriedsean-k-mooney: I appreciate that. Though it's probably not worth trying to make it perfect. I think the sweet spot will be having it work in devstack CI (again, if I end up wanting to do something there), which is of course a pretty narrow subset of usages.15:52
*** mkrai has quit IRC15:52
canori01efried: Is there a way, I coudl just force the instance to take a new allocation and spawn somewhere?  I have plenty of spare capacity15:53
efried...excluding, among other things, "works in my dirty devstack, containing who-knows-what packages, that I've restacked a zillion times"15:53
sean-k-mooneyefried: ya i would like to get a ci job in that plugin repo at a minium but i have need to use this plugin alot in the past when enableing things for openstack so i want it to be atleast usable for that15:53
*** tbachman_ has joined #openstack-nova15:53
efriedcanori01: At this point you should still be able to evacuate it, no?15:53
efriedoh, you mean YOU have use cases too?? Well, then, by all means :P15:54
*** tbachman has quit IRC15:54
*** tbachman_ is now known as tbachman15:54
sean-k-mooneywell not currently15:54
*** awalende has quit IRC15:54
sean-k-mooneyi orginally wrote the first version of that plugin back in 201415:54
sean-k-mooneyi have used it for at least 4 feature at this point15:55
canori01efried: I've tried resetting the state of the instance and evacuating again, but it consistently gives me this http://paste.openstack.org/show/785945/15:55
sean-k-mooneyso i like keping it working for when i need it15:55
efriedcanori01: Oh, hm, that's a reproducible failure? That is not expected. I think we need a bug.15:56
canori01ok, thanks15:56
sean-k-mooneycanori01: you dont have two copies of the nova compute agent running on the compute node by any chacce?15:57
*** jaosorior has joined #openstack-nova15:57
sean-k-mooneyoh never mind15:58
sean-k-mooneythis is in the scduler15:58
*** artom has joined #openstack-nova16:02
*** jaosorior has quit IRC16:04
*** jaosorior has joined #openstack-nova16:05
dansmithcdent: don't we have some guideline around not making fundamental api behaviors change based on policy or config? (well, I know config)16:07
dansmithcdent: meaning something like a user is granted some policy element, which enables a deep feature that they don't even know about nor can detect, which makes all calls to a specific API suddenly return 202 and be async, where normally they are 200 and sync16:08
cdentdansmith: there's a "what should cause a microversion bump" guideline somewhere that I can dig up. Is that what you're thinkin gof?16:08
dansmithcdent: and two users with differing policies for this element would see the different sync/async behavior16:09
cdentI don't know of anythign with that level of detail, but that sounds like a definite no no16:09
dansmithcdent: no because this would differ within one microversion for two users with different endowments16:09
dansmithcdent: seems like it to me as well16:09
*** ociuhandu has joined #openstack-nova16:10
dansmithcdent: might be more than you want to digest at the moment, but this is the reference: https://review.opendev.org/#/c/635684/53/nova/compute/api.py@385216:10
cdentdoes this apply to something you're looking at (I just got back from being away, so not in context)16:10
cdentjinx16:10
* cdent looks16:10
dansmithcdent: the shortcut is: we do a cast if allow_cross_cell_resize is true, and (in a later patch) that will be true based on your policy16:10
dansmithand that seems bad16:11
cdentso cloud A wouild be cast and B would be call, dependent on policy settings? would it be discoverable in any way?16:11
cdenthmmm. however, is a user ever supposed to that cells exist?16:12
*** ociuhandu has quit IRC16:13
cdentbecause this isn't preventing or allowing resize, correct? just resize of a particular type?16:13
*** ociuhandu has joined #openstack-nova16:14
dansmithcdent: no, it'd be *User* A and user B that see differences, in the same cloud16:14
dansmithcdent: and no, a user would not know they have this ability, nor know why they see different behavior from their peer16:14
cdenthmmm. is there a short explanation for the "why" on controlling by policy?16:15
dansmithcdent: it's literally gating whether or not the scheduler will return all hosts, or only hosts within the current cell, not something the user knows about16:15
cdent(I'm trying to read the code, but there's a lot here)16:15
dansmithcdent: the why is so you can control it per-user, where config would be per-cloud16:15
cdentyes, but _why_ would you control is per user?16:15
cdents/is/it/16:16
dansmithcdent: because it16:16
dansmithis more expensive, and/or you may have other reasons,16:16
*** nanzha has quit IRC16:16
dansmithit's a little weird that it's a policy knob, but it is that way just so that it can be controlled per user for whatever reason16:17
cdentI can see that people would want that, even though they shouldn't ;)16:17
cdenteverybody wants the knobs16:17
*** tbachman has quit IRC16:18
cdentokay, I think I have enough info to think about this with some level of legitimacy16:18
dansmithwell, that's a different argument I guess, but ... yeah.16:18
dansmithmy other argument would be that this is potentially more failure-prone,16:18
dansmithso you may not want to give it to "simple people" or something16:18
dansmithor just let admins do it so they can move things around the cloud16:18
dansmithlike the policy knob that lets you direct migrations to a specific host vs. just use the scheduler16:18
* cdent nods16:19
dansmithbut regardless, that discussion is somewhat separate from the api behavior thing16:19
* cdent nods16:19
cdentyeah, just trying to add a bit of definition so I'm not filling in the blanks poorly16:19
dansmithyup16:19
dansmithexplaining it to you here makes me feel more like my +0 should be a -116:20
*** bnemec has quit IRC16:20
cdentwhat it is doing seems like the right thing, but I agree that it should have a microversion for the reason you state: change in behavior when same code has a different user using it16:21
cdenti'll say as much on the review16:21
sean-k-mooneydansmith: i dont know if you could take extention list approch like neutron16:21
*** ociuhandu has quit IRC16:21
sean-k-mooneydansmith: and have that reterun different values based on policy16:21
cdentwhat neutron does is terrifying from an api standpoint16:21
sean-k-mooneywell at least you can detect it16:21
dansmithcdent: the microversion would have to say "after microversion 2.X, the method could be sync or async so watch your return code"16:21
sean-k-mooneybut yes16:21
sean-k-mooneyit still feels weird16:21
dansmithcdent: which I thought is one of the reasons we *don't* add a microversion16:22
*** maciejjozefczyk has quit IRC16:22
dansmithsean-k-mooney: we specifically do not version our api via extensions anymore, so I'm not sure why you'd suggest that16:22
dansmiththat's the whole point of microversions :)16:22
sean-k-mooneydansmith: neutron uses bot16:22
sean-k-mooney*both16:22
sean-k-mooneyon your return code point16:22
sean-k-mooneyit woudl be 200 for syc and 202 or 203 for async?16:23
dansmithsean-k-mooney: I said that above16:23
sean-k-mooneywould they send the same payload in both cases?16:23
dansmithsean-k-mooney: you might want to read the backscroll and the code/comment I linked above16:24
sean-k-mooneyi kind of feel like haveing a flag in the post body would be cleaner16:24
dansmithit'd be good context for the conversation16:24
dansmithsean-k-mooney: because I've said several times that this is not something user knows about or can/would opt into16:24
sean-k-mooneyam i will after my next 1:1 in 5 mins16:24
sean-k-mooneydansmith: ah ok16:24
*** tbachman has joined #openstack-nova16:25
sean-k-mooneyoh this is for cross cell resize16:26
sean-k-mooneyoh its related to the downcall to the compute. hum ya thats tricky16:27
sean-k-mooneywell maybe  "down call" is not right but the thing we were talking to matt about last week16:28
*** mkrai has joined #openstack-nova16:29
*** tbachman has quit IRC16:30
*** ociuhandu has joined #openstack-nova16:32
*** ivve has quit IRC16:32
*** ricolin has quit IRC16:34
*** ociuhandu has quit IRC16:35
*** ociuhandu has joined #openstack-nova16:36
*** mkrai has quit IRC16:36
*** maciejjozefczyk has joined #openstack-nova16:43
*** pcaruana has joined #openstack-nova16:45
*** bnemec has joined #openstack-nova16:47
*** ociuhandu has quit IRC16:52
*** ociuhandu has joined #openstack-nova16:52
*** slaweq has joined #openstack-nova16:58
*** ociuhandu has quit IRC17:00
*** slaweq has quit IRC17:02
*** cdent has quit IRC17:04
*** ociuhandu has joined #openstack-nova17:04
*** mlavalle has joined #openstack-nova17:05
*** tetsuro has quit IRC17:23
*** maciejjozefczyk has quit IRC17:25
*** rosmaita has joined #openstack-nova17:27
*** rosmaita has left #openstack-nova17:28
*** ociuhandu has quit IRC17:32
*** ociuhandu has joined #openstack-nova17:39
*** ociuhandu has quit IRC17:44
*** priteau has quit IRC17:45
*** lpetrut has quit IRC17:55
*** jaosorior has quit IRC18:06
*** ociuhandu has joined #openstack-nova18:18
*** dtantsur is now known as dtantsur|afk18:20
*** maciejjozefczyk has joined #openstack-nova18:26
*** slaweq has joined #openstack-nova18:28
*** ociuhandu has quit IRC18:31
*** nweinber_ has joined #openstack-nova18:33
*** nweinber has quit IRC18:36
*** jmlowe has joined #openstack-nova18:38
*** zhubx has quit IRC18:54
*** zhubx has joined #openstack-nova18:54
*** ivve has joined #openstack-nova18:55
*** pcaruana has quit IRC18:57
*** pcaruana has joined #openstack-nova18:58
*** maciejjozefczyk has quit IRC19:02
*** slaweq has quit IRC19:06
*** nweinber__ has joined #openstack-nova19:08
*** nweinber_ has quit IRC19:10
*** slaweq has joined #openstack-nova19:11
efriedjroll: progress: using ephemeral="yes" nothing shows up on the disk. This means we'll have to define & set the secret on the fly once per guest per service-process-instance, which seems reasonable.19:12
efriedjroll: Next sticky design point: bootstrapping the secret UUID.19:13
efriedWe don't want it to be in the flavor, because then you need a new flavor per secret.19:14
efriedI think we can do it this way: Create it with a random secret in the key manager. I guess neither the VM user nor the operator needs to know the secret value; they just have to be able to access it from the keymgr.19:15
*** slaweq has quit IRC19:16
efriedSo we just have to figure out a way to associate the secret UUID with the instance.19:16
efriedWe could use the instance UUID, but that seems fragile.19:16
*** ircuser-1 has joined #openstack-nova19:17
efriedotherwise I suppose we need to use a new field in the instance. Which makes me a little bit sad, but it's not the end of the world.19:17
efrieddansmith: do you have any other suggestions?19:17
efriedTL;DR: I'm going to generate a UUID when I create the instance, and need to be able to associate that UUID with the instance for the life of the instance.19:18
dansmithefried: system_metadata ?19:18
efriedis that a place I can stuff arbitrary key=value without a schema change?19:18
dansmithefried: yup19:18
efriedsounds perfect :)19:18
efriedthanks.19:19
efriedokay, I think I have enough answers that I can take a break from PoCing and fill in a bunch of the open questions on the spec. (<== TxGirlGeek FYI)19:19
*** awalende has joined #openstack-nova19:21
sean-k-mooneyefried: you could just use 1 the instance uuid or 1 a uuid5 based on the instance uuid as a seed19:21
efriedanything derived from the instance UUID seems brittle, because other things might assume they can do the same.19:22
efriedA separate random uuid is safest19:23
sean-k-mooneyuuid5 takes a seed uuid +  a string19:23
sean-k-mooneyso the seed uuid would be the instance uuid19:23
efriedoh, that could work.19:23
sean-k-mooneythat defines a namespace19:23
sean-k-mooneyand then you can generate uuids form that19:23
efriedyeah, the string could be "emulated tpm secret" or something unique.19:23
sean-k-mooneyya19:23
sean-k-mooneyhttps://docs.python.org/2/library/uuid.html#uuid.uuid519:24
*** ociuhandu has joined #openstack-nova19:25
*** awalende has quit IRC19:25
dansmithefried: definitely prefer generation + storing in sysmeta19:26
efriedokay19:26
*** mriedem has joined #openstack-nova19:27
sean-k-mooneyyeah you can do both19:27
efriedwow cool https://docs.openstack.org/api-ref/key-manager/19:31
dansmithefried: it's for managing secrets..including the definition of its api19:32
efriedhahahaha19:32
mriedemheh, thought that was fixed awhile ago19:32
mriedemthis page last updated: 2018-10-18 16:43:2319:32
mriedemhttps://docs.openstack.org/barbican/latest/api/19:33
sean-k-mooneydont you use castalain or something rather then barbican directly form within nova?19:33
mriedem^ is where https://docs.openstack.org/api-quick-start/ takes you19:33
mriedemcastallan is the interface, barbican is a backend implementation19:34
sean-k-mooneyya that works https://docs.openstack.org/barbican/latest/api/reference/secrets.html19:34
NostawRmI'm exploring the possibility of adding another image backend for Glance and Nova before working out a blueprint, spec, etc. Adding a driver to glance seemed straight forward, but the way nova does image backends doesn't seem as if its intended to add more backends since its all in one file. Is this the case and using boot from volume is intended or am I making bad assumptions?19:34
mriedemNostawRm: if it's adding a backend that is already supported by cinder than it's likely not going to get much traction19:35
*** henriqueof has joined #openstack-nova19:35
mriedemEMC ScaleIO was the last to try and fail at that19:35
sean-k-mooneyNostawRm: it also depends on your usecase.19:35
*** ociuhandu has quit IRC19:36
sean-k-mooneye.g. what advantage would it bring to have a new iamge backend over cinder/bfv19:36
dansmithNostawRm: are you the one that asked on the ML earlier today?19:37
*** tbachman has joined #openstack-nova19:38
NostawRmthat's a shame, but it makes. When using Ceph/RBD, we used it with ephemeral storage, but StorPool only has volumes, so having to boot from volume. Keep doing work arounds and was looking at attacking it from this angle instead19:38
*** ociuhandu has joined #openstack-nova19:38
NostawRmRight now, its because you can't rescue an instance booted from volume19:38
NostawRmdansmith, nah, wasn't me19:38
mriedemso fix rescue to work with volume-backed servers instead...19:38
mriedemlike rebuilding volume-backed servers19:38
dansmithNostawRm: ack, well, you might go look at my response then :)19:38
mriedemetc19:38
NostawRmBut if that's the case, I suppose it is just what it is19:39
dansmithmriedem: heh, exactly what I said in my response19:39
mriedemwe are sympatico19:39
*** ociuhandu has quit IRC19:39
sean-k-mooneywere is the spec for detaching a root volume19:39
*** ociuhandu has joined #openstack-nova19:39
dansmithNostawRm: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010678.html19:39
sean-k-mooneyresucing a boot form volume instance is a similar usecase19:40
dansmithsean-k-mooney: not really, rescue wouldn't require a detach19:40
dansmithjust a boot order change19:40
sean-k-mooneyya thats true19:40
NostawRmThat's hilarious19:40
NostawRmthat its also StorPool19:40
sean-k-mooneybut if you can detact it you can detach and add it to another instance19:41
dansmithsean-k-mooney: that's not what rescue is19:41
sean-k-mooneyi know rescue is booting and existing instance in place so you can fix it19:41
TxGirlGeekefried got it :)19:41
sean-k-mooneybut you can detach attach fix detach retatach to original instnace19:41
sean-k-mooney*you could19:41
sean-k-mooneyany fixing resucre for bfv is userful too19:42
*** tbachman has quit IRC19:43
*** tbachman has joined #openstack-nova19:45
sean-k-mooneyby the way anyone know if the topic of a generic cinder image backend come up again at the PTG. bfv is not quite the same thing but if we were to add another imgage backend that would be my goto approach19:47
sean-k-mooneyfixing bfv to do what you want feels like a lot less work however19:47
mriedemdansmith: didn't lee's old spec cover volume-backed rescue? https://review.opendev.org/#/c/651151/19:49
dansmithmriedem: I dunno, it'd be an important part of it though19:50
jrollefried: that all sounds good, although I don't even think the VM user or operator needs to be able to access the key in the secret manager. but yeah +1 to all the rest19:51
NostawRmSo, I'm fairly certain that the Mailing list thread was created because of me bothering StorPool. I was looking at possible solutions to propose to them, seems like they were already on it too :)19:52
sean-k-mooneyjroll: i think the operator definetly shound not be able to19:52
sean-k-mooneythe user maybe but proably not19:53
jrollyep19:53
sean-k-mooneythe user might have to be able to acess it if we use there auth token to manage it19:53
efriedjroll: I don't know what ability is conferred upon the "operator", by virtue of being the same user as is booting the instance19:53
efriedn-cpu is going to access the key manager service and be able to create & retrieve the passphrase based on being credentialed as the operator, no?19:54
sean-k-mooneyno19:54
efriedso couldn't the operator, using the same creds, simply go retrieve same?19:54
efriedoh?19:54
dansmithgenerally the user's creds would be passed to barbican right?19:54
sean-k-mooneyefried: nova compute should be useing the autoken form the boot request19:54
efriedis that not what I said?19:55
dansmithwhich means you can't start the instance in isolation, but when the user does a start, you can auth and get the deets19:55
sean-k-mooneyefried: operaor generaly refers to the cloud admin19:55
dansmithefried: you said "credentialed as the operator"19:55
efriedI thought "operator" in this context was "guy creating instance".19:55
efriedas opposed to "admin".19:55
dansmithhah19:55
dansmithno19:55
efrieddid I want "user"?19:56
dansmithyes19:56
efriedsigh, okay.19:56
efriedSo19:56
sean-k-mooneyuser or tenant19:56
efried"user" is the one with access to the secret19:56
mriedemNostawRm: fwiw i replied to the thread and linked in a bunch of pre-existing design work on several volume-backed server feature gaps in the api if you're so inclined to push your vendor to fund a developer to work on those again19:56
sean-k-mooneyyes ideally19:56
efrieddoesn't this have implications for certain lifecycle operations?19:56
sean-k-mooneyyes19:56
efried...that are done by admins?19:56
dansmithefried: see what I said above about restart19:56
sean-k-mooneyefried: this is already the case for encyped instances19:57
dansmithright19:57
efriedwhat operations are done by admins?19:57
dansmithlive migrate, restarting a compute host19:57
sean-k-mooneylive migrate might be ok19:58
dansmith(and cold migrate of course19:58
efried*can* live migrate be done by the user?19:58
sean-k-mooneyassuming libvirt woudl transfer the key19:58
dansmithefried: if granted, but not usually19:58
dansmithsean-k-mooney: yeah, fair19:58
sean-k-mooneyefried: not unless you change the policy.json19:58
dansmithbut cold migrate wouldn't work19:58
efried"libvirt would transfer the key" -- meaning the secret I loaded into the libvirtd on host1 would transfer over to the libvirtd on host2?19:59
sean-k-mooneywell cold migrate would result in the vm being ofline?19:59
sean-k-mooneywe would not be able to start it19:59
sean-k-mooneya user could do a resize19:59
sean-k-mooneythat should work19:59
dansmithsean-k-mooney: usually it is done on a running instance19:59
dansmithyeah, user-initiated resize should be doable19:59
dansmithcold migrate would need to either fail early, or else it would result in an offlined vm that the admin can't restart20:00
dansmithso not useful for maintenance without significant disruption20:00
NostawRmefried: I really appreciate that. I also don't mind putting in the work to contribute, I just like to check here to make sure the contributions wouldn't be immediately shot down if I can't find it being discussed anywhere else. Thanks!20:00
dansmithno way around that, but ideally it would refuse to move it so they don't accidentally break instances20:00
efriedI think the secret stays loaded up in libvirtd as long as that process is running, so admin could do pretty much anything that keeps the instance on the same host; as long as they don't restart libvirtd the instance will come back up.20:00
efriedI guess I'll have to test the secret-goes-with-live-migrate theory.20:01
sean-k-mooneyefried: ya maybe that would be correct20:01
dansmithefried: hopefully libvirt won't show you te secret right?20:01
sean-k-mooneyhowever hard reboot deletes the domain20:01
sean-k-mooneyand recreates it20:02
efrieddansmith: it won't, as long as private='yes'.20:02
sean-k-mooneyso that might clear the secret20:02
efriedsean-k-mooney: I'm saying specifically that doesn't matter.20:02
dansmithefried: yeah, so you can't use that to do an unauthorized cold migration, if that's what you're implying20:02
efrieddansmith: understood for cold, but still wondering about live.20:02
dansmithefried: live should work I would think20:03
dansmithefried: if not, graceful fail, otherwise you'd migrate into an unrunnable state20:03
efriedsetting up the secret and creating the instance are totally different things, from what I understand. I wouldn't expect libvirt to transfer the actual secret to the destination. However, if it's transferring the "state" of the instance, which includes an already-unencrypted vtpm, that might work. But then the VM on the target is kind of in a different condition than it was on the source, in that the secret is not loaded.20:05
dansmithefried: oh, well in that case maybe worth more thinking,20:05
efriedso like the reboot-recreate-xml scenario would *not* work in that case.20:05
sean-k-mooneyi can ask danpb tomorow when he is online if he knows what is expected to happen20:06
dansmithefried: but since the instances can't be restarted without the user's creds anyway, even if it migrates and can't reboot20:06
dansmiththat would be the same20:06
sean-k-mooneybut testing might give you an answer quiciker20:06
sean-k-mooneywhat is the state of migrations for instnace with encyption today20:07
efrieddansmith: well, what I was saying is, once the secret is loaded up in libvirtd, anybody would be able to restart, without user's creds.20:07
sean-k-mooneythe vtpm would likely have the same constraints20:07
dansmithefried: ...and break the ability to restart the instance in place you mean right?20:08
efriedunless we do a step to undefine the secret when rebooting or something.20:08
sean-k-mooneyefried: well we proably shoudl do that as part of stop20:08
*** avolkov has quit IRC20:08
sean-k-mooneyit also depens on if its a soft or hard reboot20:09
sean-k-mooneysoft reboot shoudl be fine20:09
dansmitha soft reboot would persist I'd think,20:09
dansmithbut yeah, hard reboot and stop should nuke the secret if you want it safe at rest20:09
efriedThe dom xml contains just a pointer to the secret. The secret has to be loaded up in libvirtd's memory in order to start the guest. Once it's loaded up, it stays loaded up until/unless you undefine it (or restart the service). Clearly I would undefine the secret on instance deletion. But anything where the instance needs to start, no matter who starts it, should work as long as the secret is still loaded.20:10
sean-k-mooneyefried: well you should undeife the secrete when you stop the instance without deleting it too20:10
sean-k-mooneyto not keep it in memory20:11
efriedwhy?20:11
efriedI mean sure... but why?20:11
*** slaweq has joined #openstack-nova20:11
sean-k-mooneyto prevent people reading it form libvirt memory. we could make it a config option or chose not too20:11
sean-k-mooneybut if you really care about security you want the key decyped for as short a time as possible20:12
efriedthat sounds in the same category as "delete the file from disk right quick"20:12
efriedi.e. "fake security" as dansmith calls it.20:12
sean-k-mooneynot really20:12
sean-k-mooneyyou are say if i have not deleted the isntance but stop it keep the key in memory untill either the vm is deleted or you restart libvirt20:13
dansmithefried: I think the user would expect that it's safe at rest when stopped20:13
*** tbachman has quit IRC20:13
*** slaweq has quit IRC20:16
dansmithefried: this is definitely all a bit of theater of course, because none of the places where the key lives unencrypted for any amount of time are "secure",20:17
efrieddansmith: We have apparently accepted that "compromised root" menas all bets are off anyway.20:18
dansmithbut I think it's different to say "okay you stopped it, so I forgot all the secrets until you give them to me again" vs "I wasn't very careful with your secret while I had it"20:18
sean-k-mooneyi dont think that is really true20:18
efriedthe important thing being, if someone walks off with the disk, they can't get at the secret, because they don't have the user's creds.20:19
sean-k-mooneyin some cases maybe but we should not make it easy even in that case20:19
dansmithefried: compromised root of the host you mean? that's definitely game-over I think with all the other assumptions that prop up an openstack cluster20:19
dansmithefried: yep and that's what I call 'safe at rest20:19
efriedconceivably I could delete the secret as soon as the guest starts.20:19
sean-k-mooneyefried: i have been wondering about that20:20
efriedI assume the tpm stays unlocked while the guest is up. Testing...20:20
sean-k-mooneybut that could break one usecase20:20
sean-k-mooneymaybe20:20
dansmithefried: I don't think that buys you anything real, other than avoiding forgetting/failing to clean it up, which is good of course20:20
sean-k-mooneywhat hapens when you ssh in and do a reboot in the guest20:20
efrieddansmith: yeah, that's all I meant, just simplifies some of the assumptions and operations.20:20
dansmithsean-k-mooney: IIRC modern kvm reboot is within the qemu process, not respawned by libvirt, so I would expect that works20:21
efriedMakes it harder for not-the-user to do things to the instance, but at least makes it harder consistently.20:21
dansmithefried: well, I'd see the main gain as avoiding a case where we fail during cleanup and don't get to the secret-forgetting part20:21
dansmithbut yeah20:21
sean-k-mooneydansmith: ya i thinkit sends an ahci shutdown and does nto exit the qemu process too so i think it would be ok i was just not sure20:21
dansmithsean-k-mooney: acpi you mean?20:22
sean-k-mooneysorry yes20:22
sean-k-mooneynot storage :)20:22
dansmithotherwise you're talking about usb or firewire20:22
dansmithusb I think20:22
sean-k-mooneyahci is used for sata too20:22
efried...yeah, confirmed deleting the secret after the guest starts doesn't affect ability of guest to get at the vtpm.20:22
dansmithoh sata right right20:22
dansmithmixed up my *hci acronyms20:23
dansmithefried: almost impossible for that to happen, since qemu and libvirt are fairly separate at runtime20:23
dansmithefried: try a soft reboot from inside the guest tho20:23
efriedmeaning, from within the guest, just 'reboot'?20:23
sean-k-mooneyyes20:23
sean-k-mooneya soft reboot from the api would almost be the same excpet that will escalte to a hard reboot after a time out20:24
efriedthat seems to have worked fine. Guest came back up, no errors in n-cpu, the tpm is still there and still works.20:25
dansmithfigured20:25
dansmithso delete after start is probably the way to go20:25
efriedthat's what we want though, right?20:25
sean-k-mooneyyes20:25
dansmithmakes it very clear how long we have the key20:25
dansmithyep20:26
efriedlike.20:26
efriedso assume live migration would carry the VM state, including its decrypted vtpm, and now the state on the dest is consistent: the key is not loaded on either side.20:27
dansmithexactly20:27
efriedum, meaning this is an example of an admin operation resulting in a functional vtpm on an instance that didn't "exist" before (on the dest host) where we didn't need the user creds to retrieve the key.20:27
efriedI... guess that's still fine.20:27
mriedemdansmith: replied on the cross-cell change https://review.opendev.org/#/c/635684/53/nova/compute/api.py@385220:28
dansmithefried: yes, but they still can't cold migrate or start the guest after a host reboot, but that's expected and "desired"20:28
dansmithfrom the security perspective20:28
efriedI guess it's a slightly different attack surface: if I have admin, and I have compromised root on one host in your cloud, I can get your secret by live-migrating your instance to that host and reading it out of memory.20:29
efriedum20:29
efriedI can read the *tpm* out of memory, not the secret.20:29
sean-k-mooneyit depnds on how qemu store it20:29
efriedwhich is just as bad.20:29
dansmithefried: that's  more work than you need to do to get the secret really20:29
sean-k-mooneywell the tpm data shoudl stilly be encyrpted20:30
dansmithefried: if you're root, you can read all of host memory, you can trigger a crash dump on the qemu, or ptrace it, etc20:30
sean-k-mooneybut read and writes to it shoudl be decypted by qemu20:30
efriedyeah, point is if you don't have root wherever the instance is currently running, but you do have root somewhere else in the cloud.20:30
dansmithefried: ah yes, sorry..20:30
efriedIs there already a way to mark an instance not migratable?20:31
dansmithyou don't even need root anywhere in the cluster to do that attack, fwiw20:31
dansmithall you need is the ability to connect to rabbit and it's over20:31
dansmithgiven that you can convince a compute you don't even have shell access to to send you the guest's memory contents to a non-secure port20:32
sean-k-mooneyso without going on too much of a tangent if you use SGX or other sercure memory enclave technologies you can protect form that20:33
sean-k-mooneybut i think that is out of scope fo vTPM20:33
sean-k-mooneywith SGX root cant read all memeory20:33
sean-k-mooneyand the kernel lockdown feature in 5.4 will also prevent that in some cases20:34
dansmithsean-k-mooney: my attack doesn't require root to be able to read the memory20:35
dansmithI said you don't even need shell on the box with the instance20:35
dansmithmy attack might not work with encrypted memory for other reasons, but not because of root20:35
sean-k-mooneysure that was not the point i was makeing20:36
sean-k-mooneyhttps://thenewstack.io/linux-kernel-finally-gets-its-lockdown/20:36
sean-k-mooney^ that is a cool feature in 5.4 that shoudl help20:36
sean-k-mooneybut anyway i like the idea of removing the key once we boot the instnace20:36
efriedSo is there a way to indicate I don't want my instance live-migratable?20:37
sean-k-mooneyi used to say use sriov/pinning or hugepage but we fixed that :)20:37
dansmithefried: meaning as a user asking the operator to not handle your memory contents over a migration?20:37
sean-k-mooneyill check the libvirt xml20:37
sean-k-mooneymaybe20:37
dansmithefried: the user isn't supposed to know that a live migration can or has happened really anyway, so what you're asking for is like an SLA which would happen outside of nova I think20:38
efrieddansmith: I guess I meant more like something on the instance so that n-api would simply refuse to migrate20:38
dansmithefried: I know, and I'm saying that's not an appropriate thing for nova to be doing IMHO20:38
efriedother than incidentals, like PCI devices (right?), I mean something specific.20:38
sean-k-mooneyefried: i mean we coudl just check for vtpm and reject20:39
dansmithno,20:39
dansmithsean-k-mooney: he's saying some user wanting to not be live migrated for risk,20:39
dansmithnot nova refusing to migrate vtpm-having instances20:39
sean-k-mooneyah20:39
efriedeither user (so in the flavor) or admin (conf opt)20:39
dansmithif we can migrate them (which we can I think) then we should20:39
dansmithefried: user doesn't control the flavor20:39
sean-k-mooneythats a similar discustion to the spec fo exposing post copy20:39
dansmiththe *details* of the flavor20:39
dansmithbut yeah, I'd say the cloud should say "Flavors that end in -secu will never be live migrated for security reasons" and then just handle that policy themselves20:40
sean-k-mooneyin that spec they wanted a way to mark a vm as not using post copy to avoid the risk of remote execution over a netwrok link20:40
umbSublimeThat's something I was asked today. Is there a way to make a vm/flavor _NOT_ live-migratable. Some of our users would for some reason want to opt-out of being live-migratable and understand what that means if we need to do maintenance on the host20:42
sean-k-mooneyfyi just looking at the libvirt level i dont see anything in https://libvirt.org/formatdomain.html that would alow us to mark a vm as not live migratable but we could do int at the nova level20:45
efriedhm, there's a side effect of the hack that's necessary to transfer the vtpm data meaning that you could disable migration by simply not setting those conf opts.20:45
dansmithsean-k-mooney: I assumed he was looking for a nova tunable not libvirt20:45
sean-k-mooneyya i just said i would look at both levels20:46
dansmithack20:46
sean-k-mooneyefried: which conf options?20:47
efriedhttps://review.opendev.org/#/c/639934/14/nova/conf/libvirt.py20:47
sean-k-mooneythe nova migration uri?20:47
sean-k-mooneythat would block you form using swtpm too however right20:48
sean-k-mooneywhy is swtpm_group a choices field20:49
sean-k-mooneythat kindof ties distos hands20:49
sean-k-mooneysame for user i guess20:49
sean-k-mooneyefried: with may kolla-ansible hat on it also forces me to run the swtpm as a specific user or group which is not ideal20:50
efriedsean-k-mooney: IIUC the libvirt process takes care of starting the swtpm.20:52
efriedI mean, I didn't have to do anything special to start it.20:52
sean-k-mooneyok but im not sure we should be resticting the options in nova20:53
sean-k-mooneye.g. i think that should just be a string field20:53
sean-k-mooneyif you do "ps aux | grep swtpm" what user/group is it running as20:54
efriedsome processes have a `tss` user, some have `root`.20:55
efriedsorry, the root ones are just qemu. They have swtpm CLI opts in 'em.20:56
* efried ==> school run21:01
*** efried is now known as efried_afk21:01
*** ociuhandu has quit IRC21:06
*** ociuhandu has joined #openstack-nova21:07
*** spsurya has quit IRC21:07
*** ociuhandu has quit IRC21:14
*** ociuhandu has joined #openstack-nova21:14
*** TxGirlGeek has quit IRC21:19
*** mlavalle has quit IRC21:22
*** ociuhandu has quit IRC21:24
*** mlavalle has joined #openstack-nova21:41
*** pcaruana has quit IRC21:42
*** gshippey has quit IRC21:44
*** takashin has joined #openstack-nova21:46
*** efried_afk is now known as efried21:53
*** ociuhandu has joined #openstack-nova22:07
*** slaweq has joined #openstack-nova22:11
*** ociuhandu has quit IRC22:11
*** slaweq has quit IRC22:15
*** dviroel has quit IRC22:16
efriedNot sure why I shouldn't determine the uid/gid based on the swtpm process running on the dest host.22:21
efriedI guess discovering which process that is could be a bit tricky.22:22
*** ociuhandu has joined #openstack-nova22:25
efriedoh, ahem, the swtpm processes correspond to instances; they don't exist until the tpm does, chicken/egg.22:26
sean-k-mooneyefried: is this in relattion to the config options or something else?22:27
sean-k-mooney*relation22:27
efriedyeah, I was trying to figure out a way not to need the config options.22:28
efriedbecause they suck.22:28
sean-k-mooneywell why do we need them?22:28
sean-k-mooneynova is not startign the swtpm binary22:28
sean-k-mooneyright?22:28
sean-k-mooneythat is handeled by libvirt22:28
efriedthe tpm data file needs to be owned by the uid/gid of the swtpm server process22:28
sean-k-mooneywill libvirt create that for us?22:29
efriedIt creates the process22:29
efriedit doesn't help us for it to create the file, cause it ain't the right one.22:29
sean-k-mooneyim not really a fan of nova having to manage those file permissions22:31
sean-k-mooneyespcially if the file is not own by the nova user22:31
efriedI agree. Summed up above as "suck".22:31
sean-k-mooneycould we make the files be owned by nova?22:32
sean-k-mooneyi have not really looked at how this works at a lowerer level22:32
*** ociuhandu has quit IRC22:33
sean-k-mooneyalso we really should not have a generic recursive_chown function in nova.22:34
sean-k-mooney*need to have22:34
*** dklyle has quit IRC22:35
*** david-lyle has joined #openstack-nova22:35
sean-k-mooneyi dont nessisarly know of a better way but from a privsep point of view it sucks to have generic functions like that that take a path user and group22:35
sean-k-mooneyefried: is this just needed for the save_and_migrate_vtpm_dir and restore_vtpm_dir functions22:39
efriedyes22:39
sean-k-mooneyyour kind of assuming that nova will have read acess to those folders in places22:41
sean-k-mooneyalthough you might be using privladged functions22:41
efriedI haven't really looked at that patch yet (I didn't write it) but I would assume I would need to use privileged operations on both sides.22:42
sean-k-mooneyi gues when you do  fileutils.ensure_tree(swtpm_dir) to create the folder its in the libvirt instance dir and nova should have write acess to that22:42
sean-k-mooneyno all operations would need to be privaladged22:43
sean-k-mooneyso that patch is mainly for resize since cold migrate wont work and normal users cant call it22:45
efriedOn the source, moving the data from its normal location to within the instance dir needs to be privileged to get at the data in its original location22:46
efriedConversely on the dest, creating the real location needs to be privileged; and we should really make sure it's already owned by the right user before moving the contents back out.22:46
efriedso I don't see which operations there wouldn't need to be privileged.22:46
sean-k-mooneythe ensure_tree function call here https://review.opendev.org/#/c/639934/15/nova/virt/libvirt/utils.py@65422:47
efriedcool, looks like that's the case, no?22:47
sean-k-mooneyit is i was just wondering if that need privlages and it was missing or not22:48
sean-k-mooneyi would not consider it that unusaly to assume nova can read and write to the virt driver instance folder without needing to elevate privaldges22:49
efriedyeah, that's a fair point.22:51
sean-k-mooneyits proably fine i was just trying to figure out in what context the config parmanter were used and the surronding context22:51
sean-k-mooneyin this case the reason nova has to hanel this is that in the cold migrate case libvirt wont copy these files for us. at least not by default22:52
efriedin the live migrate case it won't copy these files either.22:52
sean-k-mooneythe commit does not suggest that this is need for live migration22:53
efriedwhat's a "resize between nodes"?22:54
sean-k-mooneyyour chageing the flavor22:54
sean-k-mooneyresize is generally between hosts22:54
sean-k-mooneycold migrate is a special calse fo resize where the flaovr remains the same22:54
sean-k-mooneyat least in the libvirt driver22:54
dansmithmriedem: if you're going to flip that to an cast always, I can +W what you have and you can do the error handling cleanup and cast in a change after this but before we turn it on22:55
dansmithmriedem: if you're cool with that22:55
dansmithmriedem: also, I think you'd've needed a reno to explain how migrate _can_ be different sometimes for invisible reasons anyway, so doing one to explain that it's always a little different now is equivalent/better22:56
*** rcernin has joined #openstack-nova22:58
mriedemok, i can build that into the series before the policy change at the end. as for docs, it would have been part of the cross-cell docs patch that i haven't fleshed out yet, but yeah it's easier to just say generically 'resize / cold migrate is now a cast all the time'22:58
mriedemsean-k-mooney: the flavor always remains the same for cold migrate22:59
dansmithmriedem: I'd prefer it go early in the series the next time you rebase, just so it merges closer to this conversation for context, but.. ack22:59
mriedemthe song....that's a different story22:59
mriedemdansmith: ack, i likely need to rebase soonish anyway so i'll push it toward the bottom when i do23:00
dansmithk23:00
sean-k-mooneymriedem: yes that is what i intened to say23:01
*** tkajinam has joined #openstack-nova23:01
sean-k-mooneyi was just trying to indicate that when a commit say "When doing a resize or a cold migration between nodes, this data needs to be copied ..." its not refering to live migration23:03
sean-k-mooneyefried: this is the qemu docs on the migration process https://github.com/qemu/qemu/blob/9f2ce35dfa4ea4a31dbb765dd02bed2500891887/docs/specs/tpm.txt#L324-L38423:08
efriedIIUC all of that is handled under the covers23:09
sean-k-mooneyyes23:10
sean-k-mooneybut looking at it i am not sure they every transfer the tpm state independetly23:10
sean-k-mooneyit seams to be contained in the testvm.bin create by the migrate command23:11
openstackgerritDustin Cowles proposed openstack/nova master: WIP: Provider Config File: Enable loading and merging of provider configs  https://review.opendev.org/69346023:14
sean-k-mooneythis would also seam to mesh with our assumtion of automatic stat transfer for the live migration case https://github.com/stefanberger/libvirt-tpm/commit/72299db63690aa4b74131f54398a665a51e3592f23:16
*** mlavalle has quit IRC23:16
*** nweinber__ has quit IRC23:21
*** awalende has joined #openstack-nova23:21
*** munimeha1 has quit IRC23:25
*** awalende has quit IRC23:26
*** ociuhandu has joined #openstack-nova23:30
*** ivve has quit IRC23:32
*** ociuhandu has quit IRC23:35
*** maciejjozefczyk has joined #openstack-nova23:38
openstackgerritEric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM  https://review.opendev.org/68680423:41
efriedjroll, dansmith, sean-k-mooney: I think that's ready for review now ^23:42
sean-k-mooneyack. ill review it tomorrow23:42
*** ralonsoh has quit IRC23:44
openstackgerritEric Fried proposed openstack/nova-specs master: Spec: Ussuri: Encrypted Emulated Virtual TPM  https://review.opendev.org/68680423:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!