Tuesday, 2019-11-12

*** zbr has quit IRC00:09
*** slaweq has joined #openstack-nova00:11
*** slaweq has quit IRC00:16
*** maciejjozefczyk has quit IRC00:22
*** ociuhandu has joined #openstack-nova00:28
*** ociuhandu has quit IRC00:39
*** mriedem has quit IRC00:44
*** yoctozepto has quit IRC00:53
*** yoctozepto has joined #openstack-nova00:54
*** dviroel has joined #openstack-nova00:59
*** nanzha has joined #openstack-nova01:02
*** yoctozepto has quit IRC01:05
*** yoctozepto has joined #openstack-nova01:06
*** Liang__ has joined #openstack-nova01:10
*** ociuhandu has joined #openstack-nova01:14
*** ociuhandu has quit IRC01:18
*** lbragstad_ has joined #openstack-nova01:24
*** lbragstad has quit IRC01:26
*** ociuhandu has joined #openstack-nova01:57
*** nweinber__ has joined #openstack-nova01:59
*** ociuhandu has quit IRC02:08
*** slaweq has joined #openstack-nova02:11
*** nanzha has quit IRC02:15
*** slaweq has quit IRC02:16
*** nanzha has joined #openstack-nova02:16
*** mkrai has joined #openstack-nova02:32
openstackgerritMerged openstack/nova master: Plumb allow_cross_cell_resize into compute API resize()  https://review.opendev.org/63568402:42
*** ociuhandu has joined #openstack-nova02:46
*** nanzha has quit IRC02:47
*** nanzha has joined #openstack-nova02:48
*** ociuhandu has quit IRC02:51
*** ileixe has joined #openstack-nova02:55
*** rcernin_ has joined #openstack-nova03:00
*** tbachman has joined #openstack-nova03:03
*** rcernin has quit IRC03:03
*** tbachman_ has joined #openstack-nova03:08
*** tbachman has quit IRC03:08
*** tbachman_ is now known as tbachman03:08
*** brault has quit IRC03:13
*** brault has joined #openstack-nova03:20
*** rcernin_ has quit IRC03:26
*** rcernin has joined #openstack-nova03:26
*** tbachman has quit IRC03:26
*** tbachman has joined #openstack-nova03:29
*** bhagyashris has joined #openstack-nova03:40
*** udesale has joined #openstack-nova03:41
*** udesale has quit IRC03:45
*** udesale has joined #openstack-nova03:46
*** ivve has joined #openstack-nova03:46
*** nanzha has quit IRC03:48
*** nanzha has joined #openstack-nova03:48
*** zhubx has quit IRC03:51
*** zhubx has joined #openstack-nova03:51
*** slaweq has joined #openstack-nova04:11
*** slaweq has quit IRC04:16
*** yaawang has quit IRC04:20
*** yaawang has joined #openstack-nova04:23
*** nweinber__ has quit IRC04:28
*** tkajinam_ has joined #openstack-nova04:29
*** tkajinam has quit IRC04:32
*** tetsuro has joined #openstack-nova04:33
*** ociuhandu has joined #openstack-nova04:34
*** dviroel has quit IRC04:36
*** ociuhandu has quit IRC04:41
*** tkajinam_ has quit IRC04:58
*** tkajinam has joined #openstack-nova04:59
*** links has joined #openstack-nova05:12
*** ygk_12345 has joined #openstack-nova05:16
*** ygk_12345 has quit IRC05:19
*** ygk_12345 has joined #openstack-nova05:19
*** brinzhang_ has joined #openstack-nova05:25
*** nanzha has quit IRC05:26
*** slaweq has joined #openstack-nova05:26
*** brinzhang has quit IRC05:28
*** ratailor has joined #openstack-nova05:30
*** ociuhandu has joined #openstack-nova05:30
*** ociuhandu has quit IRC05:35
*** jraju__ has joined #openstack-nova05:36
*** links has quit IRC05:36
*** jraju__ has quit IRC05:36
*** links has joined #openstack-nova05:40
*** Luzi has joined #openstack-nova05:42
*** udesale has quit IRC05:43
*** udesale has joined #openstack-nova05:43
*** udesale has quit IRC05:44
*** udesale has joined #openstack-nova05:44
*** tbachman has quit IRC05:48
*** zhubx has quit IRC05:48
*** boxiang has joined #openstack-nova05:48
*** ivve has quit IRC05:59
*** takashin has left #openstack-nova06:33
*** lpetrut has joined #openstack-nova06:33
*** lpetrut has quit IRC06:34
*** lpetrut has joined #openstack-nova06:35
*** TxGirlGeek has joined #openstack-nova06:38
*** dpawlik has joined #openstack-nova06:49
*** jraju__ has joined #openstack-nova06:54
*** links has quit IRC06:55
*** dpawlik has quit IRC06:55
*** ivve has joined #openstack-nova07:03
*** udesale has quit IRC07:03
*** udesale has joined #openstack-nova07:04
*** chenhaw has joined #openstack-nova07:06
*** ociuhandu has joined #openstack-nova07:10
*** ociuhandu has quit IRC07:10
*** ociuhandu has joined #openstack-nova07:11
*** dpawlik has joined #openstack-nova07:14
*** ociuhandu has quit IRC07:16
*** rcernin has quit IRC07:22
*** trident has quit IRC07:37
*** mkrai has quit IRC07:42
openstackgerritLuyao Zhong proposed openstack/nova master: support live migration with vpmems  https://review.opendev.org/68785607:48
*** ociuhandu has joined #openstack-nova07:48
*** trident has joined #openstack-nova07:48
*** ociuhandu has quit IRC07:54
*** damien_r has joined #openstack-nova07:57
bauzasgood morning Nova08:02
* bauzas is back after 2 weeks :)08:02
*** adriant has quit IRC08:08
*** maciejjozefczyk has joined #openstack-nova08:09
*** tesseract has joined #openstack-nova08:11
*** mkrai has joined #openstack-nova08:14
*** ralonsoh has joined #openstack-nova08:14
*** awalende has joined #openstack-nova08:23
gibibauzas: good morning08:27
gibibauzas: how is your jetlag?08:28
*** zbr has joined #openstack-nova08:30
*** dpawlik has quit IRC08:31
*** dpawlik has joined #openstack-nova08:34
*** zbr has quit IRC08:35
bauzasgibi: quite good, just woke up today at 6.30am :)08:35
bauzasgibi: and you ?08:35
* bauzas needs to write emails FWIW08:36
*** zbr has joined #openstack-nova08:37
*** tkajinam has quit IRC08:37
*** dtantsur|afk is now known as dtantsur08:48
*** do3meli has joined #openstack-nova08:48
*** ileixe has quit IRC08:50
*** ileixe has joined #openstack-nova08:52
*** TxGirlGeek has quit IRC08:54
*** priteau has joined #openstack-nova08:58
*** rpittau|afk is now known as rpittau08:59
*** ociuhandu has joined #openstack-nova09:01
*** Luzi has quit IRC09:08
*** xek_ has joined #openstack-nova09:09
kashyaplyarwood: efried: Back today.  I haven't read the KM-long scrollback, afraid (buried under several things).09:22
*** Luzi has joined #openstack-nova09:23
*** brinzhang has joined #openstack-nova09:23
*** brinzhang_ has quit IRC09:27
*** ociuhandu has quit IRC09:30
*** mkrai has quit IRC09:31
*** ociuhandu has joined #openstack-nova09:31
*** irclogbot_0 has quit IRC09:39
*** irclogbot_1 has joined #openstack-nova09:41
*** ociuhandu has quit IRC09:42
*** ociuhandu has joined #openstack-nova09:44
*** mkrai has joined #openstack-nova09:44
*** rcernin has joined #openstack-nova09:45
*** ileixe has quit IRC09:46
*** ileixe has joined #openstack-nova09:49
*** jaosorior has joined #openstack-nova09:51
*** ociuhandu has quit IRC09:53
*** ociuhandu has joined #openstack-nova09:54
*** ociuhandu has quit IRC09:55
*** ociuhandu has joined #openstack-nova09:56
*** ociuhandu has quit IRC09:57
*** ociuhandu has joined #openstack-nova09:58
*** ociuhandu has quit IRC10:00
*** ociuhandu has joined #openstack-nova10:01
*** ociuhandu has quit IRC10:02
*** ociuhandu has joined #openstack-nova10:03
*** awalende has quit IRC10:06
*** awalende has joined #openstack-nova10:06
*** links has joined #openstack-nova10:06
openstackgerritLiang Fang proposed openstack/nova-specs master: Support volume local cache  https://review.opendev.org/68907010:07
*** jraju__ has quit IRC10:07
*** ociuhandu has quit IRC10:08
*** ygk_12345 has left #openstack-nova10:10
*** awalende has quit IRC10:11
*** rcernin has quit IRC10:11
*** mdbooth has joined #openstack-nova10:15
*** brinzhang_ has joined #openstack-nova10:16
*** Liang__ has quit IRC10:19
*** brinzhang has quit IRC10:20
*** tetsuro has quit IRC10:21
*** awalende has joined #openstack-nova10:23
*** panda has quit IRC10:24
*** ratailor_ has joined #openstack-nova10:26
*** ratailor has quit IRC10:27
*** panda has joined #openstack-nova10:29
*** vesper has quit IRC10:30
*** vesper11 has joined #openstack-nova10:31
*** brinzhang_ has quit IRC10:37
*** chenhaw has quit IRC10:42
*** dpawlik has quit IRC10:47
kashyapDoes anyone recall top off their head, AMD SEV support in Nova doesn't support live migration yet, does it?10:48
*** ociuhandu has joined #openstack-nova10:48
kashyap(I vaguely recall it doesn't.)10:48
kashyapI'm sure it's somewhere in the spec10:49
*** rcernin has joined #openstack-nova10:50
*** awalende has quit IRC10:51
*** awalende has joined #openstack-nova10:51
kashyap(Yep, my guess is correct - checked in the spec.  Also needs support for LM-with-SEV from lower layers.)10:51
*** awalende has quit IRC10:54
*** awalende has joined #openstack-nova10:54
*** ociuhandu has quit IRC10:54
*** bhagyashris has quit IRC10:54
*** ratailor__ has joined #openstack-nova10:55
*** ratailor_ has quit IRC10:57
*** gshippey has joined #openstack-nova11:00
*** awalende has quit IRC11:01
*** awalende has joined #openstack-nova11:02
gibibauzas: today was OK for me, yesterday I start my day at 4:4511:03
*** awalende has quit IRC11:06
*** awalende has joined #openstack-nova11:08
*** jraju__ has joined #openstack-nova11:09
*** jaosorior has quit IRC11:10
*** links has quit IRC11:10
*** xek_ has quit IRC11:12
*** dpawlik has joined #openstack-nova11:15
openstackgerritMerged openstack/nova master: Pass RequestContext to oslo_policy  https://review.opendev.org/67403811:17
*** udesale has quit IRC11:17
openstackgerritMerged openstack/nova master: Add func test for 'required' PCI NUMA policy  https://review.opendev.org/68294111:17
*** dpawlik has quit IRC11:19
openstackgerritMerged openstack/nova master: Resolve TODO in _remove_host_allocations  https://review.opendev.org/69363611:21
*** rcernin has quit IRC11:26
*** dpawlik has joined #openstack-nova11:27
openstackgerritBalazs Gibizer proposed openstack/nova stable/pike: Functional reproduce for bug 1852207  https://review.opendev.org/69381711:29
openstackbug 1852207 in OpenStack Compute (nova) pike "reschedule ignores that requested availability zone" [Medium,Triaged] https://launchpad.net/bugs/1852207 - Assigned to Balazs Gibizer (balazs-gibizer)11:29
gibielod: ^^11:30
*** brinzhang has joined #openstack-nova11:35
*** shilpasd has joined #openstack-nova11:44
*** bbowen has quit IRC11:46
*** do3meli has quit IRC11:50
*** dviroel has joined #openstack-nova11:58
*** maciejjozefczyk has quit IRC12:04
*** boxiang has quit IRC12:08
*** boxiang has joined #openstack-nova12:09
openstackgerritBrin Zhang proposed openstack/nova master: WIP: Support re-configure the delete_on_termination in server  https://review.opendev.org/69382812:10
*** dpawlik has quit IRC12:12
*** shilpasd has quit IRC12:18
*** dave-mccowan has joined #openstack-nova12:19
*** do3meli has joined #openstack-nova12:31
*** awalende has quit IRC12:33
*** awalende has joined #openstack-nova12:33
*** awalende has quit IRC12:34
*** awalende has joined #openstack-nova12:34
*** mdbooth has quit IRC12:36
*** mdbooth has joined #openstack-nova12:37
*** dswebb has joined #openstack-nova12:38
*** dpawlik has joined #openstack-nova12:39
*** ratailor__ has quit IRC12:39
*** openstackgerrit has quit IRC12:41
*** mdbooth has quit IRC12:41
*** efried has quit IRC12:42
*** mdbooth has joined #openstack-nova12:42
*** dpawlik has quit IRC12:44
*** ociuhandu has joined #openstack-nova12:46
*** dpawlik has joined #openstack-nova12:47
*** efried has joined #openstack-nova12:49
*** brinzhang has quit IRC12:50
*** ociuhandu has quit IRC12:50
*** brinzhang has joined #openstack-nova12:51
*** jaosorior has joined #openstack-nova12:57
*** bbowen has joined #openstack-nova13:00
*** liuyulong has joined #openstack-nova13:02
jrollefried: thanks, will hit it today13:04
*** eharney has quit IRC13:04
*** efried has quit IRC13:05
*** henriqueof has quit IRC13:05
*** awalende has quit IRC13:06
*** awalende has joined #openstack-nova13:06
*** henriqueof has joined #openstack-nova13:08
*** awalende has quit IRC13:11
*** eharney has joined #openstack-nova13:12
*** ociuhandu has joined #openstack-nova13:16
*** awalende has joined #openstack-nova13:22
*** awalende has quit IRC13:24
*** awalende has joined #openstack-nova13:25
*** awalende has quit IRC13:28
*** awalende has joined #openstack-nova13:28
kashyapgit fetch gerrit13:36
*** mriedem has joined #openstack-nova13:37
kashyapErr13:37
*** openstackgerrit has joined #openstack-nova13:37
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" spec  https://review.opendev.org/69384413:37
*** ociuhandu has quit IRC13:38
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" for Ussuri  https://review.opendev.org/69384413:38
*** ociuhandu has joined #openstack-nova13:39
*** maciejjozefczyk has joined #openstack-nova13:40
*** ociuhandu has quit IRC13:44
*** jangutter has quit IRC13:48
*** nweinber__ has joined #openstack-nova13:48
mriedemgibi: before i get started on rebasing the cross cell resize series and adding a change to just cast from api to conductor always ( https://review.opendev.org/#/c/635684/53/nova/compute/api.py@3852 ) i wanted to see if you're ok with that plan?13:49
*** efried has joined #openstack-nova13:50
efriedFun morning. About every 15 minutes, the power goes out for a half second. Just long enough to kill my router.13:50
efriedTexas: "Ohh, noooo, it's three degrees below freezing, I'm dyyyyyying!"13:51
mriedemthoughts and prayers13:52
efriedkashyap: Let me tldr it for you: what's the minimum qemu needed for *encrypted* emulated TPM?13:52
kashyapefried: Hiya; need to do some sleuthing.  Let me do that...13:52
efriedunencrypted was 2.11 (according to cfriesen)13:53
efriedthanks kashyap13:53
mriedemwe got a replacement tree put in over the weekend and you're supposed to water new trees but what about when the ground is frozen...13:53
efriednot a huge gurry.13:53
efriedmriedem: hot water?13:53
kashyapI'll not gurry.13:53
* efried goes to figure out what a gurry is13:53
efried"fish or whale offal". Nice.13:53
kashyap"the entrails of fish or whales"13:54
* efried gets entry for typo of the day in early.13:54
efriedbut we haven't seen sean-k-mooney yet, so...13:54
efried:P13:54
mriedemefried: i'm pretty sure hot water just freezes too13:54
kashyapSpeaking of "offals" ... the other week I was in Lyon, France, for KVM Forum.  They are all super big into "offals", apparently.  (/me is a vegetarian; only heard it through the grapevine)13:54
efriedThat's... awfful13:55
efried(see what I did there?)13:55
kashyapVery pun13:55
efriedMany cultures have an equivalent. Boudin sausage in Louisiana. Hot dogs in baseball stadiums...13:56
efriedhaggis in scotland13:56
openstackgerritLee Yarwood proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices  https://review.opendev.org/69384913:56
*** dpawlik has quit IRC13:57
*** dpawlik has joined #openstack-nova13:58
kashyapefried: I see.  (Yeah, heard the "haggis" term echo around when in Edinburgh last year.)13:59
*** awalende_ has joined #openstack-nova13:59
luyaoefried: hi Eric, I have a question, do I need a bp and separate spec for vpmem live migration?13:59
*** Luzi has quit IRC14:00
efriedIt must be like, at some point in their history all these cultures were starving, so they had to make use of every calorie they could scrounge up, and find a way to make it palatable. And then for reasons of... tradition? it remained in the culture long after it ceased to be necessary.14:00
efriedExcept for the hot dog thing. That's just because baseball fans are stupid and crazy.14:00
*** awalende has quit IRC14:02
*** ociuhandu has joined #openstack-nova14:02
*** priteau has quit IRC14:04
dswebbHi,have a bit of an odd issue with Livemigration on rocky.   I've got a 3 node nova cluster using iscsi -> nimble as my storage.  I can successfully migrate a VM from node 1 to node 2 (or node2 to node1) once.  If I try and migrate the vm back to the node it started on (or any node it's been on prior) it fails silently to migrate and the only logs I see that suggest any issue are the following: https://pastebin.com/Y2ccf3MR14:05
gibimriedem: sorry I have to get back to you tomorrow (my) morning14:05
*** dpawlik has quit IRC14:06
mriedemgibi: ok i'll just post the change since it'll be simple and i need to rebase the series anyway, if you have a problem with it you and dan can fight to the death on the review14:06
*** efried has quit IRC14:07
gibimriedem: sure. works for me :)14:07
*** xek_ has joined #openstack-nova14:07
kashyapAh, efried is gone ... just when I have his answer for him14:07
*** tbachman has joined #openstack-nova14:08
*** dpawlik has joined #openstack-nova14:09
*** efried has joined #openstack-nova14:11
*** ociuhandu has quit IRC14:13
*** ociuhandu has joined #openstack-nova14:14
luyaoefried: hi Eric, in case of you not seeing the msg just now,  I need you confirm if I need a blueprint and separate spec for vpmem live migration. Thanks in advance.14:15
dswebbalso worth mentioning that the same live migration (back and forth) using ceph as the cinder block store works fine14:16
efriedluyao: I don't know; let's put it on the agenda for the next meeting and ask the team.14:18
luyaoefried: OK, thanks14:19
efriedluyao: I think gibi had at least a blueprint and possibly a spec for the continuation of lifecycle ops with bandwidth providers, but I think that's a much bigger effort than for vpmem.14:19
*** ociuhandu has quit IRC14:19
kashyapefried: Hey, so on your encrypted emulated TPM question --14:19
kashyapefried: The QEMU version does not matter for encryption.  What matters is the 'swtpm'.  And you need at least libvirt 5.6.0 for the 'encryption' element14:20
kashyaphttps://libvirt.org/formatdomain.html#elementsTpm14:20
*** dpawlik has quit IRC14:20
efriedkashyap: sweet, thank you very much. I was aware of the libvirt side, but couldn't find the qemu side documented anywhere.14:21
* bauzas is done with writing ML threads for the PTG \o/14:21
*** dpawlik has joined #openstack-nova14:21
efriedbauzas, gibi: Thank you both for writing up those summaries!14:21
bauzasnp14:22
bauzasHTH14:22
*** ivve has quit IRC14:22
luyaoefried: yes, in terms of code size, the lm for vpmem is not big14:23
*** eharney has quit IRC14:23
efriedluyao: was there mention of the details of lm implementation in the original spec? (Save me digging it up)14:24
*** nweinber__ has quit IRC14:25
luyaoefried: no, and I need update the original spec, since our implementation is different from it14:25
*** nweinber has joined #openstack-nova14:26
openstackgerritGeorge proposed openstack/nova stable/ocata: Support qemu >= 2.10  https://review.opendev.org/69385114:27
*** tbachman has quit IRC14:27
efriedluyao: I think a short spec is probably a good idea, since (IIRC) there are RPC object changes, right?14:28
luyaoefried: yes, there is14:29
efriedSo yeah, please put together a brief spec. Thanks.14:30
luyaoefried: Get it, thanks14:31
*** jraju__ has quit IRC14:33
*** efried has quit IRC14:43
*** ociuhandu has joined #openstack-nova14:56
*** efried has joined #openstack-nova15:00
*** Liang__ has joined #openstack-nova15:00
*** ociuhandu has quit IRC15:00
openstackgerritArtom Lifshitz proposed openstack/nova master: Helper to start computes with different HostInfos  https://review.opendev.org/68683215:04
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259515:04
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146  https://review.opendev.org/68740415:04
openstackbug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms)15:04
artomstephenfin, if you've recovered from PTG and still have all your organs, ^^ is still waiting :)15:05
*** eharney has joined #openstack-nova15:05
*** tbachman has joined #openstack-nova15:05
*** dpawlik has quit IRC15:06
bauzasartom: stephenfin is still somewhere in China15:07
artombauzas, intentionally? ;)15:07
dansmithlyarwood: I reviewed your rescue spec.. do you have any idea what needs to happen and or what blockers there would be to making rescue work for BFV instances?15:08
bauzasartom: and I can't tell for his organs, but last time I saw him, his liver was working correctly15:08
bauzas:p15:08
artombauzas, haha15:08
bauzasartom: well, he found that Shanghai wasn't that huge so he visited another city15:08
bauzasoh, and FWIW, I'll keep my wechat account15:09
bauzasliving in China for one week makes you realize how life can be different without big G and a few other tools15:09
lyarwooddansmith: thanks for that, yeah I have a rough idea but I've not looked at this since failing to land this in Newton. I can't think of much that's changed since then with regards to the high level implementation. I also can't think of any additional blockers for BFV at the moment tbh.15:14
*** mlavalle has joined #openstack-nova15:14
dansmithlyarwood: okay, maybe I'll try removing the BFV check on a devstack and poke around with what is and isn't working in the current setup15:15
dansmithlyarwood: because BFV instances are some of the more important ones you'd want to rescue, ya'know15:15
lyarwooddansmith: right, so for rescue to work you need to switch things around so the rescue device is the final thing attached, not the first and then boot from that device.15:16
dansmithlyarwood: that's what you're proposing....15:17
*** eharney has quit IRC15:17
lyarwooddansmith: right sorry I think I'm misunderstanding what you're actually asking here.15:19
dansmithlyarwood: right now we just categorically refuse to rescue a BFV instance, and I think we should change that since BFV instances are likely pets, and likely in need of rescue15:20
*** lbragstad_ is now known as lbragstad15:20
dansmithso, with your proposed changes, in addition to the ordering, also including cinder volumes,15:20
dansmithI'm wondering what we could do to follow up your changes with a change to allow rescue of BFV instances15:20
smcginnisI've seen cattle with BFV too. Some just prefer to keep their storage on Cinder volumes. FWIW.15:21
dansmithsmcginnis: sure I know15:22
dansmithsmcginnis: I said "likely" not "always" :)15:22
lyarwooddansmith: Ah! The changes would actually allow BFV and non-BFV but I can see that isn't made clear in the spec. I totally missed this when copying it over from an older change this morning but I'll clean this up and make it clear now.15:23
dansmithlyarwood: sweet15:24
dansmithlyarwood: do you have old crusty code of up for this at all?15:24
lyarwooddansmith: there's an old abandoned series but iirc that was missing the compute service check that I think I had somewhere locally15:24
dansmithlyarwood: ack15:25
lyarwooddansmith: https://review.opendev.org/#/q/status:abandoned+topic:bp/virt-rescue-stable-disk-devices15:25
dansmithlyarwood: so you think that after these changes, if the api let them, we could rescue BFV instances no problem?15:25
lyarwooddansmith: yes15:26
dansmithif so, we'll need a compute capability to indicate this support, and could use that in place of the version check15:26
lyarwoodyup true, I also didn't cover microversions in the spec that I think came up in that series15:26
*** pcaruana has joined #openstack-nova15:28
dansmithlyarwood: no REST API changes, so why a microversion?15:29
*** eharney has joined #openstack-nova15:30
*** Liang__ has quit IRC15:31
lyarwooddansmith: https://review.opendev.org/#/c/270288/18/nova/compute/api.py@2777 - mriedem raised it here as something we might have to do, I'm also not sure if that's still the case tbh.15:31
dansmithlyarwood: oh, you even remove that in the code, I see,15:32
dansmithwell, in that case it probably does need a microversion and a rest api impact section, but in the spec I just reviewed you said "none" :)15:32
lyarwoodyeah apologies this slipped my mind when I was pushing it up for review earlier. I'll mark the spec as WIP for now while I get things back in order.15:34
dansmithlyarwood: you said you were going to work on this this cycle right?15:34
lyarwooddansmith: yup I should actually have time this cycle15:35
dansmithlyarwood: cool, I15:36
dansmithI would suggest getting this going sooner than later if you can schedule it15:36
dansmithat least rebase the code and get it restored so that others can comment/help15:36
dansmitheven if before the spec, although I think the spec is easy to merge pretty soon if you fix up the issues15:37
lyarwooddansmith: sure, I should be able to get something rebased pretty quickly.15:37
lyarwoodfamous last words15:37
* dansmith screenshots15:37
efriedjroll: randomly generated passphrase: do you think there should be a conf option for the length? If so/not, what should the default/number be?15:41
*** ociuhandu has joined #openstack-nova15:44
*** ociuhandu has quit IRC15:44
*** ociuhandu has joined #openstack-nova15:46
jrollefried: probably not worth a config until/unless someone asks. 32 or 64 characters seems reasonable15:49
efriedRoger that. (Glad I asked, I was gonna go 256 :P )15:50
jroll256 is also reasonable :D15:50
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Wire up a force disconnect_volume flag  https://review.opendev.org/58484915:50
jrollefried: I guess if nobody ever touches this, maybe 256 is a better option15:51
efriedit certainly won't be readable/typable. Like an ssh key. How big are those?15:51
*** awalende has joined #openstack-nova15:52
*** TxGirlGeek has joined #openstack-nova15:52
jroll$ wc ~/.ssh/id_rsa15:52
jroll      54      61    3326 /Users/jrollenhagen/.ssh/id_rsa15:52
*** ociuhandu has quit IRC15:53
efriedyours is bigger than mine15:53
dansmithmriedem: wha-chu-think main? https://review.opendev.org/#/c/636224/50/nova/compute/api.py15:53
jrollheh15:53
dansmithefried: whoa, wrong channel15:54
efriedthat's number of chars, b64-encoded, plus header/footer15:54
jrollright, looks like 3072 bytes15:55
mriedemdansmith: you can't compare None values can you?15:55
*** awalende_ has quit IRC15:56
dansmithmriedem: you can15:56
dansmithmriedem: >>> (None, 1) < (None, 2)15:57
dansmithTrue15:57
*** awalende has quit IRC15:57
mriedem>>> datetime.datetime.today() < None15:57
mriedemTraceback (most recent call last):15:57
mriedem  File "<stdin>", line 1, in <module>15:57
mriedemTypeError: can't compare datetime.datetime to NoneType15:57
mriedemi guess because of the tuples15:57
bauzasAFAIR, you were even able to say something like None = 215:57
dansmithoh you mean compare datetime to none15:57
bauzasin py2 something15:57
mriedemthat's what these are15:57
mriedem>>> now1 = datetime.datetime.now()15:58
mriedem>>> now2 = datetime.datetime.now()15:58
mriedem>>> now1 < now215:58
mriedemTrue15:58
mriedem>>> now1 < None15:58
mriedemTraceback (most recent call last):15:58
mriedem  File "<stdin>", line 1, in <module>15:58
mriedemTypeError: can't compare datetime.datetime to NoneType15:58
dansmithyeah, ack15:58
efriedjroll: afaict those would map to original random passphrases of ~1200-2400 chars15:58
Roamer`mriedem, you'd get the same result if you tried to compare it to "whee" or to 1715:59
dansmithmriedem: we have to find something better than what you have15:59
* efried ==> mtg15:59
Roamer`ah, but you're commenting on the patch in the review request, sorry, yeah15:59
mriedemdansmith: i looked for some builtin sorting/filtering type function but didn't find anything obvious16:00
dansmithokay I got it16:01
mriedemsorted with a key function?16:01
mdboothlyarwood: https://review.opendev.org/#/c/584850/16:02
kashyapIs this a common convention?  "os-migrateLive"16:02
dansmithmriedem: https://pastebin.com/pHtM7iEf16:02
* kashyap is reviewing a spec on "no_performance_impact"16:03
kashyap(I'll ask there)16:03
mdboothlyarwood: I know that's abandoned, but any idea where that came from? The concept of having un-flushed data after LM switchover is disturbing.16:03
Roamer`mriedem, dansmith, maybe something like (yeah, I know, quite unwieldy) `dtnull = datetime.datetime.utcfromtimestamp(0)` and then `(u1 or dtnull, a1 or dtnull) < (u1 or dtnull, a2 or dtnull)`16:03
dansmithRoamer`: yep, or that16:03
Roamer`(with the correct variable names, of course, pfheh)16:03
dansmithmine makes anything with an updated_at win and short circuit before comparing the updated_ats which means you won't ever compare non-nones16:05
dansmither, Non-None to None16:05
*** jhesketh has quit IRC16:05
lyarwoodmdbooth: os-brick had a number of bugs around multiple devices getting grouped under the same mpath device blocking the final flush/removal on disconnect16:06
Roamer`dansmith, TBH, I hadn't seen yours before writing mine :) it took me a while to test it and copy it here, yours might be better and less unwieldy :)16:06
lyarwoodmdbooth: QEMU's own flush was enough to ensure everything made it to the backend before we forced anything in that case16:06
*** belmoreira has joined #openstack-nova16:06
mdboothlyarwood: That flush was happening on the source node, after switchover?16:07
lyarwoodmdbooth: the QEMU flush?16:07
lyarwoodmdbooth: no that's before we converge iirc16:07
lyarwoodor maybe during16:07
mdboothlyarwood: os-brick's flush on disconnect16:07
mdboothSo the bug was that os-brick thought it had data to flush, but it didn't really?16:08
lyarwoodmdbooth: that's after but it's just something you have to do before disconnecting mpath devices iirc, there's no data actually cached anywhere at that point16:08
mdboothRight. It would just be really bad if there really was data to flush at that point.16:08
lyarwoodyup16:09
dansmithjroll: didn't you add some ironic grouping construct so that one ironic can serve independent pools of nodes to dedicated nova computes?16:10
dansmithah, conductor groups16:10
*** lpetrut has quit IRC16:10
belmoreiradansmith jroll how that would work?16:15
dansmithbelmoreira: did you read the spec I just sent you? :)16:15
belmoreira:) maybe... which one? :)16:15
*** gyee has joined #openstack-nova16:15
dansmithbelmoreira: https://docs.openstack.org/ironic/latest/admin/conductor-groups.html16:15
dansmithoops,16:16
dansmithbelmoreira: https://specs.openstack.org/openstack/nova-specs/specs/stein/implemented/ironic-conductor-groups.html16:16
belmoreirathis is train16:16
dansmithwhat is?16:17
belmoreirasorry. This is available in Stein16:17
dansmithright16:17
*** jaosorior has quit IRC16:18
belmoreiraI forgot about it. sorry. This is great however means that we will need to scale nova-compute in the same way as the ironic-conductor16:21
*** jhesketh has joined #openstack-nova16:21
dansmithbelmoreira: well, you probably should be doing that anyway given you're hitting nova-compute scale issues when you make ironic big :)16:21
dansmithI understand the desire to make the RT loop in nova-compute more efficient, and I'm sure we can without breaking things, but it's a change to complex crufty code that only (positively) impacts the one oddball driver that uses multiple nodes,16:22
dansmithso I'm less enthusiastic about that, especially when grouping nova-computes gets you a lot more gain too,16:23
dansmithlike better fault tolerance, mapping your existing failure domains through to ironic, etc16:23
*** bbobrov has quit IRC16:24
belmoreiradansmith I understand that. That's why the initial plan was the sharding. And I wasn't aware (forgot) about the conductor groups16:24
belmoreirathis can make it16:24
dansmithcool16:24
*** bbobrov has joined #openstack-nova16:24
belmoreiraThanks dansmith16:31
belmoreiraI will try this and let you know how it goes16:31
*** tesseract has quit IRC16:32
jroll\o/16:32
*** ociuhandu has joined #openstack-nova16:33
*** ociuhandu has quit IRC16:33
kashyapmdbooth: On that "no_performance_impact" thing for post-copy / auto-converge spec, I have to bike-shed: the name is too generic for my taste.  I suggested being more clear WTF it is doing: "no_post_copy_and_auto_converge".16:33
dansmithbelmoreira: sweet that'd  be super16:34
*** ociuhandu has joined #openstack-nova16:34
*** spatel has joined #openstack-nova16:35
openstackgerritLee Yarwood proposed openstack/nova-specs master: WIP Virtual instance rescue with stable disk devices  https://review.opendev.org/69384916:38
*** TxGirlGeek has quit IRC16:39
*** do3meli has quit IRC16:39
*** ociuhandu has quit IRC16:40
*** damien_r has quit IRC16:40
*** TxGirlGeek has joined #openstack-nova16:41
*** ociuhandu has joined #openstack-nova16:47
*** boxiang has quit IRC16:48
*** zhubx has joined #openstack-nova16:48
dansmithmriedem: on lyarwood's spec to do stable disk rescue,16:48
dansmithdo you have strong feelings about bundling the can-now-rescue-BFV feature in it,16:48
dansmithvs. a relatively tiny follow-on effort?16:49
*** mkrai has quit IRC16:51
*** ociuhandu has quit IRC16:52
*** macz has joined #openstack-nova16:55
openstackgerritsean mooney proposed openstack/nova master: block rebuild when numa topology changed  https://review.opendev.org/68795716:56
openstackgerritsean mooney proposed openstack/nova master: Disable NUMATopologyFilter on rebuild  https://review.opendev.org/68986116:56
*** ivve has joined #openstack-nova16:57
*** rpittau is now known as rpittau|afk17:00
mriedemdansmith: i haven't read it but what's the difference? you mean making the api capability separate which is what, a microversion and a min compute service version compat check?17:02
dansmithmriedem: not even a min version check I think, a compute capability check17:02
*** dtantsur is now known as dtantsur|afk17:02
mriedemso the api checking the resource provider traits?17:02
dansmither, maybe I'm confused, I thought the api could check the compute capabilities, but maybe that's stupid17:03
dansmithso yeah maybe a version check and a cap flag in the compute manager17:03
*** pcaruana has quit IRC17:03
dansmithmriedem: point being, lyarwood had it in his now-abandoned first rev to remove that api check, and his current spec says no API impact, so wondering which way you want to go17:03
mriedemi've brought up that idea before about the api checking compute capabilities via traits, e.g. https://review.opendev.org/#/c/666604/17:03
mriedemwell there is obviously an api impact17:04
dansmithmriedem: ...only if he removes that check in his series again17:04
dansmithmriedem: and, you think that needs a microversion? just removing a check in some cases?17:04
*** ociuhandu has joined #openstack-nova17:05
mriedemit's a behavior change that a user opts into, i.e. i can rescue volume-backed servers on cloud A but not cloud B,17:05
dansmithif so, I understand why, but I don't have a good gut feeling for when we do and don't need to signal removing checks17:05
mriedemwhich is usually a microversion17:05
dansmiththey can opt in but still refuse with the same error17:06
dansmithit won't work for all hypervisors,17:06
dansmithand unless the api documents that you can't do that with volume-backed instances,17:06
mriedemsame idea for multiattach volume support, same for volume-backed rebuild (if someone finished that up)17:06
dansmithit seems like an obscure thing to me17:06
mriedemi think we've said in the past (and in our docs) that the docs are not a contract, the api behavior is17:06
mriedemand yes i know we fudge these lines all the time, e.g. gibi's changes to support moving servers with qos ports17:06
dansmithabsolutely, but I'm not sure this is discoverable as the reason, which is why it doesn't seem clear-cut to me17:07
mriedemand yes i realize not all hypervisors are going to support this so just because a cloud has the microversion that allows the request it doesn't mean it's going to work17:07
dansmithif you view the current error as "You can't rescue that instance for reasons I will not divulge" then all users right now just try it and accept the fate if they can't, without knowing why"17:08
mriedemas i said in lee's original change about microversions, i said i don't know what others would think so it's likely a ML discussion17:08
dansmithin which case allowing it on more things seems fine17:08
mriedemit's not a 409 though right? it's a 40017:08
mriedemyes it's a 400 today17:08
dansmiththe distinction doesn't mean anything to me17:08
dansmith409 is transient or something?17:09
mriedemif it were a 409 that'd be a bit fudgier to saying "you can' do this right now"17:09
dansmithokay, well, that's legit17:09
dansmithprobably *should* have been a 409 then, but alas17:09
mriedemi don't think there was a lot of thought put into response code nuance back in the day17:09
dansmithno, clearly not17:10
dansmithdoes 400 mean it's a fatal error specifically or just not imply retry-ability one way or the other? I would assume the latter as a catch-all17:10
openstackgerritsean mooney proposed openstack/nova-specs master: Add spec for VM-scoped SR-IOV NUMA affinity  https://review.opendev.org/68317417:11
*** ociuhandu has quit IRC17:11
dansmithanyway, if it's a microversion, then it probably makes sense to have that be a separate effort, maybe even before this stable rescue series to open up the possibility, even with no implementors, followed by his patch that actually allows it17:11
sean-k-mooneyefried: alex_xu ^ fixed the typos but that is otherwise the same17:11
mriedem400 means don't try this again17:12
dansmith"The client SHOULD NOT repeat the request without modifications."17:14
sean-k-mooneyor at least dont try this again until the resouce is in a different state17:14
dansmithsean-k-mooney: that's  not what the w3c says for 400, but does for others, so not sure you're right about that17:14
mriedemsean-k-mooney: you just described 40917:14
sean-k-mooney400 is bad request17:14
dansmithmriedem: that's an unfortunate implication for our default error code, but fair enough17:14
*** ociuhandu has joined #openstack-nova17:15
sean-k-mooneybut it may be a bad request for the give state fo the resouce but not in general17:15
dansmithsean-k-mooney: no17:15
sean-k-mooney409 conflcit is ment for concurrent modifcation17:15
dansmithsean-k-mooney: it says the reason is "malformed syntax" and "do NOT try this again"17:15
dansmithhttps://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html17:15
dansmiththe reason is not "the thing is in the wrong state for the operation you tried"17:15
sean-k-mooneyya fair17:16
sean-k-mooneyi normally use https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/40017:16
sean-k-mooneybut yes it also highlights "The client should not repeat this request without modification."17:16
dansmithnone of those reasons imply retry17:16
sean-k-mooneyi think the only time you woudl retry is for 5xx errors17:17
dansmithmriedem: so, spec because api change, and put the api change ahead of lyarwood's patch so we can actually test through it and hold merge until both are ready yeah?17:18
dansmithsean-k-mooney: no, some of the 4xx errors are mentioned as retryable with no modification17:18
dansmithsean-k-mooney: like 40817:18
sean-k-mooneyoh ok i normaly think of 503 https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/503 when i think retry17:18
mriedemdansmith: if you're saying single spec, multiple changes (one for compute, one for api), then yes agree17:19
sean-k-mooneyoh ya request timeouts17:19
dansmithmriedem: no, I mean another spec for the api change specifically just to avoid mucking up lyarwood's clean understandable spec with the spec to change the api behavior17:19
efriedoh sean-k-mooney, you lost your +1s from Vieri and Andriy :(17:20
sean-k-mooney:( your +2 would equal both there +1s right :)17:20
dansmithmriedem: I don't want to do extra work, but it seems like having lyarwood's stable rescue spec be readable in isolation without a bunch of api semantic changes would be beneficial, but maybe the api docs required for just a status change wouldn't be too big17:21
efriedsean-k-mooney: re+2, thanks for the update.17:21
sean-k-mooneyi still need to fix the typos you noted in the code but thats next on my list17:21
sean-k-mooneythen i need to adress dansmith's feedback on another series :)17:22
mriedemdansmith: are you saying the api change would just be changing the status code from 400 to 409 rather than a microversion?17:23
dansmithmriedem: no17:23
mriedemthen i don't think a secondary api-only spec is going to be that big of a deal17:24
dansmithcripes17:24
mriedem"depends on blueprint x. in x we make y work on the compute. this spec enables it in the api with a new microversion"17:24
mriedemjust do whatever y'all think is best17:25
mriedemi haven't read the current spec17:25
dansmithmriedem: I'm trying to say I think we should separate the api change into its own small spec to keep lyarwood's focused on the disk changes17:25
mriedemand i just said i was ok with that right?17:25
dansmithmriedem: in isolation without his change, we can make the api change to remove the hard block and say "as of microversion X you may or may not be able to rescue BFVs"17:25
dansmithmriedem: it wasn't clear to me17:26
dansmithbecause that's what I've been suggesting for ten minutes and you seemed to be disagreeing, so I was trying to explain17:26
mriedemso without any compute side changes, we should be able to rescue volume-backed servers today?17:26
* dansmith headdesks17:26
mriedemi'm fine with whatever17:27
mriedemlet's just stop talking about this17:27
dansmithI'm clearly the problem here, I'll just butt out17:27
*** ociuhandu has quit IRC17:32
*** ociuhandu has joined #openstack-nova17:33
*** ociuhandu has quit IRC17:33
*** ociuhandu has joined #openstack-nova17:34
artomdonnyd, heya, what the difference between your multi-numa and multi-numa-expanded flavors? Or where can I look to find out?17:34
artom(Also, we should talk about enabling hugepages at some point, but that can wait)17:35
sean-k-mooneyartom: the expanded one has 16G of ram instead of 817:35
artomsean-k-mooney, that's it?17:35
sean-k-mooneyyep17:35
sean-k-mooneyotherwise they both have the same about of disk and cpu17:35
artomAnd NUMA nodes17:35
sean-k-mooneybut have hw:numa_nodes=217:36
sean-k-mooneyand hw:cpu_socket=2 and hw:cpu_thread=217:36
artomSo the non-expanded one is fine17:36
*** ociuhandu has quit IRC17:36
sean-k-mooneywe shoudl try to use the non expanded ones if we can to save space but we might need the expanded one on the contoler17:37
artomOhh17:37
sean-k-mooneyya i didnt want to have to super optimise the job to get it to fit intially which is one of the reason we started with the expanded lables but it should not be needed longterm17:37
artomLet's try with non-expanded I guess first17:38
*** ociuhandu has joined #openstack-nova17:38
artomAnd if we hit oom or something we can -expand :D17:38
sean-k-mooneyyep i think i was trying to allocte a low amout of hugepage in my job17:39
sean-k-mooneywell in the orginal version17:39
sean-k-mooneyi think we are generaly around the 5-6 G mark on used memroy on the contoler so you dont want to exceen more then about 1G of hugepages total17:40
artomOhh, you actually set hugepages in Ansible17:40
sean-k-mooneyyes here https://review.opendev.org/#/c/679656/13/playbooks/nfv/nfv.yaml17:40
artomSo I should probably do that instead of https://review.opendev.org/#/c/693690/17:40
sean-k-mooneybut its too much currently for the contoler17:41
artomAlthough that skip could still be useful in other envs17:41
*** david-lyle is now known as dklyle17:41
sean-k-mooneyya i would basically do this17:42
sean-k-mooney hosts: controller17:42
sean-k-mooney  tasks:17:42
sean-k-mooney    - name: allocate hugepages17:42
sean-k-mooney      shell: echo 512 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages17:42
sean-k-mooney      become: yes17:42
sean-k-mooney- hosts: subnode17:42
sean-k-mooney  tasks:17:42
sean-k-mooney    - name: allocate hugepages17:42
sean-k-mooney      shell: echo 2048 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages17:42
sean-k-mooney      become: yes17:42
*** ociuhandu has quit IRC17:42
sean-k-mooneythat will alocate 1G on the contoler and 4 on the compute of 2MB hugepages17:43
sean-k-mooneythat should be enough17:43
artomAck17:43
*** ociuhandu has joined #openstack-nova17:45
*** spatel has quit IRC17:45
*** henriqueof has quit IRC17:47
*** ociuhandu has quit IRC17:54
artomsean-k-mooney, I think the problem with using https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-copy-build-sshkey is that it's 1-user only17:55
artomIOW, 'tempest' can ssh to 'tempest'17:55
artomWe need 'tempest' to SSH to 'stack' or some other account that has limitless sudo powers17:56
*** igordc has joined #openstack-nova17:59
sean-k-mooneyartom: the same key is already copied to root and stack17:59
sean-k-mooneyroot is done in the pre job17:59
sean-k-mooneystack in the muntinode playbook17:59
sean-k-mooneyso if we use it for tempest all 3 users will have the same private and public key18:00
sean-k-mooneyyou wouldnt want to do that in production but it should be fine for ci18:00
*** bbowen has quit IRC18:01
*** bbowen has joined #openstack-nova18:01
artom*facepalm*18:01
artomI see18:01
artomThe devstack orchestrate-devstack role will put the pubkey in stack's authorized_keys for us18:01
artomAnd we can put the private key in tempest's .ssh18:02
artomIn theory, anyways18:02
* artom is sure something will explode18:02
*** bbowen_ has joined #openstack-nova18:05
artom... do we have access to another project's roles? Not sure...18:05
donnydartom: the expanded flavors have more memory18:06
*** bbowen has quit IRC18:08
*** bbowen_ has quit IRC18:11
sean-k-mooneyartom: am we can get acess to them yes there are two ways to do i but your job inherits form the tempest job yes18:11
sean-k-mooneyif so then it will do it already18:11
sean-k-mooneyyou you can just invoke the copy-build-key role twice once for stack and once for tempest18:12
artomhttps://review.opendev.org/#/c/691062/57 let's see how badly stuff explodes18:28
*** ralonsoh has quit IRC18:34
sean-k-mooneyartom: you do not need the ssh keys in that anymore18:41
sean-k-mooneybut i think it should be ok18:41
sean-k-mooneyits runing in a sperate pool so it should be pretty fast18:41
artomsean-k-mooney, something borked, it's retrying now18:42
artomHeh, oh yeah, I should get rid of the keys18:42
artomI'll wait for it to work first18:42
sean-k-mooneythe console for the job is here http://zuul.openstack.org/stream/11de9dc2d4fd4f7d931eedeea276af3e?logfile=console.log18:43
sean-k-mooneylooks like tis doing the normal multinode setup18:43
artomThat's the second attempt18:44
sean-k-mooneyyep18:44
sean-k-mooneyill leave it open on one of my monitors and see what hapens18:44
sean-k-mooneyhaha18:45
sean-k-mooneyyou have a typo18:45
sean-k-mooneyartom: http://paste.openstack.org/show/785994/18:46
sean-k-mooneyyou did not include teh whitebox-compute role in the commit18:47
sean-k-mooneyor you forgot to remove that line18:47
artom*facepalm*18:49
artom'git commit -a' isn't magic18:49
sean-k-mooneyit speciricly ignore untracted files18:49
*** henriqueof has joined #openstack-nova18:50
artomI know18:52
sean-k-mooneygood becaue i apparently cant spell specifcally and im not sure you woudl figure it out if you didnt :P18:52
sean-k-mooneyi need to try enabling spell check for irc again but the last time i tried that it was kind of depressing18:55
artom*shrug* I forgive you :)18:57
artomYou have other uses besides spelling ;)18:57
sean-k-mooneywell if you want a terible way to generate passwords just ask me to spell somthing18:58
artomlulz18:59
sean-k-mooneyso looking at the job you are trying to allocate 8G of hugepages on all nodes19:04
sean-k-mooneythat will not work with the non expanded flavors19:05
sean-k-mooney*lables19:05
artomderp19:07
*** ociuhandu has joined #openstack-nova19:08
*** ociuhandu has quit IRC19:12
*** jaosorior has joined #openstack-nova19:39
mriedemcool, the instance_actions_events table has a detail TEXT column that is never used, and when creating the InstanceActionEvent object we pass the exc_val but it gets passed down to the DB API in a 'message' kwarg rather than 'details' so that column is never populated19:46
mriedemso as a non-admin user you can see a particular event passed or failed but not why, e.g. NoValidHost19:48
mriedemand only admins can see the traceback (like a fault) by default19:48
sean-k-mooneyhum ok that sound like the behavior we want. are you going to reuse the colume for something19:49
sean-k-mooneysuch as storging the traceback?19:50
mriedemthe traceback is stored in the traceback column19:50
*** lbragstad has quit IRC19:50
mriedemthe exc_val is not stored at all19:50
mriedemb/c the object is using the wrong column name19:50
*** lbragstad has joined #openstack-nova19:51
sean-k-mooneysure but i was wondering if you planned to chagne that19:51
mriedemi think it would be useful to at least expose to the  non-admin owner of the server the exception type, like we do for a fault message19:52
sean-k-mooneyya maybe. as long as its just the type and not the body of the excption its proably safe. although im not sure all operator would be ok with that19:53
sean-k-mooneye.g. it might affect there sla depening on what the falut was19:53
mriedemif the instance goes to ERROR status we're already showing the exception type in the fault message19:54
mriedemif not the formatted message19:54
sean-k-mooneyya that is ture19:54
sean-k-mooneyi normally am logged in as root but we get the message in that case at least19:54
sean-k-mooney* admin19:54
sean-k-mooneyso im not sure what a non admin sees19:55
*** brinzhang_ has joined #openstack-nova19:57
*** brinzhang has quit IRC20:00
sean-k-mooneyartom: :) looks like it worked https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_3ae/691062/59/check/whitebox-multinode-devstack/3ae547c/testr_results.html.gz20:10
sean-k-mooneywe still need to adress the skips20:11
sean-k-mooneybut the ssh keys worked correctly and it looks liek the hugepages allocation did not cause memory issues20:12
*** jaosorior has quit IRC20:16
*** panda has quit IRC20:18
*** lbragstad has quit IRC20:19
*** lbragstad has joined #openstack-nova20:20
*** munimeha1 has joined #openstack-nova20:20
*** panda has joined #openstack-nova20:21
mriedemdansmith: your suggestions here to use the tuples for comparing times doesn't work in all cases https://review.opendev.org/#/c/636224/50/nova/compute/api.py@495620:23
mriedemand i don't really want to bake a bunch of conditional logic into unreadable tuples20:23
dansmithaight, well, do what you want20:25
dansmithinlining all that in a closure seems bad,20:25
*** damien_r has joined #openstack-nova20:25
dansmithespecially for testability, which has clearly manifested itself already20:25
dansmiththat could be a20:26
dansmithdef newer_than(obj)20:26
dansmithmethod on NovaObject and be usable for other things, as well as easier to test20:26
dansmithis there not some way we could do this without times?20:27
dansmithbecause that's kinda smelly in general,20:27
sean-k-mooneyi was going to suggest a before or after method or something but ya the tuple way would be nice if it was not for the value error20:27
dansmithassuming they're all time-synced, you don't have any delays, delayed transactions, etc20:27
sean-k-mooney*type error20:27
sean-k-mooneyoh this is in get_newer_job already20:29
sean-k-mooneyyou could define a lessthan operator on the migration object20:29
sean-k-mooneyalthough i guess that would have the same proably with None20:29
dansmithno, it wouldn't20:29
dansmithbut also, less-than can mean lots of things other than time20:30
sean-k-mooneyfor a migration object20:30
sean-k-mooneyand ya i could20:30
mriedemwe could just say f it and do like get_all and filter out by uuid after we've already processed one20:30
mriedemthis https://github.com/openstack/nova/blob/master/nova/compute/api.py#L288320:31
dansmithmriedem: yeah20:31
mriedemthe migration from the first cell processed in the results is always the one that is returned,20:32
mriedemwhich could be wrong, but i don't really care at this point20:32
dansmithdoes it matter/20:32
dansmithyou just want to not return dupes right?20:32
mriedemyes, but also figure that since we can know which is "newer" we should return that,20:33
mriedembecause the migration in the target cell is the one that's going to be getting updated as the resize progresses20:33
mriedemincluding when it goes to 'finished' status and the server is in VERIFY_RESIZE status20:33
*** damien_r has quit IRC20:34
sean-k-mooney mriedem ya i think the nestest makes sense20:34
mriedemwhich is kind of important since i think i have functional tests later in the series asserting the migratoins api returns 1 and it's the correct status20:34
dansmithmriedem: once you create the new one, couldn't you set the hidden flag on the old one or something/20:34
mriedemsure, that's the alternative mentioned in the code and the commit message20:34
mriedembut it has implications for the api because that hidden field has never been set before20:34
dansmiththe *newest* only makes sense if we never update the old migration20:35
dansmithmriedem: I don't understand what the problem with that is, as I said.. api ignores migrations with the hidden field already ...20:35
mriedemi don't think we do update the source cell migration unless we revert the resize20:35
dansmithwe just leave it pending forever?20:36
mriedemno, when you confirm the resize in the target cell we hard destroy the source cell instance which will hard destroy it's related migratoin in the source cell db,20:36
mriedemsame (in reverse) for revert20:36
dansmithwhat I meant is.. we leave it in the half-finished state until that time comes20:37
mriedemotherwise you can never resize back to that cell b/c of the duplicate instance uuid20:37
mriedempretty sure yeah20:37
sean-k-mooneyso in that case cant we have teh db return the latest version based on update  or created time20:37
mriedemsean-k-mooney: different dbs20:37
mriedemso no20:37
sean-k-mooneybut we only need to look at the one of the dbs if we only want the latest one right? i gues we would have too look at all of them during a migration20:38
*** henriqueof1 has joined #openstack-nova20:38
sean-k-mooneyya ok20:38
dansmithsean-k-mooney: maybe you should read the patch we're discussing?20:38
sean-k-mooneyi am but im also about to head off for the night20:38
sean-k-mooneythe reason i changed my mind half way true was the Verify_resize comment20:39
*** henriqueof has quit IRC20:41
mriedemso it's not just the active migration that is a duplicate either, it's all migrations linked to the instance - those get copied into the target cell db early in TargetDBSetupTask, because once we flip the instance mapping that's where the api is going to go to get migrations for that server,20:43
mriedemso if we started setting the hidden field on migrations, we'd have to do it on all of them until we're sure where the server is going to ultimately be20:43
efriedIf I initiate an operation as an admin user, does the context have is_admin set, or does that only happen when nova generates an admin context explicitly?20:44
mriedemis_admin is set20:44
mriedemthe difference is one has a token/user/project and the one nova creates for some periodics and stuff doesnt20:45
mriedemhttps://github.com/openstack/nova/blob/master/nova/context.py#L141 https://github.com/openstack/nova/blob/master/nova/policy.py#L197 https://github.com/openstack/nova/blob/master/nova/policies/base.py#L2420:46
*** slaweq has quit IRC20:47
sean-k-mooneymriedem: without your patch do we ignore the cell maping and simple cast to all cells by the way or just the cell where the instnce is20:48
mriedemfor what api?20:48
mriedemGET /os-migrations iterates all cells20:49
efriedmriedem: perhaps the better question is: can I create an instance as an admin user?20:49
sean-k-mooneybased on what you jsut said i would not need too right since we would be copying all the migrations to the new cell20:49
artomsean-k-mooney, yeah I am le much flabbergast20:50
sean-k-mooneyefried: an admin user is just a normal user with extra privladtes so it can have a project and a quota20:51
sean-k-mooneyefried: so the admin use that gets set up with devstack for example can20:52
efriedor maybe I should just say what I'm actually trying to do:20:52
efriedWhile generating libvirt xml, I need a context. The way the code is set up right now, sometimes the context comes from the user and sometimes it comes from get_admin_context(). I want to detect (and reject) the latter.20:52
efriedSo I think you're saying that I should be able to tell based on whether project_id is set?20:52
sean-k-mooneywhy do you want to reject the later20:53
sean-k-mooneyadmin users shoudl be able to create vms with vtpm20:53
sean-k-mooneyjust not on behalf of another user/project20:53
mriedemsean-k-mooney: we need to copy the migrations to the target cell db because, like i said above, if the user confirms the resize there we hard-destroy the instance and it's old migrations in the source cell db20:53
sean-k-mooneymriedem: yes im just wondering of we can optimese the /os_migration api to only look at the cell where the vm is currently20:54
mriedemefried: a context created by nova.context.get_admin_context won't have a project_id, correct20:54
sean-k-mooneythat info should be in the instnace maping in the api db right?20:54
efriedsean-k-mooney: Like we were talking about yesterday, I need to be able to bounce on code paths where $not_the_owner_of_the_instance is trying to generate xml with a vtpm secret in it.20:54
mriedemsean-k-mooney: what info?20:55
efriedI guess I could just let the key retrieval fail.20:55
sean-k-mooneymriedem: which cell the vm is in20:55
dansmithmriedem: sean-k-mooney has not read the patch and doesn't realize we're listing all migrations, not just for a specific instance20:55
mriedemcorrect20:55
sean-k-mooneydansmith: yes i have20:55
sean-k-mooneyoh i though you were listing all migrtion for the instace20:56
mriedemsean-k-mooney: but the instance mapping saying the instance is in cell X doesn't mean the migration records from cell Y are in cell X unless we copy them there20:56
sean-k-mooneynot all migrtions in general20:56
mriedemGET /os-migrations is not instance specific20:56
mriedemit's like GET /servers20:56
mriedemGET /servers/{server_id}/migrations is instance and cell specific20:56
sean-k-mooneyright sorry i was confusing it with the iserver one20:56
*** eharney has quit IRC20:57
dansmithefried: I would prefer you try and fail to get/list the key vs. inspect the context and try to detect that it isn't "real"20:57
sean-k-mooneyefried: you dont need to reject it you can let barbican do that as it will reject your token when you try to get the secret20:57
efrieddansmith: ack. I'm trying to verify whether there are situations where I want to generate the XML but am not trying to boot the instance.20:57
efriedIn such cases it would be okay to inject the secret_uuid but not define the secret itself to libvirt20:58
dansmithefried: we pass pieces of xml to libvirt in operations like attach volume, but not the whole thing I think, so probably not relevant20:58
efriedrelevant in that I would need to make sure I don't hit the key manager in such operations.20:59
dansmithefried: but still, I imagine you can't define the xml for the guest referencing a secret that you've since deleted (which you said you were going to do) right?20:59
efriedI can, as long as I don't try to boot the thing.20:59
dansmithokay I would have expected that to fail at define time21:00
efriedthe guest XML only contains the UUID, which (if the instance was ever booted) I've stored in its sysmeta.21:00
sean-k-mooneyya i dont think you will hit that in novas usage21:00
efriedI may be misunderstanding, but I think _get_guest_xml just returns an etree, doesn't actually try to libvirtify it.21:00
dansmithefried: sure21:00
dansmithefried: I guess I thought you meant generate the xml as in... update the definition in libvirt21:00
sean-k-mooneythe only case i am aware off was the requet to "create a server but not boot it" but we dont supprot that currently21:01
efriedI possibly need to rethink the flow here...21:01
efriedbecause I *actually* only want to define the secret to libvirt on boot-ish operations anyway. And then delete it right after.21:01
dansmithright so in driver.spawn()21:02
mriedemdansmith: if moving this get_newer_obj method to NovaObjectBase makes it less gross for you i can do that21:02
efriedalso cold boot?21:02
dansmithefried: cold boot? :)21:02
dansmithefried: start/21:02
efriedpower_on21:02
mriedemi thought about it but didn't since this was origially much more special case to this api method21:02
sean-k-mooneycold boot or start call power_on which for libvirt calls hard_reboot which calls spawn21:02
dansmithmriedem: honestly I just hate the way that logic is laid out, probably irrationally, but before even looking for test gaps I knew it would be annoying to validate21:03
dansmithmriedem: the fact that there was one just makes me scared, but I'm clearly the only one, so just do whatever you want, I say21:03
mriedemi expected something like this to already exist but didn't find anything21:04
mriedemif i'm left to do whatever i want, i'm going to address mdbooth's comments and keep the series going. if there are things around this to be done later, i say let's do them later.21:05
dansmithaight21:05
sean-k-mooneyefried: for libivrt you might want to create and remove the securte here https://github.com/openstack/nova/blob/7aa88029bbf6311033457c32801963da01e88ecb/nova/virt/libvirt/driver.py#L618821:06
*** aluria has quit IRC21:07
*** dosaboy has quit IRC21:07
*** ganso has quit IRC21:07
sean-k-mooneyin libvirt that is21:07
sean-k-mooneyyou would have to add the secret to the xml before21:07
sean-k-mooneyanyway night o/21:09
*** ociuhandu has joined #openstack-nova21:09
*** ganso has joined #openstack-nova21:10
*** nweinber has quit IRC21:10
*** dosaboy has joined #openstack-nova21:13
*** ociuhandu has quit IRC21:13
*** ociuhandu has joined #openstack-nova21:15
openstackgerritMatt Riedemann proposed openstack/nova master: Filter duplicates from compute API get_migrations_sorted()  https://review.opendev.org/63622421:17
openstackgerritMatt Riedemann proposed openstack/nova master: Start functional testing for cross-cell resize  https://review.opendev.org/63625321:17
openstackgerritMatt Riedemann proposed openstack/nova master: Handle target host cross-cell cold migration in conductor  https://review.opendev.org/64259121:17
openstackgerritMatt Riedemann proposed openstack/nova master: Validate image/create during cross-cell resize functional testing  https://review.opendev.org/64259221:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add zones wrinkle to TestMultiCellMigrate  https://review.opendev.org/64345021:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add negative test for cross-cell finish_resize failing  https://review.opendev.org/64345121:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add negative test for prep_snapshot_based_resize_at_source failing  https://review.opendev.org/66901321:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize_at_source compute method  https://review.opendev.org/63705821:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add ConfirmResizeTask  https://review.opendev.org/63707021:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add confirm_snapshot_based_resize conductor RPC method  https://review.opendev.org/63707521:17
openstackgerritMatt Riedemann proposed openstack/nova master: Confirm cross-cell resize from the API  https://review.opendev.org/63731621:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize_at_dest compute method  https://review.opendev.org/63763021:17
openstackgerritMatt Riedemann proposed openstack/nova master: Deal with cross-cell resize in _remove_deleted_instances_allocations  https://review.opendev.org/63945321:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add finish_revert_snapshot_based_resize_at_source compute method  https://review.opendev.org/63764721:17
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Add RevertResizeTask  https://review.opendev.org/63804621:17
openstackgerritMatt Riedemann proposed openstack/nova master: Add revert_snapshot_based_resize conductor RPC method  https://review.opendev.org/63804721:17
openstackgerritMatt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API  https://review.opendev.org/63804821:17
openstackgerritMatt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server  https://review.opendev.org/63826821:17
*** rcernin has joined #openstack-nova21:17
*** ociuhandu has quit IRC21:19
*** ociuhandu has joined #openstack-nova21:23
*** dpawlik has joined #openstack-nova21:24
*** ociuhandu has quit IRC21:28
*** dpawlik has quit IRC21:28
*** xek_ has quit IRC21:31
*** ociuhandu has joined #openstack-nova21:44
*** ociuhandu has quit IRC21:49
*** eharney has joined #openstack-nova21:58
*** slaweq has joined #openstack-nova22:11
*** mriedem has quit IRC22:12
*** slaweq has quit IRC22:15
*** dviroel has quit IRC22:15
*** ociuhandu has joined #openstack-nova22:17
*** ociuhandu has quit IRC22:22
*** brinzhang has joined #openstack-nova22:26
*** brinzhang_ has quit IRC22:29
*** eharney has quit IRC22:31
*** nweinber has joined #openstack-nova22:31
*** adriant has joined #openstack-nova22:41
*** mdbooth has quit IRC22:42
*** slaweq has joined #openstack-nova22:46
*** ociuhandu has joined #openstack-nova22:51
*** ociuhandu has quit IRC22:58
*** igordc has quit IRC23:06
*** tkajinam has joined #openstack-nova23:08
*** nweinber has quit IRC23:18
*** mmethot has quit IRC23:21
*** mmethot has joined #openstack-nova23:22
*** awalende has joined #openstack-nova23:23
*** ivve has quit IRC23:25
*** awalende has quit IRC23:28
*** ociuhandu has joined #openstack-nova23:29
*** ociuhandu has quit IRC23:30
*** ociuhandu has joined #openstack-nova23:30
*** igordc has joined #openstack-nova23:31
*** ociuhandu has quit IRC23:36
*** maciejjozefczyk has quit IRC23:42
*** klindgren_ has quit IRC23:47
*** klindgren has joined #openstack-nova23:47
*** munimeha1 has quit IRC23:49
*** klindgren has quit IRC23:50
*** klindgren has joined #openstack-nova23:51
*** brinzhang_ has joined #openstack-nova23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!