Friday, 2020-03-20

« Thursday, 2020-03-19 Index Saturday, 2020-03-21 »
*** gyee has quit IRC00:02
openstackgerritMerged openstack/nova stable/train: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph  https://review.opendev.org/71383600:08
*** tetsuro has joined #openstack-nova00:25
*** lbragstad_ has joined #openstack-nova00:28
*** tetsuro has quit IRC00:31
*** tetsuro has joined #openstack-nova00:36
*** spatel has joined #openstack-nova00:38
*** ociuhandu has joined #openstack-nova00:43
*** tetsuro has quit IRC00:44
*** macz_ has joined #openstack-nova00:45
*** ociuhandu has quit IRC00:48
*** macz_ has quit IRC00:50
*** tosky has quit IRC00:54
*** nicolasbock has quit IRC01:07
*** spatel has quit IRC01:10
*** mlavalle has quit IRC01:10
*** tetsuro has joined #openstack-nova01:11
*** Liang__ has joined #openstack-nova01:13
*** tetsuro has quit IRC01:14
*** sapd1 has joined #openstack-nova01:46
*** sapd1 has quit IRC01:48
*** sapd1 has joined #openstack-nova01:48
*** macz_ has joined #openstack-nova01:52
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147801:55
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492901:55
*** macz_ has quit IRC01:56
*** tetsuro has joined #openstack-nova02:01
*** tetsuro has quit IRC02:01
*** rcernin has quit IRC02:03
*** rcernin has joined #openstack-nova02:03
*** kaisers_ has quit IRC02:09
*** zhanglong has joined #openstack-nova02:21
*** lbragstad_ has quit IRC02:56
*** mkrai has joined #openstack-nova03:24
*** psachin has joined #openstack-nova03:26
*** macz_ has joined #openstack-nova03:40
*** macz_ has quit IRC03:44
*** macz_ has joined #openstack-nova04:11
*** ratailor has joined #openstack-nova04:28
*** macz_ has quit IRC04:34
openstackgerritArthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/68277205:00
*** links has joined #openstack-nova05:14
*** klindgren_ has joined #openstack-nova05:20
*** klindgren has quit IRC05:20
*** macz_ has joined #openstack-nova05:21
*** vishalmanchanda has joined #openstack-nova05:21
*** macz_ has quit IRC05:25
*** rcernin is now known as rcernin|brb05:30
*** evrardjp has quit IRC05:36
*** evrardjp has joined #openstack-nova05:36
*** udesale has joined #openstack-nova05:38
*** ygk_12345 has joined #openstack-nova05:47
*** dklyle has quit IRC06:15
*** yedongcan has joined #openstack-nova06:23
*** tetsuro has joined #openstack-nova06:28
*** tetsuro has quit IRC06:35
*** zhanglong has quit IRC06:37
ygk_12345can someone look into this please   https://bugs.launchpad.net/nova/+bug/186820306:49
openstackLaunchpad bug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New]06:49
*** zhanglong has joined #openstack-nova06:53
openstackgerritArthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/68277206:58
*** zhanglong has quit IRC06:58
*** ociuhandu has joined #openstack-nova07:01
*** ociuhandu has quit IRC07:05
*** macz_ has joined #openstack-nova07:09
*** zhanglong has joined #openstack-nova07:09
*** ociuhandu has joined #openstack-nova07:10
*** macz_ has quit IRC07:13
*** d34dh0r53 has quit IRC07:14
*** d34dh0r53 has joined #openstack-nova07:14
*** zhanglong has quit IRC07:15
*** ociuhandu has quit IRC07:15
*** zhanglong has joined #openstack-nova07:16
*** mkrai has quit IRC07:20
*** mkrai has joined #openstack-nova07:20
*** zhanglong has quit IRC07:21
*** dpawlik has joined #openstack-nova07:21
*** zhanglong has joined #openstack-nova07:23
*** zhanglong has quit IRC07:27
*** zhanglong has joined #openstack-nova07:29
*** psachin has quit IRC07:41
*** sapd1_x has joined #openstack-nova07:42
*** ociuhandu has joined #openstack-nova07:57
*** rpittau|afk is now known as rpittau07:57
*** ociuhandu has quit IRC07:59
*** maciejjozefczyk has joined #openstack-nova08:00
*** slaweq has joined #openstack-nova08:00
openstackgerritBrin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/68277208:01
openstackgerritBrin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/68277208:07
*** ccamacho has quit IRC08:08
*** tesseract has joined #openstack-nova08:09
brinzhang_lyarwood:https://review.opendev.org/#/c/682772/ fixed your comment, and add the test case you asked, pls review again if you are free ^^08:14
*** sapd1_x has quit IRC08:17
*** zhanglong has quit IRC08:21
*** brinzhang has joined #openstack-nova08:22
*** brinzhang_ has quit IRC08:25
*** zhanglong has joined #openstack-nova08:26
*** macz_ has joined #openstack-nova08:31
*** jawad_axd has joined #openstack-nova08:34
*** macz_ has quit IRC08:36
*** brinzhang_ has joined #openstack-nova08:37
*** brinzhang has quit IRC08:41
*** ccamacho has joined #openstack-nova08:41
*** ccamacho has quit IRC08:45
*** ralonsoh has joined #openstack-nova08:46
*** ratailor has quit IRC08:48
*** dtantsur|afk is now known as dtantsur08:50
*** ratailor has joined #openstack-nova08:50
*** tosky has joined #openstack-nova08:51
*** ociuhandu has joined #openstack-nova08:56
*** dtantsur is now known as dtantsur|learnin08:58
*** ociuhandu has quit IRC09:01
openstackgerritKevin Zhao proposed openstack/nova master: Add default cpu model for aarch64  https://review.opendev.org/70949409:06
*** brinzhang has joined #openstack-nova09:12
*** brinzhang has quit IRC09:14
*** brinzhang has joined #openstack-nova09:14
*** brinzhang_ has quit IRC09:15
nightmare_unrealhey guys, how can I run tests for python-openstackclient09:20
nightmare_unreali wanted to test it locally before submitting a patch09:20
openstackgerritLee Yarwood proposed openstack/nova master: nova-live-migration: Only stop n-cpu and q-agt during evacuation testing  https://review.opendev.org/71405709:24
lyarwoodnightmare_unreal: tox -e py36 ; tox -e pep809:24
lyarwoodsean-k-mooney: ^ btw, I missed that we are running c-vol and g-api on the subnodes for some reason09:24
* lyarwood finds out how to remove them as that's just nuts09:24
lyarwoodI was sure we removed these recently09:24
openstackgerritMerged openstack/nova master: Support unshelve with qos ports  https://review.opendev.org/70475909:29
*** xek has joined #openstack-nova09:29
nightmare_unrealthanks lyarwood  but in which directory to run ? inside devstack or inside openstackclient09:30
ygk_12345hi all09:31
ygk_12345can someone look into this once ? https://bugs.launchpad.net/nova/+bug/186820309:31
openstackygk_12345: Error: Could not gather data from Launchpad for bug #1868203 (https://launchpad.net/bugs/1868203). The error has been logged09:31
*** jraju__ has joined #openstack-nova09:33
*** Liang__ has quit IRC09:34
*** links has quit IRC09:34
lyarwoodnightmare_unreal: openstackclient sorry09:35
nightmare_unrealthanks09:35
nightmare_unrealthanks lyarwood09:41
*** brinzhang_ has joined #openstack-nova09:46
*** brinzhang has quit IRC09:48
*** zhanglong has quit IRC09:53
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147809:57
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492909:57
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147810:02
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492910:02
*** bhagyashris_ has joined #openstack-nova10:03
*** dtantsur|learnin is now known as dtantsur10:03
*** brinzhang has joined #openstack-nova10:03
*** bhagyashris_ has quit IRC10:03
*** lpetrut has joined #openstack-nova10:03
*** zhanglong has joined #openstack-nova10:04
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147810:06
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492910:06
*** brinzhang_ has quit IRC10:06
ygk_12345can someone look into this  https://bugs.launchpad.net/nova/+bug/186820310:06
openstackLaunchpad bug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New]10:06
openstackgerritMerged openstack/nova master: Add test coverage of existing flavor_access policies  https://review.opendev.org/71355610:07
*** brinzhang has quit IRC10:07
nightmare_unrealhow to add my changes to doc or how to build doc for openstackclient ?10:08
lyarwoodygk_12345: can you attach the n-cpu log to the bug? Looks like you're trying to launch qemu-system-ppc64 instances and failing?10:09
lyarwoodnightmare_unreal: tox -e docs10:09
ygk_12345lyarwood it is openstack ansible rocky. where can I find the n-cpu log ?10:10
lyarwoodnightmare_unreal: review tox.ini and https://docs.openstack.org/python-openstackclient/latest/contributor/developing.html10:10
nightmare_unrealthanks lyarwood , but how to add my doc code. for eg. i made a new flag10:10
lyarwoodnightmare_unreal: look for changes under docs (using tig and/or tig blame) for examples.10:10
jkulikCan someone close https://bugs.launchpad.net/nova/+bug/1855776 it should have been fixed by https://review.opendev.org/#/c/710214/ as it seems to be a duplicate of https://bugs.launchpad.net/nova/+bug/186504010:11
openstackLaunchpad bug 1855776 in OpenStack Compute (nova) "Aggregate ID validation" [Undecided,In progress] - Assigned to Mykola Yakovliev (vegasq)10:11
openstackLaunchpad bug 1865040 in OpenStack Compute (nova) "Able to show update and delete aggregate with invalid id" [Undecided,Fix released] - Assigned to Johannes Kulik (jkulik)10:11
nightmare_unrealthanks10:11
openstackgerritLuyao Zhong proposed openstack/nova stable/train: bug-fix: Reject live migration with vpmem  https://review.opendev.org/71406410:11
lyarwoodygk_12345: sorry no idea10:11
*** tetsuro has joined #openstack-nova10:12
lyarwoodjkulik: if you can still reproduce against master feel free to reuse https://bugs.launchpad.net/nova/+bug/1855776 and submit a fix.10:14
openstackLaunchpad bug 1855776 in OpenStack Compute (nova) "Aggregate ID validation" [Undecided,In progress] - Assigned to Mykola Yakovliev (vegasq)10:14
jkuliklyarwood, it looks exactly like the problem in the second bug, which got closed by merging. I just want that bug closed, too :)10:15
*** tetsuro has quit IRC10:16
gibiygk_12345: regarding bug #1868203 I feels like a QEMU problem on your compute. If QEMU fails to start then nova will not know the available resources on that compute10:17
openstackbug 1868203 in OpenStack Compute (nova) "nova-compute error" [Undecided,New] https://launchpad.net/bugs/186820310:17
gibiygk_12345: you should reach out the QEMU devs10:18
*** macz_ has joined #openstack-nova10:20
gibijkulik: marked the bug as duplicate of 186504010:20
gibijkulik: thanks for noticing10:20
jkulikgibi, thanks10:20
*** macz_ has quit IRC10:24
ygk_12345@gibi the libvirtd service is in error state10:25
lyarwoodygk_12345: if libvirtd isn't starting then that's a bug for the libvirt folks10:25
gibiygk_12345: based on the libvird service logs you added to the bug report the libvirtd is in error most probably due to QEMU cannot be started10:26
ygk_12345@gibi and what is that placement error I mentioned ? is it related to this libvirtd  ?10:27
gibiygk_12345: could be related yes10:27
gibiygk_12345: is reported missing RP has the same uuid as the faulty compute node uuid?10:28
gibisorry I have to jump on a call very soon10:28
kashyapygk_12345: How is libvirtd failing to start?  /me looks at the scroll10:39
ygk_12345kashyap http://paste.openstack.org/show/790930/10:39
ygk_12345@las10:41
ygk_12345kashyap it is ubuntu 1810:41
kashyapygk_12345: What libvirt and QEMU versions?10:42
kashyapThis "relocation error" looks like some dependency problem10:42
ygk_12345kashyap how to check ?10:42
kashyapygk_12345: To get versions?  Please use 'apt' command: `apt list libvirt` (or whatever Ubuntu calls it)10:44
ygk_12345kashyap libvirtd (libvirt) 5.4.010:44
kashyapAnd also QEMU version10:44
ygk_12345root@b2a-1:~# apt list qemu10:44
ygk_12345Listing... Done10:44
ygk_12345qemu/bionic-updates,now 1:4.0+dfsg-0ubuntu9.3~cloud0 amd64 [installed]10:44
stephenfinartom: When you're about, I've replied on https://review.opendev.org/#/c/631053/10:47
stephenfinartom: let me know if you want me to rework the commit message to make my point clearer10:48
*** zhanglong has quit IRC10:48
nightmare_unrealhello if i want to add something to doc for e.g. in quota docs here : https://docs.openstack.org/python-openstackclient/pike/cli/command-objects/quota.html10:50
nightmare_unrealhow can i do it10:50
nightmare_unrealI want to add doc for --force flag10:50
nightmare_unreali have just submitted a patch for it10:50
nightmare_unrealbut don't know how to add doc for it10:50
*** zhanglong has joined #openstack-nova10:50
lyarwoodnightmare_unreal: edit doc/source/cli/command-objects/quota.rst10:53
kashyapygk_12345: Your installation on Ubuntu of QEMU/libvirt is kaput with broken libraries.  How did you even install it?10:53
lyarwoodnightmare_unreal: FWIW this channel isn't for python-openstackclient btw, just being polite and trying to help.10:53
lyarwoodnightmare_unreal: #openstack might be a better place to ask these questions.10:53
ygk_12345kashyap it came with openstack ansible rocky version10:53
kashyapygk_12345: Also, your output is truncated, please post this in the bug: systemctl status libvirtd --full10:54
lyarwoodnightmare_unreal: or the mailing list.10:54
stephenfin*#openstack-sdks10:54
ygk_12345kashyap ok sure10:54
kashyapygk_12345: Can you just run this: `/usr/bin/qemu-system-ppc64` and tell what you get?10:54
nightmare_unrealhttps://www.irccloud.com/pastebin/KuPFLZ4Q/quota.rst10:54
nightmare_unreallyarwood: but that file has like no content.10:54
nightmare_unrealsee the snippet10:54
ygk_12345@kashya10:55
ygk_12345kashyap root@b2a-1:~# /usr/bin/qemu-system-ppc6410:55
ygk_12345root@b2a-1:~# /usr/bin/qemu-system-ppc6410:55
kashyapygk_12345: Does that open a QEMU window or not?10:55
*** macz_ has joined #openstack-nova10:55
ygk_12345it is commandline10:56
ygk_12345no GUI10:56
lyarwoodnightmare_unreal: ah sorry it builds that list automatically10:57
nightmare_unrealexactly, that's what I was thinking how to add10:58
lyarwoodnightmare_unreal: where's your osc change?10:58
kashyapygk_12345: That's supposed to launch a GUI window; and do you see any error on the command-line?10:59
kashyapygk_12345: Either way, your installation is broken; please look for Ubuntu's guidance for re-installing virtualization packages properly.10:59
ygk_12345kashyap /usr/bin/qemu-system-ppc64: relocation error: /usr/bin/qemu-system-ppc64: symbol fdt_check_full version LIBFDT_1.2 not defined in file libfdt.so.1 with link time reference10:59
stephenfinnightmare_unreal: if you're adding this functionality to the code then you don't need to edit the docs - they're generated automatically, as lyarwood notes10:59
stephenfinyou need to add the functionality though as it doesn't exist at the moment, as noted on openstack-discuss11:00
lyarwoodyeah just building the docs with https://review.opendev.org/#/c/714062/1 now that I assume fixes this11:00
nightmare_unrealoh okay11:00
*** macz_ has quit IRC11:00
nightmare_unrealyes11:00
kashyapygk_12345: Okay, that's broken; as I told above, please re-install Virt Packages from scratch.  Read documentation with patience, and take notes of failures.11:01
lyarwoodnightmare_unreal: yeah https://review.opendev.org/#/c/714062/ will list --force under quota set now11:02
*** nicolasbock has joined #openstack-nova11:03
nightmare_unrealthanks for clarifcation , i just submitted that patch and thought i made mistake by not writing doc11:03
nightmare_unrealfirst patch :D11:03
lyarwoodnightmare_unreal: np, there should be a docs job that builds this in CI, I'll confirm with a link once it does11:04
* lyarwood -> brb11:04
stephenfinnightmare_unreal: I left a review on that patch. There are a couple of things you need to fix. Feel free to ping me on #openstack-sdks if you have questions/need help11:06
nightmare_unrealsure stephenfin11:06
lyarwoodstephenfin: is that the channel for osc now?11:14
* lyarwood wasn't the first to direct people to #openstack11:14
stephenfinto the best of my knowledge, yes11:14
stephenfinthat's where I've always gone to discuss stuff with mordred et al11:14
openstackgerritKevin Zhao proposed openstack/nova master: Add default cpu model for aarch64  https://review.opendev.org/70949411:18
openstackgerritMerged openstack/nova stable/stein: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph  https://review.opendev.org/71383711:20
ygk_12345@kas11:26
ygk_12345kashyap now the libvirtd is running fine but still the same nova-compute resource error. I have updated the bug report with my findings11:27
*** mkrai has quit IRC11:31
sean-k-mooneylyarwood: it makes total sense to run c-vol on the subnode and we run g-api on the sub node to do multile glance store testing so dont remove either of them11:34
sean-k-mooneyc-vol is on the sub node so cinder can do testing with multiple backends too11:35
openstackgerritBrin Zhang proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available  https://review.opendev.org/68277211:35
sean-k-mooneylyarwood: im not sure we we need eitehr in our specific jobs but other project want that for there jobs11:36
*** mkrai has joined #openstack-nova11:37
*** rpittau is now known as rpittau|bbl11:37
lyarwoodsean-k-mooney: right, the way we layout the services at the moment isn't granular enough IMHO, we should really have a subnode layout *just* for compute migration testing11:40
*** mkrai has quit IRC11:43
sean-k-mooneylyarwood: im not sure i agree but we could11:45
*** ccamacho has joined #openstack-nova11:45
*** ccamacho has quit IRC11:45
lyarwoodsean-k-mooney: https://review.opendev.org/#/c/714057/ - can you hit that regardless11:45
*** ccamacho has joined #openstack-nova11:45
sean-k-mooneylyarwood: in a hyper convered deployment i wound not find it unusal to have cinder vol on a compute11:45
sean-k-mooneyglance api is a littel weired but we only have two nodes11:45
lyarwoodsean-k-mooney: our default deployment topologies shouldn't be HCI11:45
sean-k-mooneyi think it should11:46
lyarwoodsean-k-mooney: it's just adding overhead for no reason11:46
sean-k-mooneyif we use it11:46
lyarwoodduplicating c-vol and g-api is utterly pointless11:46
lyarwoodon the computes upstream11:46
lyarwoodthere's a place for that but it isn't in openstack/nova's gate11:46
lyarwoodusing devstack11:46
lyarwoodas it's no reflective of reality at all11:46
sean-k-mooneyright but i think glance or cinder made the change and we just picked it up11:46
lyarwoodright and that's fine if they need to test volume migration or multi g-api backends etc11:47
sean-k-mooneye.g. they change the common job definition and we never diabled it again11:47
lyarwoodbut it shouldn't be the default for all subnode users11:47
sean-k-mooneysure but its jsut that they didnt do it in there job they did it in either the base devstack multinode job or tempest11:48
lyarwoodit's in devstack-gate iirc in the feature matrix11:48
*** dave-mccowan has joined #openstack-nova11:48
*** jangutter has joined #openstack-nova11:48
lyarwoodthat builds the enabled services for subnodes etc11:48
sean-k-mooneyok so ya if this was zuulv3 then it would be simpelr11:49
lyarwoodyeah indeed11:49
sean-k-mooneywith devstack gate you dont have the same level of contol in project repos to alter the job11:50
sean-k-mooneyi guess thats is another thing to alter in your migration11:50
*** dpawlik has quit IRC11:50
lyarwoodright sorry my brain has just caught up to where I was 9 weeks ago on this11:50
lyarwoodand the need to migrate to control this easily per multinode job11:50
lyarwoodfriday--11:51
*** dave-mccowan has quit IRC11:53
sean-k-mooneybut but fridays are good :(11:55
lyarwoodafter 5:3011:55
lyarwoodI'm just burnt out today11:55
sean-k-mooneythen have a lazy friday11:55
stephenfinbauzas: You about? Want to rubberstamp this for me, please? https://review.opendev.org/#/c/713611/11:56
*** ccamacho has quit IRC12:10
*** macz_ has joined #openstack-nova12:18
*** dpawlik has joined #openstack-nova12:21
lyarwoodgibi: https://review.opendev.org/#/c/714057/ if you have bandwidth can you check this fup to the nova-live-migration fixes yesterday.12:23
*** zhanglong has quit IRC12:23
*** macz_ has quit IRC12:24
lyarwoodelod: https://review.opendev.org/#/q/topic:bug/1867380+(status:open+OR+status:merged) - would you mind going through the rest of this series so we can unblock stable/pike12:26
elodlyarwood: just commented one12:29
elodlyarwood: but otherwise it would be good to push through these soon :) and thanks for the fixes!12:30
openstackgerritLee Yarwood proposed openstack/nova stable/rocky: Replace ansible --sudo with --become in live_migration/hooks scripts  https://review.opendev.org/71383912:30
openstackgerritLee Yarwood proposed openstack/nova stable/rocky: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph  https://review.opendev.org/71384012:30
lyarwoodelod: yup thanks, addressed above ^12:30
elodlyarwood: cool, thanks!12:31
artomstephenfin, yep, thanks - lemme coffee up and read that12:40
gibilyarwood: done. Thanks12:45
*** kaisers_ has joined #openstack-nova12:53
*** macz_ has joined #openstack-nova12:55
*** nweinber has joined #openstack-nova12:56
*** lbragstad_ has joined #openstack-nova12:56
*** lbragstad_ has quit IRC12:57
*** macz_ has quit IRC13:00
*** lbragstad has joined #openstack-nova13:07
*** tbachman has quit IRC13:12
*** klindgren_ has quit IRC13:17
*** klindgren has joined #openstack-nova13:17
*** mkrai has joined #openstack-nova13:20
*** mriedem has joined #openstack-nova13:26
luyao dansmith, artom : are you about?13:31
*** mkrai has quit IRC13:31
artomlyarwood, I am, dansmith probably isn't, given the time zone13:31
artomErr, luyao ^^13:32
luyaoartom: I replied on https://review.opendev.org/#/c/687856, could you look it again13:33
artomluyao, is there a specific question you'd like answered though? Reading the conversations quickly, it looks like the ball is in your court to update the patch13:35
openstackgerritKevin Zhao proposed openstack/nova master: Add default cpu model for aarch64  https://review.opendev.org/70949413:35
*** udesale_ has joined #openstack-nova13:35
*** udesale has quit IRC13:38
luyaoartom: I don't update my patch since dansmith have some concern on moving cleanup before rpc call and some other details I don't know which solustion is better.13:41
luyaoartom: I saw you mentioned functional test, I run your live migration functional test on top of my patch, they runs well13:42
-openstackstatus- NOTICE: Gerrit (review.opendev.org) will be down for maintenance starting at 14:00 (in less than 20 mins), probably until 15:00 UTC13:43
*** macz_ has joined #openstack-nova13:47
*** amoralej is now known as amoralej|lunch13:47
*** ratailor has quit IRC13:49
*** derekh has joined #openstack-nova13:51
*** macz_ has quit IRC13:51
*** rpittau|bbl is now known as rpittau13:52
artomluyao, yeah, the fact that the functional test is passing is reassuring (and thanks for the review - I have it fixed locally, will push a new version soon)13:58
*** lpetrut has quit IRC14:01
*** lpetrut has joined #openstack-nova14:03
*** yedongcan has left #openstack-nova14:05
-openstackstatus- NOTICE: The Gerrit service on review.opendev.org is offline for maintenance until 15:00 UTC http://lists.opendev.org/pipermail/service-announce/2020-March/000001.html14:07
*** ChanServ changes topic to "The Gerrit service on review.opendev.org is offline for maintenance until 15:00 UTC http://lists.opendev.org/pipermail/service-announce/2020-March/000001.html"14:07
*** klindgren has quit IRC14:10
*** klindgren has joined #openstack-nova14:10
luyaoartom: If you still have questions or concerns about my reply on vpmem live migration , please leave comments there. Or you can ping me in IRC, I'll be online for about two hours I can response promply during this time.14:13
artomluyao, ok, thanks14:13
openstackgerritMerged openstack/nova master: libvirt: Correctly resize encrypted LUKSv1 volumes  https://review.opendev.org/70690014:26
*** ygk_12345 has quit IRC14:27
*** jraju__ has quit IRC14:30
*** tesseract has quit IRC14:32
*** udesale_ has quit IRC14:33
*** tesseract has joined #openstack-nova14:33
openstackgerritArtom Lifshitz proposed openstack/nova master: Increase code reuse in test_numa_servers  https://review.opendev.org/71412514:34
artomstephenfin, ^^ quick side-fix for you14:34
artom(as I work on NUMA LM func tests)14:34
*** macz_ has joined #openstack-nova14:38
*** macz_ has quit IRC14:42
*** tbachman has joined #openstack-nova14:43
luyaodansmith: Are you about?14:44
dansmithluyao: yes, but I don't really want to argue about moving that rpc call14:44
dansmithI've read your comments, I understand that it may not matter right now, but what I said is still my opinion: that changing the order of critical points of synchronization is not something to take lightly14:45
dansmithso, my opinion is still the same14:45
*** amoralej|lunch is now known as amoralej14:45
*** tbachman has quit IRC14:49
luyaodansmith: it will not break anything, so I don't  understand what's the risk14:51
luyaodansmith: I try other solution which alex_xu  help clarifing,  what's you comments about those solusion14:52
alex_xumaybe we should move the instance's state update to later, but that across the src and dest node, in the end, we have to deal with the upgrade issue.14:54
*** belmoreira has joined #openstack-nova14:55
*** dklyle has joined #openstack-nova14:56
belmoreiraHi, I need some help...14:57
belmoreiranova-compute (dedicated to ironic) market all ironic nodes (compute_nodes table) and as consequence removed all the resource_providers/inventories/allocations for those nodes14:57
belmoreirarestarting nova-compute and removing the soft delete of the compute_nodes, the resource provider/inventory is created again14:59
belmoreirabut not the allocations... I remember that this was removed from nova some time ago14:59
belmoreiraany suggestion how I can recreate thousands of allocations?14:59
dansmithbelmoreira: I assume heal instance allocations, just a sec15:00
belmoreiraI remember that mnaser and bauzas did some kind of tool15:00
belmoreiradansmith I using stein15:01
dansmithin nova-manage, I think there's an instance heal15:01
dansmithoh15:01
bauzasplacement-audit you mean ?15:02
belmoreiraplacement-audit doesn't recreate allocations if they are missing?15:02
dansmithheal_allocations is the subcmd15:03
dansmithbelmoreira: that was in stein15:03
mnaseryeah heal allocations is the thing you need to add missing entries, placement audit is what you need to remove extra entries15:03
*** ChanServ changes topic to "Current runways: https://etherpad.openstack.org/p/nova-runways-ussuri -- This channel is for Nova development. For support of Nova deployments, please use #openstack."15:04
belmoreirathanks all, let me have a look15:04
-openstackstatus- NOTICE: Gerrit maintenance is concluded at this time and requested renames have been performed15:04
*** gyee has joined #openstack-nova15:09
*** macz_ has joined #openstack-nova15:12
mriedemi think https://review.opendev.org/#/c/678100/ is still relevant to ^ for the compute side delete of allocations there15:28
mriedembut it needs love15:28
*** lpetrut has quit IRC15:30
*** toabctl has quit IRC15:31
openstackgerritBalazs Gibizer proposed openstack/nova stable/pike: Avoid circular reference during serialization  https://review.opendev.org/71414815:33
*** lbragstad has quit IRC15:34
*** ratailor has joined #openstack-nova15:36
belmoreirathanks dansmith, mnaser, bauzas. 4305 allocations recreated! In Stein it goes through all deployment. Needed to patch it to do it per cell (this will be a great addition to master).15:39
belmoreiraFor some reason nova deleted all the ironic resource providers/inventories/allocations.15:39
belmoreiraI'm investigating what trigger this... no idea for now... I will let you know when I have more info15:39
dansmithI thought it does do it per cell in master, but I'd have to go back and look15:40
belmoreirait iterates per cell but is not possible to select the cell15:42
dansmithah okay15:43
*** ccamacho has joined #openstack-nova15:46
mnaserbelmoreira: changed hostname or compute node by any chance ?15:51
belmoreiramnaser maybe... I'm configuring "conductor_groups" for ironic. To avoid this precise issue I configured only one conductor group to keep the same nova-compute... and move from there. Still looking... for now I don't understand what went wrong15:55
mnaserbelmoreira: maybe check the compute nodes table15:55
belmoreiramnaser the compute_nodes table had all the resources deleted15:56
belmoreiramnaser to recover I made the entries available again (deleted=0, deleted_at=NULL)15:59
belmoreiranew entries are not created because the uuid conflict15:59
mnaserbelmoreira: pretty sure you might look at the compute logs and see that it was deleting them15:59
mnaseralso I think I ran into that and mriedem fixed it at the time16:00
belmoreirayeah, I need to go deep into the logs... for now I'm still adding the nodes into the right placement_aggregate for them to be available16:01
*** mlavalle has joined #openstack-nova16:01
*** jawad_axd has quit IRC16:07
*** jawad_axd has joined #openstack-nova16:07
*** jawad_axd has quit IRC16:13
*** ratailor has quit IRC16:15
*** iurygregory is now known as iurygregory}brb16:19
*** iurygregory}brb is now known as iurygregory|brb16:19
mriedemsee that link above - if you're doing rebalancing of nodes when the old compute sees the nodes are no longer reported there it deletes them along with the providers and their allocations16:36
belmoreirathanks mriedem16:36
mriedemhttps://review.opendev.org/#/c/678100/3/nova/compute/manager.py@920816:36
mriedemthat cascading delete thing was added back when the computes still self-healed the allocations in the RT16:37
mriedembut that self-heal was removed a few releases ago16:37
mriedemso now we just delete your stuff on re-balance16:37
mriedemyou're welcome16:37
mriedembelmoreira: i think this is the todo you're looking for for heal_allocations yes? https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L211916:38
mriedemthat's a very easy change https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L217316:39
*** vishalmanchanda has quit IRC16:40
*** efried_gone has quit IRC16:40
belmoreiramriedem yes, it's easy to do... I did't under stress and just ran it :)16:41
belmoreirawe have a new member in the team, this is a good low hanging fruit for him16:42
*** openstack has joined #openstack-nova16:56
*** ChanServ sets mode: +o openstack16:56
*** tesseract has joined #openstack-nova16:57
*** tesseract has quit IRC16:58
*** tesseract has joined #openstack-nova16:59
*** dtantsur is now known as dtantsur|afk17:02
*** iurygregory|brb is now known as iurygregory17:14
openstackgerritMerged openstack/nova master: Introduce scope_types in os-flavor-access  https://review.opendev.org/71355917:16
*** toabctl has joined #openstack-nova17:19
*** evrardjp has quit IRC17:36
*** evrardjp has joined #openstack-nova17:36
*** maciejjozefczyk has quit IRC17:59
*** tesseract has quit IRC18:00
*** ociuhandu has joined #openstack-nova18:01
*** ociuhandu has quit IRC18:05
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147818:11
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492918:11
*** ccamacho has quit IRC18:31
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147818:32
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492918:32
openstackgerritmelanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/71311618:34
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492918:34
*** amoralej is now known as amoralej|off18:36
openstackgerritArtom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into own mixin  https://review.opendev.org/70554518:49
openstackgerritArtom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML  https://review.opendev.org/70554618:49
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259518:49
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146  https://review.opendev.org/68740418:49
openstackbug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms)18:49
artomdansmith, ^^ if you have the time and energy this Friday afternoon. Next week is fine too :) I'm going to go interact with my spawns for a bit19:03
*** KeithMnemonic has joined #openstack-nova19:06
*** ralonsoh has quit IRC19:17
openstackgerritArtom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into own mixin  https://review.opendev.org/70554519:30
openstackgerritArtom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML  https://review.opendev.org/70554619:30
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259519:30
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146  https://review.opendev.org/68740419:30
openstackbug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms)19:30
artomdansmith, so, you'd rather the class inheriting from the Mixin sets self.computes?19:50
artomMy thinking was, the mixin is a batteries-included thing to help you run_periodics()19:50
dansmithartom: I'd rather not have the mixin at all, but yes, I'd rather the classes set them and let that blow up if they don't instead effectively delegating setUp() to a later time, and in a mixin19:50
dansmithI understand19:51
artomdansmith, you mean move it directly under _IntegratedTestBase? Not everything can inherit from that...19:52
artomCan it?19:52
dansmithI dunno, you tell me.. I didn't go look at the whole inheritance tree.19:53
artomYeah, looking now19:54
dansmiththe massive mess of where and how all those tests get their functions make it super confusion to figure out where stuff is defined, which is why I don't like this19:54
dansmiththe number of places we have duplicated helper functions for checking server state, waiting for server state to change, server state to become a thing, etc is just maddening19:55
*** macz_ has quit IRC19:56
artomdansmith, I know - stephenfin's done some work to improve that19:57
artomSo, I checked and the two regression tests that I changed would be a a PITA to make inherit from the _IntegratedTestBase19:57
artomThey have their own lengthy setup19:58
artomThe mixin is probably cleaner, that at point. But OK, let's make the child setup self.computes then19:59
artom*at that point19:59
dansmithstart_service is a method on the *base* TestCase right?19:59
dansmithso why not just put all of it there?19:59
artomThe very base, yeah19:59
dansmithif that's where they meet in the inheritance tree, that's ...where they meet19:59
artomI suppose20:00
artomIsn't it weird to have func test-specific stuff in the base class?20:00
dansmithlike what's already there you mean?20:01
artomHeh, yeah :/20:01
dansmithyou could also put another layer in between the two.. _IntegratedTestBaseNoSeriouslyThisIsTheFunctionalBaseForReals20:01
artomlol20:01
*** damien_r has quit IRC20:12
*** damien_r has joined #openstack-nova20:12
*** damien_r has quit IRC20:17
*** macz_ has joined #openstack-nova20:37
*** macz_ has quit IRC20:37
*** macz_ has joined #openstack-nova20:38
*** mriedem has left #openstack-nova20:53
*** belmoreira has quit IRC20:54
openstackgerritArtom Lifshitz proposed openstack/nova master: tests: work around malformed serial XML  https://review.opendev.org/70554621:02
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259521:02
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146  https://review.opendev.org/68740421:02
openstackbug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms)21:02
openstackgerritArtom Lifshitz proposed openstack/nova master: func tests: move _run_periodics() into base class  https://review.opendev.org/71422821:02
*** nweinber has quit IRC21:05
*** KeithMnemonic has quit IRC22:02
openstackgerritArtom Lifshitz proposed openstack/nova master: Functional tests for NUMA live migration  https://review.opendev.org/67259522:09
openstackgerritArtom Lifshitz proposed openstack/nova master: NUMA LM: Add func test for bug 1845146  https://review.opendev.org/68740422:09
openstackbug 1845146 in OpenStack Compute (nova) train "NUMA aware live migration failed when vCPU pin set" [High,Fix committed] https://launchpad.net/bugs/1845146 - Assigned to Dan Smith (danms)22:09
*** kaisers_ has quit IRC22:21
*** nicolasbock has quit IRC22:24
*** hongbin has joined #openstack-nova22:33
*** mmethot_ has joined #openstack-nova22:33
*** mmethot has quit IRC22:35
*** mmethot has joined #openstack-nova22:47
*** mmethot_ has quit IRC22:49
*** jangutter has quit IRC22:55
*** macz_ has quit IRC23:02
*** macz_ has joined #openstack-nova23:03
*** macz_ has quit IRC23:08
*** CeeMac_ has joined #openstack-nova23:24
*** CeeMac has quit IRC23:25
*** portdirect has quit IRC23:25
*** evrardjp has quit IRC23:25
*** portdirect has joined #openstack-nova23:26
*** evrardjp has joined #openstack-nova23:27
*** irclogbot_2 has quit IRC23:27
*** irclogbot_2 has joined #openstack-nova23:28
*** guilhermesp_ has joined #openstack-nova23:36
*** guilhermesp has quit IRC23:37
*** guilhermesp_ is now known as guilhermesp23:37
*** jdillaman has quit IRC23:37
*** jdillaman has joined #openstack-nova23:37
*** evrardjp has quit IRC23:38
*** evrardjp has joined #openstack-nova23:43
*** slaweq has quit IRC23:49
*** xek has quit IRC23:54
*** tosky has quit IRC23:59
« Thursday, 2020-03-19 Index Saturday, 2020-03-21 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!