Tuesday, 2020-04-21

abaindurso for pre-copy, you are suggesting to try memcache for keystone tokens?00:00
sean-k-mooneyya but im not saying dont try enableing post-copy00:00
sean-k-mooneyim just sugging that memcache might impove the performace enough to not need it00:00
sean-k-mooneyi woudl test both00:00
sean-k-mooneyabaindur: how did you deploy openstack by the way00:01
abaindurthanks, yea we'll probably try both options then to see which leads to least downtime00:01
sean-k-mooneyfor both options is basically just a compute agent restart and cofig update00:01
sean-k-mooneyabaindur: at least in the case of https://bugs.launchpad.net/nova/+bug/1836642 we were seeing edgcaces wher ethe metadta api respoce could take upwards of 12 - 16  seconds and that went away when we enabeld the cache00:03
openstackLaunchpad bug 1836642 in neutron "Metadata responses are very slow sometimes" [High,Incomplete] - Assigned to Slawek Kaplonski (slaweq)00:03
sean-k-mooneyabaindur: although to be clear we enabel both the cache section and the memcahced_server in the keystone_authoken section00:05
sean-k-mooneyso  all of the highlight section is how we adressed the isseu https://zuul.opendev.org/t/openstack/build/dcde79801a624c25b195a46ead7af562/log/controller/logs/etc/nova/nova-cpu_conf.txt#62-6800:05
*** bbowen has quit IRC00:08
sean-k-mooneyabaindur: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time might be of interst00:09
*** gyee has quit IRC00:11
sean-k-mooneyi think that might only be relevent to the apis service however and might not apply to when we use the keystone auth client to talk to otehr apis but im not sure baout that. anyway its late here so im going to go o/00:11
abaindursean-k-mooney: oh wait, we have memcached_servers set in keystone_authtoken00:11
abaindurbut we dont have a [cache] section with the other configs00:12
sean-k-mooneyah the cache section is novas own internal cache config00:12
*** ttsiouts has quit IRC00:12
sean-k-mooneyso in the metadata api case it is what allows cache copies fo the metadata to be shareed between processes00:13
sean-k-mooneyi think your best path forward in that cae is likely your original one of enabling post copy. you might also want o consider auto convergence00:14
*** factor has joined #openstack-nova00:20
*** tetsuro has joined #openstack-nova00:23
abaindurok thanks for the help! will update here later if i find out more00:27
openstackgerritMerged openstack/nova master: Allocate mdevs when resizing or reverting resize  https://review.opendev.org/71274100:42
*** ociuhandu has joined #openstack-nova00:43
*** ociuhandu has quit IRC00:49
*** mkrai_ has quit IRC00:50
*** bbowen has joined #openstack-nova00:50
*** mlavalle has quit IRC01:03
*** Liang__ has joined #openstack-nova01:03
*** ttsiouts has joined #openstack-nova01:20
*** zhanglong has joined #openstack-nova01:27
openstackgerritMerged openstack/nova stable/train: Reject boot request for unsupported images  https://review.opendev.org/70857701:50
*** ttsiouts has quit IRC01:55
*** mkrai has joined #openstack-nova02:11
*** francoisp has quit IRC02:15
*** ttsiouts has joined #openstack-nova02:31
*** ttsiouts has quit IRC02:36
*** factor has quit IRC02:49
*** hongbin has joined #openstack-nova03:00
*** ttsiouts has joined #openstack-nova03:04
*** factor has joined #openstack-nova03:15
*** psachin has joined #openstack-nova03:25
*** ttsiouts has quit IRC03:38
*** tetsuro has quit IRC04:00
*** ociuhandu has joined #openstack-nova04:19
*** ociuhandu has quit IRC04:24
*** evrardjp has quit IRC04:35
*** evrardjp has joined #openstack-nova04:35
*** hongbin has quit IRC04:35
*** zhanglong has quit IRC04:48
*** ttsiouts has joined #openstack-nova04:49
*** ratailor has joined #openstack-nova04:50
*** zhanglong has joined #openstack-nova04:52
*** tetsuro has joined #openstack-nova04:59
*** vishalmanchanda has joined #openstack-nova05:09
*** udesale has joined #openstack-nova05:10
*** ratailor_ has joined #openstack-nova05:19
*** ratailor has quit IRC05:21
*** udesale_ has joined #openstack-nova05:21
*** ttsiouts has quit IRC05:23
*** udesale has quit IRC05:24
*** avolkov has joined #openstack-nova05:55
*** ttsiouts has joined #openstack-nova05:58
*** ratailor_ has quit IRC06:02
*** ratailor has joined #openstack-nova06:02
*** ttsiouts has quit IRC06:02
*** ratailor has quit IRC06:04
*** ratailor has joined #openstack-nova06:05
*** dpawlik has joined #openstack-nova06:08
*** udesale_ has quit IRC06:14
*** ttsiouts has joined #openstack-nova06:28
*** dklyle has quit IRC06:32
*** breizhkoala has joined #openstack-nova06:36
*** udesale has joined #openstack-nova06:37
gibistephenfin, gmann, artom: I'm +2 on the policy code changes.06:38
*** udesale has quit IRC06:43
*** ociuhandu has joined #openstack-nova06:48
gibiI'm tempted to squash the doc follow up into the original doc patch for more readable diff but meh I wait for others to show up06:48
*** tetsuro has quit IRC06:51
bauzasgibi: gmann: fwiw, I'll leave the prelude section change pretty small for the policy changes, but you are free to amend it for clarifying what we finally got for policy changes06:55
bauzashopefully, will be posted later this morning06:55
gibibauzas: thanks06:55
bauzas(and good morning Nova, sorry for my manners)06:55
bauzasgibi: tbh, I'm mostly paraphrasing the cycle highlights with a couple of additions (the BFV instance rescue and the vGPU multiple types support)06:57
bauzasgibi: worth mentioning others ?06:57
gibiI think that is plenty06:58
*** ociuhandu has quit IRC06:58
*** ociuhandu has joined #openstack-nova06:58
bauzasgibi: well, I'm not afraid, that will still be an order of magnitude less than the rest of the relnotes06:58
bauzashttps://blueprints.launchpad.net/nova/+spec/remove-xvpvncproxy and https://blueprints.launchpad.net/nova/+spec/remove-nova-network-ussuri should also be commented I tihnk06:59
bauzasgibi: fwiw, this is how it looks at the moment without a prelude section https://docs.openstack.org/releasenotes/nova/unreleased.html06:59
bauzasgood luck with this.06:59
bauzasanyway, late breakfast, kids are up07:00
gibihm, based on that I even highlight the fact that we dropped supporting python 2.707:01
*** ttsiouts has quit IRC07:03
*** rpittau|afk is now known as rpittau07:03
*** nightmare_unreal has joined #openstack-nova07:03
*** tetsuro has joined #openstack-nova07:04
*** maciejjozefczyk has joined #openstack-nova07:04
*** ttsiouts has joined #openstack-nova07:07
*** slaweq_ has joined #openstack-nova07:11
*** tesseract has joined #openstack-nova07:13
gibisean-k-mooney: I guess you have SRIOV env at hand. Could you check if  the bug https://bugs.launchpad.net/nova/+bug/1851545 is valid?07:25
openstackLaunchpad bug 1851545 in OpenStack Compute (nova) "Port update exception on nova unshelve for instance with PCI devices (part 2)" [Undecided,New]07:25
gibifor me it would take a day or so to set one up07:26
*** ociuhandu has quit IRC07:26
bauzasgibi: good point, okay, let's wrap this up and see what people think07:37
gibibauzas: yep, lets do the review on gerrit :)07:38
*** ttsiouts has quit IRC07:39
*** ttsiouts has joined #openstack-nova07:40
*** tosky has joined #openstack-nova07:44
*** ccamacho has joined #openstack-nova07:47
*** xek has joined #openstack-nova07:52
*** threestrands has quit IRC07:55
*** ralonsoh has joined #openstack-nova07:57
elodzigo melwitt : about https://bugs.launchpad.net/nova/+bug/1788014 , I realized that it needs maybe some refactoring... then I forgot about looking at it. :/ Will do today. I'll need some thorough review for that i guess.07:59
openstackLaunchpad bug 1788014 in OpenStack Compute (nova) rocky "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] - Assigned to Elod Illes (elod-illes)07:59
zigoelod: Thanks !08:00
*** jangutter has joined #openstack-nova08:00
*** ociuhandu has joined #openstack-nova08:01
*** lennyb has joined #openstack-nova08:03
*** KeithMnemonic has quit IRC08:05
bauzasgosh, the tox target for relnotes takes a while recalculating all the relnotes...08:06
*** tetsuro_ has joined #openstack-nova08:08
*** martinkennelly has joined #openstack-nova08:09
*** tetsuro has quit IRC08:11
*** ttsiouts has quit IRC08:14
*** links has joined #openstack-nova08:16
*** udesale has joined #openstack-nova08:16
*** jangutter has quit IRC08:19
*** jangutter has joined #openstack-nova08:19
*** ttsiouts has joined #openstack-nova08:21
*** udesale_ has joined #openstack-nova08:22
*** udesale has quit IRC08:22
*** udesale has joined #openstack-nova08:22
openstackgerritSylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section  https://review.opendev.org/72154808:38
bauzaswow, it took a while but I'm done08:38
openstackgerritMerged openstack/nova master: Add new default roles in quota class policies  https://review.opendev.org/71910008:39
bauzasgibi: stephenfin: dansmith: melwitt: alex_xu: gmann: others, the prelude section patch is up for reviews https://review.opendev.org/72154808:39
bauzaslyarwood: too ^08:40
* lyarwood clicks08:41
*** belmoreira has joined #openstack-nova08:41
bauzasgibi: can I hold the bugs lock ?08:41
gibibauzas: I'm looking at https://bugs.launchpad.net/nova/+bug/1831315 right now but after that I have to jump to your prelude patch so you can have the lock08:42
openstackLaunchpad bug 1831315 in OpenStack Compute (nova) "nova-manage cell_v2 discover_hosts fails for IPv6 - ValueError: invalid literal for int() with base 10 - db connection URI gets its brackets eaten" [Undecided,New] - Assigned to Harald Jensås (harald-jensas)08:42
bauzasgibi: you don't order bugs by Age, right?08:43
*** priteau has joined #openstack-nova08:45
gibibauzas: I skip those that I cannot do anyithing with08:47
bauzasok, I'll try to look at them and ask for more if needed08:47
* bauzas does this every week for downstream bugs08:47
gibithanks08:47
bauzasgibi: for example, https://bugs.launchpad.net/nova/+bug/1860312 looks unrelated to Nova09:00
openstackLaunchpad bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,New]09:00
gibibauzas: is it so?09:00
*** dtantsur|afk is now known as dtantsur09:00
gibibauzas: for me it seems like we somehow manage to allow a db corruption09:01
gibi"When I try to delete the old compute service(status down), it failed."09:01
gibisure I don't know how the deployment ended up in "ComputeHostNotFound: Compute host mgt-slave03 could not be found " state09:02
gibibut I feel that nova should support deleting old unused compute services09:02
gibimy problem is that I have no way to try to recreate the bug without building a kubenetes + ironic env09:05
*** zhanglong has quit IRC09:07
*** vishalmanchanda has quit IRC09:09
kashyaplyarwood: So ... one test failure seems legit here: https://review.opendev.org/#/c/708701/409:11
kashyaplyarwood: You know the reason once you see the failing test:  test_stable_device_rescue_cdrom_ide() + Q35 :)09:12
lyarwoodI've not had a look yet no09:16
lyarwoodI assume with that test that q35 doesn't have an IDE bus?09:16
lyarwoodiirc that's the case right09:16
lyarwoodI can add a configurable in tempest for that09:17
*** happyhemant has joined #openstack-nova09:17
lyarwoodthe config drive issue is still there as well btw09:17
openstackgerritLee Yarwood proposed openstack/nova master: docs: Add stable device rescue docs  https://review.opendev.org/70083709:18
kashyaplyarwood: Yeah, the issue is indeed 'q35' has no IDE bus09:18
kashyaplyarwood: Sorry, which config drive issue?09:18
lyarwoodkashyap: see test_server_basic_ops etc, blkid -L config-2 -o device', exit status: 209:19
lyarwoodah wait09:19
lyarwoodthat's because that's a cdrom09:19
lyarwoodover SCSI?09:20
kashyaplyarwood: You mean over IDE, instead of SATA (or 'virtio')?09:20
lyarwoodkashyap: we don't have IDE, I can't recall if we default to SCSI in that case or SATA09:21
kashyapI'm not sure yet ... I'll see what my non-configurable one-liner yields: https://review.opendev.org/#/c/721555/09:21
lyarwoodkashyap: nack, IDE + cdrom is still a valid test case prior to q35 IMHO09:21
kashyaplyarwood: Don't you recall the patch you made? :) We default to SATA with Q35: https://review.opendev.org/#/c/663011/09:21
kashyaplyarwood: Oh, sure; it still _is_ a valid test for PC09:22
lyarwoodah right09:22
kashyap(I'm just seeing what this not-to-be-merged yields.  And a config tunable is perhaps the way)09:22
lyarwoodso are you pulling that in anywhere to test against 135?09:23
kashyaplyarwood: Sorry, 135?09:24
lyarwoodq35 sorry09:24
bauzasgibi: the problem is that ironic still sees some node09:24
bauzasgibi: while nova doesn't09:25
kashyaplyarwood: Err, I need to make that dependent against your 'q35' job, isn' it?09:25
kashyapLet me go do that09:25
bauzasgibi: at least we can ask the reporter to tell us whether nova still sees it => Incomplete09:26
lyarwoodkashyap: make my q35 change depend on your tempest change09:26
kashyaplyarwood: Yeah.  Not sure what I'm thinking09:27
kashyapDoing ...09:27
gibibauzas: OK, I was not able to formulate any valid question to the reporter. I glad you could09:27
bauzasgibi: I'm just doing it09:27
* lyarwood rebuilds his bionic q35 env09:27
kashyaplyarwood: Can I remove the Depends-On in your 'q35' CI change against the 'blkid -L' change?09:30
kashyaplyarwood: I'll remove it for now, so as to test one thing at a time -- okay?09:31
lyarwoodkashyap: well it's seperate09:32
lyarwoodkashyap: ah wait no it isn't09:32
lyarwoodkashyap: it's in openstack/tempest so yeah feel free to09:32
lyarwoodkashyap: btw I'm planing on writing a spec for the q35 default change next week09:33
kashyapOkido; I'll add a note in the commit with a link to tit.09:33
kashyaplyarwood: FWIW, I already have it in place somewhere09:33
kashyapOne is this BP: https://blueprints.launchpad.net/nova/+spec/gracefully-handle-qemu-machine-types09:34
kashyapAnd a spec should also be around ...09:34
kashyaplyarwood: Here's the spec started 8 months ago: https://review.opendev.org/#/c/631154/09:35
*** sapd1 has joined #openstack-nova09:35
lyarwoodkk I'll repropose that for V09:35
kashyaplyarwood: I was meaning to send a note to the upstream list as a heads-up.09:37
kashyap(I have a draft e-mail, will send it sometime this week.  Giving a heads-up to prepare ... as this is a virtual 'motherboard' change)09:37
openstackgerritKashyap Chamarthy proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type  https://review.opendev.org/70870109:39
lyarwoodkashyap: lets get the spec reproposed, updated and on the agenda for PTG first09:41
* kashyap nods09:41
lyarwoodkashyap: I'm not going to be around for PTG btw would you be okay leading any sessions/discussions on the ML?09:42
bauzasgibi: FYI I think I found the problem but before triaging it to Confirmed, I need to ask the reporter to tell us whether I'm right https://bugs.launchpad.net/nova/+bug/1860312/comments/109:42
openstackLaunchpad bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete]09:42
kashyaplyarwood: Yeah, I was intending to that.  As I've been delaying this work for various reasons09:42
gibibauzas: cool09:43
gibibauzas: also I left some comments on the prelude. good job!09:44
bauzasgibi: ack09:44
gibistephenfin: could you re-review stable device rescue doc patch? https://review.opendev.org/#/c/70083709:45
bauzasI wish I could easily find bug duplicates with Launchpad, because I'm pretty sure that bug  1860312  is not the only one for the same issue09:45
openstackbug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete] https://launchpad.net/bugs/186031209:45
stephenfinsure, just finished gmann doc change09:45
stephenfin*gmann's09:45
gibistephenfin: that would have been my other request :)09:45
* gibi goes and read the policy doc pathces09:45
bauzasgibi: bingo : https://bugs.launchpad.net/nova/+bug/185300909:46
openstackLaunchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard)09:46
mgoddardbauzas, gibi: would love some reviews on those patches :)09:47
bauzasmgoddard: absolutely, will do09:48
mgoddardI think some were failing unit tests, I can tidy up if someone plans to look09:48
bauzasmgoddard: how many times ironic can rebalance ?09:48
*** sapd1_x has joined #openstack-nova09:48
bauzasbecause I think that can be a problem for a shit ton of deployers using ironic09:48
mgoddardbauzas: advance warning: prepare a cup of coffee, or three. Nasty race conditions ahead09:48
mgoddardbauzas: unlimited rebalancing09:49
stephenfinlyarwood, gibi: done https://review.opendev.org/#/c/700837/09:49
bauzasin particular when it goes on top on something abstract API like k8s...09:49
mgoddardbauzas: the hash ring just consists of the active nova computes09:49
bauzasmgoddard: just tbc, ironic does rebalance if the things below it are shuffled ?09:49
gibibauzas: ack about the duplicates09:50
gibistephenfin: thanks09:50
bauzasin our case, because pods changed or whatever, right?09:50
lyarwoodstephenfin: ah thanks, ./me respins09:50
mgoddardbauzas: there are two hash rings - one in nova, one in ironic conductor. You only need to care about the one in nova09:50
bauzasmgoddard: you do confirm that https://bugs.launchpad.net/nova/+bug/1860312 sounds dup of https://bugs.launchpad.net/nova/+bug/1853009 ?09:50
openstackLaunchpad bug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete]09:50
openstackLaunchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard)09:50
mgoddardbauzas: if it's sitting on top of a wobbly k8s jelly, with containers coming and going, it's quite likely :) will look in more detail09:51
bauzasmgoddard: I'd dare say there are two hard problems in CS09:51
bauzashash rings come with the first hard problme09:52
bauzasmgoddard: that's exactly what the reporter says "09:52
bauzasI deployed openstack with openstack-helm on kubernetes.When one of the nova-compute service(driver=ironic  replica of the deployment is 1) breakdown.It may be scheduled to  another node by kubernetes.When I try to delete the old compute  service(status down), it failed."09:52
mgoddardbauzas: although in this case I think it's a different problem. They're trying to delete the compute service, not the compute node09:53
mgoddardbauzas: I'd guess it's because they're using the service ID rather than the UUID09:53
mgoddardID no longer seems to work09:54
mgoddardneed to specify an API version to get the UUID in the list09:54
bauzasmgoddard: it's a side effect09:55
bauzasmgoddard: since the virt driver stopped reporting the node, then the RT deleted the compute node record09:55
openstackgerritLee Yarwood proposed openstack/nova master: docs: Add stable device rescue docs  https://review.opendev.org/70083709:55
lyarwoodstephenfin: ^ thanks btw09:56
mgoddardbauzas: but the service should still exist, even without a node09:57
bauzasmgoddard: we're making different assumptions in different pieces of code09:57
mgoddardbauzas: I haven't seen that happen before09:57
bauzasmgoddard: in the RT, we assume that it's reasonable to not have a CN record when the virt driver tells you no about it09:58
bauzasmgoddard: but on the other hand, we assume that compute nodes exist if you delete a compute service09:58
mgoddardbauzas: if that is true, should be easy to reproduce?10:00
mgoddardand probably easy to test my hypothesis also10:00
*** Liang__ has quit IRC10:03
bauzasmgoddard: I've asked the reporter to come back with details10:03
openstackgerritMerged openstack/nova master: Use oslo policy flag to disable default change warning instead of all  https://review.opendev.org/71788410:04
bauzasI could have some OSP crazy env that would help me to verify this hypothesis but to be frank I don't have the energy for it :)10:04
bauzasmgoddard: like I said in my last comment of bug 1860312, I'd rather spend my energy on reviewing your change10:05
openstackbug 1860312 in OpenStack Compute (nova) "compute service failed to delete" [Undecided,Incomplete] https://launchpad.net/bugs/186031210:05
bauzasmgoddard: actually, the bug is pretty nasty and has a lot of other related bugs, do you think you'd be able to give it a good shape soon ?10:08
bauzasto give *the change a good shape soon ?10:08
bauzasgibi: FYI, ^10:10
bauzaswe're too close to RC1 to consider those bugs as rc-potential but they're definitely worth backportable down the road to all our stable branches10:11
bauzasactually, mriedem thought the same https://bugs.launchpad.net/nova/+bug/184148110:12
openstackLaunchpad bug 1841481 in OpenStack Compute (nova) "Race during ironic re-balance corrupts local RT ProviderTree and compute_nodes cache" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)10:12
gibibauzas: ack, I agree that we should not try to blow up RC1 with this fix.10:12
bauzasmgoddard: sorry for interupting you again, but we have around 4 open bugs for probably the same issue(s) or related10:12
bauzasmgoddard: could we consider https://bugs.launchpad.net/nova/+bug/1853009 as the primary one ?10:12
openstackLaunchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] - Assigned to Mark Goddard (mgoddard)10:13
* bauzas will mark all the stable branches as potential targets for the fix10:13
mgoddardbauzas: I'll get the patches into shape today10:14
bauzasmgoddard: as I said, we're too short in time to deliver them for RC1, but I think this should be addressed as quicky as possible after the victoria branch10:15
mgoddardbauzas: ack10:15
*** tetsuro_ has quit IRC10:17
bauzasmgoddard: fwiw, starting to review https://review.opendev.org/#/c/695012/4/nova/tests/functional/regressions/test_bug_1853009.py10:22
mgoddardbauzas: cool. I don't expect the code to change significantly, just need to adapt to upstream changes since proposed10:23
*** ttsiouts has quit IRC10:26
*** sapd1_y has joined #openstack-nova10:26
*** ttsiouts has joined #openstack-nova10:28
*** sapd1_x has quit IRC10:30
*** rpittau is now known as rpittau|bbl10:31
*** ttsiouts has quit IRC10:33
*** ttsiouts has joined #openstack-nova10:37
gibibauzas: what do you think about https://bugs.launchpad.net/nova/+bug/1872385 it is currently marked as rc potential but it the fix is not ready. I can take the fix over this afternoon and make it complete or we can drop the rc potential tag as you also noted that this is not a clear regression for Ussuri as we have child RPs for couple of releases now10:39
openstackLaunchpad bug 1872385 in OpenStack Compute (nova) "delete sub resource provider in tree" [High,In progress] - Assigned to Wenping Song (wenping1)10:39
* gibi switch to lunch operations10:40
*** slaweq_ is now known as slaweq10:43
sean-k-mooneygibi: i dont have it deploy but i have worked with dvd on https://bugs.launchpad.net/nova/+bug/1851545 downstream https://bugzilla.redhat.com/show_bug.cgi?id=176779710:47
openstackLaunchpad bug 1851545 in OpenStack Compute (nova) "Port update exception on nova unshelve for instance with PCI devices (part 2)" [Undecided,New]10:47
openstackbugzilla.redhat.com bug 1767797 in openstack-nova "When unshelving an SR-IOV instance, the binding profile isn't reclaimed or rescheduled, and this might cause PCI-PT conflicts" [High,New] - Assigned to nova-maint10:47
sean-k-mooneygibi: it looks like all my comments downstream are private but yes this is broken on queens i memtioned this seperatly before. we are not updating the port binding profile before generating the xml on unshelve10:48
sean-k-mooneyim not sure if it affect all release but its broken on queens10:48
*** sapd1 has quit IRC10:50
sean-k-mooneygibi: ill traige the upstream bug. i can proably try and replicate it later i just have not had time to work on it since it was first reported but now might be as good a time as any one i finish rebasing my current patches10:50
*** mkrai has quit IRC11:02
*** mkrai has joined #openstack-nova11:03
*** udesale has quit IRC11:09
gibisean-k-mooney: thanks a lot!11:11
*** derekh has joined #openstack-nova11:15
openstackgerritMerged openstack/nova master: Add missing white spaces between words in log messages  https://review.opendev.org/72011211:29
*** ociuhandu has quit IRC11:34
*** ociuhandu has joined #openstack-nova11:35
*** ociuhandu has quit IRC11:40
*** priteau has quit IRC11:44
*** nweinber has joined #openstack-nova11:52
openstackgerritBalazs Gibizer proposed openstack/nova master: Revert "Temporarily skip TestNovaMigrationsMySQL"  https://review.opendev.org/71863012:03
*** ratailor has quit IRC12:06
gibibauzas and others: I have to jump on a meeting marathon for the next 2 and a half hours12:15
*** links has quit IRC12:17
*** links has joined #openstack-nova12:18
openstackgerritMerged openstack/nova-specs master: Adds spec infrastructure for Victoria  https://review.opendev.org/71002312:19
*** ttsiouts has quit IRC12:25
*** brinzhang has quit IRC12:30
*** dtantsur is now known as dtantsur|brb12:30
*** rpittau|bbl is now known as rpittau12:31
*** derekh has quit IRC12:33
*** ttsiouts has joined #openstack-nova12:39
*** derekh has joined #openstack-nova12:41
*** mriedem has joined #openstack-nova12:44
*** _kukacz has joined #openstack-nova12:45
*** ociuhandu has joined #openstack-nova12:48
bauzasgibi: ack thanks, and I'll review https://review.opendev.org/#/c/719163/12:48
*** _kukacz has quit IRC12:51
*** _kukacz has joined #openstack-nova12:52
*** psachin has quit IRC12:53
*** ociuhandu has quit IRC12:58
*** ociuhandu has joined #openstack-nova13:00
*** ociuhandu has quit IRC13:01
*** ociuhandu has joined #openstack-nova13:02
openstackgerritMerged openstack/nova master: docs: Add stable device rescue docs  https://review.opendev.org/70083713:05
*** francoisp has joined #openstack-nova13:12
*** rcernin has quit IRC13:14
kashyaplyarwood: Since I've written it partially, and been vacillating on this for a while, I'll repropose this spec for V, as that is on TODO "forever" anyway.  I know you've already got plenty on your plate13:16
*** zhanglong has joined #openstack-nova13:16
kashyaplyarwood: And I'll add you to the 'collaborators' list, that sounds okay?13:17
lyarwoodack sure13:17
kashyaplyarwood: Appreciate the virtual kick in the ***13:20
kashyapA real one will also be accepted when circumstances allow in-person gathering ;-)13:20
gmanngibi: stephenfin thanks. let me squash both policy doc patches.13:22
*** toabctl has quit IRC13:25
*** toabctl has joined #openstack-nova13:27
*** zhanglong has quit IRC13:28
*** zhanglong has joined #openstack-nova13:28
*** spatel has joined #openstack-nova13:35
openstackgerritLee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type  https://review.opendev.org/70870113:43
*** lbragstad has joined #openstack-nova13:43
*** KeithMnemonic has joined #openstack-nova13:44
lyarwoodkashyap: ^ added a virt-preview job, I think swap volume might only work once we switch to block copy13:45
*** lbragstad_ has quit IRC13:45
lyarwoodkashyap: with q3513:45
* kashyap clicks13:45
kashyaplyarwood: Ah, yes.  Indeed that's true13:46
lyarwoodthat said I can't reproduce this outside of the tempest test13:49
* kashyap nods; will investigate later. Sorry, dealing with some emergency13:54
lyarwoodyup np13:55
*** mlavalle has joined #openstack-nova13:58
*** lbragstad_ has joined #openstack-nova14:04
*** lbragstad has quit IRC14:06
*** sapd1 has joined #openstack-nova14:07
*** zhanglong has quit IRC14:18
openstackgerritMark Goddard proposed openstack/nova master: Add functional regression test for bug 1853009  https://review.opendev.org/69501214:20
openstackbug 1853009 in OpenStack Compute (nova) ussuri "Ironic node rebalance race can lead to missing compute nodes in DB" [High,In progress] https://launchpad.net/bugs/1853009 - Assigned to Mark Goddard (mgoddard)14:20
openstackgerritMark Goddard proposed openstack/nova master: Clear rebalanced compute nodes from resource tracker  https://review.opendev.org/69518714:20
openstackgerritMark Goddard proposed openstack/nova master: Invalidate provider tree when compute node disappears  https://review.opendev.org/69518814:20
openstackgerritMark Goddard proposed openstack/nova master: Prevent deletion of a compute node belonging to another host  https://review.opendev.org/69480214:20
openstackgerritMark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation  https://review.opendev.org/69518914:20
mgoddardbauzas: ^14:20
mgoddardunit test fix was simple, missing a fair=True argument to the synchronized decorator in RT, added upstream14:21
bauzasmgoddard: ack, will review it later today14:25
mgoddardthanks!14:25
bauzasstephenfin: can you please help me understand what you mean by https://review.opendev.org/#/c/721548/1/releasenotes/notes/ussuri-prelude-4b96f1244cefcdf4.yaml@12 ?14:26
bauzasstephenfin: looking at https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html#hyperlinks I can't find what you mean14:27
stephenfinbauzas:14:27
stephenfin`Hello, world`__14:27
stephenfin14:27
stephenfin__ https://example.com14:27
stephenfinOr grep for '__' in releasenotes/notes14:27
bauzasokay, but that's not something related to sphinx rST right?14:27
stephenfinIt's a general rST thing, not specific to Sphinx14:28
* bauzas trying to be a better documentator :)14:28
stephenfinbauzas: https://docutils.sourceforge.io/docs/user/rst/quickref.html#indirect-hyperlink-targets14:28
bauzasok, /me goes looking at https://docutils.sourceforge.io/docs/ref/rst/restructuredtext.html14:28
bauzasheh, jinxed14:29
openstackgerritLee Yarwood proposed openstack/nova master: WIP: nova-next: Start testing the 'q35' machine type  https://review.opendev.org/70870114:29
*** sapd1 has quit IRC14:29
bauzasah, OK, I understand better : "The reference name of the reference is not used to match the reference to its target.  Instead, the order of anonymous hyperlink references and targets within the document is significant: the first anonymous reference will link to the first anonymous target.  The number of anonymous hyperlink references in a document must match the number of anonymous targets. "14:30
stephenfinYup. Named links must be unique in their document so they chance of having conflicts in release notes is reasonably high, given the volume of them we have14:31
stephenfin*so the chance14:32
*** _erlon_ has quit IRC14:40
*** _erlon_ has joined #openstack-nova14:41
*** sapd1 has joined #openstack-nova14:41
*** dtantsur|brb is now known as dtantsur14:47
stephenfindansmith: Am I wrong, or should this not happen? https://bugs.launchpad.net/nova/+bug/186803314:47
openstackLaunchpad bug 1868033 in OpenStack Compute (nova) "Booting instance with pci_device fails during rocky->stein live upgrade" [High,Confirmed] - Assigned to Stephen Finucane (stephenfinucane)14:47
stephenfinBecause of https://github.com/openstack/oslo.versionedobjects/blob/master/oslo_versionedobjects/base.py#L555-L56414:47
dansmithstephenfin: well, he's saying that it _is_ trying to backport, but it's not backporting far enough, which makes it sound like the manifest being sent it wrong or something14:50
dansmithwhich really shouldn't happen14:50
*** dklyle has joined #openstack-nova14:50
dansmithhe's wrong to equate the upgrade_level to anything with objects though -- they are unrelated14:50
stephenfinyeah, that patch isn't a suitable fix14:50
dansmithstephenfin: I wonder if it's because that object isn't calling super()14:53
dansmithand its obj_make_compat() is only handling the sub-object backporting, which shouldn't be needed anymore anyway, so maybe this one got skipped when those were culled?14:53
stephenfinAh, good spot. That's likely it14:55
stephenfinthe superclass handles ListOfObjectField and ObjectField fields explicitly14:55
dansmithright, I'm having to reboot my dev machine,14:56
dansmithbut I think you'll find most list objects don't have an explicit backport routine anymore14:56
dansmithyeah, confirmed, but let me look up the culling patch14:57
dansmithah, I think I see14:59
dansmithwe used to use that obj_relationships mapping, and obj_make_compatible was still centralized based on that map for lists15:00
dansmithbut this one didn't use that, so it was overriding from the beginning and thus didn't get caught in the grep15:00
openstackgerritSylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section  https://review.opendev.org/72154815:02
dansmithstephenfin: I think we need to get sam to test the proper fix..happy to do that unless you've already started15:02
stephenfindansmith: Nope, go for it15:02
*** mgariepy has quit IRC15:04
*** kevinbenton has quit IRC15:05
bauzasgibi: stephenfin: if you don't mind, as I said in a comment, I don't want to depend on https://review.opendev.org/#/c/720129/15:05
*** kevinbenton has joined #openstack-nova15:06
bauzaswe had a terrible experience once in Newton when we merged the prelude reno file *after* we branched to Ocata (IIRC)15:06
bauzasbecause the prelude wasn't seen in 14.0.0 then15:06
bauzasyeah, I remembered correctly : https://docs.openstack.org/releasenotes/nova/newton.html#release-notes-14-0-1-stable-newton15:07
bauzasto avoid this, you need to first merge a prelude reno file in the master, then once you branch, you can backport a fix for the file15:08
gibibauzas: OK, then lets have a preliminary prelude merged soon, and fix it when the doc is available15:13
*** mgariepy has joined #openstack-nova15:17
openstackgerritDan Smith proposed openstack/nova master: Remove stale nested backport from InstancePCIRequests  https://review.opendev.org/72166715:20
*** lbragstad_ is now known as lbragstad15:21
gibibauzas: I still not get where we use aggregate metadata for image pre-chacing feature. I think we simply use aggregates but not aggregate metadata15:27
bauzasgibi: okay, that's maybe me15:27
bauzasI may have misunderstood the API version15:27
bauzasgibi: OK, you're right, will respin15:30
bauzashttps://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/image-precache-support.html#os-aggregates15:31
gibithanks15:33
openstackgerritSylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section  https://review.opendev.org/72154815:36
gibibauzas: +2 ^^15:39
bauzasack, thanks15:39
stephenfinbauzas: think you missed something15:39
bauzaswe should hold a bit until a handful of cores and contributors have reviewed it I think15:39
bauzasahah, see15:39
bauzasstephenfin: probably, tell me what :)15:40
stephenfinI left a comment :)15:40
*** tkajinam has quit IRC15:40
gibiOK I will only vote +1 until most of the cores leave some feedback on it15:41
stephenfinbauzas: fwiw, because you're not using Sphinx'isms there, you can use a tool like rst.ninjs.org for rendering15:41
*** udesale_ has quit IRC15:41
stephenfin(so long as you remove the yaml stuff and dedent things, obviously)15:41
bauzasstephenfin: you can test locally with the releasenotes target15:44
bauzasstephenfin: that's what I did and it worked15:44
bauzasstephenfin: also replied15:44
bauzaswith the releasenotes tox* target, my bad15:44
bauzas(but remember, you need to commit first your changes in order for reno to work15:45
stephenfinAh, you're right. My mistake.15:45
bauzas<scratch, scratch> => git commit => tox -ereleasenotes (and take a coffee cup, it'll take a while to generate)15:45
stephenfinI've another point left about putting the URLs between the bullet points too /o\15:45
stephenfinThey render really weirdly15:46
stephenfinBecause it's rendering as multiple <ul>s instead of one <ul> with multiple <li>s15:46
bauzasstephenfin: weirdo, look at what the tox target generates : http://paste.openstack.org/raw/792482/15:47
bauzas(open it as an HTML file)15:48
stephenfinyeah, see how there are multiple <ul> tags?15:48
stephenfinthere should only be one15:48
bauzasok, I can fix this, I see your point15:50
stephenfinbauzas: actually, you can do this15:50
stephenfinhttp://rst.ninjs.org/#LSBgSGVsbG9gX18KCiAgLi4gX186IFRlc3QKCi0gV29ybGQKLSBUZXN015:50
stephenfinjust indent the links15:50
bauzasyeah I was thinking on indenting, not dedenting15:50
bauzashence the <ul> tags15:51
stephenfincool15:51
bauzasif I wasn't using the bullets but an ordered list, we would get 1. 1. 1.15:51
stephenfinexactly15:51
bauzasjust verifying that's the only asked nit besides your rewordings15:52
*** gyee has joined #openstack-nova15:52
*** tbachman has quit IRC15:53
bauzasstephenfin: b/c I don't wanna get a stylistic -1 again, you okay with me saying : to configure `read-only` access15:56
*** tbachman has joined #openstack-nova15:56
bauzasand not : to configure "read-only" access15:56
bauzas(using italics instead of brakets)15:56
stephenfin*read-only* would be better15:56
bauzasbold, dude ?15:56
bauzasman, you're gross15:56
stephenfinnope, that's **read-only**15:57
* bauzas facepalms15:57
stephenfinthis ain't markdown, my man15:57
* bauzas learned German, English and a bit of Spanish, but apologies for having missed rST15:57
stephenfin:P15:57
stephenfinsingle backticks mean default role15:57
stephenfinwhich is configurable15:57
stephenfinin most environment it defaults to italics15:58
bauzasah-ha I see15:58
* stephenfin learned that the hard way writing docs for Sphinx itself :(15:58
bauzasthere is a good reason why it's named **Sphinx** AFAICT15:58
bauzasthey could have subtitled it 'old greek'15:59
bauzas(and while we speak, it's building relnotes locally...16:00
*** ttsiouts has quit IRC16:04
bauzas(still building)16:05
bauzashold my beer16:05
*** mkrai has quit IRC16:05
*** ttsiouts has joined #openstack-nova16:05
bauzasmgoddard: I haven't forgotten you, but my time was on working for release notes16:06
stephenfinwe're going to need to start compressing release notes again at some point16:06
stephenfinlike we did for e.g. mitaka16:06
mgoddardbauzas: I've been waiting long enough, I can wait a little longer :)16:06
bauzasstephenfin: that's not the problem16:06
bauzasstephenfin: it does scan all my local branches16:06
bauzasthis is what takes a while16:06
bauzasI should drop the local stable branches, that's it16:07
stephenfinnah, it'll drag them in from the remote16:08
stephenfiniirc16:08
bauzasmaybe, that's been a while I haven't really looked deep at the reno code itself16:09
bauzas(Ocata I think)16:09
*** mkrai has joined #openstack-nova16:10
openstackgerritSylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section  https://review.opendev.org/72154816:14
*** _kukacz has quit IRC16:14
bauzaswow16:14
bauzas5 mins to upload, thanks Netflix16:14
*** ociuhandu has quit IRC16:15
*** mkrai has quit IRC16:15
*** ociuhandu has joined #openstack-nova16:17
*** ociuhandu has quit IRC16:20
*** ociuhandu has joined #openstack-nova16:21
*** yaawang_ has joined #openstack-nova16:32
openstackgerritElod Illes proposed openstack/nova stable/rocky: WIP: libvirt: check job status for VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED event  https://review.opendev.org/71123316:32
*** jraju__ has joined #openstack-nova16:32
*** links has quit IRC16:33
*** yaawang has quit IRC16:33
*** evrardjp has quit IRC16:35
*** evrardjp has joined #openstack-nova16:35
*** lseki has quit IRC16:38
*** happyhemant has quit IRC16:38
*** rpittau is now known as rpittau|afk16:39
*** masayukig has quit IRC16:39
*** nightmare_unreal has quit IRC16:40
*** happyhemant has joined #openstack-nova16:40
*** jrosser has quit IRC16:41
*** klindgren has quit IRC16:41
*** klindgren has joined #openstack-nova16:41
*** mnaser has quit IRC16:41
*** lseki has joined #openstack-nova16:42
*** maciejjozefczyk has quit IRC16:42
*** bcm has quit IRC16:42
*** TheJulia has quit IRC16:42
*** csatari has quit IRC16:43
*** masayukig has joined #openstack-nova16:43
*** jrosser has joined #openstack-nova16:43
*** bcm has joined #openstack-nova16:44
*** TheJulia has joined #openstack-nova16:44
*** nightmare_unreal has joined #openstack-nova16:44
*** mnaser has joined #openstack-nova16:44
*** csatari has joined #openstack-nova16:45
*** vdrok has quit IRC16:45
*** mnaser has quit IRC16:46
*** mnaser has joined #openstack-nova16:47
*** vdrok has joined #openstack-nova16:47
*** derekh has quit IRC16:48
*** mnaser has quit IRC16:49
*** mnaser has joined #openstack-nova16:49
*** jraju__ has quit IRC16:57
*** dtantsur is now known as dtantsur|afk16:59
*** ajitha has joined #openstack-nova17:01
ajithahi all, we have running a openstack CI for cinder.. with ubuntu 18.04 and py3.6... After this commit https://opendev.org/openstack/nova/commit/f1e448d6729f674101b34b08e3c5e95fd8f75770 , our CI/CD pipeline is getting break in devstack with following error "pip._vendor.pkg_resources.ContextualVersionConflict: (oslo.policy 3.0.3 (/usr/local/lib/python3.6/dist-packages), Requirement.parse('oslo.policy>=3.1.0'), {'nova'}).."17:05
ajithahow to suppress this from local.conf?17:06
ajithaor any method to send suppress_deprecation_warnings=True from local.conf17:07
stephenfinajitha: Are you using system packages?17:08
stephenfinPython packages, that is17:09
ajithai am using python packages mostly...you are asking for oslo.policy?17:09
stephenfinYeah, is this just a bare Ubuntu 18.04 host with DevStack on it?17:10
stephenfinI'm checking whether you've installed oslo.policy from pip or apt17:10
ajithayes bare ubuntu 18.04 with devstack17:11
ajithafrom pip only not from apt17:11
stephenfinokay, can you paste the full traceback (paste.openstack.org)17:11
stephenfin?17:12
ajithasure17:12
ajithathanks17:12
openstackgerritGhanshyam Mann proposed openstack/nova master: Fix the followup comment of policy doc  https://review.opendev.org/72132217:13
*** nightmare_unreal has quit IRC17:13
ajithastephenfin: http://paste.openstack.org/show/792492/17:21
ajithapaste.openstack didnt allow to  paste the complete log..17:22
gmannstephenfin: fixed the comment in this to compare those. merging the same in original patch now https://review.opendev.org/72132217:22
stephenfinajitha: I suspect you're hitting https://github.com/pypa/pip/issues/98817:25
stephenfinand unfortunately I don't have any easy answers, you need something to install oslo.config >= 3.1.0 before 3.0.3 gets installed17:26
ajithamanual devstack execution works fine with an error throwing as ERROR: nova 20.1.0.dev1681 has requirement oslo.policy>=3.1.0, you'll have oslo-policy 3.0.3 which is incompatible... but it skips that and devstack gets completed.   and while executing devstack-gate jenkins job, devstack gets error at the above point17:26
stephenfinWe're not hitting that in the nova gate so I suspect it's something to do with a plugin that's installing a package that depends on oslo.config < 3.1.017:27
stephenfina plugin in your CI environment, that is17:27
stephenfinthe folks on #openstack-infra might be more help; they're well used to dealing with pip's brokenness /o\17:28
ajithathank you.17:29
openstackgerritGhanshyam Mann proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh  https://review.opendev.org/72012917:29
ajithait works with py3.6 too??17:29
stephenfinthe gate? yeah, I don't think this is a py3.6 issue17:30
stephenfinit's a pip issue, I'd say17:30
openstackgerritGhanshyam Mann proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh  https://review.opendev.org/72012917:30
ajithastephenfin : thanks. The error is throwing in the manual devstack execution(./stack.sh) but it skips and complete the devstack.. In case of devstack-gate, it just stops at that point17:32
*** ociuhandu has quit IRC17:34
*** ociuhandu has joined #openstack-nova17:35
*** ociuhandu has quit IRC17:40
*** tesseract has quit IRC17:59
*** hamzy__ has quit IRC17:59
*** sapd1 has quit IRC18:05
gmannstephenfin: gibi this is ready now - https://review.opendev.org/#/c/720129/1118:15
*** ralonsoh has quit IRC18:16
*** ociuhandu has joined #openstack-nova18:24
*** gmann is now known as gmann_lunch18:31
*** ociuhandu has quit IRC18:44
*** ociuhandu has joined #openstack-nova18:45
*** ociuhandu has quit IRC18:45
*** ociuhandu has joined #openstack-nova18:45
*** jangutter has quit IRC18:49
*** gmann_lunch is now known as gmann18:51
*** belmoreira has quit IRC18:53
*** jangutter has joined #openstack-nova18:59
*** ociuhandu has quit IRC19:11
*** ociuhandu has joined #openstack-nova19:11
*** slaweq has quit IRC19:14
*** ociuhandu has quit IRC19:17
*** slaweq has joined #openstack-nova19:19
*** ttsiouts has quit IRC19:21
*** ttsiouts has joined #openstack-nova19:25
*** grandchild has joined #openstack-nova19:32
*** ccamacho has quit IRC19:34
*** breizhkoala has quit IRC19:35
*** martinkennelly has quit IRC19:36
*** ociuhandu has joined #openstack-nova19:48
*** ociuhandu has quit IRC19:53
*** ajitha has quit IRC20:02
*** mgariepy has quit IRC20:09
*** mgariepy has joined #openstack-nova20:12
*** xek has quit IRC20:30
*** nweinber has quit IRC20:39
*** ociuhandu has joined #openstack-nova21:03
*** ociuhandu has quit IRC21:08
*** dlbewley has joined #openstack-nova21:10
*** bbowen has quit IRC21:15
*** bbowen has joined #openstack-nova21:16
*** spatel has quit IRC21:30
*** dpawlik has quit IRC21:34
*** sean-k-mooney has quit IRC21:40
*** slaweq has quit IRC21:42
*** sean-k-mooney has joined #openstack-nova21:43
*** mriedem has left #openstack-nova21:45
*** ttsiouts has quit IRC21:46
*** ttsiouts has joined #openstack-nova22:02
*** rcernin has joined #openstack-nova22:17
*** ttsiouts has quit IRC22:46
*** tkajinam has joined #openstack-nova22:53
*** spatel has joined #openstack-nova22:53
*** tosky has quit IRC23:03
*** avolkov has quit IRC23:25
*** ociuhandu has joined #openstack-nova23:25
*** ociuhandu has quit IRC23:30
*** grandchild has quit IRC23:36
*** mlavalle has quit IRC23:37
*** tetsuro has joined #openstack-nova23:42
*** tetsuro has quit IRC23:44

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!