Thursday, 2020-01-16

*** macz has quit IRC00:16
*** macz has joined #openstack-nova00:17
*** cz3 has quit IRC00:18
*** TheJulia has quit IRC00:18
*** donnyd has quit IRC00:18
*** bcm has quit IRC00:18
*** rajinir has quit IRC00:18
*** dave-mccowan has quit IRC00:19
*** rajinir has joined #openstack-nova00:19
*** bcm has joined #openstack-nova00:19
*** donnyd has joined #openstack-nova00:20
*** cz3 has joined #openstack-nova00:20
*** vishalmanchanda has joined #openstack-nova00:20
*** TheJulia has joined #openstack-nova00:22
*** macz has quit IRC00:22
*** csatari has joined #openstack-nova00:23
*** TxGirlGeek has quit IRC00:26
*** rpittau|afk has joined #openstack-nova00:35
*** bauzas has quit IRC00:36
*** dave-mccowan has joined #openstack-nova00:37
*** bauzas has joined #openstack-nova00:38
*** rpittau|afk has quit IRC00:42
*** knikolla has quit IRC00:42
*** samc-bbc has quit IRC00:42
*** csatari has quit IRC00:42
*** rm_work has quit IRC00:43
*** donnyd has quit IRC00:43
*** ildikov has quit IRC00:43
*** gmann has quit IRC00:43
*** vishalmanchanda has quit IRC00:43
*** TheJulia has quit IRC00:44
*** vdrok has quit IRC00:44
*** cz3 has quit IRC00:44
*** rajinir has quit IRC00:44
*** arne_wiebalck has quit IRC00:45
*** _erlon_ has quit IRC00:46
*** mnasiadka has quit IRC00:46
*** awestin1 has quit IRC00:46
*** jungleboyj has quit IRC00:46
*** dustinc has quit IRC00:46
*** portdirect has quit IRC00:46
*** coreycb has quit IRC00:46
*** guilhermesp has quit IRC00:47
*** tonyb[m] has quit IRC00:47
*** jrosser has quit IRC00:48
*** NobodyCam has quit IRC00:48
*** masayukig has quit IRC00:49
*** mnaser has quit IRC00:49
*** johnsom has quit IRC00:52
*** Liang__ has joined #openstack-nova00:55
openstackgerritMerged openstack/nova master: libvirt: Remove MIN_QEMU_FILE_BACKED_DISCARD_VERSION  https://review.opendev.org/69591500:58
*** zhanglong has joined #openstack-nova01:01
*** Li_Liu has quit IRC01:01
*** pas-ha has quit IRC01:02
*** bcm has quit IRC01:06
*** csatari has joined #openstack-nova01:16
*** mnasiadka has joined #openstack-nova01:18
*** csatari has quit IRC01:22
*** csatari has joined #openstack-nova01:24
*** dave-mccowan has quit IRC01:26
*** mnasiadka has quit IRC01:29
*** gyee has quit IRC01:30
*** csatari has quit IRC01:31
*** xiaolin has joined #openstack-nova01:33
*** xiaolin has joined #openstack-nova01:35
xiaolinhello guys, I'm trying to run tempest test "rally verify start --pattern set=volume", but got an error "Forbidden: Policy doesn't allow os_compute_api:os-assisted-volume-snapshots:create to be performed. (HTTP 403)", can anyone give me some advice ?01:39
xiaolinwhen i modify base.RULE_ADMIN_API to base.RULE_ANY in /usr/lib/python2.7/site-packages/nova/policies/assisted_volume_snapshots.py, the test passed01:39
xiaolinIs this permission setting reasonable?01:39
*** macz has joined #openstack-nova01:41
*** igordc has quit IRC01:42
*** macz has quit IRC01:45
*** Dinesh_Bhor has joined #openstack-nova01:47
openstackgerritwangjiajing proposed openstack/nova master: Optimize the comments for the "test_resize_cross_cell_weigher_filtered_to_target_cell_by_spec" method.  https://review.opendev.org/70278101:59
alex_xuxiaolin: that API I thought should be called by cinder side, probably can go to check the cinder side's nova config, whether it is admin user02:05
*** yedongcan has joined #openstack-nova02:16
*** Liang__ is now known as LiangFang02:32
*** jamesdenton has quit IRC02:40
*** jamesdenton has joined #openstack-nova02:43
*** mkrai_ has joined #openstack-nova03:12
*** knikolla has joined #openstack-nova03:18
*** zhanglong has quit IRC03:21
*** knikolla has quit IRC03:24
*** zhanglong has joined #openstack-nova03:27
*** abhishekk has quit IRC03:31
*** abhishekk has joined #openstack-nova03:32
*** macz has joined #openstack-nova03:34
*** psachin has joined #openstack-nova03:36
*** mkrai_ has quit IRC03:38
*** macz has quit IRC03:39
*** knikolla has joined #openstack-nova03:39
*** mkrai_ has joined #openstack-nova03:42
*** knikolla has quit IRC03:44
*** zhanglong has quit IRC03:47
*** bhagyashris has joined #openstack-nova03:48
*** zhanglong has joined #openstack-nova03:50
*** knikolla has joined #openstack-nova04:03
*** udesale has joined #openstack-nova04:05
*** zhanglong has quit IRC04:09
*** knikolla has quit IRC04:09
*** zhanglong has joined #openstack-nova04:10
*** openstackgerrit has quit IRC04:12
*** jungleboyj has joined #openstack-nova04:14
*** yaawang has quit IRC04:14
*** zhanglong has quit IRC04:15
*** jungleboyj has quit IRC04:20
*** masayukig has joined #openstack-nova04:23
*** masayukig has quit IRC04:26
*** openstackgerrit has joined #openstack-nova04:33
openstackgerritBrin Zhang proposed openstack/nova master: FUP Populate the 'updated_at' when aggregate metadata is updated  https://review.opendev.org/70279004:33
*** tkajinam has quit IRC04:39
*** tkajinam has joined #openstack-nova04:41
*** tkajinam_ has joined #openstack-nova05:03
*** tkajinam has quit IRC05:07
*** samc-bbc has joined #openstack-nova05:09
*** yaawang has joined #openstack-nova05:11
*** samc-bbc has quit IRC05:20
*** macz has joined #openstack-nova05:22
*** macz has quit IRC05:27
*** ociuhandu has joined #openstack-nova05:31
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-nova05:34
*** ociuhandu has quit IRC05:35
*** yedongcan has quit IRC05:52
*** yedongcan has joined #openstack-nova05:53
*** ratailor has joined #openstack-nova06:08
*** mkrai_ has quit IRC06:13
*** mkrai_ has joined #openstack-nova06:13
LiangFanghi folks, regarding the volume local cache spec. https://review.opendev.org/#/c/689070/06:15
LiangFangI'm thinking to simply the logic. If the compute node have cache capability, then leverage it. If not have the capability, then just go ahead without caching.06:16
LiangFangWe can use aggregate to group the servers that has cache capability06:17
LiangFangthis way is not perfect for every scenarios, e.g. only support libvirt, not schedule by flavor. But in some private cloud, it is simple and useful.06:20
*** rajinir has joined #openstack-nova06:23
LiangFangin the cinder spec https://review.opendev.org/#/c/684556/, lots of concerns if we fail the attach volume operation06:25
*** cz3 has joined #openstack-nova06:30
*** rajinir has quit IRC06:30
*** cz3 has quit IRC06:36
*** gmann has joined #openstack-nova06:48
*** cz3 has joined #openstack-nova06:48
*** vishalmanchanda has joined #openstack-nova06:49
*** mnaser has joined #openstack-nova06:49
*** donnyd has joined #openstack-nova06:52
*** rajinir has joined #openstack-nova06:52
*** rm_work has joined #openstack-nova06:53
*** rm_work has quit IRC06:53
*** Li_Liu has joined #openstack-nova06:54
*** portdirect has joined #openstack-nova06:59
*** csatari has joined #openstack-nova07:00
*** vdrok has joined #openstack-nova07:00
*** pcaruana has joined #openstack-nova07:00
*** johnsom has joined #openstack-nova07:02
*** coreycb has joined #openstack-nova07:02
*** jrosser has joined #openstack-nova07:03
*** TheJulia has joined #openstack-nova07:03
*** dustinc has joined #openstack-nova07:03
*** masayukig has joined #openstack-nova07:05
*** pas-ha has joined #openstack-nova07:05
*** _erlon_ has joined #openstack-nova07:06
*** awestin1 has joined #openstack-nova07:07
*** knikolla has joined #openstack-nova07:11
*** jungleboyj has joined #openstack-nova07:11
*** mnasiadka has joined #openstack-nova07:14
*** bcm has joined #openstack-nova07:15
*** NobodyCam has joined #openstack-nova07:16
*** rpittau|afk has joined #openstack-nova07:19
*** samc-bbc has joined #openstack-nova07:22
*** rchurch has quit IRC07:23
*** rcernin has quit IRC07:25
*** arne_wiebalck has joined #openstack-nova07:25
*** ildikov has joined #openstack-nova07:25
*** tonyb[m] has joined #openstack-nova07:31
*** osmanlicilegi has joined #openstack-nova07:47
*** osmanlicilegi has left #openstack-nova07:47
*** guilhermesp has joined #openstack-nova07:48
*** rm_work has joined #openstack-nova07:48
*** rm_work has quit IRC07:49
*** rm_work has joined #openstack-nova07:49
openstackgerritMarek Lyčka proposed openstack/nova-specs master: Enable REST quiescing  https://review.opendev.org/70281007:51
*** damien_r has joined #openstack-nova08:00
*** bhagyashris is now known as bhagyashris|NHO08:03
*** awalende has joined #openstack-nova08:05
openstackgerritMarek Lyčka proposed openstack/nova-specs master: Enable REST quiescing  https://review.opendev.org/70281008:07
*** yedongcan has quit IRC08:07
*** slaweq has joined #openstack-nova08:08
*** yedongcan has joined #openstack-nova08:09
*** tesseract has joined #openstack-nova08:10
*** xiaolin has quit IRC08:10
lyarwoodefried: ack thanks for that, I missed that another trait had or at least was about to land.08:21
*** maciejjozefczyk has joined #openstack-nova08:22
*** xiaolin has joined #openstack-nova08:22
*** udesale_ has joined #openstack-nova08:25
openstackgerritMarek Lyčka proposed openstack/nova-specs master: Enable REST quiescing  https://review.opendev.org/70281008:26
*** udesale has quit IRC08:28
*** rpittau|afk is now known as rpittau08:31
*** tosky has joined #openstack-nova08:32
*** ociuhandu has joined #openstack-nova08:33
*** ociuhandu has quit IRC08:33
*** jcosmao has joined #openstack-nova08:33
*** ociuhandu has joined #openstack-nova08:34
*** iurygregory has joined #openstack-nova08:36
*** alex_xu has quit IRC08:40
*** alex_xu has joined #openstack-nova08:40
*** zhanglong has joined #openstack-nova08:42
gibidansmith: hi! stephenfin would like you to look at one of the qos patch https://review.opendev.org/#/c/702261/ and if you are there already the rest is also ready for review08:47
*** ociuhandu_ has joined #openstack-nova08:48
*** ociuhandu has quit IRC08:51
*** ralonsoh has joined #openstack-nova08:52
*** sapd1 has quit IRC08:55
*** sapd1_ has joined #openstack-nova08:55
*** ociuhandu_ has quit IRC08:58
*** ociuhandu has joined #openstack-nova08:59
*** udesale_ has quit IRC09:04
*** xek_ has joined #openstack-nova09:11
openstackgerritMarek Lyčka proposed openstack/nova-specs master: Enable REST quiescing  https://review.opendev.org/70281009:14
stephenfinefried: Probably would have waited til https://review.opendev.org/#/c/702440/ was merged to mark that BP as complete, personally - I'd like to get the whole thing closed out09:22
stephenfinbut hey, the hardest work is done09:22
*** LiangFang has quit IRC09:28
*** elod has quit IRC09:28
*** ccamacho has joined #openstack-nova09:29
openstackgerritRiccardo Pittau proposed openstack/nova master: Fix typos for update_available_resource reference  https://review.opendev.org/70260209:31
stephenfinyo, sean-k-mooney, looks like you added https://github.com/openstack/nova/blob/b44b540fc70504f3869ef23022642095de0ea99e/nova/virt/libvirt/vif.py#L299-L30109:39
stephenfinsean-k-mooney: but that function has existed since it was removed in commit 92323586b5d03b31c18657ad64646a8ce3b8a74209:41
stephenfinsean-k-mooney: so that means that code is not being called by anything yet. In either case, want to work on the fix or will I?09:42
stephenfinsean-k-mooney: fwiw, I'm running pytype over the code base to see what it reveals. That's one of two actual issues it has found so far09:42
openstackgerritMarek Lyčka proposed openstack/nova-specs master: Enable REST quiescing  https://review.opendev.org/70281009:42
*** martinkennelly has joined #openstack-nova09:47
*** jangutter has joined #openstack-nova09:48
*** elod has joined #openstack-nova09:53
kashyapstephenfin: Hey, any clue how to resolve this "duplicate label" thing?:10:00
kashyapWarning, treated as error:10:00
kashyap/home/zuul/src/opendev.org/openstack/nova-specs/doc/source/specs/ussuri/approved/allow-secure-boot-for-qemu-kvm-guests.rst:72:duplicate label proposed change, other instance in /home/zuul/src/opendev.org/openstack/nova-specs/doc/source/specs/train/approved/allow-secure-boot-for-qemu-kvm-guests.rst10:00
kashyap---10:01
*** zhanglong has quit IRC10:01
stephenfinkashyap: Use anonymous labels instead10:04
stephenfin`foo`__10:04
stephenfin__ https://example.com/10:04
stephenfin(double underscore)10:04
kashyapstephenfin: Ah-ha!  you, sir, are the Main Man!10:04
kashyapOkay, good to know: "... However, this feature [Anonymous Hyperlinks] is easily abused and can result in unreadable plaintext and/or unmaintainable documents. Caution is advised."10:07
kashyapstephenfin: Oh, wait --10:08
*** zbr is now known as zbr|rover10:09
kashyapstephenfin: I see that anonymous labels are only for URLs10:09
kashyapstephenfin: I'm trying to refer to a section in the _same_ document; thus no URL here10:09
kashyapHere is a small concrete example: http://paste.openstack.org/show/788455/10:11
* kashyap just changes the label name, and sticks with existing syntax; as anonymous labels don't seem to work in this case.10:15
kashyapOkay, that works.10:20
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" for Ussuri  https://review.opendev.org/69384410:20
*** rpittau is now known as rpittau|bbl10:21
*** jawad_axd has joined #openstack-nova10:25
*** udesale has joined #openstack-nova10:38
*** macz has joined #openstack-nova10:46
*** macz has quit IRC10:50
*** udesale has quit IRC11:14
*** jangutter has quit IRC11:29
*** artom has quit IRC11:38
*** yedongcan has left #openstack-nova11:41
*** panda is now known as panda|drappt11:46
*** bhagyashris|NHO is now known as bhagyashris12:02
*** dtantsur|afk is now known as dtantsur12:07
*** mrch_ has joined #openstack-nova12:09
*** xek__ has joined #openstack-nova12:13
*** xek_ has quit IRC12:16
*** rpittau|bbl is now known as rpittau12:25
openstackgerritwaleed mousa proposed openstack/os-vif master: Add delete_netdev field in VIFPortProfileOpenVSwitch profile  https://review.opendev.org/70285712:29
*** macz has joined #openstack-nova12:35
openstackgerritwaleed mousa proposed openstack/os-vif master: Add delete_netdev field in VIFPortProfileOpenVSwitch profile  https://review.opendev.org/70285712:38
*** macz has quit IRC12:39
*** tetsuro has quit IRC12:45
openstackgerritKashyap Chamarthy proposed openstack/nova-specs master: Re-propose "Secure Boot support for KVM & QEMU guests" for Ussuri  https://review.opendev.org/69384412:53
*** ratailor has quit IRC12:58
openstackgerritIury Gregory Melo Ferreira proposed openstack/nova master: Add ironic hypervisor doc  https://review.opendev.org/70199813:00
*** mkrai_ has quit IRC13:06
*** mgariepy has quit IRC13:10
*** mgariepy has joined #openstack-nova13:13
*** dviroel has joined #openstack-nova13:21
*** panda|drappt is now known as panda13:25
efriedstephenfin: I'm confused. Is https://review.opendev.org/#/c/702440/ related to mox somehow?13:36
efriedI marked the mox blueprint complete, not the nova-net one.13:36
efried...at least that's what I meant to do...13:37
efriedlyarwood: I don't think you missed anything; that other trait merged later. I just wanted to pick it up in the same release since the timing worked out.13:38
*** vesper11 has quit IRC13:42
*** vesper11 has joined #openstack-nova13:42
*** artom has joined #openstack-nova13:47
*** eharney has quit IRC13:47
*** artom has quit IRC13:52
*** udesale has joined #openstack-nova13:52
efriedstephenfin: tidy docs patch for the ironic virt driver, should be an easy +A for you: https://review.opendev.org/#/c/701998/ (Note that more is possible, but the information there is accurate afaict and it has nod from ironic cores)13:54
*** Liang__ has joined #openstack-nova14:01
*** ociuhandu has quit IRC14:02
*** dtantsur is now known as dtantsur|brb14:03
*** ociuhandu has joined #openstack-nova14:06
iurygregoryefried, thanks for the +2 =)14:08
efriediurygregory: yahyoubetcha. Appreciate the patch.14:09
iurygregory\o/14:09
efrieddansmith: I started to look at https://review.opendev.org/#/c/580271/ and successor and they seem fine, but I don't have the confidence to approve. Would you please cast your discerning eye? Should be an easy +A for you, if it's right.14:14
*** ociuhandu has quit IRC14:15
*** nweinber has joined #openstack-nova14:16
*** ociuhandu has joined #openstack-nova14:17
*** bhagyashris has quit IRC14:18
*** ociuhandu has quit IRC14:24
dansmithefried: so I've always been told we shouldn't mess with those fields from python land, so I'm not confident to +W that either14:25
efriedglad I asked14:25
dansmithI would think we'd want someone from oslo db land to say they're comfortable with them14:25
dansmithmaybe stephenfin has already consulted the expert or is an expert himself since he's already +2?14:25
stephenfinI was expecting SQLA to handle those fields for us, tbh, but I couldn't find anything that suggested it would14:26
*** psachin has quit IRC14:26
stephenfinI didn't think about looking in oslo.db though /o\ /me goes to investigate14:28
dansmiththe update *is* handled by SQLA fwiw14:30
*** awalende has quit IRC14:30
sean-k-mooneydansmith: efried it looks like there are some issues with the arq binding on the cyborg side http://paste.openstack.org/show/788470/14:30
stephenfinYup, have spotted the use of 'TimestampMixin' in NovaBase14:31
*** awalende has joined #openstack-nova14:31
*** derekh has joined #openstack-nova14:31
efriedgibi: I'd like to queue up some review time from you for the provider config file bp if possible please. It's pretty straightforward stuff, kind of fun to look at IMO. It's very close to ready, if you want to preview: https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/provider-config-file14:32
*** eharney has joined #openstack-nova14:32
gibiefried: ack, tomorrow I can start the day with that14:32
efriedgreat, thank you.14:32
dansmithsean-k-mooney: ack14:32
belmoreiraHi, we are hitting https://review.opendev.org/#/c/684375/3 can we get an agreement on how to proceed? kashyap stephenfin14:33
*** pcaruana has quit IRC14:33
efriedsean-k-mooney: are other nova api comms working in that env? Could it just be a creds-in-config thing?14:33
sean-k-mooneyill try booting a normal vm14:34
sean-k-mooneythat shoudl trigger the neutron events14:34
*** awalende has quit IRC14:35
sean-k-mooneyya that booted fine14:35
efriedsean-k-mooney: point would be cyborg.conf's creds to [nova]14:35
sean-k-mooneyso it could be14:35
sean-k-mooneynova]14:36
sean-k-mooneyproject_domain_name = Default14:36
sean-k-mooneyproject_name = service14:36
sean-k-mooneyuser_domain_name = Default14:36
sean-k-mooneypassword = password14:36
sean-k-mooneyusername = nova14:36
sean-k-mooneyauth_url = http://192.168.135.77/identity14:36
sean-k-mooneyauth_type = password14:36
sean-k-mooneythat looks fine to me14:36
sean-k-mooneywell fine for devstack14:37
stephenfinbelmoreira: Should have been more exact about that. I'm not happy with changing the default for everyone to work around a temporary blip with a buggy kernel (which is what that appears to be)14:37
sean-k-mooneywe should not proceed with that patch unless we really really reallly have too14:38
stephenfinsean-k-mooney: are you referring to the hugepages one or something else?14:38
efriedstephenfin: would you be okay if we implemented the [workaround] opt as suggested?14:39
sean-k-mooneythe mempages mode=preferred change14:39
stephenfinefried: That depends on when the bug was introduced and when it was fixed. Latter seems to be 4.19 but kashyap's notes don't appear to capture the former14:40
belmoreirasean-k-mooney what's your main concern?14:40
stephenfinnot sean-k-mooney but I'm going to guess potentially huge performance penalties if you haven't correctly configured your host14:40
sean-k-mooneyso i originally suggested perfered to kashyap as a potentil solution but im more conserned about non hugepage guest that use hw:numa_nodes=*14:42
kashyapbelmoreira: Hi14:42
* kashyap reads 14:42
sean-k-mooneyfor hugepage we pin the pages seperately so they will be allocated form the numa node we expect14:42
sean-k-mooneyhowever setting perfered by default means we can be sure that the memory of a gues that has a numa doploty but is not backed by hugepage will come form the numa node we expect14:43
sean-k-mooneywhich will cause issue if we ever model numa in placment14:43
sean-k-mooneyi suspect that most of the time it would likely be fine but im not sure what all the sideeffect will be14:45
sean-k-mooneybelmoreira: have you tested the patch and conrimed that it fixes the issue?14:45
sean-k-mooneybelmoreira: we were not able to get our customer to do that and we did not have a plathform where we could repoduce the issue14:45
sean-k-mooneybelmoreira: so we dont acatully know that will fix it although we suspect it will14:45
kashyapstephenfin: I don't know when exactly it was introduced; I need to do kernel commit message digging (if it's not there, then worse, do bisection myself or ask a KVM dev)14:46
sean-k-mooneystephenfin: belmoreira i would be more ok with his as a workaround config option and keeping the default to strict with the expection we might remvoe it in the future.14:47
kashyapstephenfin: On forcing distros to backport — normally, that would be my approach such kernel cases.  But here, we need to bear in mind all the existing OpenStack deployments in the wild.14:48
* kashyap pulls the KVM tree; nuked it due to space reasons14:49
sean-k-mooneyi think this issue is only going to show up on centos/rhel maybe suse14:49
kashyapsean-k-mooney: I think they'd be the distros that are most likely to _backport_ those14:49
*** mkrai_ has joined #openstack-nova14:49
sean-k-mooneyif you are using the hwe kernel on ubunutu 18.04 you will be on 5.x14:50
kashyapRight14:50
*** macz has joined #openstack-nova14:50
sean-k-mooneyi supose if you stuck with the orginal ubuntu 18.04 kernel that was 4.1514:50
kashyapbelmoreira: I take it that rebooting the host kernel is not an option in your deployments?14:52
sean-k-mooneybelmoreira: o i see you noted that the patch did solve the problem14:52
kashyapYeah, he tested it14:52
kashyapsean-k-mooney: And in general, 'preferred' is what we recommend _anyway_, no?14:53
sean-k-mooneyno14:53
sean-k-mooneywe recommend strict14:54
kashyapI have to remind myself of this everytime.14:54
sean-k-mooneyprefered makes the memory allcoation of non hugepage memory a hint not a requirement14:54
kashyap"Where, 'preferred' means: disable NUMA affinity and so the14:54
kashyaprequest is turned to a "hint", i.e. "if possible, allocate from the14:54
kashyapgiven NUMA node; otherwise, fallback to other nodes"14:54
kashyapYes14:54
*** macz has quit IRC14:55
sean-k-mooneywhich will break our numa aware memory tracking14:55
sean-k-mooneyand will not work if we model memory in plamcent in the future per numa node14:55
sean-k-mooneysince we can rely on the memory being allcoated form the numa node we said14:55
sean-k-mooneyfor hugepage memeory the option will be ignored so hugepage backed vms will be fine14:55
kashyapYeah, then I don't want to muck with defaults.  With all these future caveats (which are not properly documented.)14:56
sean-k-mooneybut vm with a numa toplogy but no hugepages could have there memory allcoated from any numa node if the one we select was full14:56
* kashyap needs to head into a meeting; bbiab14:56
sean-k-mooneystephenfin: kashyap belmoreira i left a comment on the patch14:58
*** jawad_axd has quit IRC15:07
*** jawad_axd has joined #openstack-nova15:08
*** jawad_ax_ has joined #openstack-nova15:12
*** jawad_axd has quit IRC15:13
*** mkrai_ has quit IRC15:13
belmoreirasorry, needed to leave the office15:14
belmoreirasean-k-mooney yes, I can reproduce with centos 7.6 kernel: Linux XXX 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux15:15
belmoreiralibvirtd 4.5.0-2315:15
belmoreiraqemu 2.12.0-3315:16
belmoreirachanging to "preferred" fix it (can create new instances)15:16
*** jawad_ax_ has quit IRC15:16
sean-k-mooneyya as far as i know this is not fixed in 7.615:16
sean-k-mooneyi think the fix should be in 7.815:17
belmoreirakashyap we upgraded to centos 7.7 but didn't reboot the compute nodes. It's a very painful process15:17
sean-k-mooneyit might be in 7.715:17
sean-k-mooneybelmoreira: are you ok with it if we make it a workaround config option and default to strict15:18
belmoreirasean-k-mooney for now I can't reproduce it with 7.715:18
sean-k-mooneyok cool then that likely already has the backport15:18
sean-k-mooneyalthough this is hardware specific15:18
sean-k-mooneyso if its a differen model fo server then you migh not see it15:18
belmoreirasean-k-mooney yes, took me some time to isolate this15:19
sean-k-mooneyya  it took us a day or two to figure this out downstream too when it was first reported to us15:20
sean-k-mooneyits a very rare edgecase15:20
sean-k-mooneynormlaly you will have dma regions per socket15:20
sean-k-mooneyor numa node15:21
belmoreiraI think that the workaround config will be useful for everyone that is running NUMA aware VMs. Then needs to be backported to older releases15:21
sean-k-mooneyso you need a server that does not + requst a numa toploy + have a kernel with the kvm bug15:21
sean-k-mooneywell no it will be useful for the minority that have ^15:22
sean-k-mooneybut i would be ok with proceeding on that basis15:22
belmoreirasean-k-mooney ok, true :)15:22
belmoreirabut also shows that the nova community is aware of the issue and has a work around15:24
belmoreirasean-k-mooney thanks15:24
kashyapbelmoreira: We're very much aware, FWIW, see this response on 'openstack-discuss' list: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009770.html15:25
belmoreirasean-k-mooney but to clarify, "preferred" would break the future placement tracking and this should be disable as soon the new fixed kernel is loaded15:26
sean-k-mooneythat placement tracking proably wont be a thing until V15:26
sean-k-mooneyi could mess up our tracking in the resouce tracker today15:26
sean-k-mooneybut if you have the choice between not being able to use an entire socket15:27
*** ociuhandu has joined #openstack-nova15:27
sean-k-mooneyor slightly reduced performace im guess your ok with the slight perfromace hit15:27
sean-k-mooneyand slightly in accurate tracking15:27
*** Sundar has joined #openstack-nova15:28
sean-k-mooneybelmoreira: my conser is only for instace with a numa toplogy that dont use hugepages15:28
*** ociuhandu has quit IRC15:28
belmoreiragot it15:29
sean-k-mooneyhugepage guess will be tracked properly in either case15:29
belmoreirathanks a lot for this discussion and proposal15:30
Sundarsean-k-mooney: Please make sure to pull in https://review.opendev.org/698846 and https://review.opendev.org/70026115:32
*** dave-mccowan has joined #openstack-nova15:32
SundarThey are called out as dependencies in the create/bind patch15:32
Sundari.e. https://review.opendev.org/#/c/631244/15:33
sean-k-mooneyi cant have devstack pull in both if they are not in the same patch change15:33
sean-k-mooney*chain15:33
*** mvkr has quit IRC15:33
sean-k-mooneyso ill have to create a local branch and merge them in15:33
sean-k-mooneyi have https://review.opendev.org/#/c/698846/ but not https://review.opendev.org/#/c/700261/15:34
kashyapbelmoreira: stephenfin: efried: The kernel issue (https://review.opendev.org/#/c/684375/) has been latent.  (Just confirmed w/ a KVM mainatiner.)15:34
*** gyee has joined #openstack-nova15:34
SundarBoth are listed as dependencies in the same Nova patch15:34
kashyapsean-k-mooney: (Responded on the change.)  Thanks!15:34
sean-k-mooneyyes but devstack can only clone one it does not have the zuul logic to create a merge commti from both15:34
sean-k-mooneybut i can do it manually and restart the services15:35
stephenfinkashyap: I wonder why there wasn't a sudden spike so? Surely this would have been affecting belmoreira and co. for years now if it's been there forever15:35
sean-k-mooneySundar: in general depends-on should not be used within the same repo15:35
openstackgerritLee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue  https://review.opendev.org/70081115:35
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue  https://review.opendev.org/70081215:35
openstackgerritLee Yarwood proposed openstack/nova master: docs: Add stable device rescue docs  https://review.opendev.org/70083715:35
kashyapstephenfin: Yeah, I don't have a solid answer to that yet :-(15:35
sean-k-mooneyhttps://review.opendev.org/#/c/698846/ should be rebased on https://review.opendev.org/#/c/700261/15:35
kashyapstephenfin: Let me see the original kernel report I quoted in the commit, maybe there's something in there15:36
sean-k-mooneystephenfin: i have never seen hardware that actully does not have a dma regoin per numa node15:36
openstackgerritLee Yarwood proposed openstack/nova master: DNM test stable rescue tempest/devstack changes  https://review.opendev.org/70081515:37
*** ociuhandu has joined #openstack-nova15:37
sean-k-mooneystephenfin: and i mean ever in the 7+ years i have been working on this stuff15:37
*** dave-mccowan has quit IRC15:37
sean-k-mooneyso i am not surprised people have not hit this before15:37
stephenfinefried: I think I'm going to have to get you to weigh in on https://review.opendev.org/#/c/696745/, because I _think_ I've addressed the concerns but everyone else is still avoiding it :)15:38
kashyapsean-k-mooney: Ah, interesting15:38
stephenfinIt's almost all refactor noise _except_ the rename of 'nova.network.neutronv2.api' to just 'nova.network.neutron', and the subsequent removal of the 'nova.network.api' shim15:38
kashyapsean-k-mooney: I don't have the data about the hardware assmuption here (which hosts might not have the DMA32 region): https://lkml.org/lkml/2018/7/24/15215:39
sean-k-mooneykashyap: i also suspect that htis was not alway the behavior or kvm by the way15:40
sean-k-mooneyi proably had been that way for a long time but i susspect it was altered at somepoint15:41
kashyapsean-k-mooney: FWIW, a bit ago I checked w/ Paolo (one of the upstream KVM devs); he confirmed it's "latent"15:41
kashyapProbably15:41
sean-k-mooneymaybe i dont knwo why but if feel like if you grabed ubuntu 12.04 or centos 5.4 it likely wont happen15:42
belmoreirastephenfin our batch VMs (running in dedicated compute nodes with NUMA/huge pages aware) are long lived VM15:42
sean-k-mooneykashyap: i just get the feelign like there was a refactor of this at some point but i dont recall why i have that feeling15:43
belmoreiratrue that we are seeing this issue for awhile. Until now our solution has been to reboot the affected node.15:43
*** Liang__ has quit IRC15:44
*** Sundar has quit IRC15:45
*** ociuhandu has quit IRC15:45
*** ociuhandu has joined #openstack-nova15:45
*** mvkr has joined #openstack-nova15:48
*** macz has joined #openstack-nova15:49
*** zhanglong has joined #openstack-nova15:49
*** mlavalle has joined #openstack-nova15:56
openstackgerritLee Yarwood proposed openstack/nova stable/train: Avoid spurious error logging in _get_compute_nodes_in_db  https://review.opendev.org/70290216:00
*** artom has joined #openstack-nova16:00
*** dtantsur|brb is now known as dtantsur16:02
openstackgerritBalazs Gibizer proposed openstack/nova stable/pike: Error out interrupted builds  https://review.opendev.org/68791816:06
openstackgerritBalazs Gibizer proposed openstack/nova stable/pike: Fix false ERROR message at compute restart  https://review.opendev.org/69949616:06
*** TxGirlGeek has joined #openstack-nova16:09
dansmithefried: do you have any opinions on the newer literal set syntax (i.e {1, 2}) ?16:10
dansmithIMHO, it's one of the worst things ever, because it's super confusing and looks like a dict16:10
dansmithjust wondering if anyone else would be interested in a developer convention in nova of not adding code with that syntax to our tree16:10
gibidansmith: I have no problem with the literal set sytnax for me the absense of ':' is enough to notice that it is a set16:11
dansmithit's super easy to not notice that there's no : though16:12
gibiI guess we are different16:12
openstackgerritLee Yarwood proposed openstack/nova stable/train: Ensure source service is up before resizing/migrating  https://review.opendev.org/70175716:12
dansmithgibi: that's fine, I asked because I was wondering if I'm the only one16:12
artomdansmith, as per your request, I'm not letting you forget https://review.opendev.org/#/c/687404/816:18
*** mvkr has quit IRC16:20
gibidansmith: I think for me it is mostly about the fact that I used to it. I guess I would be pretty against assignment expression because that would be something I never used before in python16:20
dansmithartom: okay I'm in the middle of one of the cyborg patches, which is making me angry because apparently validating that our new methods are called like we expect them to is far too high a bar16:21
artomdansmith, to adjust nagging frequency, press 1 now ;)16:21
dansmithgibi: I think it's confusing that foo={} is a dict but foo={1} is a set, and easy to confuse foo={1:2} with foo={1,2}, and set comprehensions look a lot like dict comprehensions16:22
dansmithgibi: obviously you *can* tell the difference, it's just very easy to miss, IMHO16:22
*** belmoreira has quit IRC16:22
gibiI accept your view16:25
dansmithhah16:25
efrieddansmith: I personally have no problem with {1, 2} set syntax. A set is a dict with only keys. And I don't have trouble distinguishing between commas and colons.16:26
dansmithaight, well, that settles it then16:26
artom" A set is a dict with only keys" Umm...16:26
efrieddansmith: but jaypipes expressed a distaste for it (for the same reasons) so I avoided it... until he was gone.16:26
dansmithoh did he? damn, should have gotten my rule landed then16:26
*** artom has quit IRC16:27
*** rpittau is now known as rpittau|afk16:27
*** zhanglong has quit IRC16:28
efrieda) I think set((1, 2)) is overly verbose and unnecessary, b) my IDE warns that I should use a set literal, c) I believe set((1, 2)) is also less efficient because it first creates a tuple, then calls a method to turn it into a set.16:28
efriedobv b) I could ignore / turn off, and I'm sure c) is negligible in 99.99% of cases. But given aforementioned not-problem with the literal syntax, it's enough for me.16:29
dansmithyeah, I don't like how verbose set() has to be, and yes there's some overhead but I think it's trivial in this case16:29
dansmithack16:30
*** mvkr has joined #openstack-nova16:32
efriedstephenfin: before I dig into that: I'm of the opinion that a) we should make it look like the other service API shims. nova/$service_type/$project_name.py; b) I don't see a need for a generic abstract API layer.16:36
efriedIs that what you've done?16:36
*** tesseract has quit IRC16:39
sean-k-mooneydansmith: for what its worth i dislike {} as that allways meen a dict so i dont like useing it to initalise a default. i alway use set() if i want a set in that case16:41
sean-k-mooneywell if its empty that is16:41
dansmithwell, confirmation that I'm not crazy, at least16:41
efriedIf set(1, 2) were allowed I would be less grouchy about it.16:42
dansmithor.. "at least as crazy as sean-k-mooney" I guess :P16:42
sean-k-mooneyefried you have to do set({...}) right16:42
efriedbut having multiple meanings for the same operator/punctuation doesn't bother me as long as it's clear from the context, which IMO it is in this case, because colons.16:42
sean-k-mooneythat is annoying16:42
efriedsean-k-mooney: set($iterable)16:42
sean-k-mooneyya16:43
* stephenfin wonders why they didn't choose <>16:43
efriedso {...} if that's a set or a dict would work to get you a set of the keys; (...) for a tuple, [...] for a list etc.16:43
dansmithstephenfin: same16:43
sean-k-mooneystephenfin: becasue paython is annoying enough to parse as it is16:43
efriedI don't really get why that would be better; but I'm going to guess they chose {} because, as I said, a set is a dict without values.16:43
sean-k-mooney<> i think is technically not equal in python 216:43
dansmitha set is not a dict with no values :)16:44
stephenfinsean-k-mooney: indeed, it is16:44
stephenfinI've seen that once in my life, I think16:44
stephenfin(docutils)16:44
efrieddansmith: totally is. O(1) hashing, just no pointers out of the buckets.16:44
dansmithsean-k-mooney: as hateful as they are, I can imagine them doing that in py3 to break compatibility for fun and profit.. I guess dict literals just got added too early :)16:44
sean-k-mooneyalthough i think its also the same as != and is checking identiy not equality but i might be wrong on that16:44
dansmithefried: totally is not :)16:45
dansmithit may be the same implementation under the covers, but that does not make the statement true :)16:45
efriednova-live-migration seems to be ill. Anyone available to dig in?16:47
dansmithI would think artom would be a good person to maybe start taking on some responsibility for that stuff given his numa LM work16:53
dansmithbuilding some context for the future and what not16:54
*** mlavalle has quit IRC16:54
*** psachin has joined #openstack-nova16:54
sean-k-mooneyi got landed with some downstream stuff but in theory i am ment to be helping porting those jobs to be non legacy too16:54
sean-k-mooneyefried: is it blocking ci or just starting to fail more then we would like16:55
*** mlavalle has joined #openstack-nova16:55
efriedsean-k-mooney: It has tanked many patches in CI in the last 18h or so.16:55
efrieddoesn't seem to be 100%, but it's close.16:56
efriedI haven't looked into it at all because I haven't yet been bothered to f'ing unzip the logs.16:56
sean-k-mooneyhas my patch merged to fix that yet16:56
sean-k-mooneyyou rechecked it this morning/last night right16:56
efriedyour patch did merge, yes. Not sure if any of the failed jobs started since then.16:57
sean-k-mooneyya looks like it di16:57
efriedI don't even know if the failures are the same across the board. I just know many have been failing. (Most of the nova-next failures are due to 'multiple possible networks', which I'm fixing elsewhere.)16:57
sean-k-mooneyyes with your tempest patch16:58
*** damien_r has quit IRC16:59
sean-k-mooneythat should also be merged now. if no one else fixes it  ill try and get though these backprots then i can look at ci but proably tomorow or monday.17:00
*** slaweq has quit IRC17:01
*** artom has joined #openstack-nova17:03
*** slaweq has joined #openstack-nova17:04
efriedstephenfin: +2 on https://review.opendev.org/#/c/696745/17:08
efriedI guess controversy was in previous versions.17:08
efriedI get the gripe about making backports difficult, but the code base will just get messier and messier if we don't bite the bullet and clean this stuff up from time to time.17:08
efriedBesides, I don't see backports being hugely affected by this.17:08
efriedSo ++17:08
stephenfinthanks17:08
*** slaweq has quit IRC17:09
*** iurygregory has quit IRC17:14
*** slaweq has joined #openstack-nova17:20
*** slaweq has quit IRC17:25
efriedI can't even see why https://zuul.opendev.org/t/openstack/build/093f129de3ca47549239eed0d6acf288 is failing. The job output says devstack failed, but the devstack log looks okay to me. Anyone? Bueller?17:26
efried /opt/stack/new/devstack/lib/databases/mysql: line 64: mysql: command not found17:29
efried^ on the subnode. wtf?17:29
artomWait, that .gz compression thing hasn't been fixed?17:30
efriedartom: sean-k-mooney fixed it somewheres, but this job may have started before that fix merged.17:30
efriedReal PITA.17:30
artomSeriously, wit hummus.17:30
efriedmm, hummus.17:30
* efried feeds face17:30
*** udesale has quit IRC17:31
sean-k-mooneyartom: https://review.opendev.org/#/c/702465/ meged this morning17:31
sean-k-mooneybut if the job started before that merged then ya they would be compressed17:32
artomsean-k-mooney, were they not gziped but transparently decompressed before?17:32
sean-k-mooneyyes17:32
sean-k-mooneywell no17:32
artomSo what killed the transparent decompression?17:32
sean-k-mooneythey were gzip and serve with hearders to decompress them17:32
sean-k-mooneybut that had issues with swift17:33
artomOh, the browser did it17:33
artomClient-side17:33
sean-k-mooneyyes the borser did but the issue is they were getting double compressed wehn being served form swift17:33
sean-k-mooneyso that is why you see the garbage in your browser17:33
*** evrardjp has quit IRC17:34
artomOh, does Swift automagically compress text or something?17:34
sean-k-mooneyit is still decompressing with deflate but not then decompressing that with gzip17:34
*** evrardjp has joined #openstack-nova17:34
openstackgerritVladyslav Drok proposed openstack/nova master: Make removal of host from aggregate consistent  https://review.opendev.org/69639217:34
sean-k-mooneyartom: i did not look into it fully but fungi would be able to actully explain it17:34
sean-k-mooneyartom: for now just do curl <url>  | zcat | less17:35
sean-k-mooneyor lnav lnav is good17:35
artomI want to make a `squat` and `rack` command17:35
artomSo I can squat | curl | rack17:35
*** jcosmao has left #openstack-nova17:37
sean-k-mooney looking at the log failrue we are getting nova warning for using deprecated version of qemu and libvirt17:38
fungisean-k-mooney: we're not clear whether it's a behavior in rackspace's swift implementation or the cdn they put in front of it, but if we upload .gz files with a content-type of application/gzip then they will be transparently decompressed when browsers or other clients request them and indicate they accept that content-type. sounds convenient, right? until you save a .tar.gz file there and it's17:39
fungiquietly decompressed on retrieval without altering the extension, so you've downloaded a file whose name claims it's a gzipped tarball but which is in reality now an uncompressed tar archive. now make that an intermediate step in a release publication job and you're pushing python sdists or similar tarballs somewhere when they're not actually installable (because tools think they're compressed when17:39
fungithey're not)17:39
fungiso we altered the way we store job artifacts in swift to make it so that we transparently compress everything on upload, unless it's already a compressed file, and assume that if the file has been precompressed then it should not be transparently decompressed for requesting clients17:40
fungiartom: ^17:40
efriedclearly fungi has had to explain this before :P17:42
artomfungi, to be honest I didn't grok most of that. My takeaway is: layers do deep even onions are like "whoa dude"17:42
sean-k-mooneyyes17:42
artom*so deep17:42
sean-k-mooneybut thanks for the explaination again.17:42
fungiwe fixed the obvious cases we knew about in abstract jobs/roles but there were some in devstack which had to get fixed too17:43
sean-k-mooneyfungi: funnialy enogh the last time i tried deploying zuul at home geting the logs to render properly in the bower when loadbalnace via nginx in k8s was actully one of the hardest parts17:44
fungiand there are also some project-specific jobs which are also precompressing logs from things which will need to be adjusted to stop compressing within the job17:44
fungiwhich should be updated as we encounter them17:44
efrieddansmith: Would you please look at https://review.opendev.org/#/c/696392/ ? I'm not clear whether this counts as an API change...17:45
efriedartom: so are you investigating the failure amid all of that?17:47
artomefried, which failure? The missing MySQL?17:48
efriedyeah17:48
artomHeh, I'm being volunteered, aren't I? ;)17:48
artomI'll take a look :)17:48
efriedwhatever's causing nova-live-migration to fail17:48
efriedthanks artom. Rather you than me.17:48
efriedI'll go check the other instances of n-l-m fail and see if they're the same.17:49
artomsean-k-mooney, zless is a thing, btw :)17:50
artomDon't need to | zcat | less17:50
artomStraight up | zless17:50
sean-k-mooneyartom: sure but im actully sendign it to lnav17:50
sean-k-mooneyand you use to be able to send it to firefox17:50
sean-k-mooneybut they droped readign from standar in at some point17:51
sean-k-mooneybut good to know zless is a thing. zgrep is also a thing17:51
dansmithefried: that patch doesn't change what can be raised on error, so not an api change needing a microversion, if that's what you're asking for17:51
dansmithefried: it adds a scan of all cells on delete, which the user doesn't know about anyway17:52
efriedartom: so far 5/5 on the ones I've looked at, fails from the last 2h or so.17:52
efrieddansmith: okay. afaict it makes it so HostMappingNotFound can no longer be raised when removing host from agg. That's kind of a behavior change, so I wasn't really sure.17:54
artomefried, so, mysql-client should be getting installed17:54
efriedartom: I don't disagree. Why isn't it?17:54
artomAnd it isn't, afaict17:54
artomefried, yeah, no idea what changed yet17:54
efriedokay. Maybe -infra knows?17:54
dansmithefried: it's the same error handler as ComputeHostNotFound in the api code though so it doesn't change anything the user can see17:55
efriedright, what I mean is that with a certain set of conditions, previously there would *be* an error, but now there would *not* be.17:56
efried...with those same conditions.17:56
efriedI guess that's kind of the definition of a bug fix though17:56
efriedso I'm going to +2.17:56
dansmiththe user doesn't know about those conditions17:56
dansmithit'd be like saying we failed on mysql 4.1 and succed on 4.2 so that's user-visible17:56
efriedOkay, I must have misinterpreted that based on the bug description.17:56
*** yan0s has quit IRC17:56
efriedokay.17:57
artomefried, what was the last time that job passed? I want to compare logs, see if it installed the mysqlclient17:58
efriedI'll look for one...17:59
artomHrmm, so it installs it on the *main* node17:59
efriedartom: here's one from ~24h ago, not sure if that's the most recent https://review.opendev.org/#/c/696745/18:00
artomefried, that works, thanks18:01
*** derekh has quit IRC18:01
artomAh, so, on the main node, because we install the server, that pulls in the client18:01
openstackgerritSylvain Bauza proposed openstack/nova-specs master: Re-proposes multiple vGPU types in libvirt  https://review.opendev.org/70294318:01
efriedartom: assume this has nothing to do with cirros images http://lists.openstack.org/pipermail/openstack-discuss/2020-January/012024.html18:06
artomefried, don't think so - looks like it has something to do with whether we configure glance or not18:07
artomThe broken one runs through this:18:07
artom2020-01-16 14:30:39.880 | + ./stack.sh:echo_summary:458              :   echo -e Configuring Glance18:07
artom2020-01-16 14:30:39.883 | + ./stack.sh:main:1160                     :   init_glance18:07
artom2020-01-16 14:30:39.885 | + lib/glance:init_glance:276               :   rm -rf /opt/stack/data/glance/images18:07
artom2020-01-16 14:30:39.889 | + lib/glance:init_glance:277               :   mkdir -p /opt/stack/data/glance/images18:07
artom2020-01-16 14:30:39.893 | + lib/glance:init_glance:280               :   recreate_database glance18:07
efriedgdi, why do I suck at logstash. Shouldn't it be as simple as searching for message:"mysql: command not found" ??18:08
*** maciejjozefczyk has quit IRC18:09
efried...unless logstash is also trying to read gzipped files and getting garbage.18:11
artomHrmm, hol'up18:12
artomWorking: is_service_enabled g-reg18:12
artomBroken: is_glance_enabled18:12
artomWhich looks for g- in ENABLED_SERVICES, so g-api matches18:12
sean-k-mooneyartom: that is not somthign that would be changeing per fun but it would be different if patches had not been rebased i guess18:13
melwittdo you think we would expect to run mysql commands on the subnode? that one is compute only and shouldn't have a database on it anyway18:15
efriedartom: https://review.opendev.org/#/c/702707/18:15
artomefried, bingo18:15
artomThat eventually triggers it to attempt to recreate the glance database on the subnode18:16
artomWhich calls the missing mysql binary18:16
artomI gotta run to a quick apt18:16
dansmithmelwitt: that may be true for nova, but not all the services18:16
melwittdansmith: ack18:16
dansmithand of course "mysql -h primary" could be some use case for sequencing (just making that up)18:17
efriedartom: will this make it okay? https://review.opendev.org/#/c/702709/118:17
sean-k-mooneydansmith: while you are right i dont think cinder cyborg or neutron do direct db acess form the compute nodes eitehr18:17
dansmithsean-k-mooney: apparently glance is the service in question here?18:18
sean-k-mooneyya18:18
dansmithpresumably other projects do or could use multinode for things18:18
dansmith(or should)18:18
sean-k-mooneyglance should normally not be on the compute node but we can enable it there if we want18:19
*** ociuhandu_ has joined #openstack-nova18:19
sean-k-mooneymaybe its beign used for the multi image store testing or something like that18:19
*** ociuhandu has quit IRC18:22
*** ociuhandu_ has quit IRC18:23
efriedartom: I guess not, since they both merged at essentially the same time.18:24
openstackgerritMerged openstack/nova stable/pike: lxc: make use of filter python3 compatible  https://review.opendev.org/67650218:29
*** dtantsur is now known as dtantsur|afk18:30
*** slaweq has joined #openstack-nova18:31
efriedhttps://bugs.launchpad.net/nova/+bug/186002118:33
openstackLaunchpad bug 1860021 in OpenStack Compute (nova) "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,New]18:33
efriedCorrect me if I'm wrong: we didn't need init_glance to be happening on the subnode, cause it wasn't before?18:34
*** martinkennelly has quit IRC18:34
*** liuyulong has quit IRC18:34
sean-k-mooneyefried: well we dont need a second copy of glance running on the subnode no18:35
sean-k-mooneyunless we are specificly trying to test image that are in multiple data stores18:35
melwittis it only our multinode job that's failing? I would expect all would be18:35
sean-k-mooneyi think this was only nova livemigation?18:35
efriedmelwitt: many patches have failed with just n-l-m in the red. A few have failed with other things, but I haven't looked into them.18:35
melwittmeaning I would think all multinode jobs regardless of project would fail from this18:36
efried(other than nova-next with 'multiple possible networks' as noted, fix in the pipe)18:36
*** slaweq has quit IRC18:36
sean-k-mooneymelwitt: it could be that nova live migration is a legacy job18:36
melwitthm, that is surprising. so that means there's some other boilerplate in other jobs that we don't have in nova-live-migration if we're the only ones failing18:36
efriedperhaps somehow we're installing mysql client on the other jobs18:36
melwittyeah, that's what I'm wondering18:37
sean-k-mooneyand our other multinode jobs may be zuul v3 so the glance service might not be enabled on the sub nodes18:37
melwittsean-k-mooney: in what way is it a legacy job? I remember moving it over to zuul v318:37
efriedthe right thing would be to figure out how to make the subnode not init_glance; but I would be happy enough with a solution that got mysql client installed there so we could go back to the way we were.18:37
sean-k-mooneyoh maybe im thinking of nova-next18:37
efriedn-l-m is legacy18:37
efriedisn't it?18:37
melwittI didn't think so18:37
melwittlet me look it up because maybe I'm wrong18:38
sean-k-mooneywell there is one way to find out18:38
efriedif it weren't, wouldn't we be not having the gz problem?18:38
sean-k-mooneyhttps://github.com/openstack/nova/tree/stable/stein/playbooks/legacy18:38
melwittoh yeah, I see https://review.opendev.org/#/c/69336418:38
melwittso moving it to the .zuul.yaml does not make it zuul v3 apparently. I didn't know that18:38
sean-k-mooneyso it has playbooks in the legacy folder and https://github.com/openstack/nova/blob/stable/stein/.zuul.yaml#L114-L122 still runs them so ya18:39
melwittok. I had previously thought that migrating jobs in tree made them zuul v3 but that was a wrong thought18:39
melwittsorry about that18:39
*** ociuhandu has joined #openstack-nova18:40
sean-k-mooneymelwitt: yep matt start to move it but did not have time to finish it18:40
sean-k-mooneyalso you gerrit foo is better then mine i spent 20 minuts trying to find that last week an gave up18:40
sean-k-mooneymaybe i filtered out abandoned chagnes18:41
melwittlol. I just did "message:nova-live" nothing fancy18:41
sean-k-mooneymessage:?18:41
melwittyeah means look in commit message18:41
sean-k-mooneyoh that checks the commt message?18:41
melwitt"comment:" looks in comments18:41
sean-k-mooneysee you gerrit foo is better then mine18:41
melwittgerrit search is pretty cool18:41
sean-k-mooneyso ya i would guess that this has somethign to do with devstack-gate18:43
sean-k-mooneythe legacy jobs are going to be using devstack-gate and the non legacy ones will not18:43
melwittI'm trying to look at tempest-multinode-full-py3 to see if it's legacy but I think you could figure that out faster than me18:44
melwittthat's an example from neutron18:44
melwittwhich is passing18:44
sean-k-mooneyi think that is not a legacy job18:44
sean-k-mooneythe legacy ones usually specify run: playbook18:45
*** ociuhandu has quit IRC18:45
melwittI see, ok. so that's a data point in favor of it being related to legacy-ness18:45
melwittI don't know what it is about the non-legacy stuff that can handle the mysql commands. do they have the client available or something more involved18:46
*** psachin has quit IRC18:46
sean-k-mooneyis it related to this? https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a18:47
sean-k-mooneyspecificlly https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a#diff-987768d02fbf43ebb74ad9a359222903R2718:47
sean-k-mooneymatt enabled glance api on the subnode18:48
artomefried, back - no, it won't, because is_glance_enabled checks for 'g-' in the ENABLED_SERVICES18:48
sean-k-mooneythat was also like 27 days ago18:48
artomSo g-api triggers it too18:48
efriedLooking for some way to redefine is_glance_enabled as a stopgap18:48
artomWe might want to consult with the devstack folks18:49
efriedwherezat? -infra? -qa?18:49
sean-k-mooneyqa18:49
sean-k-mooneyartom: are you sure its not https://github.com/openstack/devstack-gate/commit/183a58eab185c2e6f48afb4d99417d49e88c0f5a#diff-987768d02fbf43ebb74ad9a359222903R2718:49
artomsean-k-mooney, I'm never sure of anything, but I don't think so18:50
sean-k-mooneyi guess that is GRENADE_SUBNODE_MASTER18:50
efriedcould be that, and then the glance-enabled check exposed it.18:50
artomWait, what did that change?18:51
artomAdded g-api to the subnode?18:51
*** ralonsoh has quit IRC18:51
sean-k-mooneyyes18:52
sean-k-mooneyalthough maybe only in the grenade jobs18:53
*** openstackgerrit has quit IRC18:58
*** openstackgerrit has joined #openstack-nova19:05
openstackgerritEric Fried proposed openstack/nova master: Test fix for bug 1860021  https://review.opendev.org/70296119:05
openstackbug 1860021 in devstack "nova-live-migration fails 100% with "mysql: command not found" on subnode" [Undecided,In progress] https://launchpad.net/bugs/1860021 - Assigned to Radosław Piliszek (yoctozepto)19:05
efriedartom: ^19:05
artomefried, thanks!19:08
efriedartom: thanks for helping triage this.19:09
artom\o/ I'm useful!19:09
*** ccamacho has quit IRC19:18
openstackgerritMerged openstack/nova stable/rocky: Cache security group driver  https://review.opendev.org/69750519:22
*** nweinber_ has joined #openstack-nova19:30
*** nweinber has quit IRC19:33
*** nweinber__ has joined #openstack-nova19:47
*** nweinber_ has quit IRC19:49
openstackgerritMerged openstack/nova stable/rocky: Block deleting compute services with in-progress migrations  https://review.opendev.org/69811319:57
openstackgerritsean mooney proposed openstack/nova stable/stein: Remove 'test_cold_migrate_with_physnet_fails' test  https://review.opendev.org/70297119:59
openstackgerritsean mooney proposed openstack/nova stable/stein: Block rebuild when NUMA topology changed  https://review.opendev.org/70297219:59
openstackgerritsean mooney proposed openstack/nova stable/stein: Disable NUMATopologyFilter on rebuild  https://review.opendev.org/70297319:59
openstackgerritsean mooney proposed openstack/nova stable/stein: FUP for in-place numa rebuild  https://review.opendev.org/70297419:59
sean-k-mooneythat totally did not take an hour...20:00
sean-k-mooneyalso night o/20:00
*** tosky has quit IRC20:04
efriedsean-k-mooney: FYI I'm still getting gzips in https://7cb0f59696603595f18f-52bbbaaadc4f7624d8dbaffca646d87d.ssl.cf1.rackcdn.com/702961/1/check/nova-live-migration/f6ff2d6/logs/20:21
*** tbachman has joined #openstack-nova20:29
gmannefried: this job does not use the devstack roles sean-k-mooney modified to not gzips the logs. somewhere legacy base jobs need to do the same.20:30
efriedokay. I wasn't sure.20:30
melwittsame in here https://zuul.opendev.org/t/openstack/build/fb368e8af1854170b701dac772f53bc8/log/logs/mysql/error.txt.gz20:31
melwittgrenade-py320:32
gmannyeah that is also legacy job. i am not sure where exactly those logs are defined20:33
efriedI will fast approve patches that fix that.20:33
efriedand do a little dance20:34
melwittdang, everything legacy20:35
*** mriedem has joined #openstack-nova20:36
melwittI guess I don't understand that bc it says legacy jobs should have migrated with a legacy- prefix in the name https://docs.openstack.org/infra/manual/zuulv3.html#legacy-job-migration-details20:37
melwittlike why are all the jobs secret legacy somehow20:37
*** slaweq has joined #openstack-nova20:38
melwittok, should have read further before typing. once moved in-tree they had to be renamed to not collide with naming, now this is the next step https://docs.openstack.org/infra/manual/zuulv3.html#reworking-legacy-jobs-to-be-v3-native20:39
openstackgerritArtom Lifshitz proposed openstack/nova master: Clarify fitting hugepages log message  https://review.opendev.org/70298120:39
melwittlooks like there's some work in-flight for the grenade jobs https://review.opendev.org/#/q/topic:grenade_zuulv3+(status:open+OR+status:merged)20:42
*** artom has quit IRC20:45
*** amuller has joined #openstack-nova20:51
amullerhi folks, looking at nova.conf... [glance] -> api_servers was deprecated in https://github.com/openstack/nova/commit/cffbc2e431e90f7b6a67cf4a82349ee1d8de202a20:51
amullerI was wondering what option replaced it?20:51
efriedamuller: Use standard ksa opts to get at glance20:52
amullerefried: do you remember the section and conf key off the top of your head?20:52
efriedwe don't support rotating IPs (poor man's load balancer hack) there anymore20:52
amuller[keystone_authtoken] ?20:53
efriedamuller: If you simply remove api_servers, it should just pick up what's in your service catalog. If you really need to specify something different, you can use [glance]endpoint_override.20:53
efriedBut I would suggest if you find yourself needing that ^ then your service catalog is busted :)20:53
efriedI guess some edge deployments might want to override, I think I heard that from someone.20:53
dansmithfrom me20:55
amullerefried++20:55
amullerthank you20:55
efriedno problem, good luck.20:55
amullerI see that TripleO Stein still sets [glance] api_servers20:56
amullerthough the option was deprecated in Queens =p20:56
efriedif only someone from tripleo was paying attention20:56
*** nweinber__ has quit IRC20:57
efriednova meeting now ish in #openstack-meeting20:59
*** alistarle has joined #openstack-nova21:04
*** slaweq has quit IRC21:09
openstackgerritmelanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929  https://review.opendev.org/70147821:23
openstackbug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/184492921:23
*** eharney has quit IRC21:26
*** alistarle has quit IRC21:38
openstackgerritDustin Cowles proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs  https://review.opendev.org/67602921:50
openstackgerritDustin Cowles proposed openstack/nova master: Provider Config File: Functions to merge provider configs to provider tree  https://review.opendev.org/67652221:50
openstackgerritDustin Cowles proposed openstack/nova master: Provider Config File: Enable loading and merging of provider configs  https://review.opendev.org/69346021:50
dustincefried: it is ready for re-review now :)21:58
efriedack21:58
*** gentoorax is now known as gentoorax_away21:59
*** gentoorax_away is now known as gentoorax21:59
donnydsean-k-mooney: FN is back in operation22:00
* donnyd waves to everyone in nova 22:01
*** brault has joined #openstack-nova22:07
*** rcernin has joined #openstack-nova22:07
efriedo/ donnyd!22:10
*** amuller has quit IRC22:14
*** xek__ has quit IRC22:17
*** artom has joined #openstack-nova22:18
*** TxGirlGeek has quit IRC22:21
*** zhanglong has joined #openstack-nova22:37
efrieddustinc: if you write one more meaty test case, I'll be +2 on that top patch.22:49
dustincefried: happy to, did you post the case in gerrit?22:50
efriedand I convinced gibi to put the series in his queue for tomorrow morning, so you could conceivably come back next week to a completed bp :)22:50
efrieddustinc: yeah22:50
dustincok, maybe I will drop my other task to get that done..22:50
*** smcginnis is now known as smcginnis|PTO22:52
*** zhanglong has quit IRC22:53
*** dviroel has quit IRC22:54
*** TxGirlGeek has joined #openstack-nova23:08
*** awalende has joined #openstack-nova23:24
*** awalende has quit IRC23:29
*** tbachman has quit IRC23:47
*** mlavalle has quit IRC23:52
openstackgerritMerged openstack/nova stable/rocky: Join migration_context and flavor in Migration.instance  https://review.opendev.org/69657223:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!