Wednesday, 2018-09-19

*** takashin has joined #openstack-nova00:01
openstackgerritTetsuro Nakamura proposed openstack/nova-specs master: Spec: Support filtering by forbidden aggregate  https://review.openstack.org/60335200:02
*** brinzhang has joined #openstack-nova00:02
*** hamzy has joined #openstack-nova00:05
*** janki has quit IRC00:09
*** janki has joined #openstack-nova00:09
*** janki has quit IRC00:10
*** janki has joined #openstack-nova00:11
*** gyee has quit IRC00:19
*** jiapei has quit IRC00:24
*** new-comer has joined #openstack-nova00:37
new-comerhttps://bugs.launchpad.net/nova/+bug/179267400:39
openstackLaunchpad bug 1792674 in OpenStack Compute (nova) "[Rocky] fail to get VNC console" [Undecided,Confirmed]00:39
*** yingjun has joined #openstack-nova00:43
*** new-comer has left #openstack-nova01:01
*** new-comer has quit IRC01:02
*** slaweq has joined #openstack-nova01:11
*** evrardjp has quit IRC01:12
*** slaweq has quit IRC01:15
*** mhen has quit IRC01:19
*** mhen has joined #openstack-nova01:21
*** Dinesh_Bhor has joined #openstack-nova01:29
*** mrsoul has quit IRC01:29
*** litao has joined #openstack-nova01:33
*** tetsuro has quit IRC01:35
*** hongbin has joined #openstack-nova01:37
openstackgerritChen proposed openstack/nova master: doc trivial: additional info to admin-password-injection  https://review.openstack.org/60341401:43
*** dpawlik has joined #openstack-nova01:56
*** dpawlik has quit IRC02:00
*** sapd1_ has joined #openstack-nova02:04
*** sapd1 has quit IRC02:07
*** Dinesh_Bhor has quit IRC02:22
*** yingjun has quit IRC02:28
*** Dinesh_Bhor has joined #openstack-nova02:31
openstackgerritMatt Riedemann proposed openstack/nova master: DNM: test resize reschedule up-calls  https://review.openstack.org/60338902:38
*** tetsuro has joined #openstack-nova02:43
*** rnoriega has quit IRC02:45
*** rnoriega has joined #openstack-nova02:46
*** vivsoni has joined #openstack-nova02:56
*** jistr has quit IRC03:00
*** jistr has joined #openstack-nova03:00
*** afazekas has quit IRC03:00
*** BlackDex has quit IRC03:01
*** BlackDex has joined #openstack-nova03:03
*** slaweq has joined #openstack-nova03:11
*** slaweq has quit IRC03:16
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66  https://review.openstack.org/60354903:18
*** Dinesh_Bhor has quit IRC03:28
*** pvc has quit IRC03:28
openstackgerritMerged openstack/nova stable/rocky: Handle binding_failed vif plug errors on compute restart  https://review.openstack.org/59531703:32
openstackgerritMerged openstack/nova stable/rocky: Restart scheduler in TestNovaManagePlacementHealAllocations  https://review.openstack.org/59815203:32
*** Dinesh_Bhor has joined #openstack-nova03:34
*** vivsoni has quit IRC03:37
*** Dinesh_Bhor has quit IRC03:42
*** udesale has joined #openstack-nova03:52
*** cfriesen has quit IRC03:55
*** tetsuro has quit IRC03:58
*** vivsoni has joined #openstack-nova04:04
*** yingjun has joined #openstack-nova04:04
*** vivsoni has quit IRC04:06
*** vivsoni has joined #openstack-nova04:06
*** jaosorior_ is now known as jaosorior04:13
*** yingjun has quit IRC04:18
*** Dinesh_Bhor has joined #openstack-nova04:32
gmannalex_xu:  i will skip today API office hour due to fever. we will continue from next week.04:57
*** hamzy_ has joined #openstack-nova05:01
*** hamzy has quit IRC05:02
*** hongbin has quit IRC05:06
alex_xugmann: ok, no problem05:06
*** hamzy has joined #openstack-nova05:09
*** hamzy_ has quit IRC05:11
*** slaweq has joined #openstack-nova05:11
*** slaweq has quit IRC05:15
*** hamzy has quit IRC05:17
*** hamzy has joined #openstack-nova05:18
*** tetsuro has joined #openstack-nova05:20
*** hamzy has quit IRC05:23
*** hamzy has joined #openstack-nova05:24
*** rcernin has quit IRC05:30
*** rcernin has joined #openstack-nova05:30
*** ratailor has joined #openstack-nova05:31
*** yingjun has joined #openstack-nova05:36
*** luksky11 has joined #openstack-nova05:51
*** yingjun has quit IRC05:54
*** rcernin_ has joined #openstack-nova06:04
*** rcernin has quit IRC06:06
*** holser_ has joined #openstack-nova06:12
*** Luzi has joined #openstack-nova06:16
*** janki has quit IRC06:17
*** holser_ has quit IRC06:18
*** slaweq has joined #openstack-nova06:18
*** holser_ has joined #openstack-nova06:21
*** slaweq has quit IRC06:23
*** maciejjozefczyk has joined #openstack-nova06:26
*** dpawlik has joined #openstack-nova06:27
*** dpawlik has quit IRC06:28
*** dpawlik_ has joined #openstack-nova06:28
*** maciejjozefczyk has quit IRC06:30
*** dpawlik_ has quit IRC06:30
*** dpawlik has joined #openstack-nova06:30
*** dpawlik has quit IRC06:31
*** dpawlik_ has joined #openstack-nova06:31
*** rcernin has joined #openstack-nova06:34
*** rcernin_ has quit IRC06:36
*** yingjun has joined #openstack-nova06:36
*** deepak_mourya_ has quit IRC06:40
openstackgerritOpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata  https://review.openstack.org/60104706:53
*** slaweq has joined #openstack-nova06:55
*** janki has joined #openstack-nova06:55
*** slaweq has quit IRC07:00
*** rcernin has quit IRC07:06
*** Dinesh_Bhor has quit IRC07:09
dpawlik_is someone using cellv2 but not as "simple" cell but multiple cells?07:11
*** hamdyk has joined #openstack-nova07:11
*** slaweq has joined #openstack-nova07:16
*** hoonetorg has quit IRC07:17
*** cfriesen has joined #openstack-nova07:18
sorrisondpawlik_: I know CERN are07:22
dpawlik_sorrison: thx07:22
dpawlik_sorrison: I got a feeling when I read documentation and the code on github, that all uses simple cell setup xD07:23
sorrisonWe (Nectar) have some cellsv2 stuff running in testing, still cellsv1 in production07:23
dpawlik_there are no starts scripts for RDO/Debian Salsa/Ubuntu cloud archive to start super conductor and also conductor that take configuration from different file07:23
sorrisonI'm *think* that the super conductor is just a normal conductor but it listens on the api level rabbit queue07:25
sorrisonIf you were running multiple cells then you'd have lots of different controllers running different conductors07:26
bauzasgood morning stackers07:26
bauzassorrison: this is correct07:27
bauzasdpawlik_: a super conductor is just a normal conductor with a specific configuration07:27
dpawlik_bauzas: but its reading amqps queue from vhost / and its also talking with nova_cell0 db yes?07:30
*** hoonetorg has joined #openstack-nova07:30
dpawlik_and the second conductor e.g. cell1 is reading from another vhost e.g. nova_cell1 and its also puting data to nova_cell1 db07:31
dpawlik_I just describe my point of knowledge how it should be configured, is it true?07:31
dpawlik_I just take it from devstack07:32
dpawlik_and Im wondering why after executing the same commands like it  is done on devstack, on my controller I have duplicated nova services07:32
*** Dinesh_Bhor has joined #openstack-nova07:36
*** Gorian has quit IRC07:37
*** jamiec has quit IRC07:38
*** Gorian has joined #openstack-nova07:39
*** cfriesen has quit IRC07:39
*** helenafm has joined #openstack-nova07:39
bauzasdpawlik_: sorry, I was on mute so I missed your ping07:40
* bauzas facepalms07:40
bauzasdpawlik_: hold on, we have some docs that explain this07:40
dpawlik_bauzas: thx07:40
bauzasdpawlik_: https://docs.openstack.org/nova/latest/user/cells.html#setup-of-cells-v207:41
bauzasdpawlik_: and https://docs.openstack.org/nova/latest/user/cellsv2-layout.html07:42
bauzashope that will help you understand the cells v2 concepts07:42
dpawlik_So im doing the same how its described and TBH its not working well07:42
dpawlik_Im still have duplication of nova services07:42
dpawlik_yesterday I send it  to dansmith how my cell mappings looks like07:43
dpawlik_and let me quote "looks like only one mapping in there, so I guess you're good"07:43
*** jamiec has joined #openstack-nova07:43
dpawlik_hmm, clean host, new day07:45
dpawlik_bauzas:07:45
dpawlik_bauzas: pro tip to setup cells07:45
bauzashow many services are you running in parallel ?07:46
dpawlik_stop nova-conductor services before start setup cells_v207:46
bauzasdpawlik_: you're in an upgrade case or greenfields ?07:46
*** maciejjozefczyk has joined #openstack-nova07:47
dpawlik_bauzas: for now Im just testing deployment queens release on new infra07:47
dpawlik_for now I don't want to touch "upgrade" procedure07:47
bauzasbut yeah, I'm assuming you populate the mappings first before starting the services07:47
bauzassince those services use the mappings you need to set07:48
bauzasdpawlik_: but we could make it clearer in docs, I presume07:48
dpawlik_Now I will know07:48
bauzasdpawlik_: so yeah, that's why you had duplicate services07:48
dpawlik_bauzas: Small notification on the top of the doc will be helpful for other people07:48
bauzasthose first registered on startup07:48
bauzasdpawlik_: yeah, that makes sense in https://docs.openstack.org/nova/latest/user/cells.html#cells-v207:49
bauzasI could spin a change07:49
dpawlik_dansmith: ^^ above is description why I have duplicated services on good cell mapping xD07:49
dpawlik_bauzas: please :)07:50
dpawlik_bauzas: if you can, write it please07:50
openstackgerritBrin Zhang proposed openstack/python-novaclient master: Add support changes-before for microversion 2.66  https://review.openstack.org/60354907:51
*** sahid has joined #openstack-nova07:57
bauzasdpawlik_: lemme stash my current work :)07:58
*** yingjun has quit IRC07:59
*** takashin has left #openstack-nova08:02
*** maciejjozefczyk has quit IRC08:02
openstackgerritTetsuro Nakamura proposed openstack/nova-specs master: Spec: allocation candidates in tree  https://review.openstack.org/60358508:02
*** maciejjozefczyk has joined #openstack-nova08:05
*** yingjun has joined #openstack-nova08:05
openstackgerritSylvain Bauza proposed openstack/nova master: cells: Be explicit in docs about service restarts  https://review.openstack.org/60358808:08
bauzasdpawlik_: ^08:10
*** tetsuro has quit IRC08:12
kashyapSigh, copy-pasting Git commit messages seem to break the line-wrapping in LaunchPad: https://bugs.launchpad.net/nova/+bug/178986808:16
openstackLaunchpad bug 1789868 in OpenStack Compute (nova) "RFE: Add a virtio-rng device to Nova instances by default" [Medium,Triaged] - Assigned to Kashyap Chamarthy (kashyapc)08:16
kashyapThe formatting is now fugly08:16
dpawlik_bauzas: checking08:18
*** mdbooth_ has joined #openstack-nova08:19
*** mdbooth_ is now known as mdbooth08:20
dpawlik_bauzas: maybe warning will be better than info08:22
dpawlik_but it looks good08:22
bauzasnot really08:23
dpawlik_bauzas: im reading it from my perspective where I was wondering half day why my deployment doesn't work correctly08:24
*** jpena|off is now known as jpena08:31
*** skatsaounis has quit IRC08:34
*** ralonsoh has joined #openstack-nova08:34
kashyapbauzas: johnthetubaguy: Hey there, this trivial thing is ready for merge: https://review.openstack.org/#/c/602592/08:38
*** alex_xu has quit IRC08:38
*** luksky11 has quit IRC08:39
*** alexchadin has joined #openstack-nova08:43
*** skatsaounis has joined #openstack-nova08:44
*** mdbooth has quit IRC08:47
*** mdbooth has joined #openstack-nova08:48
*** derekh has joined #openstack-nova08:49
*** owalsh has quit IRC08:51
*** mdbooth_ has joined #openstack-nova08:54
*** owalsh has joined #openstack-nova08:56
*** mdbooth has quit IRC08:56
*** dtantsur|afk is now known as dtantsur08:57
*** mdbooth_ has quit IRC09:06
*** mdbooth has joined #openstack-nova09:06
*** luksky11 has joined #openstack-nova09:09
*** dpawlik_ has quit IRC09:13
*** dpawlik has joined #openstack-nova09:13
*** mdbooth_ has joined #openstack-nova09:20
*** mdbooth has quit IRC09:23
*** markvoelker has quit IRC09:25
*** elod has quit IRC09:26
openstackgerritVlad Gusev proposed openstack/nova stable/queens: WIP libvirt: Reduce calls to qemu-img during update_available_resource  https://review.openstack.org/60335809:26
*** elod has joined #openstack-nova09:27
openstackgerritVlad Gusev proposed openstack/nova stable/queens: libvirt: Reduce calls to qemu-img during update_available_resource  https://review.openstack.org/60335809:33
*** s10 has joined #openstack-nova09:33
*** maciejjozefczyk has quit IRC09:40
*** yingjun has quit IRC09:41
bauzaskashyap: sorry, was on meeting09:46
*** owalsh has quit IRC09:46
bauzaskashyap: do we recreate the guest XML on start() ?09:48
bauzaskashyap: I'm confused by https://review.openstack.org/#/c/602592/3/releasenotes/notes/Use-virt-as-machine-type-for-ARMv7-cd2c252336057ec8.yaml09:48
bauzasbut I can check09:48
bauzaskashyap: nevermind, I check09:49
bauzaschecked*09:49
*** Dinesh_Bhor has quit IRC09:49
bauzashttps://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L293109:50
bauzasoops, bad link https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L289509:51
*** owalsh has joined #openstack-nova09:52
*** Dinesh_Bhor has joined #openstack-nova09:52
kashyapbauzas: Sorry, was out to get some groceries09:58
bauzasnah no worries09:58
kashyapbauzas: Would you like me to update the text here: https://review.openstack.org/#/c/602592/3/releasenotes/notes/Use-virt-as-machine-type-for-ARMv7-cd2c252336057ec8.yaml09:59
kashyapbauzas: Since your question is answered (by yourself), or is the text good as-is for you?09:59
kashyapAh, you +2ed it09:59
*** maciejjozefczyk has joined #openstack-nova09:59
kashyapThanks for looking!10:00
*** Dinesh_Bhor has quit IRC10:04
*** tbachman has quit IRC10:10
*** markvoelker has joined #openstack-nova10:30
*** tbachman has joined #openstack-nova10:36
*** Dinesh_Bhor has joined #openstack-nova10:37
*** hoangcx has quit IRC10:38
*** hoangcx has joined #openstack-nova10:38
*** Dinesh_Bhor has quit IRC10:39
*** pooja_jadhav has quit IRC10:40
*** tbachman has quit IRC10:40
*** tbachman has joined #openstack-nova10:48
*** pooja_jadhav has joined #openstack-nova10:56
*** pooja-jadhav has joined #openstack-nova10:57
*** markvoelker has quit IRC11:00
*** erlon has joined #openstack-nova11:02
*** tbachman has quit IRC11:09
*** ShilpaSD has quit IRC11:10
*** priteau has joined #openstack-nova11:16
*** imacdonn has quit IRC11:19
*** imacdonn has joined #openstack-nova11:20
*** jpena is now known as jpena|lunch11:20
*** cdent has joined #openstack-nova11:25
*** sambetts|afk is now known as sambetts11:35
*** med_ has joined #openstack-nova11:42
*** owalsh has quit IRC11:42
*** pvc has joined #openstack-nova11:47
*** tssurya has joined #openstack-nova11:48
pvcgood evening bauzas is vgpu supported on ocata?11:49
bauzasnope, queens only11:49
pvcqueens and rocky right?11:50
bauzasbut you can passthru in ocata11:50
bauzasqueens and later, yes11:50
pvcthank you so much11:50
*** udesale has quit IRC11:56
*** brinzhang has quit IRC11:56
*** markvoelker has joined #openstack-nova11:57
tssuryajohnthetubaguy or alex_xu: could you review https://review.openstack.org/#/c/596285/ whenever you have the time ?12:01
sean-k-mooneytssurya: thanks for working on https://review.openstack.org/#/c/603352/2/specs/stein/approved/alloc-candidates-negative-member-of.rst. i saw it yesterday when i went to write up the same spec :)12:11
sean-k-mooneytssurya: i have not had time to review properly yet but at a glance you seam to have captured the important points12:12
tssuryasean-k-mooney: you are surely talking to the wrong person :)12:13
jaypipesstephenfin: do you really have someone on your internal team named Jon Snow?12:13
*** tbachman has joined #openstack-nova12:17
stephenfinjaypipes: Within the company, yes, and the puns come thick and fast12:18
*** med_ has quit IRC12:20
jaypipesstephenfin: that is... well, that is awesome.12:20
jaypipesstephenfin: do you require him to carry a Valorian steel blade around the office?12:21
* jaypipes thinks you should12:21
tssuryaI am thinking he must have the hardest time with the sentence "You know no'ting Jon Snow"12:22
cdentsean-k-mooney: you're looking tetsuro12:22
stephenfinjaypipes: No, but it appears he is the least knowledgeable person in the company12:22
stephenfintssurya: Indeed :D12:22
tssurya:D12:22
jaypipestssurya: ++12:23
*** jpena|lunch is now known as jpena12:24
*** markvoelker has quit IRC12:31
*** skatsaounis has quit IRC12:36
sean-k-mooneycdent: tssurya sorry yes12:36
sean-k-mooneyjaypipes: quick question for the device whitelist spec im working on.12:42
openstackgerritLee Yarwood proposed openstack/nova stable/queens: libvirt: Always escape IPv6 addresses when used in migration URI  https://review.openstack.org/60373712:42
sean-k-mooneydid you want versioning, a schma or both.  if both would you like it to also cover migrations. trying to refine scope12:42
openstackgerritLee Yarwood proposed openstack/nova stable/pike: libvirt: Always escape IPv6 addresses when used in migration URI  https://review.openstack.org/60373812:43
openstackgerritLee Yarwood proposed openstack/nova stable/ocata: libvirt: Always escape IPv6 addresses when used in migration URI  https://review.openstack.org/60374012:44
jaypipessean-k-mooney: doesn't have to cover the evolution of the schema, just needs to have a schema and a version attribute in the schema so that we *can* evolve the schema over time.12:46
sean-k-mooneyjaypipes: that is much simpler :) i was hopping you would say that.12:46
jaypipessean-k-mooney: i.e. you don't need to go into the implementation details of evolving the schema over time. just make sure there's a schema and version.12:46
sean-k-mooneyjaypipes: im debating about proposing jsonscma as the validation language for the schmea but it does not have built in migration support. oslo.config also can do the validation but forces ini and ovo gives the versioning but is way to verbose and annoying for configfiles. so of the eaisly accessable tools there was not one i could tink of that nativly was file format independent, did validation and12:49
sean-k-mooneyhand native migration/versioning12:49
jaypipessean-k-mooney: go with JSONSchema.12:50
sean-k-mooneyjaypipes: thats what it current assumes12:50
jaypipessean-k-mooney: you can always use YAML as the serialization format later if you want.12:50
jaypipessean-k-mooney: but keep JSONSchema for validation.12:50
sean-k-mooneyjaypipes: yes i also assumed that would be the default12:50
sean-k-mooneyif we use json scema you can use json/toml/yaml as the file format and we dont need to care12:51
sean-k-mooneyalso i cant spell schema apparently12:51
jaypipessean-k-mooney: well, let's face it, spelling isn't your strong suit. :P12:52
openstackgerritJay Pipes proposed openstack/os-traits master: clean up CUDA traits  https://review.openstack.org/59717012:53
jaypipesstephenfin: ^12:53
stephenfinjaypipes: ack12:53
*** alexchadin has quit IRC12:53
*** alexchadin has joined #openstack-nova12:54
*** vivsoni has quit IRC12:56
*** vivsoni has joined #openstack-nova12:56
*** kukacz_ is now known as kukacz13:00
openstackgerrithuanhongda proposed openstack/nova stable/ocata: Cleanup RP and HM records while deleting a compute service.  https://review.openstack.org/60374913:02
bauzasfolks, bailing out for 20-ish minutes to see first plays of basketball from my oldest13:02
bauzas++13:02
*** ratailor has quit IRC13:03
*** gbarros has joined #openstack-nova13:03
*** holser_ has quit IRC13:04
sean-k-mooneyjaypipes: :) grammerly to the rescure. at least for the spec.13:04
*** holser__ has joined #openstack-nova13:04
*** Kevin_Zheng has quit IRC13:08
*** alexchadin has quit IRC13:11
openstackgerrithuanhongda proposed openstack/nova stable/ocata: Cleanup RP and HM records while deleting a compute service.  https://review.openstack.org/60374913:11
jaypipessean-k-mooney: grammarly you mean? :)13:15
jaypipessean-k-mooney: and I think you meant rescue, not rescure. P13:15
*** alexchadin has joined #openstack-nova13:15
jaypipessean-k-mooney: don't MAKE me come over to Ireland to be your personal spelling assistant.13:15
*** janki has quit IRC13:16
sean-k-mooneyjaypipes: you see if only they had a rest api that could be intergrated with my irc client life would be so much simpler. which reminds me i need to reinstall the aspell plugin...13:17
dimsLOLOL13:17
sean-k-mooneyjaypipes: :) im gong to push the very rough draft for some early feedback. ill be converting it to spec format and then starting on the actuall schema in json schema format in the next revision or two.13:19
*** mriedem has joined #openstack-nova13:21
*** mdbooth_ has quit IRC13:22
openstackgerritWenran Xiao proposed openstack/nova-specs master: Add suport selecting subnet when createing vm  https://review.openstack.org/60334413:24
*** mdbooth has joined #openstack-nova13:26
*** claudiub has joined #openstack-nova13:28
BlackDexHello there. I'm trying to manipulate the libvirt/qemu/kvm parameters without needing to cold-migrate/hard-reboot an instances13:28
BlackDexI changed the xml of an instances using `virsh edit`, i do a live-migrate of that instances and the xml on the destination host contains the correct values, but looking at the process `ps fauxw` it still uses the old values.13:29
BlackDexHow and where does nova determine which parameters to use for the live-migrate to spawn the instance to live-migrate to?13:30
claudiubyou're basically trying to resize the libvirt/qemu/kvm instance without restarting it?13:30
BlackDexclaudiub: yes13:30
claudiubheh, that sounds like live-resize13:30
BlackDexyea kinda is13:31
BlackDexbut i haven't seen that feature yet ;)13:31
BlackDexi need it only for the iotune parameters13:31
claudiubit's because the proposal for that feature didn't merge: https://review.openstack.org/#/c/141219/13:31
BlackDexthe iotune parameters only seem to be changed when using volume-types13:31
BlackDexthose seem te be dynamic when i change the min/max iops13:32
BlackDexbut not with ephemeral disks13:32
claudiubhmm, i'm not sure those are set for ephemeral disks13:34
BlackDexyou can set that in the flavor13:34
BlackDexquota:disk_read_iops_sec for instnace13:34
BlackDexthat works for an empemeral disk13:35
BlackDexbut, if you want to change that you need to resize/cold-migrate the instace for it to work13:35
BlackDexwhile volumes using a volume-type which are live-migrated will get a new iops setting if the volume-type has a qos set13:36
claudiubyeah, indeed, they are being set during resize.13:36
BlackDexi tried to change the mysql entries of the instance_extras table13:36
BlackDexthe volume-type qos is also set during a live-migrate!13:37
BlackDexif it previously wasn't13:37
*** udesale has joined #openstack-nova13:38
BlackDexso, i changed in the mysql everything of the instance metadata info, but it doesn't use those values during a live-migrate13:38
BlackDexso i wondered where the parameters for qemu are comming from durning the live-migrate13:39
BlackDexit seems they do not come from the mysql database, or the libvirt xml13:39
BlackDexthere are pulled from somewhere els13:39
BlackDexe13:39
*** munimeha1 has joined #openstack-nova13:42
*** maciejjozefczyk has quit IRC13:46
*** maciejjozefczyk has joined #openstack-nova13:46
*** sapd1 has joined #openstack-nova13:52
openstackgerritsean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy  https://review.openstack.org/60380513:53
openstackgerritSurya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down  https://review.openstack.org/56778513:55
openstackgerritSurya Seetharaman proposed openstack/nova master: Add scatter-gather-single-cell utility  https://review.openstack.org/59494713:55
openstackgerritMatt Riedemann proposed openstack/nova master: Resource retrieving: add changes-before filter  https://review.openstack.org/59927613:57
*** janki has joined #openstack-nova13:59
openstackgerritJay Pipes proposed openstack/os-traits master: clean up CUDA traits  https://review.openstack.org/59717014:04
*** janki has quit IRC14:04
*** mlavalle has joined #openstack-nova14:04
*** munimeha1 has quit IRC14:09
*** awaugama has joined #openstack-nova14:12
*** Luzi has quit IRC14:19
*** cfriesen has joined #openstack-nova14:22
mriedemdansmith: still humping approved stable changes through the gate, fyi14:23
dansmithack14:23
*** alexchadin has quit IRC14:25
*** s10 has quit IRC14:25
*** alexchadin has joined #openstack-nova14:30
*** gouthamr_ is now known as gouthamr14:35
*** udesale has quit IRC14:37
*** hamdyk has quit IRC14:38
mriedemjaypipes: i'm +2 on the 2.66 changes-before change now https://review.openstack.org/#/c/599276/14:39
jaypipesmriedem: yeah, I +W'd that a while ago :)14:39
jaypipesmriedem: "a while ago" == whole minutes ago.14:39
mriedemah heh14:40
mriedemcool14:40
*** alexchadin has quit IRC14:43
mriedemgmann: is https://review.openstack.org/#/c/596285/ the last change for https://blueprints.launchpad.net/nova/+spec/api-extensions-merge-stein ?14:48
*** munimeha1 has joined #openstack-nova14:49
cdentmnaser: I think you mentioned you had a huge database that you could test a nova->placement migration script on, to get a sense of timing and the like? If that's true dansmith has made a thing: https://review.openstack.org/#/c/603234/14:51
mnaseryup, i can try that out14:51
mriedemlooks like it is, the only other extension with @wsgi.extends for servers is hide_server_addresses and i think that's deprecated?14:51
dansmithmnaser: it just dumps/imports, so it should be safe to run against a live source deployment.. it doesn't delete anything from the source14:51
dansmithbut also, you could just measure the size of the tables and have a pretty good idea anyway14:52
lbragstadgmann  is https://review.openstack.org/#/c/547850/ still being pursued?14:52
*** marvin_mhg has quit IRC14:53
*** pvc has quit IRC14:55
mriedemmelwitt: btw, if you don't like geddy's voice, it's no crime to enjoy https://www.youtube.com/watch?v=iB4uwO1Dmf4 which is purely instrumental14:56
*** alexchadin has joined #openstack-nova14:57
johnthetubaguygmann: good question, he did say he was planning on doing something about that14:59
*** marvin_mhg has joined #openstack-nova15:00
mnaserso looks like the biggest table is consumers15:01
mnaseri tried it on a relatively large ~100ish deployment and it took 2 minutes with a triple replicated galera cluster with pretty beefy controllers15:02
bauzasstupid question, but say I'm migrating some libvirt instance and would like to introspect its domain XML, should I lookup the source or the destination ? (in other words, shall I look at the migration allocation or the instance allocation) ?15:02
*** janki has joined #openstack-nova15:03
mnaserbauzas: afaik the domain xml should be the same for live migrations at least afaik15:03
bauzasfor the moment, this is going to be a TODO since we don't support migrations for VGPU, but I need to make sure the code I'm writing for the reshape works15:03
mnaserdansmith: ^ fyi for some numbers, it's relatively painless comparing to something like.. cells v2 stuff15:03
bauzasmnaser: yup, but then the allocated PCI devices could differ15:03
mnaserbauzas: ah yes. that's a qurik i didn't think of15:04
dansmithmnaser: ack15:04
mnaserIt would be nice if it could pull in environment variables15:04
mnaserBut we could iterate on that later15:04
mnaserSo we don’t have to write credentials on disk when we don’t have to15:05
dansmithmnaser: you should be able to do that as-is15:05
dansmithwrite an empty config file and make those variables be in the environment already15:06
*** janki has quit IRC15:08
openstackgerritMerged openstack/nova master: Fix some typos in nova api ref doc  https://review.openstack.org/60330615:08
*** tbachman has quit IRC15:08
mriedembauzas: the answer depends on what you need to know,15:10
mriedemif you're cleaning up something on the source during live migratoin, then look at the source xml, else look at the dest xml15:10
bauzasmriedem: no, I just want to reshape the existing allocations onto the right physical device15:10
bauzasmriedem: for that, I need to get the corresponding mediated devices15:11
mriedemumm,15:11
mriedemreshape *during* a live migration?15:11
dansmithmnaser: I can make the db and host things not clobber environment too and I guess add a flag that will allow the file to be missing or something15:11
dansmithcredentials in environment is easy, but not more secure necessarily, so I guess I'm not sure why that's better, but alas :)15:12
mriedembauzas: at the start of live migration, conductor moves the existing allocations from the instance to the migration record, and then the scheduler is going to allocate resources from the dest (tree) for the instance15:12
mriedemso i'm not sure why you'd need to reshape at all15:12
bauzasmriedem: I know about how we manage allocations during a migration15:13
dansmithmriedem: assume he means a symbolic reshape of allocations right?15:13
dansmithnot an actual POST /reshaper operation15:13
bauzasthe case I'm concerned is a cold migrate (because live migrating vGPUs is YAGNI)15:13
mriedemmy point is, why?15:13
dansmithbut a self-healing given the opportunity provided by a move15:13
bauzasright15:13
dansmithI dunno, I don't know what he's trying to heal exactly15:13
bauzasso, say cold migrate is a thing15:13
mriedemyou can't heal a broken heart15:13
bauzasI mean, cold migrate a VGPU15:14
bauzas(which is not something there yet, but a bugfix I have targeted for Stein)15:14
bauzasthen cold migrating the instance would mean 2 allocations for this15:14
*** luksky11 has quit IRC15:15
bauzasone having the consumer UUID being the migration UUID on the source host, the other one being the real allocation15:15
bauzasright?15:15
mriedemand during the cold migrate you want to move the existing vgpu allocation from the root node provider on the source host to the child vgpu provider on the dest node?15:15
mriedemthe scheduler should take care of the latter15:15
mriedemand after we confirm the resize, we'll drop the former15:15
mriedemso i'm not sure why it matters15:15
bauzasmriedem: no, I'd then just move the allocation on the source host to be on the right vGPU provider on the same host15:15
dansmithwait what/15:16
*** munimeha1 has quit IRC15:16
dansmithwe will do the same migration-holds-source-allocation yeah?15:16
bauzasthat's what I think yeah15:16
* dansmith is confused15:16
mriedemme too15:17
mriedemi don't see the issue15:17
*** udesale has joined #openstack-nova15:17
bauzasI'm probably confused too by what means a reshape15:17
mriedemit sounds like you're trying to auto-heal/reshape duringa cold migrate so we don't have to run the maybe more expensive reshape-on-compute-startup upgrade thing15:17
bauzasduring the migration, we have 2 allocations on two different hosts, right?15:17
bauzasah shit, that's what I fucked15:18
dansmithum, what? :)15:18
dansmithlol.15:18
mriedemmaybe you want to just block az renames while the az has instances?15:18
mriedemthat's probably easier15:18
bauzascan we have a migration running, and both services to be restarted ?15:18
mriedemsure15:18
bauzasin that case, it would trigger a reshape non?15:18
mriedemsure15:18
bauzasfor both ?15:18
bauzasso in that interim period, we have two allocations, nope ?15:19
*** alexchadin has quit IRC15:19
mriedemif the allocations have moved from the instance to the migration record on the source host and the source host is restarted and a reshape happens, we should still move the allocations for the migration record15:19
mriedemthe reshape in this case happens on distinct provider trees15:19
bauzasthat's my point15:19
bauzasso in that case, I have a migration UUID that is the consumer15:20
bauzason the source host15:20
mriedemif the resize is confirmed, we drop the source node tree allocations for the migration record,15:20
mriedemon revert we drop the target node tree allocations for the instance and move them back to the instance for the source tree15:20
mriedembut the reshape should be ok15:20
bauzasok, but then I have to figure out the real instance UUID for the migration record15:20
bauzasthat's a special case15:21
mriedemthe migration consumer is just a fill in so someone doesn't quack quack seat back the source node resources15:21
mriedemhonestly i don't know what problem you're trying to solve15:21
mriedemwe should probably identify that a problem exists before discussing designs on how to fix it15:21
bauzasprobably15:21
bauzasand I'm unclear15:22
bauzasI'll just leave a comment in my patch and I move on15:22
dansmithmnaser: how much of your two minutes is writing and reading that dumpfile on disk? would you prefer we just pipe it between the export and import (optionally?) and maybe tee it out for forensics? might be faster15:28
*** dtantsur is now known as dtantsur|brb15:28
openstackgerritMatt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option  https://review.openstack.org/60383115:30
mriedemgmann: ^15:31
mnaserdansmith: like 3-5 seconds for the dump. The write took the longest15:34
dansmithmnaser: 3-5 seconds overhead for writing the file? or 3-5 seconds to do the dump and the rest of the minute to write it out?15:35
dansmithon my tiny database it's immeasurable of course15:36
dansmithI get a constant 3.6s to do the dump and import regardless15:36
dansmithmnaser: I have a diff that does it pipely if you want to try it15:36
openstackgerritMatt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option  https://review.openstack.org/60383115:36
*** gyee has joined #openstack-nova15:37
dansmithmnaser: https://termbin.co/FXJg15:37
dansmithI also put a "tee $tmpfile |" in the middle of that pipe (hence the comment) but removed it for your test15:38
*** helenafm has quit IRC15:40
*** tbachman has joined #openstack-nova15:44
*** dtantsur|brb is now known as dtantsur15:46
*** pcaruana has joined #openstack-nova15:47
openstackgerritBen Nemec proposed openstack/nova master: WIP: Migrate upgrade checks to oslo.upgradecheck  https://review.openstack.org/60349915:56
dansmithtssurya: mriedem shall we meet about cells? we just talked about things last week and melwitt is not around this week16:00
mnaserdansmith: sorry i was in a meeting, the dump was really quick but the push was much slower because of (what i assume) replication16:01
dansmithah the load, I see16:01
dansmithyeah I'm sure that's the heavy part16:02
dansmithit's heavy on my toy devstack even16:02
dansmithin that case, I'd just say we should leave separate the dump/load like it is now so that one can complete without the other, leaving a file you can just import if you want it16:03
*** mdbooth_ has joined #openstack-nova16:04
jaypipesstephenfin: so, unless I'm mis-reading your comments on the cpu-resource-tracking spec, you'd actually favor an "opt-in" approach to hyperthread usage. Is that correct? i.e. the virt driver defaults to *not* counting hyperthreads as CPU resources for guests unless some knob is turned on?16:05
*** mdbooth has quit IRC16:05
stephenfinjaypipes: For instances with dedicated CPUs, yes. However, we need to avoid breaking users16:06
stephenfinSo that knob would have to default to on16:06
stephenfinThe reason I want that is because a hyperthread != a core and I think the decision to model it as one was a mistake on day one16:06
jaypipesstephenfin: ack. no disagreement from me on that16:06
jaypipesstephenfin: agreed.16:07
*** mdbooth_ has quit IRC16:10
openstackgerritsean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy  https://review.openstack.org/60380516:10
*** mdbooth_ has joined #openstack-nova16:18
openstackgerritMerged openstack/nova stable/queens: Fix DB archiver AttributeError due to wrong table name attribute used  https://review.openstack.org/59988216:20
openstackgerritMerged openstack/nova stable/queens: VMware: fix TypeError while get console log  https://review.openstack.org/59136516:20
openstackgerritMerged openstack/nova stable/queens: Move conductor wait_until_ready() delay before manager init  https://review.openstack.org/59920016:20
*** mdbooth_ has quit IRC16:22
*** dtantsur is now known as dtantsur|afk16:23
*** sahid has quit IRC16:24
*** pcaruana has quit IRC16:27
*** udesale has quit IRC16:28
mriedemdansmith: i'm good to skip; plan on starting my poc for cross-cell resize today16:28
dansmithoh boy16:29
openstackgerritMerged openstack/nova stable/ocata: Return 400 when compute host is not found  https://review.openstack.org/59064916:30
stephenfinjaypipes: If you don't have one already, a placement cheatsheet (in PDF and therefore printable) would be super useful16:31
jaypipesstephenfin: also, I agree with you about getting rid of emulator_threads_policy eventually, but is it 100% critical that I update the cpu-resource-tracking spec with information about that?16:31
jaypipesstephenfin: you mean cheatsheet about the REST API?16:32
jaypipesstephenfin: or something else?16:32
jaypipesstephenfin: or you mean a glossary of placement terms?16:32
stephenfinjaypipes: No, it can be be handled separately. However, I imagine you do need to take cpu overhead, which is incremented for each dedicated emulator thread core, into account16:33
openstackgerritJack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots  https://review.openstack.org/60384416:33
stephenfinjaypipes: Both. Glossary of terms followed by examples of either the REST API or using osc-placement16:33
jaypipesstephenfin: ack16:34
stephenfinjaypipes: Just an idea, obviously. I might even start drafting something myself some point this week16:34
jaypipesstephenfin: cool. let me know when you get to the "what is a consumer?" part...16:34
*** holser__ has quit IRC16:36
stephenfinjaypipes: FYI, the reason I brought up the whole "let's kill 'isolate' idea" was that that approach seemed easier than handling the CPU overhead thing *and* it reduced complexity for the user in the process, which feels like one of the goals16:36
stephenfinBut again, not a blocker :)16:37
*** panda has joined #openstack-nova16:38
jaypipesstephenfin: yes, agreed. glad it's not a blocker though.16:39
openstackgerritSylvain Bauza proposed openstack/nova master: libvirt: mdevs returning parent and vendor PCI info  https://review.openstack.org/56230416:39
bauzasjaypipes: stephenfin: can one of you reapprove https://review.openstack.org/562304 ? I just rebased it out of the existing series and I need it for the reshaper change16:40
bauzasI'm surprised I lost my +2s on the rebase but meh16:40
bauzasprobably because I changed the commit id16:41
bauzasoh yeah, I changed the message, hence why16:41
jaypipesstephenfin: feel free to re-+W bauzas patch16:41
bauzasjaypipes: thanks16:41
bauzasI need the parenting relationship for reshaping the VGPU resources to the right PGPU RP16:42
stephenfinbauzas: Done16:42
bauzasstephenfin: ack, thanks16:43
*** derekh has quit IRC16:51
*** rabel has joined #openstack-nova16:53
*** gbarros has quit IRC16:53
openstackgerritJay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking  https://review.openstack.org/55508117:01
tobias-urdinping on a possible upgrade path bug q->r https://bugs.launchpad.net/nova/+bug/179335317:09
openstackLaunchpad bug 1793353 in OpenStack Compute (nova) "broken upgrade path q->r requirement for oslo.db" [Undecided,New]17:09
*** ralonsoh has quit IRC17:19
*** alex_xu has joined #openstack-nova17:26
*** sambetts is now known as sambetts|afk17:27
*** jiteka has joined #openstack-nova17:29
*** jpena is now known as jpena|off17:30
openstackgerritDavid Rabel proposed openstack/nova master: Really use source image format as default for snapshot_image_format  https://review.openstack.org/60385517:37
jitekahello, could someone confirm that migrate feature is working with flavor using vCPU pinning and Numa in Mitaka ?17:51
*** munimeha1 has joined #openstack-nova17:53
*** gbarros has joined #openstack-nova18:00
*** gbarros has quit IRC18:01
*** sapd1 has quit IRC18:01
openstackgerritMerged openstack/nova-specs master: Propose configurable maximum number of volumes to attach  https://review.openstack.org/59730618:01
mriedemjiteka: cfriesen might know18:10
mriedemthere were some patches that mellanox and windriver worked on to make that work, but i can't remember which release in which those patches landed18:11
mriedemjiteka: cold migrate i mean, not live migration18:11
mriedemnuma/pinned cpus does not work with live migration18:11
mriedemhttps://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-live-migration.html18:11
*** tbachman has quit IRC18:11
*** claudiub has quit IRC18:14
*** gbarros has joined #openstack-nova18:16
openstackgerritMerged openstack/nova stable/ocata: Add recreate test for RT.stats bug 1784705  https://review.openstack.org/58807618:28
openstackbug 1784705 in OpenStack Compute (nova) ocata "ResourceTracker.stats can leak across multiple ironic nodes" [High,In progress] https://launchpad.net/bugs/1784705 - Assigned to Matt Riedemann (mriedem)18:28
*** tbachman has joined #openstack-nova18:30
mriedemdansmith: would be good to get this pike backport in https://review.openstack.org/#/c/599883/18:31
dansmithmriedem: hmm, did I do that? ISTR some reasoning there.. maybe sqla version?18:32
mriedemit was originally your code yes, but table.name is standard18:32
*** gbarros has quit IRC18:36
mriedemdansmith: does anything jump out at you for the source of the IndexError in this failure? http://logs.openstack.org/72/600372/1/gate/openstack-tox-py35/dcfd363/job-output.txt.gz18:44
mriedemlooks like something between oslo.db/sqla/pymysql/eventlet switches context and we timeout after some huge amount of time18:44
mriedemnova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [664.512994s] ... FAILED18:45
mriedemnova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [39.644814s] ... ok18:45
mriedemclearly we're not just missing some short timeout window18:45
dansmithstill waiting for the damn log to load18:46
dansmithwow18:48
mriedemyeah we have a lot of these failures in the gate across master, rocky and queens,18:48
mriedemhttps://bugs.launchpad.net/cinder/+bug/179336418:48
openstackLaunchpad bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed]18:48
mriedemnova and cinder b/c they are using the same test fixtures18:49
dansmithhmm18:49
openstackgerritEric Fried proposed openstack/nova master: Add contributor guide for upgrade status checks  https://review.openstack.org/59690218:50
mriedemmaybe "connection.scalar(select([1]))"?18:50
dansmithdoes the index complaint surprise you? because it looks like a nonsense name18:50
mriedemwell i see this too18:51
mriedemsqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically18:51
dansmithyeah18:51
mriedemhttp://status.openstack.org/elastic-recheck/data/integrated_gate.html Overall Categorization Rate: 15.4%18:52
mriedemmeaning we (openstack) have a shit load of uncategorized failures killing stuff in the gate18:52
mriedemwhich is why it's taking us days to merge code18:52
*** BlackDex has quit IRC18:57
mriedemhttp://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20connection.scalar(select(%5B1%5D))'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d18:57
*** BlackDex has joined #openstack-nova18:58
cfriesenjiteka: I think cold migration should work in mitaka.  I think there are some gotchas where resource tracking isn't accurate until the next audit, and maybe there are some resource tracking issues if you revert a resize.19:04
*** gbarros has joined #openstack-nova19:04
*** munimeha1 has quit IRC19:05
tssuryadansmith: ack, let's skip19:06
cfriesenjiteka: a bit more digging shows that PCI devices were not cold migratable until Newton.19:06
mriedemcfriesen: did you find a specific change that makes that work?19:07
mriedemb/c we could make a note of that on https://docs.openstack.org/nova/latest/admin/pci-passthrough.html19:07
cfriesenmriedem: 5129f48226c I think19:09
cfriesenactually, looks like there are multiple:  https://bugs.launchpad.net/nova/+bug/151288019:10
openstackLaunchpad bug 1512880 in OpenStack Compute (nova) newton "Failed cold migration with SR-IOV" [Medium,Fix released]19:10
mriedemyeah was just going to say that https://review.openstack.org/#/q/topic:bug/1512880+(status:open+OR+status:merged)19:10
mriedemi think it's fair to say it didn't work until newton19:11
*** lucidguy has joined #openstack-nova19:27
lucidguyI believe this bug applies to me, unfortunately I don't know how to resolve, assistance?  https://bugs.launchpad.net/tripleo/+bug/178556819:28
openstackLaunchpad bug 1785568 in OpenStack Compute (nova) "Multiple migration requests for same vm might fail" [Undecided,Incomplete]19:28
mriedemcfriesen: so if you have a minute, could you just update the note at the top of https://docs.openstack.org/nova/latest/admin/pci-passthrough.html about sriov to also mention that cold migration of servers with sriov ports attached didn't work until newton and reference that bug?19:29
*** awaugama has quit IRC19:38
openstackgerritSurya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down  https://review.openstack.org/59165819:42
openstackgerritSurya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down  https://review.openstack.org/58482919:46
*** tssurya has quit IRC19:51
openstackgerritMerged openstack/nova-specs master: fix spelling mistake  https://review.openstack.org/57347920:12
*** erlon has quit IRC20:24
*** cdent has quit IRC20:27
cfriesenanyone know anything about "bad magic number in nova.conf.crypto" when trying to build nova docs?20:32
cfriesenhttp://paste.openstack.org/show/730383/20:33
mriedemi've seen that i think...20:35
*** hemna_ has quit IRC20:36
mriedempy34 isn't supported...20:37
mriedemnot sure if that would be related20:37
mriedemalso,20:37
mriedemnova.conf.crypto isn't in master20:37
mriedemis that a starlingx special?20:38
mriedemoh pike20:38
mriedemhttps://github.com/openstack/nova/blob/stable/pike/nova/conf/crypto.py20:38
cfriesenmy regular "python" is 2.7.5.  not sure why tox would use 3.4 unless it's searching for whether it's present.   I'm on the master branch of upstream nova, though I have checked out pike in this repo.20:39
*** priteau has quit IRC20:39
*** dklyle has quit IRC20:47
cfriesenfound a nova/conf/crypto.pyc file left over from a previous branch.  deleted and retrying20:51
*** gbarros has quit IRC20:53
openstackgerritMatt Riedemann proposed openstack/nova master: Mention SR-IOV cold migration limitation in admin docs  https://review.openstack.org/60390920:54
mriedemcfriesen: huh, i thought we used to have a thing in all tox runs that would clean out pycs20:55
mriedembut i also seem to remember someone removing that years ago20:55
mriedemoh right it's in the base testenv,20:55
mriedembut commands is overridden in the actual testenv:docs run20:55
mriedemcfriesen: oh were you building those docs ^20:56
mriedemfor the thing i just pushed?20:56
cfriesenheh...yep20:58
mriedemsorry, you didn't say anything so i assumed you were busy with something else20:58
cfriesenno worries.20:58
openstackgerritMohammed Naser proposed openstack/nova master: Default zero disk flavor to RULE_ADMIN_API in Stein  https://review.openstack.org/60391020:59
cfriesenmriedem: I almost feel like we should use PCI in there instead of SR-IOV, since it's the more generic form.   Not all PCI passthrough is SR-IOV, but all SR-IOV is a form of PCI passthrough.21:00
openstackgerritMohammed Naser proposed openstack/nova master: Drop migration pre-check error for file_backed_memory  https://review.openstack.org/60391121:02
mnasermriedem: ^ i thought we were in openstack-nova oops21:02
cfriesenmnaser: you are. :)21:02
mnaseroh no i was talking in #o-operators21:03
mnaserpushing stuffa nd wondering why the hell the bot wasn't notifying21:03
cfriesenthat'll do it21:03
mriedemcfriesen: feel free to -1 my change and break my heart21:04
mnaserthats my nova commit for the cycle21:05
mriedemmnaser: i'm guessing you didn't run this through tests yet right?21:05
* mnaser &21:05
cfriesenmriedem: I commented on it21:05
mnasermriedem: nope21:05
mnaseri probably should21:05
mnaserwell it looks like tests should pass because the tests use a fixture of using RULE_ADMIN_API and making sure it fails21:07
mnaserso it was testing the then-future case21:07
mriedemmnaser: at least EnforceVolumeBackedForZeroDiskFlavorTestCase in tox -e functional21:07
mnasermriedem: running21:08
mriedemcomments inline21:09
mnaserlemme address after seeing if test breaks21:09
openstackgerritMohammed Naser proposed openstack/nova master: Default zero disk flavor to RULE_ADMIN_API in Stein  https://review.openstack.org/60391021:15
*** _pewp_ has quit IRC21:18
*** cseader has quit IRC21:19
*** _pewp_ has joined #openstack-nova21:19
mriedemlbragstad: are dashes in policy rule names frowned upon?21:19
mriedeme.g. os_compute_api:servers:cross-cell-resize?21:19
mriedemshould be os_compute_api:servers:cross_cell_resize?21:19
lbragstadi don't think so21:19
lbragstadbut when i went through a bunch of the projects last week there isn't really a standard21:20
mriedemshocking21:20
lbragstadinoright?21:20
lbragstadso - i guess if we're going to try and do this, i'd like to whack all the moles at once...21:20
mriedemsince the prefix is os_compute_api i assume suffix should also be underscores21:20
mriedemoh and we have things like network:attach_external_network and create:zero_disk_flavor so nvm21:21
mriedemanswered my own question21:21
lbragstaddid you see my note about using the service name?21:21
mriedemno, where?21:21
* lbragstad fetches link21:21
mriedemi have'nt read the ML thread21:21
lbragstadhttp://lists.openstack.org/pipermail/openstack-dev/2018-September/134597.html21:22
lbragstadit starts there...21:22
lbragstadsorry if you we waiting to parse the lists later :)21:22
mriedem:(21:22
lbragstadbut i'm trying to update that with all context of what i've found across projects (since operators are on that note, too)21:22
mriedemso you'd like to see compute:server:cross_cell_resize21:24
mriedemnote this is an action,21:24
mriedemso it won't cleanly map to a method21:25
lbragstadright21:25
lbragstadwhich i guess is another reason to not put http methods in policy names?21:25
mriedemi mean, we could do compute:servers:resize:cross_cell21:25
lbragstadyeah - i don't think nova would be super special in that case... there are other projects have more than just <service>:<resource>:<action>21:26
lbragstadironic and heat for example21:26
mriedemalright, this is months/releases away from landing so i'll just do that and move on21:27
mriedemthis is the amount of progress i've made on this since last week21:27
lbragstadwhat's months away from landing?21:27
mriedemcross-cell resize21:27
lbragstadoh - sure21:28
mriedemnot your thing21:28
mriedembeing cross project, your thing is *years* away from landing :)21:28
lbragstadi'd like to get the convention "established" as soon as possible, because I think it's going to affect gmann's work too21:28
lbragstadright - let's be real here21:28
mnaseris there a documented process for when you end up with: "Instance 13a09ff5-7124-4b3c-8229-a8691a2f43b4 has allocations against this compute host but is not found in the database."21:41
mnasercontext: dns was borked on a compute, live migration would complete, but the placement migration allocation thing fails because compute cant talk to placement21:41
*** dklyle has joined #openstack-nova21:42
mriedemthat doesn't sound like a scenario in which you'd see that message21:43
mnaserwell, it happened after fixing dns21:44
mnaserthe compute node finally started talking to placement properly and now complains forever21:44
mriedemis that instance in the db?21:44
mnaserhmm21:44
mriedemthe code calls out 2 cases that could happen:21:45
mnaserno, i think the exception i was seeing was unrelated21:45
mnaseri think it was no placement + vm deleted21:45
mnaseri think i jumped to a conclusion there21:45
mriedem1. compute RT is racing with the scheduler where the scheduler created allocations in placement but didn't yet create the instance in the cell db,21:45
mriedem2. the instance was deleted and archived/purged from the db, but the alloctions are still in placement for that node21:45
mriedemyeah,21:45
mriedemso if the compute couldn't talk to placement when the vm was deleted, we'd fail to cleanup the allocations21:46
mriedemhttps://docs.openstack.org/osc-placement/latest/cli/index.html#resource-provider-allocation-delete21:46
mnaserthanks for making life easy and writing those clie stuf21:46
mriedemopenstack resource provider allocation delete 13a09ff5-7124-4b3c-8229-a8691a2f43b421:46
mriedemif the instance is truly deleted21:46
mriedemif it were moved and allocations were just messed up, doing ^ would be bad21:47
mnaserill double check they're all deleted21:47
mriedemmight be good to update that log message with the same info21:47
mnasertail -1000 /var/log/nova/nova-compute.log | grep 'has allocations against this compute host but is not found in the database.' | cut -d' ' -f13 | sort | uniq | xargs -n1 echo nova show21:47
mnasermagic21:47
mriedem"If you continue to see this message and have confirmed the instance is truly gone from the database, you can run 'openstack resource provider allocation delete <uuid>' to remove the allocation from the placement service."21:48
mnaseryeah just even document it somewhere i guess21:48
mnaseri mean21:48
mriedemwell,21:48
mnasercouldnt nova confirm that it doesnt exist in the db21:48
mriedemnot if it's a race21:49
mnaserand technically just delete it when it polls21:49
mnaserah21:49
mnaseri guess we'd have to check N times that it's not there but that starts becoming silly i guess21:49
mriedemas noted, we create the allocations in scheduling before the instance is created in the cell db21:49
mriedemright21:49
mnaseror if its been running for a while21:49
mriedemputting it into some kind of faq could work, but people might not see that and come here asking the same thing21:49
mnaserlike 300s age21:49
mnaser300s isn't a race condition at that point i guess21:50
mnaserbut i guess log is the easiest path21:50
mriedemrather than nova auto-delete your stuff, i'd prefer to just document weird edge cases for manual intervention21:50
mriedemwe've also talked about (last week it came up too) a command to compare placement allocations against nova instances,21:51
mriedemthe opposite of heal_allocations,21:51
*** takashin has joined #openstack-nova21:51
mriedemmore of a "tell me what garbage is in the placement db b/c nova didn't clean up after itself"21:51
mriedembut we can't reliably do that until we have consumer types in placement...21:51
mriedemtakashin: is this todo still valid https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3509 given we check the specified host is in the same cell above? https://github.com/openstack/nova/blob/master/nova/compute/api.py#L340521:54
mriedemi think we can remove that todo; when we call objects.ComputeNode.get_first_node_by_host_for_old_compat the context is targeted to the cell in which the instance currently exists21:55
mriedemso if the specified host is in another cell, we'll get ComputeHostNotFound there21:56
*** owalsh has joined #openstack-nova21:57
*** owalsh has quit IRC21:58
mriedemjohnthetubaguy: we should remove this rebuild parameter from the conductor migrate_server task API :) https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L26321:58
mriedemthat must have been something someone talked about way back in 2013 to unify the various move APIs21:59
*** vishwanathj has joined #openstack-nova22:02
takashinmriedem: you are right. We can remove the 'todo'.22:10
mriedemtakashin: please push a change to remove it and i'll +222:10
takashinmriedem: okay. I will submit the patch.22:11
vishwanathjhi looking for guidance on how to limit the number of instances on certain set of hosts...22:13
vishwanathjnot clear on sequence of steps to be executed on controller and compute nodes to get this working22:15
*** mlavalle has quit IRC22:22
mriedemthere is a NumInstancesFilter22:31
mriedemvishwanathj: https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#numinstancesfilter22:32
mriedemand https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#aggregatenuminstancesfilter for aggregates22:32
openstackgerritJack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots  https://review.openstack.org/60384422:33
mriedemthere is a known bug with those filters https://bugs.launchpad.net/nova/+bug/174032022:33
openstackLaunchpad bug 1740320 in OpenStack Compute (nova) "nova-scheduler does not honor max_instances_per_host set to a host aggregate" [Undecided,Confirmed]22:33
mriedemrace bug22:34
*** hoonetorg has quit IRC22:41
*** vishwanathj has quit IRC22:41
*** rcernin has joined #openstack-nova22:44
openstackgerritTakashi NATSUME proposed openstack/nova master: Remove an unnecessary comment  https://review.openstack.org/60392422:47
takashinmriedem: I submitted https://review.openstack.org/#/c/603924 .22:49
sorrisonAnyone know how to re kick a gate job that failed due to a timeout. https://review.openstack.org/#/c/526558/ I tried a recheck but that aint it22:57
sorrisonmriedem: ^22:57
*** hoonetorg has joined #openstack-nova22:58
mriedemsorrison: the gate has been eating a ton of dairy23:02
mriedemplug 526558 into http://zuul.openstack.org/23:02
mriedemit's queued up23:03
mriedemhttp://lists.openstack.org/pipermail/openstack-dev/2018-September/134867.html23:03
sorrisonok thanks23:04
*** mvkr has joined #openstack-nova23:10
mriedemtakashin: got it, thanks23:19
*** slaweq has quit IRC23:19
takashinmriedem: Thank you for your review.23:20
*** dklyle has quit IRC23:29
openstackgerritMatt Riedemann proposed openstack/nova master: WIP: Cross-cell resize  https://review.openstack.org/60393023:40
mriedemship it!23:40
*** mriedem is now known as mriedem_away23:44
gmannlbragstad: i will check the ML today and get back to you.23:52

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!