*** mvkr has quit IRC | 00:00 | |
*** hamzy has joined #openstack-nova | 00:00 | |
*** gyee has quit IRC | 00:18 | |
*** ganso has quit IRC | 00:21 | |
*** betherly has joined #openstack-nova | 00:23 | |
*** ganso has joined #openstack-nova | 00:24 | |
*** brinzhang has joined #openstack-nova | 00:26 | |
*** betherly has quit IRC | 00:27 | |
*** BjoernT has joined #openstack-nova | 00:28 | |
*** dklyle has joined #openstack-nova | 00:30 | |
*** BjoernT has quit IRC | 00:35 | |
*** BjoernT has joined #openstack-nova | 00:35 | |
*** mriedem has quit IRC | 00:41 | |
*** ricolin has joined #openstack-nova | 00:48 | |
*** BjoernT has quit IRC | 00:50 | |
*** bhagyashris has joined #openstack-nova | 00:58 | |
*** BjoernT has joined #openstack-nova | 00:58 | |
*** JamesBenson has joined #openstack-nova | 00:59 | |
*** BjoernT has quit IRC | 01:02 | |
*** JamesBenson has quit IRC | 01:05 | |
*** rcernin has quit IRC | 01:16 | |
*** rcernin has joined #openstack-nova | 01:17 | |
*** rcernin has quit IRC | 01:18 | |
*** rcernin has joined #openstack-nova | 01:19 | |
*** BjoernT has joined #openstack-nova | 01:28 | |
*** ttsiouts has joined #openstack-nova | 01:29 | |
*** whoami-rajat has joined #openstack-nova | 01:34 | |
*** betherly has joined #openstack-nova | 01:34 | |
*** macza has joined #openstack-nova | 01:35 | |
*** macza has quit IRC | 01:39 | |
*** betherly has quit IRC | 01:40 | |
*** nicolasbock has quit IRC | 01:42 | |
*** boxiang has joined #openstack-nova | 01:48 | |
*** tetsuro has joined #openstack-nova | 01:49 | |
openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix failure to boot instances with qcow2 format images https://review.opendev.org/640271 | 01:54 |
---|---|---|
*** ttsiouts has quit IRC | 02:02 | |
openstackgerrit | Merged openstack/nova master: Add zvm driver image type capabilities https://review.opendev.org/655732 | 02:05 |
*** cfriesen has quit IRC | 02:05 | |
*** betherly has joined #openstack-nova | 02:06 | |
*** betherly has quit IRC | 02:10 | |
openstackgerrit | Brin Zhang proposed openstack/nova master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659201 | 02:26 |
openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659202 | 02:30 |
openstackgerrit | Merged openstack/nova master: Disable limit if affinity(anti)/same(different)host is requested https://review.opendev.org/658110 | 02:35 |
*** tetsuro_ has joined #openstack-nova | 02:37 | |
*** betherly has joined #openstack-nova | 02:37 | |
*** yedongcan has joined #openstack-nova | 02:37 | |
*** hongbin has joined #openstack-nova | 02:39 | |
*** tetsuro has quit IRC | 02:39 | |
*** betherly has quit IRC | 02:41 | |
*** tetsuro has joined #openstack-nova | 02:42 | |
openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659205 | 02:44 |
*** tetsuro_ has quit IRC | 02:46 | |
*** bbowen has quit IRC | 02:46 | |
openstackgerrit | Tetsuro Nakamura proposed openstack/nova stable/stein: Skip _exclude_nested_providers() if not nested https://review.opendev.org/659206 | 02:46 |
openstackgerrit | Tetsuro Nakamura proposed openstack/nova stable/rocky: Skip _exclude_nested_providers() if not nested https://review.opendev.org/659207 | 02:51 |
*** tkajinam has quit IRC | 02:55 | |
*** tkajinam has joined #openstack-nova | 02:56 | |
*** betherly has joined #openstack-nova | 02:57 | |
openstackgerrit | Boxiang Zhu proposed openstack/nova master: Add host and hypervisor_hostname flag to create server https://review.opendev.org/645520 | 02:58 |
*** betherly has quit IRC | 03:02 | |
*** bbowen has joined #openstack-nova | 03:02 | |
*** BjoernT has quit IRC | 03:15 | |
*** hongbin has quit IRC | 03:18 | |
*** betherly has joined #openstack-nova | 03:28 | |
*** betherly has quit IRC | 03:33 | |
*** cfriesen has joined #openstack-nova | 03:38 | |
openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659202 | 03:43 |
*** tetsuro has quit IRC | 03:50 | |
*** tetsuro has joined #openstack-nova | 03:52 | |
*** hemna has joined #openstack-nova | 03:53 | |
*** psachin has joined #openstack-nova | 03:56 | |
*** udesale has joined #openstack-nova | 03:58 | |
*** betherly has joined #openstack-nova | 03:59 | |
*** ttsiouts has joined #openstack-nova | 03:59 | |
*** cfriesen has quit IRC | 04:01 | |
*** betherly has quit IRC | 04:04 | |
*** ttsiouts has quit IRC | 04:31 | |
*** hemna has quit IRC | 04:36 | |
*** tetsuro has quit IRC | 04:37 | |
*** mvkr has joined #openstack-nova | 04:37 | |
*** abhishekk has joined #openstack-nova | 04:42 | |
*** janki has joined #openstack-nova | 04:43 | |
*** Luzi has joined #openstack-nova | 04:46 | |
*** betherly has joined #openstack-nova | 04:51 | |
*** brinzh has joined #openstack-nova | 04:54 | |
*** betherly has quit IRC | 04:56 | |
*** brinzhang has quit IRC | 04:58 | |
*** vincentchong1 has joined #openstack-nova | 05:24 | |
*** alex_xu has quit IRC | 05:26 | |
*** ratailor has joined #openstack-nova | 05:27 | |
*** betherly has joined #openstack-nova | 05:32 | |
*** betherly has quit IRC | 05:37 | |
*** spsurya has joined #openstack-nova | 05:49 | |
*** betherly has joined #openstack-nova | 05:53 | |
*** psachin has quit IRC | 05:54 | |
*** psachin has joined #openstack-nova | 05:57 | |
*** betherly has quit IRC | 05:58 | |
*** awalende has joined #openstack-nova | 06:04 | |
*** awalende has quit IRC | 06:08 | |
*** threestrands has joined #openstack-nova | 06:12 | |
*** dpawlik has joined #openstack-nova | 06:23 | |
*** lpetrut has joined #openstack-nova | 06:23 | |
*** kaisers has joined #openstack-nova | 06:24 | |
*** vincentchong1 has quit IRC | 06:24 | |
*** vincentchong1 has joined #openstack-nova | 06:25 | |
*** ttsiouts has joined #openstack-nova | 06:29 | |
*** jaosorior has quit IRC | 06:42 | |
*** mvkr has quit IRC | 06:43 | |
*** luksky has joined #openstack-nova | 06:43 | |
*** maciejjozefczyk has joined #openstack-nova | 06:48 | |
*** jaosorior has joined #openstack-nova | 06:51 | |
*** vincentchong1 has quit IRC | 06:52 | |
*** awalende has joined #openstack-nova | 07:01 | |
*** ttsiouts has quit IRC | 07:02 | |
*** yonglihe has joined #openstack-nova | 07:04 | |
*** awalende_ has joined #openstack-nova | 07:04 | |
*** awalende has quit IRC | 07:05 | |
*** abhishekk has quit IRC | 07:05 | |
*** alex_xu has joined #openstack-nova | 07:05 | |
openstackgerrit | Brin Zhang proposed openstack/python-novaclient master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659202 | 07:06 |
*** rpittau|afk is now known as rpittau | 07:07 | |
*** tssurya has joined #openstack-nova | 07:07 | |
*** aarents has joined #openstack-nova | 07:08 | |
*** tesseract has joined #openstack-nova | 07:13 | |
*** awalende_ has quit IRC | 07:17 | |
*** awalende has joined #openstack-nova | 07:17 | |
*** helenafm has joined #openstack-nova | 07:18 | |
*** awalende_ has joined #openstack-nova | 07:20 | |
*** awalende has quit IRC | 07:22 | |
*** awalende_ has quit IRC | 07:22 | |
*** zhongjun2_ has joined #openstack-nova | 07:23 | |
*** awalende has joined #openstack-nova | 07:25 | |
*** rcernin has quit IRC | 07:25 | |
*** zigo has quit IRC | 07:28 | |
*** ttsiouts has joined #openstack-nova | 07:30 | |
*** imacdonn has quit IRC | 07:47 | |
*** imacdonn has joined #openstack-nova | 07:51 | |
*** boxiang has quit IRC | 07:51 | |
*** ivve has joined #openstack-nova | 07:54 | |
sean-k-mooney | bauzas: have you seen http://logs.openstack.org/97/652197/17/check/grenade-py3/7b3cac4/logs/grenade.sh.txt.gz#_2019-05-15_02_34_39_701 recently | 08:08 |
*** tkajinam has quit IRC | 08:08 | |
sean-k-mooney | im not sure if our grenade job is broken after the change to os-dev | 08:08 |
sean-k-mooney | *opendev | 08:08 |
*** awalende has quit IRC | 08:11 | |
*** awalende has joined #openstack-nova | 08:11 | |
*** awalende has quit IRC | 08:16 | |
*** lennyb has joined #openstack-nova | 08:16 | |
*** threestrands has quit IRC | 08:18 | |
sean-k-mooney | ya... 337 hits in the last 12 hours http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Could%20not%20install%20packages%20due%20to%20an%20EnvironmentError:%20HTTPSConnectionPool(host%3D'git.openstack.org',%20port%3D443)%5C%22 | 08:19 |
bauzas | sean-k-mooney: nope, not yet | 08:23 |
*** awalende has joined #openstack-nova | 08:24 | |
bauzas | sean-k-mooney: there is a redirect for https://git.openstack.org/cgit/openstack/requirements/plain/upper-constraints.txt | 08:25 |
sean-k-mooney | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22Could%20not%20install%20packages%20due%20to%20an%20EnvironmentError:%20HTTPSConnectionPool(host%3D'git.openstack.org',%20port%3D443)%5C%22 | 08:25 |
bauzas | sean-k-mooney: so we shouldn't get the exception | 08:25 |
sean-k-mooney | bauzas: we should not but there have been 337 filures since in the last week | 08:26 |
sean-k-mooney | so i think its not always working | 08:26 |
sean-k-mooney | i just mention this on the infra channel | 08:26 |
bauzas | ok | 08:26 |
sean-k-mooney | i might create an elastic recheck query and file a bug but im not sure where to file it | 08:26 |
bauzas | good question | 08:27 |
bauzas | we used to create bugs in Nova even if they were related to infra | 08:27 |
bauzas | and we marked them as critical | 08:27 |
bauzas | we also had a tag for them | 08:27 |
sean-k-mooney | ok i can do that. its affecting neutron jobs too but the grenade jobs seam to be the onse teh hit it most | 08:28 |
bauzas | sean-k-mooney: https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure | 08:28 |
sean-k-mooney | ah cool ill file one and add the gate-failure tag | 08:29 |
*** ttsiouts has quit IRC | 08:34 | |
*** dtantsur|afk is now known as dtantsur | 08:35 | |
*** ralonsoh has joined #openstack-nova | 08:46 | |
openstackgerrit | Surya Seetharaman proposed openstack/nova stable/stein: Disable limit if affinity(anti)/same(different)host is requested https://review.opendev.org/659239 | 08:48 |
sean-k-mooney | bauzas: should i set it to critical by the way or just high. i have hit it twice in a row but i dont think its failing in all builds its just annoying | 08:49 |
bauzas | sean-k-mooney: on 1:1, sorry | 08:49 |
sean-k-mooney | no worries | 08:49 |
*** pcaruana has joined #openstack-nova | 08:51 | |
*** derekh has joined #openstack-nova | 08:57 | |
openstackgerrit | Alex Xu proposed openstack/nova master: Remove the request_destination check in the scheduler https://review.opendev.org/659243 | 08:58 |
alex_xu | ^ efried, is it something we can removed? | 08:59 |
*** luksky has quit IRC | 09:01 | |
bhagyashris | efried, bauzas: Hi, I just want to discuss about http://logs.openstack.org/81/555081/28/check/openstack-tox-docs/6314f69/html/specs/train/approved/cpu-resources.html#configuration-options regarding the reserved_host_cpu calculation for PCPU resource, | 09:02 |
bhagyashris | f I have total 16 cpus and 4 are assigned to cpu_dedicated_set and 2 are assign to cpu_shared_set then in that case reserved_cpus for PCPU resource class will be (16- (4+2)) = 10 and for VCPU resource class it will be 0 . | 09:03 |
bhagyashris | Then in that case if I request instance which requires 1 PCPU then placement returns the 0 allocation_candiacted () (reason is it fails at _capacity_check_clause()) | 09:03 |
bhagyashris | Please correct me if I am wrong while calculating the reserved host cpu. | 09:03 |
kashyap | sean-k-mooney: dansmith: Yesterday you said: just like how you don't want Nova to not land instances based on "fan speed, or cpu temperature, or firmware patch levels..." you also don't want to stop launching instances on Compute hosts with vulnerable hypervisors. | 09:04 |
openstackgerrit | Surya Seetharaman proposed openstack/nova stable/rocky: Disable limit if affinity(anti)/same(different)host is requested https://review.opendev.org/659246 | 09:04 |
kashyap | sean-k-mooney: dansmith: Fan speed & CPU temp are really different, as they don't really affect the guest. | 09:05 |
*** jangutter has quit IRC | 09:06 | |
bhagyashris | sean-k-mooney: ^^ | 09:07 |
openstackgerrit | Surya Seetharaman proposed openstack/nova stable/queens: Disable limit if affinity(anti)/same(different)host is requested https://review.opendev.org/659247 | 09:08 |
*** brinzhang has joined #openstack-nova | 09:08 | |
*** brinzh has quit IRC | 09:10 | |
sean-k-mooney | bhagyashris: no | 09:23 |
sean-k-mooney | if you have 4 cpus in cpu_dedicated_set then that capasity of the inventory will b 4 and the resrved value will be 0 | 09:24 |
*** mdbooth_ has joined #openstack-nova | 09:25 | |
sean-k-mooney | similarly for vcpus the capasity will be defiened by the total number of cores defiened in the cpu_shared_set | 09:25 |
sean-k-mooney | bhagyashris: so the reserved value will effectivly always be 0 unless you set it to something else via the api | 09:25 |
bhagyashris | sean-k-mooney: How? but What formula says (set(all_cpus) - (set(dedicated) - set(shared))) | 09:26 |
sean-k-mooney | that is not waht the current version fo the spec say if i rememeber correctly | 09:27 |
bhagyashris | So in my case the total cpu count on host is 16 and I have define dedicated set as 4 cpus and shared as 2 cpus | 09:27 |
bhagyashris | Ok then in my case what will be the all_cpus value | 09:27 |
sean-k-mooney | then the inventory will be PCUP total 4 reserved 0 | 09:27 |
sean-k-mooney | no | 09:27 |
sean-k-mooney | we will only be reporting the consumable cpus to placement | 09:28 |
sean-k-mooney | not all the cpus on the plathform | 09:28 |
bhagyashris | ok | 09:28 |
*** mdbooth has quit IRC | 09:28 | |
sean-k-mooney | let me just double check the spec to ensure that is correct but i belive that is what we settled on | 09:29 |
bhagyashris | ok check and let me help to understand | 09:29 |
bhagyashris | Sorry help me to understand | 09:30 |
sean-k-mooney | looks like that has not been updated yet https://review.opendev.org/#/c/555081/28/specs/train/approved/cpu-resources.rst@201 | 09:30 |
sean-k-mooney | ill leave a comment | 09:30 |
sean-k-mooney | stephenfin: ^ is the above correct | 09:30 |
bhagyashris | I am still confuse what will be the all_cpus value | 09:31 |
sean-k-mooney | there wont be one | 09:32 |
bhagyashris | if you check the Upgrade section there they have mentioned the formula | 09:32 |
sean-k-mooney | at least i dont think there shoudl be | 09:32 |
bhagyashris | To calculate the reserved hot | 09:32 |
bhagyashris | Host cpu* | 09:32 |
sean-k-mooney | that is not present in the current verion as far as i can see | 09:33 |
*** mrch_ has joined #openstack-nova | 09:33 | |
bhagyashris | http://logs.openstack.org/81/555081/28/check/openstack-tox-docs/6314f69/html/specs/train/approved/cpu-resources.html#configuration-options | 09:34 |
bhagyashris | Here they have mentioned | 09:34 |
bhagyashris | sean-k-mooney: ^^ | 09:35 |
*** zigo has joined #openstack-nova | 09:36 | |
sean-k-mooney | ya i think that is in correct. this was one of the part of the spec that i really disliked but we still need to get agreement on that | 09:36 |
bhagyashris | Yeah and there I am still unclear what will be the all_cpus value | 09:36 |
sean-k-mooney | i am suggstion there should not be one | 09:37 |
sean-k-mooney | i think it is incorrect to track cpus that cant be consumed in placement | 09:37 |
sean-k-mooney | it give an incorrect assement of your capastity | 09:37 |
bhagyashris | yeah | 09:39 |
bhagyashris | Then how will calculate the reserved value | 09:41 |
sean-k-mooney | we dont we always set it to 0 when creting the rp | 09:41 |
bhagyashris | Okay mean for both the PCPU and VCPU resource class inventory it will be 0 | 09:42 |
sean-k-mooney | yes | 09:42 |
bhagyashris | But didn't get why this is added in spec | 09:43 |
sean-k-mooney | that is not what the spec says but i think we shoudl updated it to say that | 09:43 |
bhagyashris | As I am implementing the spec so it's till bit confusing to me | 09:43 |
*** luksky has joined #openstack-nova | 09:43 | |
sean-k-mooney | well we have changed the design a bit form last cycle | 09:43 |
sean-k-mooney | this was one of the point that there was still some disagrement on | 09:44 |
bhagyashris | Ohh ok. | 09:45 |
bhagyashris | stephenfin: Hi, do you give me some suggestion and opinion about it | 09:46 |
sean-k-mooney | i think the UX of what was prosped in the spec is incrroect and i fully expect to get bug report because of it which is why i wanted to change it | 09:46 |
bhagyashris | Regarding the reserved_host_cpu calculation | 09:46 |
bhagyashris | sean-k-mooney: Yeah , still waiting for final decision how will be the reserved value for cpus | 09:47 |
stephenfin | sean-k-mooney: Yeah, that's a mistake that should be updated | 09:47 |
stephenfin | reserved = 0 by default since we already have a way to "reserve" CPUs (by not including them in either mask) | 09:48 |
sean-k-mooney | stephenfin: are we going to merge the spec as is and fix it in a follow up patch | 09:48 |
stephenfin | sean-k-mooney: Yeah, I'd prefer that, personally | 09:48 |
stephenfin | I'll draft the follow-up now | 09:49 |
sean-k-mooney | so whould i to finaly get it merged | 09:49 |
stephenfin | 'zactly | 09:49 |
stephenfin | bhagyashris: Does that make sense/ | 09:49 |
stephenfin | *? | 09:49 |
bhagyashris | sean-k-mooney, stephenfin: yes , so for both the VCPU and PCPU resource class inventory the reserved value will be 0 and will deprecate the configuration option anyways right? | 09:50 |
sean-k-mooney | yes we are deprecating resrved_host_cpus and sured really raise an error if its used with the new cpu_dedicated_set also set in the config | 09:53 |
bhagyashris | sean-k-mooney, stephenfin: okay | 09:54 |
bhagyashris | sean-k-mooney, stephenfin: Thank you for your inputs :) | 09:54 |
openstackgerrit | Matthew Booth proposed openstack/nova master: Fix retry of instance_update_and_get_original https://review.opendev.org/658845 | 09:56 |
*** bhagyashris has quit IRC | 09:56 | |
*** dpawlik has quit IRC | 09:58 | |
*** dpawlik has joined #openstack-nova | 10:03 | |
*** faizy98 has joined #openstack-nova | 10:07 | |
NewBruce | howdy all - as i mentioned last night, ive been digging into a few other issues in the migration process - hitting a wall, thinking about a bug report but figure its best to get a second opinion before doing so | 10:07 |
NewBruce | live migrations with volumes and volume-snapshots are failing - two errors | 10:08 |
NewBruce | 1. 'serial' is deprecated, please use the corresponding option of '-device’ instead | 10:08 |
NewBruce | 2. Could not open backing file: Could not open '/var/lib/nova/mnt/… : Permission denied: | 10:08 |
NewBruce | in the first, an error due to deprecation is odd, but also according to libvirt docs, serial is only deprecated for type block and we have type ‘file’ in the domain xml | 10:09 |
*** tesseract has quit IRC | 10:12 | |
*** tesseract has joined #openstack-nova | 10:12 | |
openstackgerrit | Merged openstack/os-vif master: Fix Kuryr-Kubernetes job name https://review.opendev.org/656492 | 10:16 |
openstackgerrit | Hamdy Khader proposed openstack/nova master: [WIP] OVS DPDK port representors support https://review.opendev.org/658785 | 10:27 |
*** zigo has quit IRC | 10:30 | |
*** ttsiouts has joined #openstack-nova | 10:31 | |
*** yedongcan has left #openstack-nova | 10:34 | |
openstackgerrit | Hamdy Khader proposed openstack/os-vif master: [WIP] OVS DPDK port representors support https://review.opendev.org/658786 | 10:35 |
*** zigo has joined #openstack-nova | 10:42 | |
*** tbachman has quit IRC | 10:53 | |
*** ttsiouts has quit IRC | 11:04 | |
*** udesale has quit IRC | 11:08 | |
*** udesale has joined #openstack-nova | 11:09 | |
*** jangutter has joined #openstack-nova | 11:14 | |
*** awalende has quit IRC | 11:50 | |
*** awalende has joined #openstack-nova | 11:51 | |
*** awalende_ has joined #openstack-nova | 11:54 | |
*** nicolasbock has joined #openstack-nova | 11:54 | |
*** awalende has quit IRC | 11:55 | |
*** dpawlik has quit IRC | 11:59 | |
*** panda is now known as panda|launch | 12:01 | |
*** janki has quit IRC | 12:04 | |
*** tbachman has joined #openstack-nova | 12:06 | |
*** tbachman has quit IRC | 12:07 | |
*** tbachman has joined #openstack-nova | 12:07 | |
*** spsurya has quit IRC | 12:08 | |
aarents | Hi there, can some one check relevance of this please https://review.opendev.org/#/c/659054/ ? | 12:11 |
sean-k-mooney | mdbooth_: ^ seams to be something you coudl weigh in on | 12:12 |
sean-k-mooney | aarents: the image cache code is rather arcane | 12:14 |
mdbooth_ | sean-k-mooney: It is stretched beyond all reasonable limits. | 12:14 |
*** mdbooth_ is now known as mdbooth | 12:14 | |
sean-k-mooney | aarents: hardcoding size for the disk is not a good approch | 12:15 |
sean-k-mooney | oh that is the test code... | 12:15 |
aarents | :) | 12:15 |
sean-k-mooney | so the change is just to pass the size parmater to the copy from host | 12:16 |
aarents | yep that's all | 12:16 |
*** cdent has joined #openstack-nova | 12:16 | |
aarents | if not the qemu img disk have the size of backing file | 12:17 |
sean-k-mooney | so size in this case is the flavor root disk | 12:17 |
aarents | yes it is | 12:17 |
sean-k-mooney | well we are passing it like 10 lines above so it seams sane | 12:18 |
*** dpawlik has joined #openstack-nova | 12:21 | |
*** dpawlik has quit IRC | 12:21 | |
sean-k-mooney | but honestly i dont really know the full implication but at first glance it looks like an apporpiate fix for the bug you reported | 12:21 |
*** mriedem has joined #openstack-nova | 12:23 | |
mdbooth | Could somebody hit https://review.opendev.org/#/c/658845/ for me, please? It has 1.5 +2s. It's the db retry bug. | 12:24 |
aarents | sean-k-mooney: ok thks, it fixs the issue in my environment | 12:24 |
mdbooth | mriedem: I merged the comment you asked for you at the request of stephenfin, btw. | 12:24 |
* mdbooth was going to leave it in a follow-up. | 12:25 | |
sean-k-mooney | aarents: it also passed ci so it dose not break the happy path although i dont think we have a function or tempest test that validates teh behavior | 12:25 |
mriedem | mdbooth: yeah looking | 12:25 |
mdbooth | mriedem: Thanks | 12:26 |
sean-k-mooney | mdbooth: oh this is the modifyicion of input parmaters thing you sent to the mailing list | 12:26 |
mdbooth | sean-k-mooney: Yeah. | 12:26 |
mriedem | sean-k-mooney: the sriov live migration blueprint isn't closed yet b/c surely there are docs updates to be made, right? | 12:27 |
mriedem | at least this https://docs.openstack.org/neutron/latest/admin/config-sriov.html#known-limitations | 12:27 |
mriedem | melwitt: are you ready for https://blueprints.launchpad.net/nova/+spec/count-quota-usage-from-placement to go back into a runway slot? | 12:27 |
sean-k-mooney | that in neutreon but yes i should go update that. | 12:28 |
mriedem | mdbooth: coreycb: btw, starlingx is saying since we merged the eventlet thing on stable/stein that they are broken ow | 12:28 |
mriedem | *now | 12:28 |
mriedem | so that's fun | 12:28 |
mriedem | https://bugs.launchpad.net/nova/+bug/1829062 | 12:28 |
openstack | Launchpad bug 1829062 in StarlingX "nova placement api non-responsive due to eventlet error" [Critical,New] - Assigned to Bruce Jones (brucej) | 12:28 |
sean-k-mooney | mriedem: dpending on how its run there were issues with some rabbitmq timeouts or somthing right | 12:29 |
mriedem | granted if starlingx moves to extracted placement in stein they won't have to deal with eventlet | 12:29 |
mdbooth | mriedem: Yeah, we're seeing similar downstream. However, I'm pretty sure we didn't. | 12:29 |
mdbooth | mriedem: Honestly what I really want to do is prioritise un-eventleting the multi-cell instance list code, and let them switch eventlet off. | 12:29 |
*** dpawlik has joined #openstack-nova | 12:30 | |
mriedem | which is mel's series isn't it? | 12:30 |
sean-k-mooney | ya althoeugh we have an open question of how to we kill long running task that will timeout due to a down cell | 12:30 |
sean-k-mooney | mriedem: yes | 12:30 |
mdbooth | mriedem: Let me clarify 'we didn't', btw: the eventlet band-aid thing didn't break it, it was already broken. | 12:31 |
mdbooth | I believe them that it's broken, though. | 12:31 |
mdbooth | mriedem: Yes. We discussed it at PTG and the feeling was that we'd address it later. However, I think that was when we believe we'd applied an effective band-aid already. | 12:32 |
sean-k-mooney | if we dont have a path forward with melwitt's serise we might want to consider reverting the scater gatehr stuff temporally and dropping eventlets | 12:32 |
*** tbachman has quit IRC | 12:32 | |
mdbooth | sean-k-mooney: Also worthy of discussion. | 12:32 |
* mdbooth would definitely like to bump the priority of that series. | 12:34 | |
*** _erlon_ has joined #openstack-nova | 12:35 | |
sean-k-mooney | so looks like https://review.opendev.org/#/c/657750/1 is the last patch in that series and it passing ci but i think we are currently stuck on the open question in https://review.opendev.org/#/c/650172/14 | 12:36 |
sean-k-mooney | is the startx issue seperate for teh amqp issue reported on the mailing list | 12:37 |
sean-k-mooney | http://lists.openstack.org/pipermail/openstack-discuss/2019-May/005822.html | 12:38 |
*** awalende_ has quit IRC | 12:40 | |
* mdbooth added a comment to that review. | 12:42 | |
mdbooth | Thanks for the reminder, sean-k-mooney | 12:42 |
*** sridharg has joined #openstack-nova | 12:43 | |
*** janki has joined #openstack-nova | 12:44 | |
*** awalende has joined #openstack-nova | 12:47 | |
*** awalende has quit IRC | 12:51 | |
*** awalende has joined #openstack-nova | 12:53 | |
mriedem | tssurya: we need a novaclient release before osc can use the locked reason stuff https://review.opendev.org/#/c/659124/ | 12:54 |
mriedem | i'll propose a release | 13:01 |
*** ttsiouts has joined #openstack-nova | 13:01 | |
*** panda|launch is now known as panda | 13:05 | |
openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Cap sphinx for py2 to match global requirements https://review.opendev.org/659202 | 13:06 |
*** sapd1_x has quit IRC | 13:07 | |
*** BjoernT has joined #openstack-nova | 13:13 | |
*** BjoernT has quit IRC | 13:14 | |
*** artom has quit IRC | 13:14 | |
openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Cap sphinx for py2 and drop keyring dependency https://review.opendev.org/659202 | 13:14 |
*** jchhatbar has joined #openstack-nova | 13:15 | |
*** janki has quit IRC | 13:17 | |
*** lbragstad has joined #openstack-nova | 13:18 | |
*** tbachman has joined #openstack-nova | 13:18 | |
mriedem | the requirements-check job is happy with ^ now if someone wants to approve so we can get that into the novaclient release i'm pushing | 13:19 |
*** sapd1_x has joined #openstack-nova | 13:20 | |
*** BjoernT has joined #openstack-nova | 13:21 | |
mdbooth | mriedem: Thanks. | 13:22 |
mdbooth | Incidentally, MySQL's repeatable read isolation level (which we use), is seriously obtuse when trying to handle concurrency. If I hadn't had to handle it when we were working on this code previously I would never have believe anybody would have intentionally created this behaviour. However, MySQL goes to great lengths to be broken. | 13:25 |
mdbooth | They apparently see value in it. | 13:25 |
mriedem | job security | 13:28 |
dansmith | kashyap: I'm entirely against (a) nova automatic scheduling based on perceived threats from vulnerabilities and (b) putting traits in placement about which vulnerabilities the host has, has been patched for, or has enabled/disabled. Hopefully that clears it up. | 13:29 |
*** brinzhang has quit IRC | 13:30 | |
kashyap | Sure, I don't care about these things much at all; I wanted to put in a missing thing in traits and move on. | 13:30 |
kashyap | dansmith: Also: you write "perceived threats" -- they're not. | 13:31 |
*** Luzi has quit IRC | 13:31 | |
kashyap | You can _choose_ to take a policy stance, as "this is out of scope". That's perfectly fine :-) | 13:31 |
sean-k-mooney | the add the pcid and 1ghugepage tratis and drop the other form the patch | 13:31 |
*** lpetrut has quit IRC | 13:32 | |
dansmith | kashyap: if I am running a single-tenant cloud full of workloads I control, then the threats are entirely perceived | 13:32 |
*** lpetrut has joined #openstack-nova | 13:32 | |
kashyap | dansmith: Fair, that's one valid example. That's why I only said: should we consider "warn". | 13:33 |
*** awalende has quit IRC | 13:33 | |
kashyap | I also noted a few more valid cases where people willingly want to run on vulnerable hosts in my second reply on the 'openstack-security' thread. | 13:33 |
dansmith | we most certainly should not put traits in placement for the purposes of warning, but I also do not agree that nova should be doing the warning | 13:33 |
*** awalende has joined #openstack-nova | 13:34 | |
coreycb | mriedem: i wonder if they could work around it by running with python3 | 13:34 |
*** ttsiouts has quit IRC | 13:35 | |
kashyap | dansmith: Yeah, I agre with the former (not put them in traits). The latter is a 'policy' decision -- so I'm fine with whatever the consensus of the community is. | 13:35 |
*** awalende_ has joined #openstack-nova | 13:37 | |
*** awalende has quit IRC | 13:38 | |
*** ratailor has quit IRC | 13:39 | |
amorin | hey all, can someone take a look at that: https://review.opendev.org/#/c/658362/ | 13:39 |
*** awalende_ has quit IRC | 13:42 | |
mdbooth | coreycb: Are you talking about the eventlet thing? | 13:45 |
coreycb | mdbooth: ye | 13:45 |
coreycb | yes | 13:45 |
mdbooth | coreycb: Unfortunately not. We're hitting this in OSP15, which is py3 only. | 13:46 |
sean-k-mooney | coreycb: if you dont use uswign and run the nova-api via the console script it shoudl not have this issue | 13:46 |
sean-k-mooney | *uwsgi | 13:46 |
mdbooth | However, IIRC there's another project also hitting it? | 13:46 |
coreycb | mdbooth: ok. i just saw py2.7 in the traceback and that was my thought. sean-k-mooney: ok | 13:46 |
mdbooth | sean-k-mooney: Do you happen to remember which one it was? | 13:46 |
sean-k-mooney | monasca i think | 13:47 |
mdbooth | It's a library thing, anyway. It's nothing specific to what Nova's doing. | 13:47 |
mdbooth | It's just that eventlet + wsgi is bad. | 13:47 |
sean-k-mooney | as i said the workaound currently is to not run the nova api under wsgi and fall back to using the python nova-api command and fronting that with a load blancer like ha proxy | 13:49 |
* mdbooth would prefer to expedite melwitt's patches and remove the eventlet requirement asap. | 13:50 | |
* sean-k-mooney likes mdbooth ideas and want to subsribe to his newsletter | 13:51 | |
mdbooth | I mentioned in the review that I don't think the lack of ability to cancel a thread is a showstopper in practice, and certainly not as big an issue as eventlet + wsgi. | 13:51 |
sean-k-mooney | ah it was mistral | 13:51 |
kashyap | sean-k-mooney: What does "monasca" even do? Okay, let me go look up | 13:51 |
mdbooth | sean-k-mooney: That sounds better. | 13:52 |
sean-k-mooney | monsaca is a montioring service | 13:52 |
kashyap | Ah, thx. | 13:52 |
sean-k-mooney | but i just rememebre it began with m just fount th lin in the mailing list post | 13:52 |
mriedem | amorin: you should have a test | 13:52 |
sean-k-mooney | but ya long story short as mdbooth said eventlet+wsgi is just a bad idea in general | 13:53 |
amorin | mriedem: ok | 13:53 |
*** mlavalle has joined #openstack-nova | 13:59 | |
*** awalende has joined #openstack-nova | 14:04 | |
*** awalende has quit IRC | 14:08 | |
efried | alex_xu: looking... | 14:12 |
tssurya | mriedem: oh cool thansk :) | 14:21 |
aarents | mdbooth: thanks for reviewing https://review.opendev.org/#/c/659054/, you catch the test logic, I will add a comment in test as suggested | 14:23 |
*** artom has joined #openstack-nova | 14:27 | |
openstackgerrit | Stephen Finucane proposed openstack/nova-specs master: Follow up for I9200c2b18369da8b9194a8ccbbf6f1baf6a9613c https://review.opendev.org/659300 | 14:31 |
stephenfin | bauzas: Time to merge https://review.opendev.org/#/c/555081/ ? | 14:31 |
stephenfin | bauzas: I've posted a follow-up to address some of the outstanding comments and issues | 14:32 |
bauzas | cool then, I'll review those :) | 14:32 |
bauzas | but yeah I feel we agreed on saying cool cool at the PTG | 14:32 |
*** dpawlik has quit IRC | 14:32 | |
*** jchhatbar has quit IRC | 14:34 | |
*** lpetrut has quit IRC | 14:37 | |
efried | alex_xu: How does that play with https://review.opendev.org/#/c/645520/ ? | 14:41 |
*** luksky has quit IRC | 14:41 | |
*** dpawlik has joined #openstack-nova | 14:45 | |
*** dpawlik has quit IRC | 14:49 | |
*** maciejjozefczyk has quit IRC | 15:09 | |
openstackgerrit | Alexandre arents proposed openstack/nova master: Fix live-migration when glance image deleted https://review.opendev.org/659054 | 15:10 |
*** ociuhandu has joined #openstack-nova | 15:13 | |
*** ivve has quit IRC | 15:15 | |
*** liuyulong has quit IRC | 15:21 | |
openstackgerrit | Arnaud Morin proposed openstack/nova master: Always Set dhcp_server in network_info https://review.opendev.org/658362 | 15:22 |
openstackgerrit | Merged openstack/nova-specs master: Standardize CPU resource tracking https://review.opendev.org/555081 | 15:24 |
*** panda is now known as panda|ruck | 15:26 | |
*** cdent has quit IRC | 15:28 | |
*** cdent has joined #openstack-nova | 15:29 | |
*** dpawlik has joined #openstack-nova | 15:30 | |
openstackgerrit | Matthew Booth proposed openstack/nova stable/stein: Fix retry of instance_update_and_get_original https://review.opendev.org/659317 | 15:30 |
openstackgerrit | Matthew Booth proposed openstack/nova stable/rocky: Fix retry of instance_update_and_get_original https://review.opendev.org/659318 | 15:31 |
*** ttsiouts has joined #openstack-nova | 15:32 | |
*** READ10 has joined #openstack-nova | 15:32 | |
openstackgerrit | Matthew Booth proposed openstack/nova stable/queens: Fix retry of instance_update_and_get_original https://review.opendev.org/659320 | 15:33 |
openstackgerrit | Arnaud Morin proposed openstack/nova master: Always Set dhcp_server in network_info https://review.opendev.org/658362 | 15:33 |
*** helenafm has quit IRC | 15:33 | |
*** dpawlik has quit IRC | 15:34 | |
*** cdent has quit IRC | 15:35 | |
*** tesseract has quit IRC | 15:35 | |
amorin | hey mriedem and all | 15:37 |
amorin | I added some unit tests there: | 15:37 |
amorin | https://review.opendev.org/#/c/658362 | 15:37 |
*** macza has joined #openstack-nova | 15:37 | |
*** davidsha has joined #openstack-nova | 15:38 | |
*** cfriesen has joined #openstack-nova | 15:39 | |
*** cdent has joined #openstack-nova | 15:41 | |
*** BjoernT has quit IRC | 15:44 | |
*** dpawlik has joined #openstack-nova | 15:46 | |
*** BjoernT has joined #openstack-nova | 15:46 | |
*** sridharg has quit IRC | 15:46 | |
*** ttsiouts has quit IRC | 15:50 | |
mriedem | ack it's in the tab queue | 15:50 |
*** dpawlik has quit IRC | 15:50 | |
*** ccamacho has quit IRC | 15:56 | |
*** imacdonn has quit IRC | 16:01 | |
*** wwriverrat has joined #openstack-nova | 16:06 | |
*** rpittau is now known as rpittau|afk | 16:06 | |
*** imacdonn has joined #openstack-nova | 16:13 | |
*** xek has quit IRC | 16:14 | |
*** tesseract has joined #openstack-nova | 16:24 | |
melwitt | mriedem: yes, should be ready. I just looked through the latest comments again and replied. and it looks like it hasn't fallen into merge conflict yet | 16:29 |
*** dklyle has quit IRC | 16:31 | |
mriedem | ok | 16:33 |
mriedem | into the runway slot vat of acid it will be thrown | 16:33 |
*** BjoernT has quit IRC | 16:33 | |
melwitt | haha, thanks | 16:34 |
*** dklyle has joined #openstack-nova | 16:37 | |
*** BjoernT has joined #openstack-nova | 16:38 | |
*** gyee has joined #openstack-nova | 16:38 | |
openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Add functional confirm_migration_error test https://review.opendev.org/658834 | 16:39 |
openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Delete allocations even if _confirm_resize raises (part 2) https://review.opendev.org/659338 | 16:39 |
mriedem | ganso: ^ | 16:39 |
*** hemna has joined #openstack-nova | 16:42 | |
ganso | mriedem: thanks! will rebase on top | 16:44 |
ganso | mriedem: oh you already rebased | 16:45 |
ganso | mriedem: thanks! | 16:45 |
*** bbowen has quit IRC | 16:45 | |
imacdonn | mriedem: pls add https://review.opendev.org/#/c/653065/ to your tab queue for revisit? I have a feeling that the issue with the functional tests has something to do with sqlalchemy sessions, or something, but may be off on the wrong track | 16:45 |
*** bbowen has joined #openstack-nova | 16:45 | |
imacdonn | (if there's a sqlalchemy guru available for a consult, that might work too) | 16:45 |
weshay | kashyap ping.. looking to add the qemu-agent rpm to nova, I think that would be on a container and not bm.. not sure also wanted to verify w/ you which container that rpm should be installed on | 16:45 |
weshay | kashyap context is: nova.exception.QemuGuestAgentNotEnabled: QEMU guest agent is not enabled | 16:46 |
weshay | https://bugs.launchpad.net/tripleo/+bug/1824315 | 16:46 |
openstack | Launchpad bug 1824315 in tripleo "periodic fedora28 standalone job failing at test_volume_boot_pattern" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 16:46 |
weshay | melwitt maybe you would know? ^ | 16:47 |
melwitt | weshay: the qemu guest agent is something that would have to be installed on the guest, which is currently not done as part of any CI that I'm aware of | 16:50 |
weshay | melwitt ok.. some folks were saying the container but the bm makes more sense to me | 16:50 |
melwitt | that is, if you create a nova instance, you'd then have to install the qemu guest agent on that guest after it comes up | 16:50 |
melwitt | but I'm wondering why it should be needed | 16:52 |
*** mvkr has joined #openstack-nova | 16:52 | |
dansmith | if they're trying to do something that requires it, like quiesce disks (I think) | 16:52 |
melwitt | weshay: that error message is just saying it's going to skip quiescing the instance bc the guest agent isn't present, but it shouldn't fail the job because of that I don't think | 16:53 |
weshay | melwitt it does not fail consistently, maybe 1/10 times | 16:53 |
weshay | maybe this is a red herring | 16:53 |
dansmith | if it's failing on the guest agent really not being there 1/10 times, I'd suspect the guest isn't booting fully 1/10 times or something like that | 16:54 |
melwitt | weshay: ok. I don't think this is the root cause of the failure because AFAICT it's just skipping that. it should say that every time I think | 16:54 |
melwitt | dansmith: I don't think any CI installs the guest agent, at least I was not aware of any | 16:55 |
*** mvkr has quit IRC | 16:55 | |
dansmith | melwitt: if they are using a fedora guest... | 16:55 |
openstackgerrit | Merged openstack/python-novaclient master: Cap sphinx for py2 and drop keyring dependency https://review.opendev.org/659202 | 16:55 |
melwitt | dansmith: fedora guest has it by default? | 16:55 |
weshay | dansmith ya.. we this in fedora deployments.. but w/ cirros guest | 16:56 |
dansmith | melwitt: no idea, I'm just saying, different from a cirros guest | 16:56 |
dansmith | okay | 16:56 |
dansmith | the bug shows a big stack trace.. melwitt you think we're logging that trace just while checking the skip? | 16:56 |
melwitt | oh. yeah, anecdotally I've not had the qemu guest agent come with any cloud image, like centos for example I've used | 16:56 |
melwitt | dansmith: I think so, opening up the code to confirm | 16:56 |
dansmith | okay we should not be logging a trace for a check thing, amirite? | 16:57 |
*** ociuhandu has quit IRC | 16:57 | |
*** arxcruz has joined #openstack-nova | 16:57 | |
melwitt | should not, no. I see a LOG.info for it but no re-raise, so not yet sure what's going on | 16:58 |
dansmith | that's why I'm wondering if we're actually leaking the exception out and not just skipping | 16:58 |
melwitt | yeah, let's see, the trace is happening in the driver, so that must be where | 16:58 |
melwitt | there's no re-raise in compute/api, I meant | 16:59 |
*** ociuhandu has joined #openstack-nova | 16:59 | |
*** dtantsur is now known as dtantsur|afk | 17:01 | |
*** davidsha has left #openstack-nova | 17:01 | |
*** derekh has quit IRC | 17:02 | |
cdent | efried: is ocata nova still an active concern? there's a logging buglet in the reasource tracker + report client that can be pretty misleading but the issue goes away with pike (or maybe queens) | 17:06 |
*** ricolin has quit IRC | 17:07 | |
cdent | (goes away with pike, just checked) | 17:09 |
*** luksky has joined #openstack-nova | 17:10 | |
*** BjoernT_ has joined #openstack-nova | 17:11 | |
*** BjoernT has quit IRC | 17:12 | |
*** psachin has quit IRC | 17:15 | |
melwitt | weshay: on your latest comment on LP, the log shows the code continuing on to do the volume snapshot http://logs.openstack.org/84/655684/2/check/tripleo-ci-fedora-28-standalone/eda3002/logs/undercloud/var/log/containers/nova/nova-api.log.txt.gz#_2019-05-13_16_03_50_120 so I think that trace is unrelated | 17:16 |
melwitt | weshay: I don't yet find how that trace is getting logged (it shouldn't be logged), so I'm still looking for how that's happening | 17:17 |
weshay | k k.. thanks.. so melwitt we're saying it's a red herring and the problem is elsewhere right? | 17:17 |
weshay | confirming that I guess | 17:17 |
melwitt | weshay: right. the logging of the trace is bad and I'll have to find how that's occurring to remove it, but AFAICS the code is continuing on after skipping the quiesce and shouldn't affect the pass/fail of the CI | 17:18 |
melwitt | "Creating snapshot from volume" | 17:19 |
weshay | k | 17:19 |
weshay | arxcruz ^ | 17:20 |
melwitt | I'll look at the rest of those logs to see if I can tell where it's failing | 17:20 |
*** udesale has quit IRC | 17:21 | |
melwitt | weshay: looking at the test results of that run, the test_volume_boot_pattern passed but test_network_basic_ops failed due to a ssh timeout http://logs.openstack.org/84/655684/2/check/tripleo-ci-fedora-28-standalone/eda3002/logs/tempest.html.gz | 17:22 |
weshay | aye | 17:22 |
weshay | ya.. probably not same thing | 17:23 |
weshay | paste happy | 17:23 |
weshay | I guess | 17:23 |
melwitt | definitely not in this particular run. that run ^ failed because of a ssh timeout and test_volume_boot_pattern passed with the log trace | 17:24 |
tssurya | mriedem: trivial doc change: https://review.opendev.org/#/c/659067/ | 17:24 |
*** dklyle has quit IRC | 17:29 | |
*** Vek has joined #openstack-nova | 17:31 | |
openstackgerrit | Merged openstack/os-resource-classes master: Replace git.openstack.org URLs with opendev.org URLs https://review.opendev.org/658519 | 17:32 |
*** maciejjozefczyk has joined #openstack-nova | 17:33 | |
*** ttsiouts has joined #openstack-nova | 17:35 | |
*** maciejjozefczyk has quit IRC | 17:39 | |
*** dpawlik has joined #openstack-nova | 17:47 | |
*** tssurya has quit IRC | 17:47 | |
mriedem | cdent: ocata is in extended maintenance meaning we can fix things if we want, but if it's not a big deal then meh | 17:47 |
* cdent has been wrong many times about what various folk think is or is not a big deal | 17:48 | |
*** dklyle has joined #openstack-nova | 17:49 | |
mriedem | i thought VIO was off ocata (and pike for that matter)? | 17:50 |
cdent | there are lots of supported customers back on ocata, and a few that are -2 or -4 releases from that.... | 17:51 |
*** dpawlik has quit IRC | 17:51 | |
mriedem | just tell those banks to upgrade | 17:51 |
mriedem | do you know if there was a specific fix in pike that fixes it? or was it some big refactor we wouldn't want to backport? | 17:52 |
mriedem | if it's something dumb easy and backports cleanly than whatever | 17:52 |
cdent | there was a refactor: the introduction of _ensure_resource_provider | 17:53 |
mriedem | well then that's probably not something we'll want to backport to ocata at this point | 17:53 |
cdent | quite | 17:53 |
*** dklyle has quit IRC | 17:56 | |
*** tesseract has quit IRC | 18:01 | |
*** READ10 has quit IRC | 18:02 | |
*** dpawlik has joined #openstack-nova | 18:02 | |
*** dpawlik has quit IRC | 18:07 | |
*** BjoernT_ has quit IRC | 18:11 | |
openstackgerrit | Merged openstack/nova master: [Trivial doc change] Admin can overwrite the locked_reason of an owner https://review.opendev.org/659067 | 18:16 |
*** dklyle has joined #openstack-nova | 18:20 | |
*** BjoernT has joined #openstack-nova | 18:20 | |
*** tbachman has quit IRC | 18:27 | |
*** ttsiouts has quit IRC | 18:27 | |
*** BjoernT has quit IRC | 18:29 | |
*** luksky has quit IRC | 18:32 | |
*** ab-a has quit IRC | 18:40 | |
*** ttsiouts has joined #openstack-nova | 18:43 | |
*** bbowen has quit IRC | 18:44 | |
*** ttsiouts has quit IRC | 18:48 | |
*** BjoernT has joined #openstack-nova | 18:52 | |
*** BjoernT_ has joined #openstack-nova | 18:58 | |
*** BjoernT has quit IRC | 18:59 | |
*** dklyle has quit IRC | 19:01 | |
*** cdent has quit IRC | 19:02 | |
*** tssurya has joined #openstack-nova | 19:02 | |
*** BjoernT_ has quit IRC | 19:03 | |
*** bbowen has joined #openstack-nova | 19:06 | |
melwitt | mriedem: wondering if you might know anything about this... is it known or expected that the traceback will become part of the printed exception object when that exception occurred on the other side of an rpc call? example http://logs.openstack.org/72/650172/14/check/tempest-full-py3/61914af/controller/logs/screen-n-api.txt.gz#_May_08_08_52_36_106235 | 19:08 |
melwitt | in nova/compute/api.py, we're catching the exception and logging the exception object, which I thought should normally only contain the short message and not the traceback. but the traceback seems to also be part of that | 19:09 |
melwitt | I looked at exception_wrapper (from our @wrap_exception decorator) and expected_exceptions in oslo.messaging and didn't see how they could be adding the traceback. I wanted to ask you in case you've seen this before and already know what's going on | 19:10 |
mriedem | i want to say it's a thing in oslo.log if there is an exception in scope | 19:18 |
mriedem | which is what this is, it's logging the exception within an except block | 19:19 |
melwitt | hm, I think I found where it's added in oslo.messaging https://github.com/openstack/oslo.messaging/blob/40c25c2bde6d2f5a756e7169060b7ce389caf174/oslo_messaging/_drivers/common.py#L212 | 19:19 |
mriedem | remember this? https://github.com/openstack/oslo.log/commit/a63438cb5b12e2a6973fb0e1b0b9ebce2f8116b6#diff-e121e929b5792521de104c2db0d14d94 | 19:20 |
mriedem | oh, so, | 19:20 |
melwitt | if I'm understanding this code right, when oslo.messaging deserializes an exception, it tacks on the traceback to the normal message | 19:20 |
mriedem | if it's an rpc call, which this is, we need to use expected_exceptions on the compute manager method | 19:20 |
mriedem | are we doing that? | 19:20 |
melwitt | we are | 19:20 |
melwitt | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L8513 | 19:21 |
melwitt | oh hah, I forgot about that commit. reading.. | 19:21 |
mriedem | so my guess is it's related to that magic stuff oslo.log does when you're logging from within an except block | 19:22 |
mriedem | even if you're not passing exc_info=True | 19:22 |
*** BjoernT has joined #openstack-nova | 19:24 | |
melwitt | thanks, this gives me more to look into | 19:25 |
mriedem | https://github.com/openstack/oslo.log/blob/master/oslo_log/formatters.py#L120 should handle that i'd think, but idk | 19:25 |
mriedem | light up the doug signal | 19:25 |
*** ivve has joined #openstack-nova | 19:25 | |
melwitt | yeah... I don't think this has to do with the error summary (I'm having trouble remembering what that is, even with seeing the old commit). but rather that if we catch a remote exception, the 'message' attribute of the object is going to contain the traceback, so if we don't want to log a traceback, we can't be doing "LOG.info('stuff %s', exp)" | 19:30 |
melwitt | this log line is explicitly doing that | 19:31 |
*** BjoernT has quit IRC | 19:35 | |
efried | cdent: pike just went em, so I'm not sure how much attention we are (or should be) paying to ocata at this point. Probably only security vulnerabilities, kind of thing. | 19:39 |
*** BjoernT has joined #openstack-nova | 19:39 | |
*** dpawlik has joined #openstack-nova | 20:03 | |
*** dpawlik has quit IRC | 20:08 | |
*** BjoernT has quit IRC | 20:12 | |
*** BjoernT has joined #openstack-nova | 20:12 | |
*** pcaruana has quit IRC | 20:14 | |
*** dpawlik has joined #openstack-nova | 20:21 | |
*** whoami-rajat has quit IRC | 20:24 | |
*** dpawlik has quit IRC | 20:25 | |
*** hemna has quit IRC | 20:35 | |
*** dklyle has joined #openstack-nova | 20:39 | |
mriedem | i'm still fixing shit back in pike so yeah it's still active-ish | 20:45 |
*** ttsiouts has joined #openstack-nova | 20:45 | |
dansmith | is it easier to land things in pike than master? | 20:46 |
dansmith | because based on master these days, I'm guessing so | 20:46 |
artom | What's the company cores policy on stable stuff? | 20:47 |
artom | So for instance, would Mel be allowed to +W https://review.opendev.org/#/c/657125/ ? | 20:48 |
*** tbachman has joined #openstack-nova | 20:50 | |
*** hongbin has joined #openstack-nova | 20:54 | |
openstackgerrit | melanie witt proposed openstack/nova master: Stop logging traceback when skipping quiesce https://review.opendev.org/659374 | 21:04 |
*** ralonsoh has quit IRC | 21:07 | |
*** panda|ruck has quit IRC | 21:10 | |
melwitt | mriedem: now I know what format_message() is for ^ and also I found this, from 6 years ago https://github.com/openstack/nova/commit/86031940b313532606449c400bca64707be1797a | 21:12 |
*** BjoernT has quit IRC | 21:13 | |
*** BjoernT_ has joined #openstack-nova | 21:13 | |
*** panda has joined #openstack-nova | 21:14 | |
*** dklyle has quit IRC | 21:14 | |
*** ttsiouts has quit IRC | 21:16 | |
mriedem | oh geez | 21:16 |
melwitt | yeah xD | 21:16 |
mriedem | if we should always be using format_message(), shouldn't we override __str__ or __repr__ or something for NovaExceptions? | 21:16 |
melwitt | I _think_ that the exception type for remote exceptions is actually generated dynamically by oslo.messaging, | 21:17 |
melwitt | *_Remote exceptions, and it overrides __str__ and __unicode__ to have the traceback in it | 21:18 |
mriedem | if it's not a NovaException then we can't call format_message on it can we? | 21:19 |
melwitt | yeah, maybe it's that NovaException is the base class it uses | 21:19 |
mriedem | er i guess you're talking about this https://github.com/openstack/oslo.messaging/blob/40c25c2/oslo_messaging/_drivers/common.py#L247 | 21:20 |
melwitt | yes | 21:20 |
mriedem | f what a mess | 21:21 |
melwitt | but you're right that it must be deriving somehow from NovaException else format_message wouldn't work | 21:21 |
mriedem | so NovaException override __str__ won't work b/c we aren't using a NovaException? | 21:21 |
mriedem | python black magic | 21:21 |
melwitt | I think probably not, based on this https://github.com/openstack/oslo.messaging/blob/40c25c2/oslo_messaging/_drivers/common.py#L244 | 21:22 |
melwitt | I was also looking at these tests which seem to know all about this issue :P https://github.com/openstack/nova/blob/master/nova/tests/unit/test_exception.py#L203 | 21:23 |
melwitt | FakeNovaException_Remote is presumably simulating what oslo.messaging does | 21:23 |
melwitt | I'm not seeing though in oslo.messaging code how it's subclassing the original exception type though | 21:25 |
mriedem | i think that's https://github.com/openstack/oslo.messaging/blob/40c25c2/oslo_messaging/_drivers/common.py#L243 | 21:25 |
mriedem | https://docs.python.org/3.6/library/functions.html#type | 21:26 |
melwitt | oh, I see yeah | 21:26 |
mriedem | the first arg is the new type _Remote, 2nd is the parent class which is the NovaException | 21:26 |
melwitt | the second argument is the base classes | 21:26 |
mriedem | python | 21:26 |
mriedem | black | 21:26 |
mriedem | magic | 21:26 |
melwitt | right | 21:26 |
melwitt | yeah, so that's fun | 21:26 |
mriedem | NotImplementedError doesn't have format_message()... | 21:28 |
mriedem | i know what we can do! | 21:28 |
mriedem | monkey patch oslo.messaging, | 21:28 |
mriedem | extend their dynamic type, | 21:28 |
mriedem | and override with our own __str__ that does format_message! | 21:28 |
melwitt | brilliant! | 21:28 |
mriedem | and somehow do it with eventlet | 21:28 |
*** dklyle has joined #openstack-nova | 21:29 | |
openstackgerrit | Hamdy Khader proposed openstack/nova master: [WIP] OVS DPDK port representors support https://review.opendev.org/658785 | 21:31 |
*** BjoernT_ has quit IRC | 21:46 | |
openstackgerrit | melanie witt proposed openstack/nova master: Stop logging traceback when skipping quiesce https://review.opendev.org/659374 | 21:50 |
melwitt | mriedem: just did a fugly update ^ better ideas welcome | 21:53 |
openstackgerrit | Merged openstack/nova master: Add image type request filter https://review.opendev.org/656413 | 21:53 |
*** dklyle has quit IRC | 22:00 | |
*** takashin has joined #openstack-nova | 22:00 | |
*** xek has joined #openstack-nova | 22:07 | |
mriedem | replied | 22:10 |
*** artom has quit IRC | 22:11 | |
*** mlavalle has quit IRC | 22:17 | |
melwitt | thanks | 22:18 |
openstackgerrit | melanie witt proposed openstack/nova master: Stop logging traceback when skipping quiesce https://review.opendev.org/659374 | 22:18 |
openstackgerrit | Merged openstack/nova master: Enable image type query support in nova-next https://review.opendev.org/656903 | 22:18 |
openstackgerrit | Merged openstack/nova master: Add docs for image type support request filter https://review.opendev.org/657025 | 22:18 |
openstackgerrit | Merged openstack/nova stable/queens: Fix nova-grenade-live-migration run book for opendev migration https://review.opendev.org/655800 | 22:19 |
*** dpawlik has joined #openstack-nova | 22:21 | |
*** dpawlik has quit IRC | 22:26 | |
*** ttsiouts has joined #openstack-nova | 22:32 | |
mriedem | dansmith: it is done https://blueprints.launchpad.net/nova/+spec/request-filter-image-types | 22:36 |
* dansmith drops to his knees | 22:36 | |
*** dpawlik has joined #openstack-nova | 22:37 | |
*** dpawlik has quit IRC | 22:42 | |
*** mriedem has quit IRC | 22:51 | |
*** macza has quit IRC | 22:59 | |
*** mdbooth_ has joined #openstack-nova | 23:01 | |
*** mdbooth has quit IRC | 23:04 | |
*** tkajinam has joined #openstack-nova | 23:05 | |
*** ttsiouts has quit IRC | 23:06 | |
*** rcernin has joined #openstack-nova | 23:14 | |
*** panda has quit IRC | 23:30 | |
*** panda has joined #openstack-nova | 23:31 | |
*** tssurya has quit IRC | 23:32 | |
*** brinzhang has joined #openstack-nova | 23:41 | |
*** d34dh0r53 has joined #openstack-nova | 23:43 | |
*** sapd1_x has quit IRC | 23:55 | |
*** BjoernT has joined #openstack-nova | 23:57 | |
*** d34dh0r53 has quit IRC | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!