| *** tbachman has quit IRC | 00:01 | |
| *** tbachman has joined #openstack-nova | 00:02 | |
| *** tbachman has quit IRC | 00:02 | |
| *** itlinux has quit IRC | 00:03 | |
| *** yamamoto has joined #openstack-nova | 00:03 | |
| *** slaweq has joined #openstack-nova | 00:04 | |
| *** gbarros has quit IRC | 00:06 | |
| *** gjayavelu has joined #openstack-nova | 00:07 | |
| *** slaweq has quit IRC | 00:09 | |
| *** yamamoto has quit IRC | 00:09 | |
| *** gbarros has joined #openstack-nova | 00:10 | |
| *** AlexeyAbashkin has joined #openstack-nova | 00:19 | |
| *** stakeda has joined #openstack-nova | 00:22 | |
| *** lbragstad has quit IRC | 00:23 | |
| *** AlexeyAbashkin has quit IRC | 00:23 | |
| *** munimeha has quit IRC | 00:24 | |
| *** slaweq has joined #openstack-nova | 00:25 | |
| *** Zames has joined #openstack-nova | 00:26 | |
| *** slaweq has quit IRC | 00:29 | |
| *** hshiina|afk is now known as hshiina | 00:30 | |
| *** edmondsw has quit IRC | 00:31 | |
| *** vladikr has joined #openstack-nova | 00:32 | |
| *** chyka_ has joined #openstack-nova | 00:33 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Support shared and dedicated VMs in one host https://review.openstack.org/543805 | 00:35 |
|---|---|---|
| *** tbachman has joined #openstack-nova | 00:36 | |
| *** chyka has quit IRC | 00:36 | |
| *** mlavalle has quit IRC | 00:37 | |
| *** chyka_ has quit IRC | 00:37 | |
| *** r-daneel has quit IRC | 00:43 | |
| *** vladikr has quit IRC | 00:44 | |
| *** threestrands has joined #openstack-nova | 00:44 | |
| *** Zames has quit IRC | 00:45 | |
| *** slaweq has joined #openstack-nova | 00:46 | |
| *** moshele has joined #openstack-nova | 00:48 | |
| *** slaweq has quit IRC | 00:50 | |
| *** tbachman has quit IRC | 00:50 | |
| *** tiendc has joined #openstack-nova | 00:54 | |
| *** moshele has quit IRC | 00:55 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/queens: [placement] Add sending global request ID in put (3) https://review.openstack.org/543113 | 00:55 |
| *** tbachman has joined #openstack-nova | 00:56 | |
| *** vladikr has joined #openstack-nova | 00:56 | |
| *** takashin has joined #openstack-nova | 00:56 | |
| *** claudiub has joined #openstack-nova | 00:58 | |
| *** amodi has quit IRC | 01:02 | |
| *** phuongnh has joined #openstack-nova | 01:03 | |
| *** tbachman has quit IRC | 01:05 | |
| *** yamamoto has joined #openstack-nova | 01:05 | |
| *** zhaochao has joined #openstack-nova | 01:05 | |
| *** slaweq has joined #openstack-nova | 01:06 | |
| *** XueFeng_ is now known as XueFeng | 01:07 | |
| *** gjayavelu has quit IRC | 01:08 | |
| *** hieulq has joined #openstack-nova | 01:09 | |
| *** yamamoto has quit IRC | 01:11 | |
| *** slaweq has quit IRC | 01:11 | |
| *** wolverineav has quit IRC | 01:16 | |
| *** Tom-Tom has joined #openstack-nova | 01:24 | |
| *** slaweq has joined #openstack-nova | 01:27 | |
| *** Tom-Tom has quit IRC | 01:29 | |
| *** claudiub has quit IRC | 01:30 | |
| *** phuongnh has quit IRC | 01:31 | |
| *** phuongnh has joined #openstack-nova | 01:32 | |
| *** slaweq has quit IRC | 01:32 | |
| *** yamahata has quit IRC | 01:32 | |
| *** moshele has joined #openstack-nova | 01:35 | |
| *** hieulq has quit IRC | 01:38 | |
| *** tbachman has joined #openstack-nova | 01:40 | |
| *** itlinux has joined #openstack-nova | 01:46 | |
| *** gbarros has quit IRC | 01:47 | |
| *** gbarros has joined #openstack-nova | 01:48 | |
| *** slaweq has joined #openstack-nova | 01:48 | |
| *** slaweq has quit IRC | 01:53 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:53 | |
| *** mriedem has quit IRC | 02:06 | |
| *** phuongnh has quit IRC | 02:06 | |
| *** yamamoto has joined #openstack-nova | 02:07 | |
| *** phuongnh has joined #openstack-nova | 02:07 | |
| *** slaweq has joined #openstack-nova | 02:09 | |
| *** slaweq_ has joined #openstack-nova | 02:09 | |
| *** fragatina has quit IRC | 02:12 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** slaweq_ has quit IRC | 02:14 | |
| *** slaweq has quit IRC | 02:14 | |
| *** edmondsw has joined #openstack-nova | 02:17 | |
| *** Tom-Tom has joined #openstack-nova | 02:18 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/queens: [placement] Add sending global request ID in get https://review.openstack.org/543116 | 02:20 |
| *** edmondsw has quit IRC | 02:22 | |
| *** fragatina has joined #openstack-nova | 02:22 | |
| *** bhujay has joined #openstack-nova | 02:24 | |
| *** acormier has quit IRC | 02:26 | |
| *** fragatina has quit IRC | 02:26 | |
| *** zhenguo has joined #openstack-nova | 02:27 | |
| *** itlinux has quit IRC | 02:28 | |
| *** slaweq has joined #openstack-nova | 02:30 | |
| *** yamahata has joined #openstack-nova | 02:31 | |
| *** acormier has joined #openstack-nova | 02:32 | |
| openstackgerrit | Merged openstack/nova stable/pike: Drop extra loop which modifies Cinder volume status https://review.openstack.org/546218 | 02:34 |
| *** slaweq has quit IRC | 02:35 | |
| *** itlinux has joined #openstack-nova | 02:38 | |
| *** hieulq has joined #openstack-nova | 02:39 | |
| *** acormier has quit IRC | 02:40 | |
| *** acormier has joined #openstack-nova | 02:40 | |
| *** xnox has quit IRC | 02:46 | |
| *** slaweq has joined #openstack-nova | 02:46 | |
| *** salv-orl_ has joined #openstack-nova | 02:46 | |
| *** acormier has quit IRC | 02:48 | |
| *** salv-orlando has quit IRC | 02:49 | |
| *** annp has joined #openstack-nova | 02:49 | |
| *** hongbin has joined #openstack-nova | 02:51 | |
| *** slaweq has quit IRC | 02:51 | |
| *** slaweq has joined #openstack-nova | 02:51 | |
| *** slaweq has quit IRC | 02:55 | |
| *** acormier has joined #openstack-nova | 02:56 | |
| *** tinwood has quit IRC | 03:05 | |
| *** tinwood has joined #openstack-nova | 03:06 | |
| *** yamamoto has joined #openstack-nova | 03:09 | |
| *** takashin has left #openstack-nova | 03:09 | |
| *** acormier has quit IRC | 03:09 | |
| *** acormier has joined #openstack-nova | 03:10 | |
| *** slaweq has joined #openstack-nova | 03:10 | |
| *** bhujay has quit IRC | 03:10 | |
| *** Faster-Fanboi_ has quit IRC | 03:10 | |
| *** slaweq_ has joined #openstack-nova | 03:12 | |
| *** Faster-Fanboi has joined #openstack-nova | 03:12 | |
| *** yamamoto has quit IRC | 03:14 | |
| *** slaweq has quit IRC | 03:15 | |
| *** slaweq_ has quit IRC | 03:16 | |
| *** yamamoto has joined #openstack-nova | 03:27 | |
| *** slaweq has joined #openstack-nova | 03:28 | |
| *** itlinux has quit IRC | 03:30 | |
| *** slaweq has quit IRC | 03:32 | |
| mnaser | i'm seeing more and more cinder snapshot failures in stable/pike | 03:36 |
| *** acormier has quit IRC | 03:43 | |
| *** slaweq has joined #openstack-nova | 03:43 | |
| *** slaweq has quit IRC | 03:48 | |
| *** vivsoni__ has joined #openstack-nova | 03:55 | |
| *** gyee has quit IRC | 03:56 | |
| *** vivsoni_ has quit IRC | 03:58 | |
| *** slaweq has joined #openstack-nova | 03:59 | |
| *** zhaochao has quit IRC | 03:59 | |
| *** zhaochao has joined #openstack-nova | 04:00 | |
| *** slaweq has quit IRC | 04:04 | |
| *** edmondsw has joined #openstack-nova | 04:06 | |
| *** janki has joined #openstack-nova | 04:06 | |
| *** edmondsw has quit IRC | 04:11 | |
| *** chergd has quit IRC | 04:11 | |
| *** AlexeyAbashkin has joined #openstack-nova | 04:18 | |
| *** Dinesh_Bhor has quit IRC | 04:18 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:19 | |
| *** psachin has joined #openstack-nova | 04:20 | |
| *** links has joined #openstack-nova | 04:21 | |
| *** AlexeyAbashkin has quit IRC | 04:23 | |
| *** slaweq has joined #openstack-nova | 04:23 | |
| *** udesale has joined #openstack-nova | 04:25 | |
| *** abhishekk has joined #openstack-nova | 04:27 | |
| *** slaweq has quit IRC | 04:29 | |
| *** slaweq has joined #openstack-nova | 04:31 | |
| *** links has quit IRC | 04:33 | |
| *** Dinesh_Bhor has quit IRC | 04:34 | |
| *** slaweq has quit IRC | 04:35 | |
| *** itlinux has joined #openstack-nova | 04:36 | |
| *** zhenguo has quit IRC | 04:37 | |
| *** sree_ has joined #openstack-nova | 04:40 | |
| *** andreas_s has joined #openstack-nova | 04:40 | |
| *** sree_ is now known as Guest16541 | 04:40 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:41 | |
| *** ratailor has joined #openstack-nova | 04:42 | |
| *** Dinesh_Bhor has quit IRC | 04:42 | |
| alex_xu | mriedem, dansmith, working on eric patch now | 04:43 |
| *** andreas_s has quit IRC | 04:44 | |
| *** Faster-Fanboi has quit IRC | 04:46 | |
| *** slaweq has joined #openstack-nova | 04:47 | |
| *** links has joined #openstack-nova | 04:47 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:48 | |
| *** slaweq has quit IRC | 04:51 | |
| *** hieulq has quit IRC | 04:52 | |
| *** Faster-Fanboi has joined #openstack-nova | 04:52 | |
| *** fragatina has joined #openstack-nova | 04:55 | |
| *** hongbin has quit IRC | 04:55 | |
| *** fragatina has quit IRC | 04:59 | |
| *** slaweq has joined #openstack-nova | 05:02 | |
| *** baoli has quit IRC | 05:03 | |
| *** fragatina has joined #openstack-nova | 05:03 | |
| *** baoli has joined #openstack-nova | 05:03 | |
| *** Dinesh_Bhor has quit IRC | 05:07 | |
| *** slaweq has quit IRC | 05:07 | |
| *** Dinesh_Bhor has joined #openstack-nova | 05:07 | |
| *** Faster-Fanboi has quit IRC | 05:14 | |
| *** liuyulong has joined #openstack-nova | 05:16 | |
| *** slaweq has joined #openstack-nova | 05:18 | |
| *** Faster-Fanboi has joined #openstack-nova | 05:20 | |
| *** chyka has joined #openstack-nova | 05:21 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Support shared/dedicated vCPUs in one instance https://review.openstack.org/545734 | 05:21 |
| *** slaweq has quit IRC | 05:23 | |
| *** baoli has quit IRC | 05:25 | |
| openstackgerrit | Nakanishi Tomotaka proposed openstack/nova master: Test Compute API in multiple cells https://review.openstack.org/547273 | 05:26 |
| *** chyka has quit IRC | 05:26 | |
| *** inara has quit IRC | 05:31 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova-specs master: Support shared/dedicated vCPUs in one instance https://review.openstack.org/545734 | 05:34 |
| *** slaweq has joined #openstack-nova | 05:34 | |
| *** inara has joined #openstack-nova | 05:34 | |
| *** slaweq has quit IRC | 05:39 | |
| *** lajoskatona has joined #openstack-nova | 05:40 | |
| *** slaweq has joined #openstack-nova | 05:41 | |
| *** moshele has quit IRC | 05:45 | |
| *** slaweq has quit IRC | 05:46 | |
| *** mdnadeem has joined #openstack-nova | 05:49 | |
| *** slaweq has joined #openstack-nova | 05:50 | |
| *** abhishekk is now known as abhishekk|afk | 05:50 | |
| *** claudiub has joined #openstack-nova | 05:50 | |
| *** vladikr has quit IRC | 05:53 | |
| *** vladikr has joined #openstack-nova | 05:53 | |
| *** pengdake_ has joined #openstack-nova | 05:54 | |
| *** slaweq has quit IRC | 05:54 | |
| *** edmondsw has joined #openstack-nova | 05:55 | |
| *** udesale_ has joined #openstack-nova | 05:58 | |
| *** vishwana_ has joined #openstack-nova | 05:59 | |
| *** edmondsw has quit IRC | 05:59 | |
| *** udesale has quit IRC | 06:00 | |
| *** vishwanathj has quit IRC | 06:03 | |
| *** slaweq has joined #openstack-nova | 06:05 | |
| *** gokhan has joined #openstack-nova | 06:10 | |
| *** slaweq has quit IRC | 06:10 | |
| *** Spazmotic has joined #openstack-nova | 06:11 | |
| Spazmotic | Morning Novaers | 06:11 |
| *** slaweq has joined #openstack-nova | 06:11 | |
| *** namnh has joined #openstack-nova | 06:12 | |
| *** threestrands has quit IRC | 06:13 | |
| *** namnh has quit IRC | 06:16 | |
| *** tiendc has quit IRC | 06:16 | |
| *** annp has quit IRC | 06:16 | |
| *** phuongnh has quit IRC | 06:16 | |
| *** slaweq has quit IRC | 06:16 | |
| *** phuongnh has joined #openstack-nova | 06:16 | |
| *** namnh has joined #openstack-nova | 06:16 | |
| *** tiendc has joined #openstack-nova | 06:16 | |
| *** annp has joined #openstack-nova | 06:16 | |
| tetsuro | is cfriesen around? | 06:16 |
| itlinux | hello nova team.. i have a question will it be better to set the hypervisor to kvm or just leave them to qemu? | 06:17 |
| itlinux | thanks for the advice.. | 06:17 |
| *** udesale_ is now known as udesale | 06:18 | |
| *** Zames has joined #openstack-nova | 06:19 | |
| cfriesen | tetsuro: I'm here. | 06:19 |
| tetsuro | itlinux: KVM has more functionality. Let me quote from the document https://docs.openstack.org/ocata/config-reference/compute/hypervisors.html “QEMU - Quick EMUlator, generally only used for development purposes.” | 06:20 |
| itlinux | ok. | 06:20 |
| tetsuro | cfriesen thanks for reply | 06:20 |
| cfriesen | tetsuro: and I just added a bunch of comments to the review | 06:20 |
| itlinux | thanks.. | 06:20 |
| tetsuro | Yup, I didn’t understand your comment… | 06:20 |
| tetsuro | so what is it from the placement view? | 06:21 |
| tetsuro | alway 9 VCPUs in the inventory, right? | 06:21 |
| *** slaweq has joined #openstack-nova | 06:21 | |
| cfriesen | I'm not an expert on placement, but I think so | 06:21 |
| cfriesen | I'm not sure how it tracks allocated cpus though, whether it factors in cpu_allocation_ratio or not | 06:23 |
| tetsuro | okay, if you want (8*1+1*(1/2))=9, who will exclude the host if you claim the third shared vcpu ? | 06:23 |
| tetsuro | I mean (1/2)*2+(1/2)>1 | 06:24 |
| tetsuro | where is the logic of “(1/2)*2+(1/2)>1”? | 06:25 |
| *** slaweq has quit IRC | 06:26 | |
| cfriesen | in our internal patch we allocate "dedicated" pCPUs dynamically, so we would start with 9, allocate 8 dedicated ones (reporting a usage of 8), then allocate a "shared" one (and report a usage of 8.5) | 06:26 |
| tetsuro | Maybe corefilter is extended and he is aware of shared cpus and shared dedicated cpus? | 06:26 |
| tetsuro | oops | 06:26 |
| cfriesen | then at that point CoreFilter would prevent you from allocating another dedicated pCPU since that would put usage at 9.5 which is larger than 9 | 06:27 |
| cfriesen | but you could still allocate another shared pCPU since that would put usage right at 9 | 06:28 |
| cfriesen | If we statically determine which pCPUs are shared/dedicated, then I think CoreFilter or the placement equivalent would need to be modified. | 06:29 |
| tetsuro | Ah, so you mean your approach doesn’t need to pre-set something like “cpu_dedicate_set” ? | 06:31 |
| *** phuongnh has quit IRC | 06:31 | |
| itlinux | ok thanks tetsuro: my nova.conf says virt_type = kvm but my openstack hypervisor list show qemu type.. confused | 06:32 |
| cfriesen | yes. anything in NUMACell.cpuset that is not in NUMACell.pinned_cpus is available to run "shared" vCPUs | 06:32 |
| tetsuro | You are taking dynamically allocation approach that I put into alternative. | 06:32 |
| tetsuro | Okay. | 06:32 |
| cfriesen | yes, that's why I mentioned it originally | 06:33 |
| tetsuro | cleared | 06:33 |
| tetsuro | itlinux: that’s the default behavior https://bugs.launchpad.net/nova/+bug/1195361 | 06:34 |
| openstack | Launchpad bug 1195361 in OpenStack Compute (nova) "QEMU hypervisor type returned when libvirt_type = kvm" [Low,In progress] - Assigned to Tetsuro Nakamura (tetsuro0907) | 06:34 |
| cfriesen | the way we avoid needing to re-affine all the VMs when we allocate a new "dedicated" pCPU is that we put all the vCPU tasks into linux cpusets on the host. that way we can just reaffine the linux cpuset and all the VMs are affected | 06:35 |
| cfriesen | this does mean we don't use the libvirt cpusets though, so I'm not sure that can be used as a generic solution for upstream | 06:35 |
| cfriesen | with statically-allocated divisions between "shared" and "dedicated" pCPUs we might want to report two separate "cpus" and "cpus_used" numbers in the hypervisor details, one for "shared" and one for "dedicated" | 06:36 |
| *** slaweq has joined #openstack-nova | 06:37 | |
| *** sridharg has joined #openstack-nova | 06:38 | |
| *** Zames has quit IRC | 06:38 | |
| tetsuro | itlinux: because KVM only works with QEMU hypervisor, but agree this is confusing. We will fix this at least in Rocky realease. | 06:39 |
| itlinux | thanks tesuro: I am in OOO | 06:39 |
| tetsuro | your welcome | 06:41 |
| *** slaweq has quit IRC | 06:42 | |
| *** andreas_s has joined #openstack-nova | 06:42 | |
| *** acormier has joined #openstack-nova | 06:44 | |
| tetsuro | cfriesen: Ah, I see. I think I got you. So you are not re-configure xml files but re-pinning them in linux layer. | 06:44 |
| *** andreas_s has quit IRC | 06:45 | |
| cfriesen | itlinux: the complication arises because both kvm and qemu end up calling the "qemu" binary as the hypervisor, but in the kvm case libvirt passes something like "-machine pc-i440fx-rhel7.3.0,accel=kvm" to tell it to use the hardware acceleration | 06:47 |
| *** slaweq has joined #openstack-nova | 06:47 | |
| cfriesen | tetsuro: yes, that's correct | 06:47 |
| *** acormier has quit IRC | 06:48 | |
| itlinux | cfriesen: how can I verify that? | 06:48 |
| cfriesen | from the compute node you can just look at the "qemu" commandline using "ps -ef" or similar | 06:49 |
| tetsuro | cfriesen: Okay, I’ll add some words in the alternative and discuss with cores how they think. BTW, are you coming to PTG? | 06:49 |
| itlinux | is that in the /etc/libvirt/qemu has a file I will chk that.. | 06:50 |
| cfriesen | yes, I'll be there starting monday afternoon | 06:50 |
| cfriesen | itlinux: no, when you have a guest up and running, then look at the commandline for the running process | 06:50 |
| itlinux | looking now | 06:50 |
| itlinux | http://paste.openstack.org/show/682875/ | 06:51 |
| itlinux | looks correct to me but could you confirm it cfriesen: | 06:51 |
| itlinux | thanks | 06:51 |
| *** tpatil has joined #openstack-nova | 06:52 | |
| cfriesen | tetsuro: my concern about not reporting usage for "dedicated" pCPUs is that it'll make it tricky to debug scheduler failures if we don't have any way to see how "full" the compute nodes are | 06:52 |
| *** slaweq has quit IRC | 06:52 | |
| cfriesen | itlinux: looks good to me | 06:52 |
| itlinux | thanks | 06:52 |
| tetsuro | I’m coming to Dublin, too. looking forward to see ya. :) | 06:53 |
| cfriesen | likewise. I should get to bed, it's 1am | 06:53 |
| tetsuro | OMG | 06:54 |
| cfriesen | :) | 06:54 |
| tetsuro | I got your concern, too. | 06:54 |
| cfriesen | cool. should be a fun discussion | 06:54 |
| cfriesen | later | 06:54 |
| tetsuro | have a good night! I gotta go now, too. | 06:55 |
| *** cfriesen has quit IRC | 06:55 | |
| *** pcaruana has joined #openstack-nova | 06:56 | |
| *** andreas_s has joined #openstack-nova | 07:00 | |
| *** pcaruana has quit IRC | 07:02 | |
| *** pcaruana has joined #openstack-nova | 07:02 | |
| *** sahid has joined #openstack-nova | 07:03 | |
| *** Tom-Tom has quit IRC | 07:03 | |
| *** Tom-Tom has joined #openstack-nova | 07:03 | |
| *** Tom-Tom has quit IRC | 07:03 | |
| *** Tom-Tom has joined #openstack-nova | 07:04 | |
| *** udesale_ has joined #openstack-nova | 07:12 | |
| *** udesale has quit IRC | 07:15 | |
| *** lajoskatona has quit IRC | 07:16 | |
| *** josecastroleon has joined #openstack-nova | 07:16 | |
| *** fragatina has quit IRC | 07:18 | |
| *** lajoskatona has joined #openstack-nova | 07:19 | |
| *** slaweq has joined #openstack-nova | 07:23 | |
| *** josecastroleon has quit IRC | 07:25 | |
| *** josecastroleon has joined #openstack-nova | 07:25 | |
| *** slaweq has quit IRC | 07:27 | |
| *** Zames has joined #openstack-nova | 07:28 | |
| *** cz2 has quit IRC | 07:30 | |
| *** McNinja has quit IRC | 07:30 | |
| *** vdrok has quit IRC | 07:30 | |
| *** khappone has quit IRC | 07:30 | |
| *** homeski has quit IRC | 07:30 | |
| *** itlinux has quit IRC | 07:31 | |
| *** trozet has quit IRC | 07:31 | |
| *** dgonzalez has quit IRC | 07:31 | |
| *** ebbex has quit IRC | 07:31 | |
| *** Zames has quit IRC | 07:31 | |
| *** slaweq has joined #openstack-nova | 07:37 | |
| *** tpatil has quit IRC | 07:40 | |
| openstackgerrit | Andreas Jaeger proposed openstack/osc-placement master: DNM: Testing py35 https://review.openstack.org/547313 | 07:40 |
| *** slaweq has quit IRC | 07:42 | |
| *** slaweq has joined #openstack-nova | 07:42 | |
| *** edmondsw has joined #openstack-nova | 07:43 | |
| *** slaweq has quit IRC | 07:44 | |
| *** slaweq has joined #openstack-nova | 07:44 | |
| *** edmondsw has quit IRC | 07:48 | |
| *** jichen has joined #openstack-nova | 07:48 | |
| *** josecastroleon has quit IRC | 07:51 | |
| *** vladikr has quit IRC | 07:52 | |
| *** vladikr has joined #openstack-nova | 07:52 | |
| *** itlinux has joined #openstack-nova | 07:53 | |
| *** trozet has joined #openstack-nova | 07:53 | |
| *** dgonzalez has joined #openstack-nova | 07:53 | |
| *** ebbex has joined #openstack-nova | 07:53 | |
| *** Dinesh_Bhor has quit IRC | 07:55 | |
| *** josecastroleon has joined #openstack-nova | 07:56 | |
| *** cz2 has joined #openstack-nova | 07:57 | |
| *** McNinja has joined #openstack-nova | 07:57 | |
| *** vdrok has joined #openstack-nova | 07:57 | |
| *** khappone has joined #openstack-nova | 07:57 | |
| *** homeski has joined #openstack-nova | 07:57 | |
| *** vivsoni__ has quit IRC | 08:05 | |
| *** claudiub has quit IRC | 08:07 | |
| *** claudiub|2 has joined #openstack-nova | 08:08 | |
| *** vivsoni has joined #openstack-nova | 08:08 | |
| *** claudiub has joined #openstack-nova | 08:10 | |
| *** claudiub|2 has quit IRC | 08:11 | |
| *** cz2 has quit IRC | 08:11 | |
| *** McNinja has quit IRC | 08:11 | |
| *** vdrok has quit IRC | 08:11 | |
| *** khappone has quit IRC | 08:11 | |
| *** homeski has quit IRC | 08:11 | |
| *** itlinux has quit IRC | 08:11 | |
| *** trozet has quit IRC | 08:11 | |
| *** dgonzalez has quit IRC | 08:11 | |
| *** ebbex has quit IRC | 08:11 | |
| *** yangyapeng has joined #openstack-nova | 08:13 | |
| *** amoralej|off is now known as amoralej | 08:14 | |
| *** jichen_ has joined #openstack-nova | 08:14 | |
| *** fusmu has joined #openstack-nova | 08:14 | |
| *** jichen has quit IRC | 08:15 | |
| *** jichen_ is now known as jichen | 08:15 | |
| *** jpena|off is now known as jpena | 08:16 | |
| *** rcernin has quit IRC | 08:16 | |
| *** yangyapeng has quit IRC | 08:19 | |
| *** tesseract has joined #openstack-nova | 08:19 | |
| *** acormier has joined #openstack-nova | 08:19 | |
| *** acormier has quit IRC | 08:20 | |
| *** acormier has joined #openstack-nova | 08:20 | |
| *** dgonzalez has joined #openstack-nova | 08:22 | |
| *** ebbex has joined #openstack-nova | 08:22 | |
| *** trozet has joined #openstack-nova | 08:22 | |
| sahid | jaypipes: i saw you interest on some libvirt related specs, can you have look at these specs too: https://review.openstack.org/#/c/511188/ | 08:23 |
| sahid | https://review.openstack.org/#/c/539605/ | 08:23 |
| sahid | they address important use cases related to NFV | 08:24 |
| *** edmondsw has joined #openstack-nova | 08:28 | |
| *** phuongnh has joined #openstack-nova | 08:30 | |
| *** tetsuro has left #openstack-nova | 08:32 | |
| *** homeski has joined #openstack-nova | 08:33 | |
| *** vdrok has joined #openstack-nova | 08:33 | |
| *** McNinja has joined #openstack-nova | 08:33 | |
| *** cz2 has joined #openstack-nova | 08:33 | |
| *** khappone has joined #openstack-nova | 08:34 | |
| *** ttsiouts has quit IRC | 08:36 | |
| *** ttsiouts has joined #openstack-nova | 08:37 | |
| *** danpawlik has joined #openstack-nova | 08:38 | |
| *** tssurya has joined #openstack-nova | 08:40 | |
| *** damien_r has joined #openstack-nova | 08:44 | |
| *** slaweq_ has joined #openstack-nova | 08:48 | |
| *** ccamacho has joined #openstack-nova | 08:49 | |
| *** slaweq_ has quit IRC | 08:53 | |
| *** acormier has quit IRC | 08:53 | |
| *** udesale_ is now known as udesale | 08:53 | |
| *** vks1 has joined #openstack-nova | 08:55 | |
| bauzas | good morning Nova | 08:57 |
| Spazmotic | Morning bauzas | 08:57 |
| Spazmotic | Any cores that could pu this on their radar for this week? https://review.openstack.org/#/c/538415/ Been sitting for about 3 weeks now. | 08:57 |
| *** slaweq_ has joined #openstack-nova | 08:58 | |
| Spazmotic | I am back from Korea btw, good to see you all again. | 08:59 |
| bauzas | Spazmotic: most of the folks are preparing to travel for the PTG | 08:59 |
| bauzas | maybe you could ask for reviewing after the next week ? | 09:00 |
| Spazmotic | Yeah I know it, just hoping to get some eyes on it I guess | 09:00 |
| bauzas | today, I'll look at specs | 09:00 |
| *** blkart has quit IRC | 09:00 | |
| *** tianhui has quit IRC | 09:00 | |
| *** tianhui has joined #openstack-nova | 09:01 | |
| *** liuyulong has quit IRC | 09:01 | |
| *** blkart has joined #openstack-nova | 09:03 | |
| *** yamahata has quit IRC | 09:03 | |
| Spazmotic | But asking is about all that I can do for now, and would remiss if I didn't do it :) | 09:03 |
| bauzas | sure, just explaining | 09:07 |
| Spazmotic | Yeah I understand of course :D | 09:08 |
| *** rcernin has joined #openstack-nova | 09:09 | |
| *** slaweq_ has quit IRC | 09:09 | |
| *** slaweq_ has joined #openstack-nova | 09:10 | |
| *** slunkad has quit IRC | 09:13 | |
| *** slunkad has joined #openstack-nova | 09:14 | |
| *** slaweq_ has quit IRC | 09:14 | |
| *** lucas-pto is now known as lucasagomes | 09:15 | |
| *** yangyapeng has joined #openstack-nova | 09:16 | |
| *** moshele has joined #openstack-nova | 09:18 | |
| *** yamamoto_ has joined #openstack-nova | 09:21 | |
| *** yamamoto has quit IRC | 09:24 | |
| *** slunkad has quit IRC | 09:27 | |
| *** slunkad has joined #openstack-nova | 09:28 | |
| *** moshele has quit IRC | 09:29 | |
| *** vivsoni has quit IRC | 09:31 | |
| *** rubasov has joined #openstack-nova | 09:33 | |
| *** liuyulong has joined #openstack-nova | 09:35 | |
| *** slunkad has quit IRC | 09:38 | |
| openstackgerrit | Lajos Katona proposed openstack/nova master: WIP: ServerMovingTests with custom resources https://review.openstack.org/497399 | 09:39 |
| *** vivsoni has joined #openstack-nova | 09:39 | |
| *** belmoreira has joined #openstack-nova | 09:42 | |
| *** mgoddard_ has joined #openstack-nova | 09:44 | |
| ttsiouts | johnthetubaguy: Goodmorning, are you around? | 09:44 |
| *** liuyulong has quit IRC | 09:47 | |
| johnthetubaguy | ttsiouts: I am | 09:48 |
| openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | 09:48 |
| * johnthetubaguy unplugs headphones so he can hear when people say hello | 09:49 | |
| *** slunkad has joined #openstack-nova | 09:50 | |
| ttsiouts | johnthetubaguy: Hello! :) we were thinking about the second call to placement | 09:51 |
| johnthetubaguy | ttsiouts: what was the plan before, just returning what you had previously fetched from placement? | 09:51 |
| *** Tom-Tom has quit IRC | 09:52 | |
| ttsiouts | johnthetubaguy: hmmm if the freeing of the resources is successful then we can call placement for a second time. | 09:52 |
| *** Tom-Tom has joined #openstack-nova | 09:52 | |
| ttsiouts | we chose to form the response in the service just to save time | 09:53 |
| ttsiouts | and not having to trigger placement again.. | 09:53 |
| *** derekh has joined #openstack-nova | 09:53 | |
| johnthetubaguy | ttsiouts: I think that proxy is bad really, you don't know what microversion nova wants to ask for, etc | 09:53 |
| johnthetubaguy | ttsiouts: simpler to tell Nova if you were successful or not | 09:53 |
| ttsiouts | johnthetubaguy: Great. I'll do that. | 09:54 |
| johnthetubaguy | ttsiouts: I think its worth doing a cheaky import of your client in there for now, or a link to the code at least, just so its possible to follow the breadcrumbs | 09:55 |
| ttsiouts | johnthetubaguy: yes, seems better | 09:56 |
| ttsiouts | johnthetubaguy: what do you think is better? an api call or importing the service? | 09:56 |
| johnthetubaguy | ttsiouts: cool, getting back to the spec updates (oils delete key) | 09:56 |
| johnthetubaguy | ttsiouts: I quite like the API call, copying vendor data | 09:57 |
| *** Tom-Tom has quit IRC | 09:57 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Support disabling a cell https://review.openstack.org/546684 | 09:57 |
| ttsiouts | johnthetubaguy: Great!! thanks John, I'll follow that up | 09:57 |
| johnthetubaguy | ttsiouts: traditionally any cross service communication in OpenStack is via a REST API, so good not to break that rule | 09:58 |
| *** rubasov has left #openstack-nova | 09:58 | |
| johnthetubaguy | ttsiouts: being an API on the other end will making caching easier anyways | 09:58 |
| *** rcernin has quit IRC | 09:59 | |
| ttsiouts | johnthetubaguy: cool! thanks John! | 10:00 |
| johnthetubaguy | no worries | 10:01 |
| *** masber has quit IRC | 10:01 | |
| bauzas | oh man, FF | 10:04 |
| *** bauzas is now known as bauwser | 10:04 | |
| *** liuyulong has joined #openstack-nova | 10:06 | |
| openstackgerrit | Arvind Nadendla proposed openstack/nova-specs master: Support traits in Glance https://review.openstack.org/541507 | 10:07 |
| *** pengdake_ has quit IRC | 10:09 | |
| *** Guest16541 has quit IRC | 10:13 | |
| *** ispp has quit IRC | 10:15 | |
| *** vivsoni has quit IRC | 10:19 | |
| *** vivsoni has joined #openstack-nova | 10:19 | |
| *** mdnadeem has quit IRC | 10:22 | |
| *** sree_ has joined #openstack-nova | 10:24 | |
| *** sree_ is now known as Guest82581 | 10:25 | |
| *** stakeda has quit IRC | 10:27 | |
| *** udesale_ has joined #openstack-nova | 10:28 | |
| *** Guest82581 has quit IRC | 10:29 | |
| *** udesale has quit IRC | 10:29 | |
| *** bnemec-pto has quit IRC | 10:32 | |
| *** mdnadeem has joined #openstack-nova | 10:34 | |
| *** janki has quit IRC | 10:35 | |
| *** namnh has quit IRC | 10:44 | |
| *** chyka has joined #openstack-nova | 10:45 | |
| openstackgerrit | Hironori Shiina proposed openstack/nova-specs master: Ironic: Instance switchover https://review.openstack.org/449155 | 10:47 |
| *** chyka has quit IRC | 10:50 | |
| *** tianhui has quit IRC | 10:58 | |
| redondo-mk | Hi. Is there anyone there that would be willing to give me some info how nova compute pulls NUMA memory info...would save me some time just trying to figure it out from nova codebase... | 11:00 |
| *** xnox has joined #openstack-nova | 11:02 | |
| *** xnox has joined #openstack-nova | 11:02 | |
| redondo-mk | I'm looking at info that I get in `at /proc/meminfo | grep Huge` and `grep Huge /sys/devices/system/node/node*/meminfo` and I see numbers adding up...what I don't get is the following....Total-Free should give you Used, right but then I see different numbers for used in compute_nodes db table (`select numa_topology from compute_nodes`)? | 11:02 |
| openstackgerrit | Hironori Shiina proposed openstack/nova master: ironic: Support resize and cold migration https://review.openstack.org/500677 | 11:04 |
| *** abhishekk|afk has quit IRC | 11:05 | |
| *** gouthamr has joined #openstack-nova | 11:08 | |
| *** phuongnh has quit IRC | 11:09 | |
| *** pooja_jadhav has quit IRC | 11:09 | |
| *** annp has quit IRC | 11:11 | |
| *** udesale__ has joined #openstack-nova | 11:15 | |
| *** moshele has joined #openstack-nova | 11:16 | |
| *** udesale_ has quit IRC | 11:18 | |
| *** moshele has quit IRC | 11:19 | |
| *** tiendc has quit IRC | 11:20 | |
| *** moshele has joined #openstack-nova | 11:23 | |
| *** janki has joined #openstack-nova | 11:28 | |
| *** mdnadeem_ has joined #openstack-nova | 11:35 | |
| *** mdnadeem has quit IRC | 11:35 | |
| *** udesale__ has quit IRC | 11:36 | |
| *** dtantsur|afk is now known as dtantsur | 11:40 | |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Spec on preemptible servers https://review.openstack.org/438640 | 11:41 |
| *** liuyulong has quit IRC | 11:41 | |
| *** BryanS68 has joined #openstack-nova | 11:42 | |
| *** links has quit IRC | 11:42 | |
| johnthetubaguy | ttsiouts: got a first go at the updates to the spec done: ^ | 11:43 |
| *** vks1 has quit IRC | 11:47 | |
| *** sahid has quit IRC | 11:50 | |
| *** fusmu has quit IRC | 11:52 | |
| *** fusmu_ has joined #openstack-nova | 11:52 | |
| *** tbachman has quit IRC | 11:55 | |
| *** links has joined #openstack-nova | 11:55 | |
| *** moshele has quit IRC | 12:01 | |
| strigazi | johnthetubaguy: thanks for the update | 12:01 |
| *** mdnadeem_ has quit IRC | 12:06 | |
| *** fragatina has joined #openstack-nova | 12:09 | |
| *** srf has joined #openstack-nova | 12:15 | |
| srf | Hello i want to ask when i install devstack i meet nova error | 12:16 |
| Spazmotic | Feel free to let us know the error and i'm sure someone has seen it before, or you may also want to try in #openstack for deployment help if you don't get any help as many people are preparing to travel to Irelund | 12:17 |
| *** edmondsw has quit IRC | 12:18 | |
| *** pengdake_ has joined #openstack-nova | 12:18 | |
| *** mdnadeem_ has joined #openstack-nova | 12:18 | |
| *** edmondsw has joined #openstack-nova | 12:18 | |
| *** edmondsw has quit IRC | 12:23 | |
| *** lucasagomes is now known as lucas-hungry | 12:24 | |
| srf | I want to send but i can't paste or copy or send the pic | 12:24 |
| Spazmotic | Well devstack is a command line installer so many copy and paste the error from terminal into a github gist :) | 12:25 |
| openstackgerrit | Elod Illes proposed openstack/nova master: Functional test: cold migrate to compute down https://review.openstack.org/496280 | 12:27 |
| *** bhagyashris has quit IRC | 12:29 | |
| srf | Oke i'll try. | 12:29 |
| *** links has quit IRC | 12:30 | |
| *** sahid has joined #openstack-nova | 12:34 | |
| *** masber has joined #openstack-nova | 12:38 | |
| *** links has joined #openstack-nova | 12:43 | |
| *** ratailor has quit IRC | 12:46 | |
| *** tbachman has joined #openstack-nova | 12:54 | |
| *** jpena is now known as jpena|lunch | 12:55 | |
| jaypipes | bauwser: is sahid going to PTG? | 12:56 |
| *** jaypipes is now known as leakypipes | 12:56 | |
| *** tbachman has quit IRC | 12:59 | |
| *** tbachman has joined #openstack-nova | 13:00 | |
| bauwser | leakypipes: no, mostly all our team but him :) | 13:00 |
| leakypipes | bauwser: shame :( there's a bunch of specs I wanted to discuss with him. | 13:03 |
| *** gouthamr has quit IRC | 13:04 | |
| *** liverpooler has joined #openstack-nova | 13:04 | |
| *** gouthamr has joined #openstack-nova | 13:04 | |
| leakypipes | dansmith: still around? | 13:05 |
| leakypipes | dansmith: question for you on that IO thread vs vCPU thread thing... | 13:05 |
| sean-k-mooney | leakypipes: e.g. qemu emulator treads? | 13:06 |
| sean-k-mooney | they are the ones that od io right? | 13:06 |
| *** fusmu_ has quit IRC | 13:06 | |
| leakypipes | sean-k-mooney: are you making fun of me? :) | 13:07 |
| sean-k-mooney | haha no it geunely works differently depend on kernel vs dpdk vhost or kvm vs qeum with tci backend so it never keep this strait in my head | 13:07 |
| leakypipes | sean-k-mooney: no, AFAIK, the emulator threads are the QEMU control process threads -- they send communication events to the guest I think? | 13:08 |
| *** fusmu has joined #openstack-nova | 13:08 | |
| leakypipes | sean-k-mooney: the I/O threads are for doing device read/write operations on behalf of the guest. And the vCPU threads are for the guest's userspace code to run. | 13:08 |
| leakypipes | sean-k-mooney: but I had a question about whether libvirt allows each of those three types of threads to be pinned to a specific pCPU or whether only emulator and vCPU threads were possible to pin... | 13:09 |
| johnthetubaguy | you know I assumed it pinned them all together, but that is a good question | 13:10 |
| *** edmondsw has joined #openstack-nova | 13:10 | |
| *** srf has quit IRC | 13:10 | |
| leakypipes | sean-k-mooney: and further to that, can you pin the emulator thread(s) to the same pCPU as the I/O threads... | 13:10 |
| sahid | leakypipes: you can pin i/o threads with emulator threads on a same pCPUs but it can have bad effects, and increase I/O lantency | 13:12 |
| leakypipes | sahid: ok, cool. thanks :) | 13:12 |
| leakypipes | sahid: the emulator thread doesn't get a whole lot of control process "traffic" though, right? | 13:13 |
| leakypipes | sahid: pretty minimal compared to IO or vCPU threads, yeah? | 13:13 |
| leakypipes | sahid: I mean, I understand for RT environments wanting to have guaranteed execution and all that, I'm just curious about the internals of it all, nothing more. | 13:14 |
| sahid | honnestly i don't know exaclty, it's just that we know the emulator threads can lock the full CPU so we want avoid to have vCPUs running sensitive apps | 13:15 |
| sahid | i would say we want the same for i/o threads so it's probably better to avoid to pin them together | 13:16 |
| leakypipes | got it. | 13:16 |
| leakypipes | thanks :) | 13:16 |
| *** srf has joined #openstack-nova | 13:18 | |
| srf | i'll try to install devstack in my computer but always error in nova http://paste.openstack.org/show/682946/ anybody can help me what wrong this | 13:21 |
| *** lucas-hungry is now known as lucasagomes | 13:21 | |
| *** Tom-Tom has joined #openstack-nova | 13:21 | |
| *** udesale has joined #openstack-nova | 13:22 | |
| hshiina | leakypipes: hi, i have a question about your note at https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L855 | 13:24 |
| hshiina | InventoryInUse exception is periodically logged with stacktrace in ironic job: http://logs.openstack.org/19/546919/2/check/ironic-tempest-dsvm-ipa-partition-pxe_ipmitool-tinyipa-python3/2737ab0/logs/screen-n-cpu.txt.gz?level=ERROR | 13:25 |
| hshiina | is there anything to fix here? | 13:25 |
| *** acormier has joined #openstack-nova | 13:26 | |
| *** acormier has quit IRC | 13:29 | |
| *** acormier has joined #openstack-nova | 13:29 | |
| leakypipes | hshiina: one sec, reading :) | 13:33 |
| *** lajoskatona has left #openstack-nova | 13:34 | |
| leakypipes | hshiina: no, that looks to be a different scenario than the one described in that comment. lemme dig some more into the log there. | 13:35 |
| *** jichen has quit IRC | 13:36 | |
| hshiina | leakypipes: sure, thanks. | 13:37 |
| leakypipes | hshiina: if I had to guess, what is happening in your case is the following: | 13:37 |
| leakypipes | 1) nova boots an Ironic flavor | 13:37 |
| leakypipes | 2) nova-scheduler finds an available Ironic compute node that has CUSTOM_BAREMETAL inventory of 1, used of 0. | 13:38 |
| leakypipes | 3) nova-compute calls the Ironic virt driver to launch the instance | 13:38 |
| leakypipes | 4) Ironic virt driver launches the instance. All good. | 13:39 |
| leakypipes | 5) At some point, the Ironic node goes into an "unavailable" status | 13:39 |
| leakypipes | 6) The resource tracker in nova-compute runs update_available_resource() | 13:39 |
| leakypipes | 7) the Ironic virt driver returns an inventory total of 0 CUSTOM_BAREMETAL for the unavailable node | 13:39 |
| leakypipes | 8) The nova-compute resource tracker dutifully tries to remove the inventory for that node | 13:40 |
| leakypipes | 9) the Placement API returns an InventoryInUse exception because there is an instance running on that Ironic node | 13:40 |
| leakypipes | hshiina: so what I would check in your job logs is whether that Ironic node that the instance ended up on was set to unavailable at some point between 11:17:05 and 11:18:13 | 13:41 |
| leakypipes | hshiina: from what I can tell, the resource tracker and placement API are doing "the right thing" and preventing inventory from being deleted when there is an active instance on that Ironic node. | 13:41 |
| hshiina | leakypipes: thank you. i will dig it later. | 13:43 |
| hshiina | leakypipes: i am interested in. but, i have to leave office shortly. | 13:43 |
| leakypipes | np | 13:43 |
| *** pchavva has joined #openstack-nova | 13:48 | |
| *** edleafe is now known as figleaf | 13:50 | |
| *** gouthamr has quit IRC | 13:50 | |
| srf | i'll try to install devstack in my computer but always error in nova http://paste.openstack.org/show/682946/ anybody can help me, ? | 13:50 |
| *** acormier has quit IRC | 13:52 | |
| Spazmotic | srf: Quick google revealed this.. did you try to search around a bit? https://bugs.launchpad.net/devstack/+bug/1726260 | 13:54 |
| openstack | Launchpad bug 1726260 in devstack "devstack fails while running ./stack.sh in master/queens in ubuntu 16.04 while starting n-cpu" [Undecided,New] | 13:54 |
| *** jpena|lunch is now known as jpena | 13:54 | |
| Spazmotic | Give that a try :) | 13:54 |
| *** lyan has joined #openstack-nova | 13:54 | |
| *** vks1 has joined #openstack-nova | 13:55 | |
| *** links has quit IRC | 13:56 | |
| *** hshiina is now known as hshiina|afk | 13:57 | |
| srf | Spazmotic : I've tried to adding ENABLED_SERVICES=placement-api but still error | 13:58 |
| *** pengdake_ has quit IRC | 13:59 | |
| *** moshele has joined #openstack-nova | 14:02 | |
| *** vks1 has quit IRC | 14:04 | |
| *** psachin has quit IRC | 14:04 | |
| *** hshiina|afk has quit IRC | 14:04 | |
| *** dklyle has quit IRC | 14:04 | |
| Spazmotic | Not too sure then, but i'm sure someone may have seen it before.. also again you may wish to ask in #openstack as that's more designed for deployment support | 14:05 |
| *** imacdonn has quit IRC | 14:06 | |
| *** imacdonn has joined #openstack-nova | 14:07 | |
| *** eharney has joined #openstack-nova | 14:08 | |
| *** vks1 has joined #openstack-nova | 14:09 | |
| *** fragatina has quit IRC | 14:09 | |
| *** Tom-Tom has quit IRC | 14:11 | |
| *** Tom-Tom has joined #openstack-nova | 14:13 | |
| *** baoli has joined #openstack-nova | 14:14 | |
| *** moshele has quit IRC | 14:15 | |
| *** saphi_ has joined #openstack-nova | 14:18 | |
| *** esberglu has joined #openstack-nova | 14:21 | |
| *** lyan has quit IRC | 14:21 | |
| *** moshele has joined #openstack-nova | 14:23 | |
| *** Guest70 has joined #openstack-nova | 14:24 | |
| Guest70 | hello | 14:24 |
| *** lyan has joined #openstack-nova | 14:25 | |
| *** Guest70 has left #openstack-nova | 14:26 | |
| *** awaugama has joined #openstack-nova | 14:28 | |
| *** udesale_ has joined #openstack-nova | 14:28 | |
| *** udesale has quit IRC | 14:29 | |
| *** Tom-Tom has quit IRC | 14:31 | |
| *** fusmu has quit IRC | 14:33 | |
| *** fusmu_ has joined #openstack-nova | 14:33 | |
| *** moshele has quit IRC | 14:35 | |
| *** acormier has joined #openstack-nova | 14:35 | |
| *** acormier has quit IRC | 14:37 | |
| *** acormier has joined #openstack-nova | 14:37 | |
| *** lbragstad has joined #openstack-nova | 14:37 | |
| *** yamahata has joined #openstack-nova | 14:41 | |
| *** amoralej is now known as amoralej|lunch | 14:42 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: Network interface attach/detach https://review.openstack.org/546813 | 14:44 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/nova master: WIP: Add the reaper entry point https://review.openstack.org/547450 | 14:45 |
| *** cfriesen has joined #openstack-nova | 14:51 | |
| *** Zames has joined #openstack-nova | 14:51 | |
| *** acormier has quit IRC | 14:52 | |
| bauwser | if folks have specs, I'm hungry to eat them | 14:53 |
| bauwser | like, you're going to the PTG and would like to discuss about your spec, I'm your man | 14:53 |
| *** gbarros has quit IRC | 14:53 | |
| figleaf | bauwser: after you eat them, you know what they turn into :) | 14:54 |
| *** r-daneel has joined #openstack-nova | 14:54 | |
| bauwser | if they are made of cheese, I'm +2 | 14:54 |
| *** Zames has quit IRC | 14:55 | |
| bauwser | even if https://www.thelocal.fr/20180223/french-cheese-wars-are-the-days-of-the-real-normandy-camembert-numbered | 14:55 |
| figleaf | guess it will be like taking a microbrew beer and then mass-producing it | 14:57 |
| *** tidwellr has joined #openstack-nova | 14:59 | |
| *** mriedem has joined #openstack-nova | 14:59 | |
| *** edmondsw has quit IRC | 15:00 | |
| *** edmondsw has joined #openstack-nova | 15:00 | |
| dansmith | leakypipes: I'm really not the right person to answer detailed questions about that | 15:04 |
| *** edmondsw has quit IRC | 15:05 | |
| *** masber has quit IRC | 15:05 | |
| *** amodi has joined #openstack-nova | 15:07 | |
| *** gbarros has joined #openstack-nova | 15:08 | |
| *** gbarros has quit IRC | 15:14 | |
| *** gbarros has joined #openstack-nova | 15:15 | |
| bauwser | dansmith: I don't know your current TZ, but it's Friday here :) | 15:15 |
| dansmith | bauwser: I don't know my current TZ either, but I'm pretty sure it's not friday in my body | 15:15 |
| bauwser | still on the air ? | 15:16 |
| dansmith | no, you caught me, I know my current TZ | 15:16 |
| dansmith | but my body doesn't | 15:16 |
| bauwser | heh | 15:16 |
| *** mdnadeem_ has quit IRC | 15:17 | |
| mriedem | why do we have https://github.com/openstack/nova/blob/ef4000a0d326deb004843ee51d18030224c5630f/nova/db/sqlalchemy/api_models.py#L409 ? | 15:19 |
| mriedem | seems silly to have an entire table to hold one string | 15:19 |
| mriedem | maybe it goes back to the idea that server groups would have >1 policy | 15:19 |
| mriedem | but the API doesn't allow that | 15:19 |
| mriedem | https://github.com/openstack/nova/blob/ef4000a0d326deb004843ee51d18030224c5630f/nova/api/openstack/compute/schemas/server_groups.py#L29-L39 | 15:20 |
| *** srf has quit IRC | 15:20 | |
| *** david-lyle has joined #openstack-nova | 15:21 | |
| dansmith | mriedem: yeah it's 1:N I think | 15:21 |
| dansmith | not that we can do that, I don't think | 15:21 |
| mriedem | the api doesn't allow it | 15:22 |
| mriedem | but that's the legacy reason i guess, | 15:22 |
| mriedem | the schema makes you pass a single-item list for the policy | 15:22 |
| dansmith | but even still, none of the stuff we do to honor the policy considers multiple options right? | 15:22 |
| mriedem | and it's called "policies" but doesn't allow >1 | 15:22 |
| mriedem | correct | 15:22 |
| mriedem | it's just weird | 15:22 |
| leakypipes | cfriesen: am I gonna have to get out the beating stick for you? :P | 15:23 |
| mriedem | dansmith: context is https://review.openstack.org/#/c/546925 which is tied to an old blueprint from sgordon | 15:23 |
| leakypipes | cfriesen: well, I suppose we could rent a pair of those fake Sumo wrestler costumes and go at it. | 15:23 |
| dansmith | mriedem: ah yes, affrinity | 15:25 |
| mriedem | i'm currently in the process of ripping up the spec | 15:25 |
| dansmith | mriedem: ugh, that max_number_per_host thing is just silly.. that's never going to do something useful, IMHO | 15:26 |
| *** BryanS68 has quit IRC | 15:27 | |
| mriedem | i can see the use, | 15:27 |
| mriedem | it makes the soft policy less soft | 15:28 |
| mriedem | jello-like if you will | 15:28 |
| dansmith | what's the point in that? | 15:28 |
| mriedem | there is some nfv use case in the spec | 15:28 |
| *** amoralej|lunch is now known as amoralej | 15:28 | |
| dansmith | I think from a modeling point of view, it'd be hard to reason about that unless you knew "I'm going to start ten of the exact same thing for throughput and I want those spread across three nodes" | 15:29 |
| mriedem | anti-affinity to a point, so they don't need all VMs in the group on separate hosts, but they can only tolerate up to a limit of VMs on the same host | 15:29 |
| *** dave-mccowan has joined #openstack-nova | 15:29 | |
| mriedem | otherwise today if you have 10 VMs in a soft-anti-affinity group, and only 2 available hosts, you could get like 1 on 1 host and 9 on the other | 15:30 |
| dansmith | right, so a more useful thing would be to say "instances in this group should spread across N hosts before considering doubling up" | 15:30 |
| dansmith | that's an easier to reason-about model from the outside I think | 15:30 |
| dansmith | and it doesn't depend on the size of the group | 15:30 |
| mriedem | i can't say if that's easier to reason about from the outside or not, | 15:31 |
| dansmith | it focuses on the amount of redundancy you need regardless | 15:31 |
| mriedem | since i've had this in my head the way it's proposed | 15:31 |
| mriedem | dansmith: mention it in the spec? | 15:31 |
| mriedem | either way i need to bring it up at the ptg b/c of some questions about how to model the thing in the api and db | 15:31 |
| dansmith | I really hate the idea of adding extra_specs to anything, but especially groups | 15:32 |
| mriedem | i said that on the spec | 15:32 |
| *** gbarros has quit IRC | 15:32 | |
| *** gbarros has joined #openstack-nova | 15:33 | |
| dansmith | I guess their use-case actually specifically requires this as the model | 15:34 |
| *** BryanS68 has joined #openstack-nova | 15:36 | |
| mriedem | is sgordon still a man about town? | 15:37 |
| mriedem | isn't he in the UK/Ireland area? | 15:37 |
| sgordon | mriedem, nah canadia | 15:37 |
| mriedem | drats | 15:37 |
| mriedem | sgordon: can you take a look at that spec regardless of your physical location? | 15:37 |
| sgordon | mriedem, yeah i just got the email notification and opened it up | 15:38 |
| mriedem | gibi might also have some input here since ericsson added the soft affinity stuff | 15:38 |
| *** jpena is now known as jpena|brb | 15:39 | |
| sgordon | mriedem, i have to admit i havent had anyone clamouring for this in a fair while, but that may just mean folks are working around it using host aggregates and higher level orchestration | 15:39 |
| *** kenperkins has joined #openstack-nova | 15:41 | |
| *** fusmu_ has quit IRC | 15:42 | |
| *** BryanS68 has quit IRC | 15:42 | |
| *** sahid has quit IRC | 15:43 | |
| bauwser | mriedem: dansmith: we have a clear distinction between filters which hard-check possibilities and weighers which soft-check | 15:43 |
| bauwser | having some middleground is something more than just for affinity | 15:44 |
| bauwser | tbc, I'm -2 on the spexc | 15:44 |
| bauwser | mriedem: dansmith: but johnthetubaguy had an idea to deprecate filters/weighers and have a different semantic for saying "here I want to hard-stop or fail-safe" | 15:45 |
| * bauwser will try to find the backlog spec for that before -2ing the change | 15:45 | |
| johnthetubaguy | bauwser: got a pointer to the new one | 15:46 |
| *** sahid has joined #openstack-nova | 15:46 | |
| johnthetubaguy | might be better than my previous re-write the world plan | 15:46 |
| dansmith | bauwser: this is harder than soft | 15:46 |
| bauwser | soft is weighting | 15:46 |
| bauwser | hard is filtering | 15:46 |
| bauwser | here they want to have a soft-but-hard | 15:46 |
| dansmith | bauwser: right, this spec specifies something hard, which is filtering, | 15:46 |
| bauwser | bref | 15:47 |
| dansmith | but it weakens the normally hard anti-affinity for some value of allowed overlap | 15:47 |
| dansmith | I really don't like it | 15:47 |
| bauwser | it's unclear and I don't think modifying filters or weighers is a good thing | 15:47 |
| bauwser | yeah exactly that | 15:47 |
| bauwser | instead of hijacking what we currently have, we should discuss on a better approach | 15:47 |
| bauwser | which is what was johnthetubaguy exactly trying to do | 15:47 |
| bauwser | johnthetubaguy: what is the new proposal, tbh I haven't gone thru all specs yet | 15:48 |
| mriedem | i'm not following you | 15:49 |
| mriedem | you can do hard anti affinity and fail if you can't place the VMs on separate hosts, | 15:49 |
| mriedem | you can do soft anti affinity and get more VMs on a host than you'd really like | 15:49 |
| mriedem | this is saying, i want soft but i can only tolerate up to a limit | 15:49 |
| *** Spazmotic has quit IRC | 15:49 | |
| mriedem | or, soft is ok but to a limit | 15:49 |
| dansmith | mriedem: I see it as a variant of hard not soft | 15:50 |
| mriedem | sure, it's in between | 15:50 |
| dansmith | mriedem: it's the same filter has hard but instead of ==0, it's <=$max | 15:50 |
| dansmith | you can't do this with a weigher is my point | 15:50 |
| johnthetubaguy | dansmith: +1 | 15:50 |
| dansmith | and that's what soft anti-affinity is | 15:50 |
| mriedem | a weigher is not being proposed | 15:50 |
| dansmith | I know | 15:50 |
| *** dklyle has joined #openstack-nova | 15:50 | |
| bauwser | mriedem: soft anti-affinity should never hard-stop | 15:50 |
| bauwser | hard anti-affinity is something we already have | 15:51 |
| mriedem | umm | 15:51 |
| dansmith | mriedem: I'm saying this is proposing changing our hard filter from ==0 to <$max | 15:51 |
| mriedem | so flacid anti affinity as a new policy? | 15:51 |
| bauwser | what the proposer asks is a way to have a mechanism that would describe a 'max" limit | 15:51 |
| mriedem | yes i understand | 15:51 |
| bauwser | dansmith: and I don't like that | 15:51 |
| bauwser | because it would confuse a lot of people | 15:52 |
| dansmith | bauwser: neither do I, I think that's a confusing model | 15:52 |
| dansmith | I would rather a policy of min-redundancy, with a value of the minimum | 15:52 |
| mriedem | so your issue is creating a group with the soft policy but then having this attribute which makes it more hard than soft, | 15:52 |
| bauwser | my point is, instead of trying to hack our current model, rather try to describe a better model | 15:52 |
| mriedem | so it's a conception issue | 15:52 |
| bauwser | exactly | 15:52 |
| dansmith | mriedem: but it wouldn't be that | 15:52 |
| mriedem | so a new policy | 15:52 |
| dansmith | mriedem: it would be hard with a grace limit | 15:52 |
| bauwser | sometimes people want to hard-fail on a filter, but sometimes they prefer to soft-fail | 15:53 |
| dansmith | bauwser: I'm not sure what that has to do here, | 15:53 |
| dansmith | because this is still hard fail on this filter, it's just at a nonzero level of duplication | 15:53 |
| *** Spazmotic has joined #openstack-nova | 15:53 | |
| dansmith | if you go over that limit, which is currently zero, you hard fail all the time | 15:53 |
| mriedem | ok, so create a server group with the anti-affinity policy (hard) with some limit on overlap | 15:53 |
| dansmith | with what is proposed here | 15:53 |
| *** david-lyle has quit IRC | 15:53 | |
| *** Spazmotic is now known as Guest33405 | 15:53 | |
| dansmith | mriedem: that's what is being proposed here | 15:54 |
| mriedem | no it's not, it's the opposite | 15:54 |
| mriedem | the spec is talking about the soft policy | 15:54 |
| mriedem | you're saying soft policy is bad optics, | 15:54 |
| mriedem | so flip it to be hard policy, | 15:54 |
| mriedem | with the overlap limit | 15:54 |
| mriedem | either way we get to the same thing | 15:54 |
| dansmith | mriedem: but the spec is wrong in how things work | 15:54 |
| *** slaweq has quit IRC | 15:54 | |
| leakypipes | did someone say flaccid affinity? | 15:54 |
| * leakypipes jumps in | 15:54 | |
| mriedem | leakypipes: yes, word of the day | 15:54 |
| dansmith | mriedem: they think they're adding a limit to soft, but you can't do that | 15:54 |
| dansmith | you can increase the limit on hard though | 15:55 |
| mriedem | dansmith: b/c soft is enforced via weigher | 15:55 |
| mriedem | sure, | 15:55 |
| dansmith | mriedem: exactly | 15:55 |
| mriedem | so yeah let's make that point in the spec | 15:55 |
| *** zhaochao has quit IRC | 15:55 | |
| bauwser | I'm just commenting it | 15:55 |
| *** slaweq has joined #openstack-nova | 15:55 | |
| bauwser | dansmith: are you proposing to have a max limit defined on the filter ? I don't like that | 15:55 |
| * dansmith is writing this on the spec | 15:56 | |
| dansmith | bauwser: no, I'm saying that is what they are proposing here | 15:56 |
| jroll | (disclaimer, haven't read the spec) so, use case: I want to be able to say "I want 20 instances spread across at least 5 hosts, but preferably across more" - I think that's what is proposed here (based on this conversation), is that correct? | 15:56 |
| dansmith | they just don't know it | 15:56 |
| dansmith | jroll: no, that's what I was saying though | 15:56 |
| dansmith | that's what I would prefer I mean | 15:56 |
| jroll | dansmith: and this is what the spec thinks it is proposing, but it's really not? :) | 15:56 |
| bauwser | jroll: that's exactly what weighers aim to achieve | 15:56 |
| dansmith | jroll: no, that's not what the spec thinks its proposing | 15:57 |
| bauwser | for that usecase | 15:57 |
| mriedem | jroll: dansmith: well one thing is when you create the group, you don't say how many instances are going to be in it | 15:57 |
| bauwser | ie. ponder your list of hosts based on some criterias that are given by weighers | 15:57 |
| dansmith | jroll: it's proposing that you know how big the group will ever be, and set the allowed overlap per host to some number so you get the desired amount of minimum redundancy | 15:57 |
| mriedem | so "I want 20 instances spread across at least 5 hosts" is a bit chicken and egg to me | 15:57 |
| dansmith | which is much harder to reason about | 15:57 |
| *** slaweq_ has joined #openstack-nova | 15:57 | |
| *** slaweq_ has quit IRC | 15:57 | |
| jroll | dansmith: ah, ok, will read again | 15:57 |
| mriedem | but i guess if you already know when you create the group how many VMs are going to be in it, then that works | 15:57 |
| dansmith | mriedem: that's what this spec requires, is you to know how many instances will be in there | 15:58 |
| *** slaweq_ has joined #openstack-nova | 15:58 | |
| dansmith | mriedem: but, if you say "I want minimum redundancy of three" then you don't get any overlap until you boot the fourth and now you can start to double up because you've achieved that minimum level | 15:58 |
| *** slaweq_ has quit IRC | 15:59 | |
| *** slaweq_ has joined #openstack-nova | 16:00 | |
| leakypipes | dansmith: I think you saw the use case section begin with "As a NFV user, ..." and immediately threw up in your mouth ;) | 16:00 |
| *** slaweq_ has quit IRC | 16:00 | |
| bauwser | dansmith: it's very different | 16:00 |
| bauwser | dansmith: but I agree, describing for anti-affinity how many groups you want is somehow reasonable | 16:00 |
| jroll | leakypipes: there's vomit everywhere in here from that | 16:00 |
| leakypipes | :) | 16:00 |
| *** slaweq_ has joined #openstack-nova | 16:00 | |
| dansmith | bauwser: how many groups? you lost me there :) | 16:01 |
| mriedem | i think he means group of VMs per host in the group? | 16:01 |
| bauwser | man | 16:01 |
| bauwser | mriedem: bonus point | 16:01 |
| dansmith | oh min-redundancy you mean? | 16:01 |
| jroll | dansmith: I read through it, I agree with what you're suggesting there, thanks | 16:01 |
| dansmith | I think that's how most people will want to define a thing like this | 16:01 |
| dansmith | and jroll does, so.. end of story | 16:02 |
| * jroll likes the thought of "best effort" affinity that's halfway between hard and soft | 16:02 | |
| bauwser | call it failure zone | 16:02 |
| * leakypipes thinks this reminds him of something.... OH THAT'S RIGHT... HEAT. | 16:02 | |
| bauwser | but if we call it failure zone, then leakypipes would kill me | 16:02 |
| bauwser | and I'm scared | 16:02 |
| leakypipes | bauwser: indeed I would. | 16:02 |
| * leakypipes readies giant block of cheese to throw at bauwser | 16:03 | |
| bauwser | why people are doing instance groups besides the idea that they expect groups of failure domains | 16:03 |
| bauwser | ? | 16:03 |
| bauwser | leakypipes: you don't imagine my mood now... restating https://www.thelocal.fr/20180223/french-cheese-wars-are-the-days-of-the-real-normandy-camembert-numbered | 16:03 |
| leakypipes | bauwser: because... N. F. V. | 16:03 |
| leakypipes | hahaha | 16:04 |
| * bauwser runs | 16:04 | |
| leakypipes | that's awesome. | 16:04 |
| leakypipes | only in La France | 16:04 |
| *** jaosorior has quit IRC | 16:05 | |
| *** slaweq_ has quit IRC | 16:05 | |
| leakypipes | mriedem: did you say that https://review.openstack.org/#/c/546925/2/specs/rocky/approved/allow-specifying-limit-for-affrinity-group.rst was somehow related to something sgordon put together? | 16:08 |
| mriedem | see the bp | 16:09 |
| mriedem | https://blueprints.launchpad.net/nova/+spec/complex-soft-anti-affinity-policies | 16:09 |
| leakypipes | oh... 113355 | 16:09 |
| leakypipes | ok, not 113355 | 16:10 |
| mriedem | https://review.openstack.org/#/c/546925/ | 16:10 |
| mriedem | https://review.openstack.org/#/c/224325/ is the old backlog spec | 16:10 |
| leakypipes | mriedem: yeah, was confused since the spec is called something different than the blueprint. | 16:10 |
| *** mgoddard_ has quit IRC | 16:11 | |
| leakypipes | mriedem: ftr, k8s does this very well. | 16:13 |
| leakypipes | mriedem: because, you know, it's an *orchestrator*. | 16:14 |
| leakypipes | mriedem: would be much better, IMHO, to recommend to your group to install k8s on the nova compute hosts in the NFVi and just have at it. | 16:14 |
| dansmith | leakypipes: do we provide heat enough control to do this externally? | 16:14 |
| jroll | k8s does this well because the orchestrator and the scheduler can work together, imo | 16:15 |
| leakypipes | dansmith: initially, yes. but k8s will do it on an ongoing basis -- i.e. if the state of the pods on a node changes, k8s will re-evaluate the tolerances and scheduling conditions and place pods on different nodes accordingly. | 16:15 |
| leakypipes | jroll: actually, they don't. | 16:16 |
| dansmith | right, sure | 16:16 |
| dansmith | I guess I'm not sure how heat would do this with our api | 16:16 |
| *** yamamoto_ has quit IRC | 16:18 | |
| leakypipes | jroll: what actually happens in k8s is the descheduler (an orchestrator piece, not part of the scheduler) will evict pods from nodes that no longer meet scheduling constraints and the scheduler will try to re-place those evicted pods somewhere else (or the same node if conditions happen to be right at that specific moment) | 16:19 |
| leakypipes | jroll: the descheduler was originally called the rescheduler, but there's actually a rescheduler thing in k8s that does a different type of eviction, and that rescheduler piece is being deprecated. | 16:19 |
| jroll | leakypipes: sure, but the scheduler is aware of the orchestrator's requirements, right? or does it randomly choose and the descheduler continues to evict until things are right? | 16:20 |
| leakypipes | jroll: the reason the k8s works well is specifically because there is no coupling between the orchestrator and the resource tracker/scheduler. | 16:20 |
| leakypipes | jroll: scheduler is not aware of descheduler, no. | 16:21 |
| *** danpawlik has quit IRC | 16:21 | |
| jroll | leakypipes: I probably need to do some reading to fully understand | 16:21 |
| leakypipes | jroll: descheduler is basically just a daemon that runs a periodic loop that checks to see if the constraints that were met on initial placement are still met. and if not, simply evicts the pod. that puts the pod into a "ready for scheduling" state and the scheduler, on its next loop, will scheduler that pod again. | 16:22 |
| *** yamamoto has joined #openstack-nova | 16:22 | |
| openstackgerrit | Claudiu Belu proposed openstack/nova master: hyper-v: autospec classes before they are instantiated https://review.openstack.org/342211 | 16:23 |
| jroll | curious how heat would accomplish this affinity model, though. does heat know about compute nodes, which instances are on them, the affinity characteristics of said compute nodes (to know if two instances are actually in a separate power domain, for example?) | 16:23 |
| leakypipes | incidentally, this is precisely how I proposed that spot instances be handled in Nova instead of a highly-coupled-to-the-scheduler architecture that the guys at Inria proposed. | 16:23 |
| *** vks1 has quit IRC | 16:23 | |
| *** jpena|brb is now known as jpena | 16:23 | |
| jroll | leakypipes: is the scheduler aware of the constraints, like affinity, that need to be met? (and therefore that the descheduler is trying to meet?) | 16:23 |
| leakypipes | jroll: sure. the constraints are simply part of the pod scheduling policy that is associated with the pod when it's originally created. | 16:25 |
| bauwser | okay, so let's pretend we have "affinity knobs" :p | 16:25 |
| bauwser | leakypipes: I guess you know that some other openstack projects are trying to achieve the descheduler thing in OpenStack right? | 16:26 |
| bauwser | the reconciliation of resource usage IIUC what the k8s descheduler is trying to achieve | 16:26 |
| jroll | leakypipes: cool, so does nova expose sufficient data (instances, instance groups, affinity policies, compute node information, etc) for an external thing like heat to accomplish things like this spec proposes? | 16:26 |
| leakypipes | jroll: yes. | 16:26 |
| leakypipes | jroll: for the most part :) | 16:27 |
| jroll | heh | 16:27 |
| leakypipes | jroll: but yes. | 16:27 |
| * jroll is suspect about that, for a non-admin user | 16:27 | |
| * bauwser is about to call it a day | 16:27 | |
| leakypipes | jroll: heat is admin user, no? | 16:27 |
| bauwser | leakypipes: heat can have trusted users | 16:27 |
| bauwser | I mean trusts | 16:27 |
| jroll | leakypipes: I've no clue, honestly | 16:27 |
| bauwser | it's user delegation | 16:28 |
| bauwser | but anyway | 16:28 |
| leakypipes | bauwser: and trusts don't work with federated keystone, so... | 16:28 |
| bauwser | it's 5pm here and I need to pack my stuff :) | 16:28 |
| leakypipes | :) | 16:28 |
| dansmith | leakypipes: so how would heat go about doing this? | 16:28 |
| bauwser | so, see you folks in a pub or somewhere else | 16:28 |
| bauwser | like in a stadium | 16:28 |
| dansmith | leakypipes: it could boot the first three instances in an anti-affinity group and then after that, the fourth and later go anywhere? | 16:28 |
| jroll | leakypipes: but, this means that an openstack user needs to rely on their cloud provider to provide heat to be able to do this, rather than being able to write a little program to diy it | 16:28 |
| *** bauwser is now known as bauzas | 16:28 | |
| jroll | which might be fine but kinda sucks | 16:29 |
| leakypipes | dansmith: are you asking about heat or k8s? | 16:29 |
| dansmith | jroll: does it? I kinda expected people would be able to use heat themselves with just their creds | 16:29 |
| dansmith | leakypipes: heat and nova | 16:29 |
| leakypipes | jroll: "a little program"? :) | 16:29 |
| jroll | :P | 16:29 |
| jroll | dansmith: not if you need admin rights to nova to get compute node information | 16:29 |
| dansmith | jroll: you don't need that, and even if you had it I don't think it would help | 16:30 |
| leakypipes | dansmith: no.. if heat specifies the same instance group for all 5 instances, Nova guarantees they will go on different compute hosts... | 16:30 |
| dansmith | jroll: users can tell compute nodes apart, just not the actual hostnames | 16:30 |
| leakypipes | dansmith: but you already knew that, so I suspect I am missing yoru point. | 16:30 |
| dansmith | leakypipes: right, but how would heat implement this "no more than three instances per host" requirement with nova? | 16:31 |
| jroll | dansmith: fair point, I guess I'm thinking ahead to affinity around power/network domains, sorry | 16:31 |
| dansmith | jroll: yeah, but unless you let heat create aggregates, I still don't think being admin is helpful | 16:31 |
| dansmith | jroll: and each tenant creating aggregates via heat would be kinda crazy I think | 16:31 |
| dansmith | maybe just using them with custom flavors? I dunno | 16:32 |
| jroll | dansmith: do regular users have access to aggregate info? | 16:32 |
| dansmith | I'm legit asking because I don't know much about heat | 16:32 |
| leakypipes | dansmith: how would it implement it in a transactional/atomic way? it couldn't. but it could certainly call a compute API to get the compute node -> instance association for all instances in a group and do the calculation itself, no?' | 16:32 |
| dansmith | if this is doable outside of nova with heat, then I'm super -2 on the spec, I just don't know that it's possible | 16:32 |
| jroll | I don't either, so I hope you aren't asking me :) | 16:32 |
| dansmith | jroll: no, but that's what I'm saying, I don't know that you'd give your heat admin user ability to do much with them either | 16:32 |
| dansmith | leakypipes: but it can't control placement of those things | 16:33 |
| *** openstackgerrit has quit IRC | 16:33 | |
| leakypipes | dansmith: hmm, true nuf | 16:34 |
| leakypipes | dansmith: yeah, you're right. it would need to be a part of the request spec sent to placement/scheduler :( | 16:34 |
| jroll | dansmith: I would think you'd need enough info to be able to map instances -> aggregates, decide if the spread is sufficient (>3 power domains or whatever), and I was going to say whatever jay is thinking from there :) | 16:34 |
| dansmith | that's why I'm saying I expect this is a primitive we need to provide so heat can provide the automatic kill-one-spawn-one rebalancing sort of behavior | 16:34 |
| dansmith | jroll: well, you still can't control where a new instance lands enough to effect this policy I think | 16:35 |
| jroll | just boot and delete instances until it looks right :P | 16:35 |
| dansmith | jroll: monte carlo scheduling? I'm in | 16:35 |
| jroll | dansmith: right, I was hoping jay was going to solve that problem as I typed | 16:35 |
| *** claudiub|2 has joined #openstack-nova | 16:35 | |
| jroll | :P | 16:35 |
| leakypipes | dansmith: there will be a point (already reached?) where this stuff is just too expensive to try and calculate for each scheduling request and makes the interface between scheduler and placement overly cumbersome. | 16:36 |
| dansmith | leakypipes: I don't think this has anything to do with what scheduler asks of placement | 16:37 |
| dansmith | leakypipes: what is described in the spec is a simple change to the current hard affinity filter | 16:37 |
| dansmith | and what I was proposing would be similar but with some different math | 16:37 |
| dansmith | the biggest change is "oh now we're going to have extra specs on instance groups" | 16:37 |
| dansmith | which I think is a very big step we should not take lightly | 16:38 |
| dansmith | (or maybe at all) | 16:38 |
| *** claudiub has quit IRC | 16:38 | |
| dansmith | especially my point about changing that value on an existing group, because people will expect that if they change that from 3 to 2, existing instances get moved around | 16:38 |
| dansmith | and that ain't hap'nan | 16:38 |
| *** gbarros has quit IRC | 16:42 | |
| leakypipes | dansmith, jroll: if you want some mind-bending reading from k8s on this subject: https://github.com/kubernetes/kubernetes/pull/18265/files | 16:48 |
| leakypipes | dansmith, jroll: note that k8s affinity stuff is still in beta... | 16:48 |
| *** slaweq_ has joined #openstack-nova | 16:48 | |
| jroll | oh my | 16:48 |
| * jroll puts tab away for later | 16:48 | |
| mriedem | dansmith: i think the 'extra spec' term in his spec is a mistake, or just a WIP thing, | 16:49 |
| mriedem | i don't think we should add a random bag of extra specs to instance groups either | 16:49 |
| mriedem | dansmith: i also said updating the max-per-host value on an existing group isn't going to happen | 16:50 |
| mriedem | for the same reasons we don't allow changing policy on an existing group, or changing membership | 16:50 |
| leakypipes | what about an awkward bag of flaccid specs? | 16:51 |
| mriedem | if it's as simple as putting an attribute on a group in a hard affinity policy that says, instead of allowing no more than 1 vm from this group per host, you can allow up to 3 (or whatever), then that seems to fit the bill | 16:51 |
| mriedem | leakypipes: they have pills for that | 16:51 |
| leakypipes | lol | 16:51 |
| mriedem | let me get you frank thomas' number | 16:51 |
| *** sahid has quit IRC | 16:51 | |
| mriedem | actually, jimmy johnson sells them too and he already lives in FL | 16:52 |
| *** udesale__ has joined #openstack-nova | 16:52 | |
| *** udesale_ has quit IRC | 16:52 | |
| mriedem | sorry i got distracted, was hastily throwing together a PBC... | 16:52 |
| cfriesen | dansmith: we originally had metadata in instance groups, but it got pulled out due to not really having any users | 16:52 |
| mriedem | cfriesen: yeah that was linked into the spec and was something i didn't even know existed | 16:52 |
| mriedem | metadata in general makes our lives terrible | 16:53 |
| mriedem | like aggregate meta, and flavor extra specs | 16:53 |
| mriedem | i realize it's use though | 16:53 |
| cfriesen | dansmith: it'd be possible to prohibit changing the value on an existing group to something that would result in the current spread being invalid....alternately you could just allow that and document that it'll only affect the *next* scheduling decision. | 16:54 |
| *** gbarros has joined #openstack-nova | 16:54 | |
| mriedem | cfriesen: to determine if the new requested value would invalidate things would mean running through the scheduler all over again | 16:54 |
| mriedem | and you could still get it wrong | 16:54 |
| cfriesen | mriedem: there has been that recurring spec for allowing instances to be added/subtracted from a group | 16:54 |
| mriedem | which is why we have the late affinity check on the compute hosts | 16:54 |
| mriedem | cfriesen: i remember powervc pushing it back in kilo but that's been abandoned for a long time | 16:55 |
| cfriesen | arguably we've got races all over with instance group affinity, so what's one more. :) | 16:55 |
| dansmith | cfriesen: yeah then the group is in violation of policy with no way of getting it out, unless you do your own shuffling | 16:55 |
| *** slaweq_ has quit IRC | 16:56 | |
| dansmith | mriedem: and yeah I'd rather a real attribute, but I also think we're just going to end up with unlimited attributes for other things like this | 16:56 |
| dansmith | mriedem: so I dunno.. it's a can of worms | 16:56 |
| *** yamamoto has quit IRC | 16:57 | |
| mriedem | idk, if this is the first time somtehing like this has come up in the last what 5 years? | 16:57 |
| mriedem | it would get annoying if, over time, we had a bunch of attributes on groups that only applied to specific policies | 16:58 |
| cfriesen | for what it's worth, internally we added a "best-effort" flag to the "hard" affinity/antiaffinity policies to allow us to migrate instances off a compute node for maintenance | 16:58 |
| mriedem | cfriesen: so soft affinity? | 16:58 |
| cfriesen | no, you can turn the best-effort flag on and off dynamically | 16:58 |
| * dansmith goes to dinner | 16:59 | |
| cfriesen | so you'd normally run with it strict, but if you need to take down a compute node you can set best-effort, move everything, then turn it back off | 16:59 |
| *** mgoddard_ has joined #openstack-nova | 16:59 | |
| cfriesen | otherwise if you've got hard-affinity you can't migrate any of them | 16:59 |
| mriedem | which is why people want a force flag on evacuate, live migrate (and now cold migrate) | 17:00 |
| leakypipes | mriedem: and the late affinity check is (the only?) remaining upcall from a cell to API, no? | 17:00 |
| mriedem | leakypipes: hells no | 17:00 |
| leakypipes | mriedem: it's not an upcall? | 17:00 |
| mriedem | https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls | 17:00 |
| mriedem | it is an upcall | 17:00 |
| mriedem | but it's not the only one | 17:00 |
| leakypipes | oh, not the only reminaing.. | 17:00 |
| mriedem | we've got aggregates too | 17:00 |
| mriedem | so aggregates and affinity are the remaining upcall issues | 17:01 |
| mriedem | but right now there are 2 each | 17:01 |
| mriedem | so 4 upcall issues | 17:01 |
| *** yamamoto_ has joined #openstack-nova | 17:01 | |
| leakypipes | mriedem: then you'll LOVE my aggregate affinity spec! :) now with moar AFFINITY and moar AGGREGATES! | 17:01 |
| *** damien_r has quit IRC | 17:01 | |
| mriedem | leakypipes: i already said 'upcall upcall upcall' on that spec several times :) | 17:01 |
| leakypipes | I know :) | 17:01 |
| mriedem | i think i also hedged with something like, 'but we already do this in a few other places so people already have to rely on it, so maybe another log on the fire doesn't kill us' | 17:02 |
| cfriesen | On a totally different topic...has anyone ever heard of nova allocating duplicate network interfaces? (So the user boots while asking for 2 network interfaces, and nova allocates two ports on each network.) | 17:02 |
| mriedem | yes | 17:03 |
| mriedem | that's old news | 17:03 |
| mriedem | tempest has a test for it also i think | 17:03 |
| *** chyka has joined #openstack-nova | 17:03 | |
| mriedem | been around since juno? | 17:03 |
| cfriesen | I mean nova is allocating twice as many as were asked for. | 17:05 |
| mriedem | double your pleasure | 17:05 |
| mriedem | idk, going to lunch | 17:05 |
| *** fragatina has joined #openstack-nova | 17:06 | |
| *** ttsiouts has quit IRC | 17:06 | |
| *** lucasagomes is now known as lucas-afk | 17:10 | |
| *** gyee has joined #openstack-nova | 17:10 | |
| *** yamamoto_ has quit IRC | 17:12 | |
| *** efried has joined #openstack-nova | 17:13 | |
| efried | Greetings from JFK airport | 17:14 |
| *** jpena is now known as jpena|off | 17:15 | |
| *** dave-mccowan has quit IRC | 17:15 | |
| *** dave-mccowan has joined #openstack-nova | 17:16 | |
| *** dklyle has quit IRC | 17:17 | |
| *** wolverineav has joined #openstack-nova | 17:17 | |
| *** edmondsw has joined #openstack-nova | 17:18 | |
| *** sree_ has joined #openstack-nova | 17:21 | |
| *** sree_ is now known as Guest93643 | 17:22 | |
| *** jafeha has quit IRC | 17:24 | |
| *** belmoreira has quit IRC | 17:25 | |
| *** Guest93643 has quit IRC | 17:25 | |
| * mnaser grumbles at grendae | 17:25 | |
| *** sridharg has quit IRC | 17:26 | |
| *** yamamoto has joined #openstack-nova | 17:27 | |
| mnaser | ok | 17:27 |
| mnaser | im convinced grenade is broken | 17:27 |
| mnaser | for stable/pike | 17:27 |
| efried | Isn't that mnaser guy known for being johnny-on-the-spot for grenade fixes? | 17:28 |
| mnaser | only when i have to :( | 17:28 |
| mnaser | Host 'ubuntu-xenial-rax-dfw-0002683360' is not mapped to any cell | 17:28 |
| mnaser | we keep getting this in multinode | 17:28 |
| *** fragatina has quit IRC | 17:28 | |
| efried | It would seem odd that cell discovery isn't being run. Like, nothing would ever work. | 17:29 |
| efried | And that's pretty much the only thing I know about cells. | 17:30 |
| *** slaweq_ has joined #openstack-nova | 17:30 | |
| mnaser | efried: indeed seems to be the case. looks like there was a change to add 'CELLSV2_SETUP=singleconductor' in there, so not sure if that might have affected it | 17:30 |
| efried | mnaser: You're going to have something of a hard time finding a core today, but I'll +1 your fix :) | 17:31 |
| mnaser | efried: looks like there's only 3 cores for grenade too.. | 17:31 |
| *** hongbin has joined #openstack-nova | 17:32 | |
| *** yamamoto has quit IRC | 17:32 | |
| efried | Looks like qa-release is included by inheritance. | 17:32 |
| efried | So seven | 17:33 |
| mnaser | ok looks like this runs => nova-manage cell_v2 simple_cell_setup --transport-url rabbit://stackrabbit:secretrabbit@10.209.130.218:5672/ | 17:35 |
| mnaser | but discover_hosts is never called | 17:35 |
| efried | And simple_cell_setup doesn't run discovery itself? | 17:36 |
| efried | I remember having to fix this around pike timeframe. | 17:36 |
| mnaser | efried: going through the code it looks like it does call _map_cell_and_hosts() | 17:37 |
| *** amoralej is now known as amoralej|off | 17:37 | |
| *** swamireddy has quit IRC | 17:40 | |
| *** yamamoto has joined #openstack-nova | 17:42 | |
| *** yamamoto has quit IRC | 17:42 | |
| *** mlavalle has joined #openstack-nova | 17:44 | |
| *** danpawlik has joined #openstack-nova | 17:45 | |
| *** yamahata has quit IRC | 17:47 | |
| mnaser | sigh https://bugs.launchpad.net/grenade/+bug/1708039 looks like it was 'supposed' to be fixed | 17:47 |
| openstack | Launchpad bug 1708039 in devstack "gate-grenade-dsvm-neutron-multinode-ubuntu-xenial fails with "No host-to-cell mapping found for selected host"" [Medium,Fix released] - Assigned to Sean Dague (sdague) | 17:47 |
| *** udesale_ has joined #openstack-nova | 17:47 | |
| *** lpetrut has joined #openstack-nova | 17:48 | |
| mnaser | it looks like it regressed | 17:49 |
| mnaser | and its all stable/pike hits | 17:49 |
| *** mlavalle has quit IRC | 17:49 | |
| *** udesale__ has quit IRC | 17:50 | |
| *** danpawlik has quit IRC | 17:51 | |
| *** tidwellr has quit IRC | 17:53 | |
| mnaser | "Didn't find service registered by hostname after 60 seconds" .. found it | 17:54 |
| mnaser | it's actually listed but the bash for some reason doesnt find it | 17:54 |
| efried | leakypipes: You around today? | 17:55 |
| efried | Hoho, it's Friday | 17:55 |
| *** efried is now known as fried_rice_jfk | 17:56 | |
| andreaf | mriedem hey I'm setting up a zuul-v3 multinode job, and everything works fine apart from nova that gives me "Host is not mapped to any cell" http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/ara/result/525a60bd-ac22-4fc4-9db7-e61fce8ac1f5/ | 17:56 |
| andreaf | mriedem: I compared configs in localrc and nova and I don't see anything obvious - do you have any idea about what this could be? | 17:56 |
| leakypipes | fried_rice_jfk: ues | 17:56 |
| leakypipes | yes | 17:56 |
| fried_rice_jfk | leakypipes: It looks like alex_xu may be right. I wrote a gabbit for it. | 17:57 |
| mnaser | andreaf: im actually looking into this right ow | 17:57 |
| mnaser | im seeing this issue in stable/pike | 17:57 |
| *** fragatina has joined #openstack-nova | 17:58 | |
| andreaf | mnaser oh ok at least it's not just me :P | 17:58 |
| *** dtantsur is now known as dtantsur|afk | 17:58 | |
| mnaser | andreaf: i'm seeing the devstack start waiting for compute to go up at "2018-02-23 04:24:33.575" (with a 60s timeout) and the compute record get created at "2018-02-23 04:25:44.182" | 17:59 |
| mnaser | with a 60 second timeout, it means that devstack gives up at 04:25:33, but the compute record actually gets created 11 seconds later | 17:59 |
| * mnaser investigates moar | 18:00 | |
| fried_rice_jfk | leakypipes: Trying to figure out why. Is the order in which I construct my query supposed to not matter? | 18:01 |
| *** hongbin has quit IRC | 18:01 | |
| *** derekh has quit IRC | 18:03 | |
| fried_rice_jfk | leakypipes: Maybe query.select_from() overwrites any previous .select_from()? | 18:03 |
| mnaser | so it takes 42 seconds to go from "Connecting to libvirt: qemu:///system _get_new_connection /opt/stack/old/nova/nova/virt/libvirt/host.py:366" => "Registering for lifecycle events" | 18:04 |
| mnaser | and those 42 seconds are enough for devstack to timeout waiting for compute | 18:04 |
| mnaser | so somehow "wrapped_conn = self._connect(self._uri, self._read_only)" takes 42 seconds here https://github.com/openstack/nova/blob/stable/pike/nova/virt/libvirt/host.py#L368 | 18:05 |
| *** rm_work has quit IRC | 18:06 | |
| *** dansmith has quit IRC | 18:07 | |
| mnaser | libvirtd starts at "2018-02-23 04:23:59.741" from devstack | 18:07 |
| *** StevenK has quit IRC | 18:07 | |
| *** homeski has quit IRC | 18:07 | |
| fried_rice_jfk | leakypipes: Gah, my apologies; I was misreading my test failure. It's fine, it's cumulative as we expected. | 18:07 |
| *** openstackgerrit has joined #openstack-nova | 18:08 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: rp: GET /resource_providers?required=<traits> https://review.openstack.org/546837 | 18:08 |
| leakypipes | fried_rice_jfk: k | 18:08 |
| fried_rice_jfk | leakypipes, alex_xu: Added a gabbit to prove ANDness with resources ^ | 18:08 |
| *** mgoddard_ has quit IRC | 18:11 | |
| *** gjayavelu has joined #openstack-nova | 18:14 | |
| *** salv-orl_ has quit IRC | 18:18 | |
| mnaser | who's the person to bug for libvirt related questions | 18:18 |
| *** salv-orlando has joined #openstack-nova | 18:18 | |
| mnaser | i'm seeing 1116 "device-list-properties" on service start | 18:19 |
| *** tesseract has quit IRC | 18:19 | |
| *** tidwellr has joined #openstack-nova | 18:19 | |
| *** yamahata has joined #openstack-nova | 18:20 | |
| *** tssurya has quit IRC | 18:20 | |
| figleaf | fried_rice_jfk: looks good! | 18:22 |
| fried_rice_jfk | Thanks figleaf | 18:22 |
| *** salv-orlando has quit IRC | 18:22 | |
| *** bnemec has joined #openstack-nova | 18:23 | |
| *** r-daneel has quit IRC | 18:24 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094 | 18:25 |
| *** fragatina has quit IRC | 18:27 | |
| *** fragatina has joined #openstack-nova | 18:28 | |
| *** bnemec is now known as bnemec-pto | 18:30 | |
| *** lbragstad has quit IRC | 18:31 | |
| *** esberglu has quit IRC | 18:31 | |
| *** pcaruana has quit IRC | 18:33 | |
| *** dansmith has joined #openstack-nova | 18:33 | |
| *** dansmith is now known as Guest17026 | 18:33 | |
| *** danpawlik has joined #openstack-nova | 18:33 | |
| *** udesale_ has quit IRC | 18:36 | |
| *** rajinir_ has joined #openstack-nova | 18:37 | |
| *** xyang_ has joined #openstack-nova | 18:37 | |
| *** vdrok_ has joined #openstack-nova | 18:37 | |
| *** hogepodge_ has joined #openstack-nova | 18:37 | |
| *** mwhahaha_ has joined #openstack-nova | 18:37 | |
| *** coreycb_ has joined #openstack-nova | 18:37 | |
| *** khappone_ has joined #openstack-nova | 18:37 | |
| *** gus_ has joined #openstack-nova | 18:38 | |
| *** andreykurilin_ has joined #openstack-nova | 18:43 | |
| *** yamamoto has joined #openstack-nova | 18:43 | |
| *** logan_ has joined #openstack-nova | 18:43 | |
| *** McNinja_ has joined #openstack-nova | 18:43 | |
| *** mgoddard_ has joined #openstack-nova | 18:44 | |
| *** johnthetubaguy_ has joined #openstack-nova | 18:44 | |
| *** khappone has quit IRC | 18:44 | |
| *** vdrok has quit IRC | 18:44 | |
| *** McNinja has quit IRC | 18:44 | |
| *** mwhahaha has quit IRC | 18:44 | |
| *** rajinir has quit IRC | 18:44 | |
| *** mgoddard has quit IRC | 18:44 | |
| *** logan- has quit IRC | 18:44 | |
| *** andreykurilin has quit IRC | 18:44 | |
| *** coreycb has quit IRC | 18:44 | |
| *** hogepodge has quit IRC | 18:44 | |
| *** xyang has quit IRC | 18:44 | |
| *** gus has quit IRC | 18:44 | |
| *** johnthetubaguy has quit IRC | 18:44 | |
| *** vdrok_ is now known as vdrok | 18:44 | |
| *** rajinir_ is now known as rajinir | 18:44 | |
| *** mwhahaha_ is now known as mwhahaha | 18:44 | |
| *** mgoddard_ is now known as mgoddard | 18:44 | |
| *** xyang_ is now known as xyang | 18:44 | |
| *** hogepodge_ is now known as hogepodge | 18:44 | |
| *** coreycb_ is now known as coreycb | 18:45 | |
| *** danpawlik has quit IRC | 18:47 | |
| *** logan_ is now known as logan- | 18:47 | |
| *** wolverineav has quit IRC | 18:49 | |
| *** yamamoto has quit IRC | 18:49 | |
| *** wolverineav has joined #openstack-nova | 18:49 | |
| *** rm_work has joined #openstack-nova | 18:51 | |
| *** rm_work has quit IRC | 18:51 | |
| *** rm_work has joined #openstack-nova | 18:51 | |
| *** wolverineav has quit IRC | 18:54 | |
| * fried_rice_jfk waves. See y'all in Dublin! | 18:56 | |
| *** lbragstad has joined #openstack-nova | 18:57 | |
| *** fried_rice_jfk has quit IRC | 18:57 | |
| *** artom has quit IRC | 19:03 | |
| *** mgoddard_ has joined #openstack-nova | 19:04 | |
| *** mchlumsky has quit IRC | 19:06 | |
| *** mchlumsky has joined #openstack-nova | 19:07 | |
| *** pchavva has quit IRC | 19:14 | |
| mnaser | o/ | 19:17 |
| *** andreas_s has quit IRC | 19:18 | |
| *** salv-orlando has joined #openstack-nova | 19:19 | |
| mriedem | andreaf: mnaser: maybe on a slow test node, | 19:20 |
| mriedem | devstack times out waiting for the compute host to get mapped to the cell, which won't happen until after the compute node record gets auto-created when the nova-compute service starts up | 19:20 |
| mnaser | mriedem: exactly, but libvirt takes 42 seconds to start up and seems to be doing a ton of commands before starting up | 19:21 |
| mnaser | sorry, not 42 | 19:21 |
| mnaser | it takes 1m22s | 19:21 |
| mnaser | i wonder if we should add wait_for_libvirt :\ | 19:22 |
| mnaser | seeing a tooon of "Send command '{"execute":"device-list-properties","arguments":{"typename":"kvm-pci-assign"},"id":"libvirt-19"}' for write with FD -1" | 19:22 |
| *** itlinux has joined #openstack-nova | 19:22 | |
| mriedem | not sure why libvirt would take a long time to startup in pike and not ocata...or is it the old side of grenade failing, meaning it's ocata? | 19:22 |
| mnaser | mriedem: old side is failing | 19:23 |
| mriedem | ok so that's ocata | 19:23 |
| mriedem | but not sure why anything would have changed there wrt libvirt | 19:23 |
| *** salv-orlando has quit IRC | 19:23 | |
| mnaser | looks like uca published libvirt-bin_2.5.0-3ubuntu5.6~cloud2_amd64.deb on 9th of february | 19:25 |
| mnaser | prior to that libvirt-bin_2.5.0-3ubuntu5.6~cloud1_amd64.deb was released 23 jan 2018 | 19:26 |
| mnaser | and before that.. nothing in 2.x series | 19:26 |
| mriedem | ocata and pike are both using the ocata UCA https://github.com/openstack-dev/devstack/blob/stable/ocata/tools/fixup_stuff.sh#L84 | 19:27 |
| mriedem | https://github.com/openstack-dev/devstack/blob/stable/pike/tools/fixup_stuff.sh#L87 | 19:27 |
| mnaser | mriedem: interesting, the issue doesnt seem to be there with the master which uses 3.6.0 | 19:29 |
| mnaser | mriedem: i dunno, i feel like bumping that timeout to 120s, but that might mask regressions with nova taking too long to register for whatever reason | 19:31 |
| mnaser | i guess i can write a wait_for_libvirt | 19:31 |
| mriedem | i'd be ok with i think bumping the timeout on pike/ocata | 19:33 |
| *** mgoddard_ has quit IRC | 19:33 | |
| mriedem | we know we have slower nodes now, | 19:34 |
| mriedem | and if there were some changes to libvirt on stable that makes it take longer to start in the ocata UCA, we can workaround it with a slower timeout; ocata is only around for a few more weeks anyway | 19:34 |
| *** itlinux has quit IRC | 19:35 | |
| mriedem | looks like some recent cve fixes went into the libvirt packages | 19:36 |
| mriedem | so that's probably the cause | 19:36 |
| mriedem | https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/ocata-staging/+sourcepub/8774271/+listing-archive-extra | 19:37 |
| mriedem | SECURITY UPDATE: Add support for Spectre mitigations | 19:38 |
| mriedem | so there is your slowdown | 19:38 |
| *** chergd has joined #openstack-nova | 19:38 | |
| mriedem | looks like the pike package has some security fixes but not that same one https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/pike-staging/+sourcepub/8806001/+listing-archive-extra | 19:39 |
| *** danpawlik has joined #openstack-nova | 19:40 | |
| *** itlinux has joined #openstack-nova | 19:44 | |
| *** yamamoto has joined #openstack-nova | 19:47 | |
| *** danpawlik has quit IRC | 19:48 | |
| *** itlinux has quit IRC | 19:50 | |
| *** burt has joined #openstack-nova | 19:51 | |
| *** moshele has joined #openstack-nova | 19:52 | |
| *** pchavva has joined #openstack-nova | 19:52 | |
| *** yamamoto has quit IRC | 19:52 | |
| *** awaugama has quit IRC | 19:55 | |
| *** sree__ has joined #openstack-nova | 19:57 | |
| openstackgerrit | Logan V proposed openstack/nova stable/pike: Allow os_interface and os_region_name on Keystone reqs https://review.openstack.org/547654 | 19:59 |
| *** sree__ has quit IRC | 20:02 | |
| *** figleaf is now known as edleafe | 20:08 | |
| *** lpetrut has quit IRC | 20:12 | |
| *** wolverineav has joined #openstack-nova | 20:17 | |
| *** itlinux has joined #openstack-nova | 20:17 | |
| *** salv-orlando has joined #openstack-nova | 20:19 | |
| *** moshele has quit IRC | 20:21 | |
| *** wolverineav has quit IRC | 20:22 | |
| *** saphi_ has quit IRC | 20:23 | |
| *** salv-orlando has quit IRC | 20:24 | |
| *** weshay is now known as weshay_bbin30 | 20:25 | |
| *** david-lyle has joined #openstack-nova | 20:25 | |
| *** weshay_bbin30 is now known as weshay | 20:28 | |
| *** hongbin has joined #openstack-nova | 20:28 | |
| *** READ10 has joined #openstack-nova | 20:30 | |
| *** tssurya has joined #openstack-nova | 20:31 | |
| *** pramodrj07 has joined #openstack-nova | 20:34 | |
| *** itlinux has quit IRC | 20:36 | |
| *** itlinux has joined #openstack-nova | 20:42 | |
| andreaf | mriedem thanks for the follow up | 20:43 |
| *** danpawlik has joined #openstack-nova | 20:43 | |
| andreaf | mriedem I see that same issue on a new multinode job I'm working on, which runs on master | 20:44 |
| andreaf | mriedem two runs twice the same issue, but maybe I've been unlucky with slow nodes... | 20:44 |
| *** sree__ has joined #openstack-nova | 20:45 | |
| *** yamamoto has joined #openstack-nova | 20:48 | |
| *** sree__ has quit IRC | 20:49 | |
| *** slaweq_ has quit IRC | 20:50 | |
| *** slaweq_ has joined #openstack-nova | 20:51 | |
| *** wolverineav has joined #openstack-nova | 20:52 | |
| *** yamamoto has quit IRC | 20:53 | |
| *** slaweq_ has quit IRC | 20:55 | |
| *** wolverineav has quit IRC | 20:57 | |
| mriedem | i see a neutron-grenade-multinode job of mnaser's fail on pike, and it's running on rax-dfw nodes which i've always seen to be slows since the spectre patches http://logs.openstack.org/19/546219/3/check/neutron-grenade-multinode/ae7d292/zuul-info/inventory.yaml | 20:57 |
| mriedem | *slow | 20:57 |
| *** amodi has quit IRC | 20:58 | |
| mnaser | im writing a short wait_for_libvirt | 20:58 |
| mriedem | we could just hack it on pike and ocata for now and set https://github.com/openstack-dev/devstack/blob/stable/pike/lib/nova#L966 to 120 | 20:59 |
| *** r-daneel has joined #openstack-nova | 21:00 | |
| mriedem | and site https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/ocata-staging/+sourcepub/8774271/+listing-archive-extra | 21:00 |
| mnaser | mriedem: i can do that too, probably easier and quicker | 21:01 |
| mriedem | *cite ? | 21:01 |
| mriedem | mnaser: it would be easier and quicker for now yeah, | 21:01 |
| mriedem | could get mtreinish to take a gander | 21:01 |
| mriedem | alternatively we could run the pike UCA on stable/pike... | 21:01 |
| andreaf | mriedem mnaser there's a patch up to make that value configurable already | 21:02 |
| mriedem | https://review.openstack.org/#/c/536798/ | 21:02 |
| *** pramodrj07 has quit IRC | 21:02 | |
| andreaf | https://review.openstack.org/#/c/547431/ | 21:02 |
| andreaf | mriedem mnaser ^^^ | 21:02 |
| mnaser | mriedem: grenade uses stable/ocata though | 21:02 |
| mriedem | mnaser: yeah the old side does | 21:02 |
| mriedem | true | 21:03 |
| mnaser | andreaf: thats a nice patch, what does the gate use for SERVICE_TIMEOUT ? | 21:03 |
| mriedem | it defaults to 60 | 21:03 |
| mnaser | so i guess we use that patch+backport+bump up SERVICE_TIMEOUT in stable/ocata + stable/pike ? | 21:04 |
| *** weshay is now known as weshay_bbiab | 21:04 | |
| *** tbachman has quit IRC | 21:04 | |
| mriedem | i guess...i'm not sure how much people are going to want to bump the default timeout | 21:04 |
| mriedem | globally i mean | 21:04 |
| *** wolverineav has joined #openstack-nova | 21:05 | |
| mriedem | that seems worse than just hard-coding the wait_for_compute timeout to 120 | 21:05 |
| mnaser | yeah i think i'd want to bump up wait_for_compute because then everything will wait 120s which might be hiding regressions | 21:05 |
| mriedem | so i suggest a new stackrc variable that defaults to $SERVICE_TIMEOUT, but that we can bump to be higher in stable/ocata/pike if we want | 21:07 |
| mriedem | noted that in frickler's patch | 21:07 |
| *** amodi has joined #openstack-nova | 21:10 | |
| *** pchavva has quit IRC | 21:11 | |
| *** Guest98116 has quit IRC | 21:12 | |
| *** janki has quit IRC | 21:14 | |
| andreaf | mriedem this is the log line in devstack where there is some sign of the issue going on I think http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/devstacklog.txt.gz#_2018-02-23_17_22_21_709 | 21:14 |
| andreaf | mriedem but I few lines later it seems to be fine | 21:15 |
| mnaser | mriedem, andreaf, mtreinish: https://review.openstack.org/547431 Create NOVA_COMPUTE_SERVICE_TIMEOUT in is_nova_ready function | 21:15 |
| mriedem | that's normal | 21:15 |
| mriedem | andreaf: n-cpu barfs that the compute node that represents the local host doesn't exist, and then it creates it | 21:15 |
| mnaser | if that can be approved, i can backport it and then bump it in stable/ocata and stable/pike | 21:15 |
| mriedem | and stops the barfing | 21:15 |
| andreaf | mriedem so I don't know why I get http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/job-output.txt.gz#_2018-02-23_17_22_40_367118 on every single VM created in that job | 21:16 |
| *** itlinux has quit IRC | 21:16 | |
| mriedem | is it every one or just the subnode? | 21:18 |
| mriedem | ubuntu-xenial-inap-mtl01-0002692907 is the subnode | 21:18 |
| andreaf | mriedem actually every single VM on the 907 node | 21:19 |
| mriedem | andreaf: so this is the warning you get on first startup | 21:19 |
| mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/screen-n-cpu.txt.gz?level=INFO#_Feb_23_17_22_20_954537 | 21:19 |
| mriedem | No compute node record found for host ubuntu-xenial-inap-mtl01-0002692907. If this is the first time this service is starting on this host, then you can ignore this warning.: ComputeHostNotFound_Remote: Compute host ubuntu-xenial-inap-mtl01-0002692907 could not be found. | 21:19 |
| mriedem | then the compute node record is created | 21:20 |
| mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/screen-n-cpu.txt.gz?level=INFO#_Feb_23_17_22_21_021412 | 21:20 |
| mriedem | Feb 23 17:22:21.021412 ubuntu-xenial-inap-mtl01-0002692907 nova-compute[21947]: INFO nova.compute.resource_tracker [None req-5ceeacfd-f9f6-4b19-9044-6fd13c9adadc None None] Compute node record created for ubuntu-xenial-inap-mtl01-0002692907:ubuntu-xenial-inap-mtl01-0002692907 with uuid: 85868727-6173-4345-bd2e-a92bfaf1de8a | 21:20 |
| openstackgerrit | Matthew Edmonds proposed openstack/nova master: Fix N358 hacking check https://review.openstack.org/547670 | 21:20 |
| *** salv-orlando has joined #openstack-nova | 21:20 | |
| mriedem | on that subnode, wait_for_compute starts here: | 21:21 |
| mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/compute1/logs/devstacklog.txt.gz#_2018-02-23_17_22_18_880 | 21:21 |
| mriedem | 2018-02-23 17:22:18.880 | + lib/nova:is_nova_ready:1003 : wait_for_compute 60 | 21:21 |
| mriedem | and ends here | 21:21 |
| mriedem | 2018-02-23 17:22:21.607 | + functions:wait_for_compute:463 : return 0 | 21:21 |
| andreaf | so it's successful | 21:21 |
| mriedem | so in that case, it looks like the compute node record was created before devstack started waiting | 21:21 |
| *** salv-orlando has quit IRC | 21:22 | |
| mnaser | mriedem: round 2 https://review.openstack.org/#/c/547431/ | 21:22 |
| andreaf | right... so I'm missing what's going wrong... | 21:23 |
| mnaser | its friday and the servers are lazy | 21:23 |
| mriedem | the host mapping for the n-cpu on the controller node is created here: | 21:23 |
| mriedem | http://logs.openstack.org/24/545724/9/check/tempest-multinode-full/1bbec81/controller/logs/devstacklog.txt.gz#_2018-02-23_17_13_29_978 | 21:23 |
| mriedem | 2018-02-23 17:13:29.978 | Creating host mapping for compute host 'ubuntu-xenial-inap-mtl01-0002692906': adbcb697-55fc-4750-9242-b13aa75cab07 | 21:23 |
| mriedem | discover_hosts doesn't run on the subnode | 21:24 |
| mriedem | and i don't see discover_hosts running after the subnode compute is created to discover it | 21:24 |
| *** salv-orlando has joined #openstack-nova | 21:25 | |
| mnaser | oh that's why its different | 21:25 |
| mriedem | andreaf: you'd likely need to compare to an existing multinode job and see where discover_hosts runs, | 21:25 |
| mnaser | grenade uses simple_cell_setup which does discover_hosts | 21:25 |
| mriedem | because discover_hosts *has* to be run every time you startup a new n-cpu | 21:25 |
| andreaf | mriedem yeah that's what I was about to do :) | 21:25 |
| andreaf | should it run on the controller? | 21:26 |
| andreaf | discover_host is a devstack function right? | 21:26 |
| mnaser | andreaf: discover_hosts is a nova-manage cli func | 21:26 |
| mnaser | nova-manage cell_v2 discover_hosts | 21:26 |
| mnaser | it runs on the controller after you add new computes | 21:27 |
| mriedem | andreaf: OH I KNOW THE PROBLEM!!! | 21:27 |
| mriedem | your problem is zuulv3 | 21:27 |
| mriedem | :) | 21:27 |
| andreaf | heh | 21:27 |
| mnaser | i thought that was the solution for everything | 21:27 |
| mriedem | devstack-gate would run discover_hosts on the primary after the subnode was setup | 21:27 |
| mriedem | the devstack multinode guide says this too | 21:27 |
| mriedem | i'll look for the d-g code | 21:27 |
| andreaf | mriedem oh, right | 21:28 |
| andreaf | mriedem I missed that in d-g | 21:28 |
| mriedem | https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L783 | 21:28 |
| mriedem | andreaf: ^ | 21:28 |
| mriedem | d-g would run discover_hosts after the subnode was setup | 21:28 |
| mriedem | so you'll need an ansible task to run discover_hosts on the primary node once the subnode is setup | 21:28 |
| mriedem | that should do the trick | 21:28 |
| andreaf | mriedem heh the one line I did not port to ansible! | 21:29 |
| andreaf | mriedem I guess I kind of assumed cells did not matter for the base job :( but it's cells v2 | 21:29 |
| mriedem | mnaser: need to fix the commit message | 21:29 |
| mriedem | cells v2 is king | 21:29 |
| mnaser | oops | 21:30 |
| andreaf | mriedem cool thanks a lot, much appreciated!! | 21:30 |
| mriedem | yw | 21:30 |
| andreaf | mriedem # NOTE(mriedem): We want to remove this if/when nova supports auto-registration of computes with cells, but that's not happening in Ocata. | 21:31 |
| mnaser | ok fixed the commit msg, andreaf i'd borrow your +2 back please https://review.openstack.org/#/c/547431/ :) | 21:31 |
| andreaf | mriedem I guess it's not happening in pike or queens either :D | 21:31 |
| mnaser | s/Ocata/a while/ | 21:31 |
| andreaf | mnaser +2 | 21:31 |
| mnaser | thank you | 21:32 |
| *** NostawRm has quit IRC | 21:32 | |
| *** NostawRm has joined #openstack-nova | 21:32 | |
| mriedem | andreaf: right | 21:33 |
| mriedem | andreaf: also we don't run discover_hosts from the subnode because it's not configured to have access to the nova_api db | 21:34 |
| mriedem | on purpose | 21:34 |
| *** dave-mcc_ has joined #openstack-nova | 21:34 | |
| mriedem | we want to make sure that when people like leakypipes try to add code to nova-compute that hits the API it blows up :P | 21:34 |
| *** dave-mccowan has quit IRC | 21:36 | |
| *** edmondsw has quit IRC | 21:40 | |
| leakypipes | mriedem: ha. ha. | 21:47 |
| *** mriedem has quit IRC | 21:49 | |
| *** yamamoto has joined #openstack-nova | 21:50 | |
| cfriesen | if we tell nova to boot an instance (specifying that we want interfaces on two networks), then the instance fails to start, so we reschedule to another node, would we expect to re-use the ports that were created for the first compute node? | 21:52 |
| *** yamamoto has quit IRC | 21:55 | |
| *** hongbin has quit IRC | 21:56 | |
| *** hongbin has joined #openstack-nova | 21:57 | |
| cfriesen | answering my own question, based on a comment in ComputeManager._build_networks_for_instance() the answer seems to be "yes" | 21:58 |
| cfriesen | or at least "maybe" | 21:59 |
| *** danpawlik has quit IRC | 22:00 | |
| *** danpawlik has joined #openstack-nova | 22:01 | |
| *** danpawlik has quit IRC | 22:02 | |
| *** danpawlik has joined #openstack-nova | 22:02 | |
| *** tbachman has joined #openstack-nova | 22:03 | |
| *** slaweq has quit IRC | 22:09 | |
| *** danpawlik has quit IRC | 22:10 | |
| *** StevenK has joined #openstack-nova | 22:15 | |
| *** FL1SK has quit IRC | 22:17 | |
| *** weshay_bbiab is now known as weshay | 22:18 | |
| *** yamamoto has joined #openstack-nova | 22:18 | |
| *** hongbin has quit IRC | 22:20 | |
| *** salv-orlando has quit IRC | 22:21 | |
| *** lyan has quit IRC | 22:21 | |
| *** salv-orlando has joined #openstack-nova | 22:21 | |
| *** salv-orlando has quit IRC | 22:25 | |
| *** tbachman has quit IRC | 22:25 | |
| *** tssurya has quit IRC | 22:27 | |
| *** masber has joined #openstack-nova | 22:27 | |
| *** masber has quit IRC | 22:30 | |
| *** masber has joined #openstack-nova | 22:30 | |
| *** dave-mcc_ has quit IRC | 22:34 | |
| *** salv-orlando has joined #openstack-nova | 22:43 | |
| *** dave-mccowan has joined #openstack-nova | 22:46 | |
| *** FL1SK has joined #openstack-nova | 22:52 | |
| *** amodi has quit IRC | 22:58 | |
| *** yamamoto has quit IRC | 23:00 | |
| *** burt has quit IRC | 23:05 | |
| *** masuberu has joined #openstack-nova | 23:07 | |
| *** masber has quit IRC | 23:09 | |
| *** dave-mccowan has quit IRC | 23:11 | |
| *** mchlumsky has quit IRC | 23:12 | |
| *** yamamoto has joined #openstack-nova | 23:13 | |
| *** yamamoto has quit IRC | 23:14 | |
| *** FL1SK has quit IRC | 23:20 | |
| *** itlinux has joined #openstack-nova | 23:22 | |
| *** swamireddy has joined #openstack-nova | 23:23 | |
| *** r-daneel has quit IRC | 23:27 | |
| *** itlinux has quit IRC | 23:28 | |
| *** slaweq has joined #openstack-nova | 23:30 | |
| *** vladikr has quit IRC | 23:30 | |
| *** tbachman has joined #openstack-nova | 23:33 | |
| *** slaweq has quit IRC | 23:35 | |
| *** tbachman_ has joined #openstack-nova | 23:36 | |
| *** tbachman has quit IRC | 23:37 | |
| *** tbachman_ is now known as tbachman | 23:37 | |
| *** salv-orlando has quit IRC | 23:39 | |
| *** salv-orlando has joined #openstack-nova | 23:39 | |
| *** pengdake_ has joined #openstack-nova | 23:39 | |
| *** salv-orlando has quit IRC | 23:44 | |
| *** pengdake_ has quit IRC | 23:44 | |
| *** tbachman has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!