| *** suresh12 has joined #openstack-nova | 00:00 | |
| *** germs has joined #openstack-nova | 00:01 | |
| *** germs has quit IRC | 00:01 | |
| *** germs has joined #openstack-nova | 00:01 | |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_extra_specs_delete db api method https://review.openstack.org/539702 | 00:04 |
|---|---|---|
| openstackgerrit | Merged openstack/nova master: Report client: Remove version discovery comment https://review.openstack.org/554253 | 00:04 |
| openstackgerrit | Merged openstack/nova master: Remove version/date from CLI documentation https://review.openstack.org/553903 | 00:05 |
| *** germs has quit IRC | 00:05 | |
| *** pooja has joined #openstack-nova | 00:07 | |
| *** suresh12 has quit IRC | 00:11 | |
| *** ircuser-1 has joined #openstack-nova | 00:13 | |
| pooja | Hi.. I am seeing an issue with NumInstancesFilter in nova scheduler (Newton release) when provisioning multiple instances in parallel (not in one batch api call) | 00:13 |
| pooja | The scheduler's view of host isn't updated and so multiple instances get placed on a host, which exceeds the max_instances value set for that host. | 00:14 |
| pooja | Is this a known issue and is there a solution for it in ocata/pike release? | 00:14 |
| pooja | Appreciate any pointers or change links. Thanks! | 00:15 |
| melwitt | pooja: are you running a single scheduler? there was a change in pike to do resource claims in the scheduler via placement. I'm not yet familiar with the "max_instances" value you mentioned though | 00:16 |
| melwitt | okay, so it's a config option | 00:16 |
| pooja | melwitt: Yes, I'm running a single instance of nova-scheduler | 00:21 |
| pooja | This is the filter I'm referring to - https://github.com/openstack/nova/blob/master/nova/scheduler/filters/num_instances_filter.py#L28 | 00:21 |
| *** itlinux has joined #openstack-nova | 00:23 | |
| melwitt | I found it too, currently looking through the code. it looks like the problem you described should be fixed as of pike with the claims in the scheduler. let me see if I can find a patch related to that specific area | 00:23 |
| pooja | Great! thanks for your help looking into it, melwitt! | 00:24 |
| Spaz-Home | Morning | 00:25 |
| *** yamamoto has quit IRC | 00:26 | |
| *** gjayavelu has quit IRC | 00:27 | |
| *** yamamoto has joined #openstack-nova | 00:28 | |
| *** yamamoto has quit IRC | 00:28 | |
| *** liverpooler has joined #openstack-nova | 00:30 | |
| melwitt | pooja: to be honest, I'm not sure if the issue is fixed as of the new code. to be sure, it would be better to ask someone like bauzas or edleafe. here's a link to where I started tracing, if that might help in the meantime https://github.com/openstack/nova/blob/master/nova/scheduler/host_manager.py#L283-L284 | 00:30 |
| melwitt | and the consume_from_request is used in filter_scheduler.py, looking at it more in filter_scheduler.py, it does seem like it would not be resilient to the issue of parallel requests | 00:33 |
| melwitt | it looks like the report of "num_instances" comes from a compute node stat report, which may not be updating in real-time | 00:35 |
| *** gjayavelu has joined #openstack-nova | 00:35 | |
| *** Dinesh_Bhor has joined #openstack-nova | 00:37 | |
| *** hongbin has joined #openstack-nova | 00:40 | |
| *** jichen has joined #openstack-nova | 00:41 | |
| *** amodi has quit IRC | 00:44 | |
| *** suresh12 has joined #openstack-nova | 00:44 | |
| *** elmaciej_ has quit IRC | 00:46 | |
| *** lifeless_ has joined #openstack-nova | 00:46 | |
| *** lifeless has quit IRC | 00:46 | |
| *** elmaciej has joined #openstack-nova | 00:46 | |
| *** gjayavelu has quit IRC | 00:48 | |
| pooja | melwitt: Sure, let me check with bauzas or edleafe too. | 00:48 |
| *** suresh12 has quit IRC | 00:49 | |
| pooja | Yes, the problem is that stats get updated asynchronously and num_instances value used by scheduler would be inaccurate based on that timing. | 00:49 |
| melwitt | pooja: I see. I think I understand now, and based on that, it's probably still a problem in the current code now | 00:50 |
| pooja | Oh okay.. should I file a bug for it? | 00:51 |
| *** itlinux has quit IRC | 00:51 | |
| pooja | Do these filters work the same way with the new Placement API? | 00:51 |
| melwitt | some do, some don't. the [Core|Ram|Disk]Filter became obsolete in the filter scheduler once we started calling placement. because we pre-filter based on answers from placement and the scheduler claims with placement along the way | 00:52 |
| melwitt | but the rest of the filters run after the placement call, as they did before | 00:54 |
| *** tbachman has joined #openstack-nova | 00:54 | |
| *** wolverineav has quit IRC | 00:54 | |
| *** phuongnh has joined #openstack-nova | 00:54 | |
| *** hiro-kobayashi has joined #openstack-nova | 00:55 | |
| melwitt | again, bauzas, edleafe, and co are the people to chat with about that | 00:56 |
| melwitt | if you want to open a bug, I can point them to it and ask them to comment on it. it's up to you, however you want to do it | 00:56 |
| *** wolverineav has joined #openstack-nova | 00:57 | |
| *** Dinesh_Bhor has quit IRC | 00:57 | |
| *** Dinesh_Bhor has joined #openstack-nova | 00:57 | |
| *** odyssey4me has quit IRC | 00:59 | |
| *** odyssey4me has joined #openstack-nova | 00:59 | |
| *** Dinesh_Bhor has quit IRC | 00:59 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:02 | |
| *** wolverineav has quit IRC | 01:03 | |
| *** wolverineav has joined #openstack-nova | 01:04 | |
| *** elmaciej has quit IRC | 01:07 | |
| *** yamamoto has joined #openstack-nova | 01:07 | |
| pooja | Sounds good! I will connect with them and see if I need to file a bug for this. Will let you know if I do that. Thanks again! | 01:09 |
| melwitt | cool, thanks pooja | 01:09 |
| *** wxy has joined #openstack-nova | 01:10 | |
| *** wolverineav has quit IRC | 01:11 | |
| *** Kevin_Zheng has joined #openstack-nova | 01:14 | |
| *** gjayavelu has joined #openstack-nova | 01:15 | |
| *** tiendc has joined #openstack-nova | 01:15 | |
| *** gjayavelu has quit IRC | 01:17 | |
| *** suresh12 has joined #openstack-nova | 01:19 | |
| *** suresh12 has quit IRC | 01:23 | |
| *** yamamoto has quit IRC | 01:27 | |
| *** Dinesh_Bhor has quit IRC | 01:49 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:52 | |
| *** Dinesh_Bhor has quit IRC | 01:52 | |
| *** suresh12 has joined #openstack-nova | 01:54 | |
| *** annp has joined #openstack-nova | 01:56 | |
| *** suresh12 has quit IRC | 01:58 | |
| *** suresh12 has joined #openstack-nova | 01:59 | |
| openstackgerrit | Merged openstack/nova master: VMware: fix TypeError while get console log https://review.openstack.org/549182 | 01:59 |
| *** artom_ has joined #openstack-nova | 02:01 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:01 | |
| *** dikonoor has joined #openstack-nova | 02:02 | |
| *** Dinesh_Bhor has quit IRC | 02:02 | |
| *** germs has joined #openstack-nova | 02:02 | |
| *** artom has quit IRC | 02:03 | |
| *** germs has quit IRC | 02:06 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:08 | |
| *** wolverineav has joined #openstack-nova | 02:08 | |
| *** suresh12 has quit IRC | 02:10 | |
| *** dikonoor has quit IRC | 02:11 | |
| *** Nil_ has quit IRC | 02:12 | |
| *** fragatina has quit IRC | 02:15 | |
| *** chyka_ has joined #openstack-nova | 02:15 | |
| *** fragatina has joined #openstack-nova | 02:15 | |
| *** fragatin_ has joined #openstack-nova | 02:18 | |
| *** chyka has quit IRC | 02:18 | |
| *** fragatina has quit IRC | 02:19 | |
| *** yamahata has quit IRC | 02:21 | |
| *** fragatin_ has quit IRC | 02:22 | |
| *** chyka_ has quit IRC | 02:22 | |
| *** gongysh has joined #openstack-nova | 02:25 | |
| *** dave-mccowan has quit IRC | 02:31 | |
| *** hoangcx has joined #openstack-nova | 02:31 | |
| *** r-daneel has joined #openstack-nova | 02:32 | |
| *** r-daneel_ has joined #openstack-nova | 02:34 | |
| *** r-daneel has quit IRC | 02:36 | |
| *** r-daneel_ is now known as r-daneel | 02:36 | |
| *** suresh12 has joined #openstack-nova | 02:37 | |
| *** suresh12 has quit IRC | 02:39 | |
| *** suresh12 has joined #openstack-nova | 02:39 | |
| *** links has joined #openstack-nova | 02:40 | |
| *** itlinux has joined #openstack-nova | 02:42 | |
| *** suresh12 has quit IRC | 02:45 | |
| *** pooja has quit IRC | 02:46 | |
| *** salv-orl_ has joined #openstack-nova | 02:50 | |
| alex_xu_ | jaypipes: what do you think about the preferred_traits, that resovled the cyborg weigher problem | 02:52 |
| *** salv-orlando has quit IRC | 02:52 | |
| *** hoangcx has quit IRC | 02:55 | |
| *** wolverineav has quit IRC | 02:58 | |
| *** hoangcx has joined #openstack-nova | 03:00 | |
| *** psachin has joined #openstack-nova | 03:02 | |
| *** suresh12 has joined #openstack-nova | 03:02 | |
| *** wolverineav has joined #openstack-nova | 03:04 | |
| *** yingjun has joined #openstack-nova | 03:07 | |
| *** suresh12 has quit IRC | 03:07 | |
| *** bkopilov has quit IRC | 03:08 | |
| *** sree has joined #openstack-nova | 03:13 | |
| *** sree has quit IRC | 03:14 | |
| *** sree has joined #openstack-nova | 03:14 | |
| *** wolverineav has quit IRC | 03:18 | |
| *** wolverineav has joined #openstack-nova | 03:22 | |
| *** chyka has joined #openstack-nova | 03:23 | |
| *** yingjun has quit IRC | 03:27 | |
| *** chyka has quit IRC | 03:28 | |
| *** gongysh has quit IRC | 03:42 | |
| *** suresh12 has joined #openstack-nova | 03:46 | |
| *** itlinux has quit IRC | 03:49 | |
| *** yamamoto has joined #openstack-nova | 03:52 | |
| *** suresh12 has quit IRC | 03:56 | |
| *** hongbin has quit IRC | 04:00 | |
| *** yamamoto has quit IRC | 04:01 | |
| *** germs has joined #openstack-nova | 04:03 | |
| *** germs has quit IRC | 04:03 | |
| *** germs has joined #openstack-nova | 04:03 | |
| *** hiro-kobayashi has quit IRC | 04:04 | |
| *** yamamoto has joined #openstack-nova | 04:04 | |
| *** germs has quit IRC | 04:07 | |
| *** yamamoto has quit IRC | 04:08 | |
| *** yamamoto has joined #openstack-nova | 04:09 | |
| *** yamamoto has quit IRC | 04:14 | |
| *** fragatina has joined #openstack-nova | 04:14 | |
| *** wolverineav has quit IRC | 04:15 | |
| *** yamamoto has joined #openstack-nova | 04:16 | |
| *** fragatina has quit IRC | 04:16 | |
| *** fragatina has joined #openstack-nova | 04:16 | |
| *** suresh12 has joined #openstack-nova | 04:17 | |
| *** suresh12 has quit IRC | 04:22 | |
| *** yamamoto has quit IRC | 04:27 | |
| openstackgerrit | Merged openstack/nova master: Add placeholder migrations for Queens backports https://review.openstack.org/553831 | 04:28 |
| *** yamamoto has joined #openstack-nova | 04:28 | |
| *** vivsoni__ has joined #openstack-nova | 04:29 | |
| *** andreas_s has joined #openstack-nova | 04:29 | |
| *** andreas_s has quit IRC | 04:34 | |
| *** bkopilov has joined #openstack-nova | 04:41 | |
| *** wolverineav has joined #openstack-nova | 04:42 | |
| *** yamamoto has quit IRC | 04:43 | |
| *** suresh12 has joined #openstack-nova | 04:46 | |
| *** yamamoto has joined #openstack-nova | 04:46 | |
| *** vivsoni__ has quit IRC | 04:51 | |
| *** vivsoni__ has joined #openstack-nova | 04:53 | |
| *** Dinesh__Bhor has joined #openstack-nova | 04:58 | |
| *** Dinesh_Bhor has quit IRC | 04:58 | |
| *** wolverineav has quit IRC | 05:00 | |
| *** wolverineav has joined #openstack-nova | 05:01 | |
| *** lpetrut has joined #openstack-nova | 05:02 | |
| *** wolverineav has quit IRC | 05:05 | |
| *** ratailor has joined #openstack-nova | 05:06 | |
| *** imacdonn has quit IRC | 05:14 | |
| *** imacdonn has joined #openstack-nova | 05:14 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | 05:18 |
| *** gjayavelu has joined #openstack-nova | 05:18 | |
| *** jmlowe has quit IRC | 05:18 | |
| *** hiro-kobayashi has joined #openstack-nova | 05:19 | |
| *** Aditya_ has joined #openstack-nova | 05:21 | |
| *** vivsoni__ has quit IRC | 05:25 | |
| *** hoangcx has quit IRC | 05:26 | |
| openstackgerrit | Naichuan Sun proposed openstack/nova master: xenapi: Use XAPI pool instead of aggregate pool for shared SR migration https://review.openstack.org/554154 | 05:27 |
| *** suresh12 has quit IRC | 05:29 | |
| *** hoangcx has joined #openstack-nova | 05:29 | |
| *** Zames has joined #openstack-nova | 05:31 | |
| *** mdnadeem has joined #openstack-nova | 05:36 | |
| *** jmlowe has joined #openstack-nova | 05:37 | |
| *** sridharg has joined #openstack-nova | 05:38 | |
| *** Zames has quit IRC | 05:39 | |
| *** jmlowe has quit IRC | 05:42 | |
| *** yufei has joined #openstack-nova | 05:44 | |
| *** claudiub has joined #openstack-nova | 05:45 | |
| *** sidx64 has joined #openstack-nova | 05:51 | |
| *** sidx64 has quit IRC | 05:57 | |
| *** fragatina has quit IRC | 05:58 | |
| *** lpetrut has quit IRC | 06:00 | |
| openstackgerrit | Pranab proposed openstack/os-vif master: Add abstract OVSDB API https://review.openstack.org/476612 | 06:03 |
| *** sidx64 has joined #openstack-nova | 06:03 | |
| *** sidx64 has quit IRC | 06:04 | |
| *** germs has joined #openstack-nova | 06:04 | |
| *** dineshbhor__ has joined #openstack-nova | 06:07 | |
| *** germs has quit IRC | 06:08 | |
| *** OctopusZhang__ has joined #openstack-nova | 06:09 | |
| *** Dinesh__Bhor has quit IRC | 06:09 | |
| *** OctopusZhang__ has quit IRC | 06:09 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/548772 | 06:10 |
| *** sidx64 has joined #openstack-nova | 06:11 | |
| *** lpetrut has joined #openstack-nova | 06:12 | |
| *** masber has quit IRC | 06:12 | |
| *** yufei has quit IRC | 06:12 | |
| *** sidx64 has quit IRC | 06:13 | |
| openstackgerrit | Jianghua Wang proposed openstack/nova master: libvirt: Improve 'qemu-img convert' performance https://review.openstack.org/522067 | 06:15 |
| *** OctopusZhang__ has joined #openstack-nova | 06:15 | |
| *** OctopusZhang__ is now known as yufei | 06:15 | |
| *** Aditya_ has quit IRC | 06:18 | |
| *** trinaths has joined #openstack-nova | 06:21 | |
| *** Jack_Iv has joined #openstack-nova | 06:22 | |
| *** Jack_Iv has quit IRC | 06:26 | |
| *** psachin has quit IRC | 06:27 | |
| *** ccamacho has quit IRC | 06:29 | |
| *** yufei has quit IRC | 06:37 | |
| *** sidx64 has joined #openstack-nova | 06:38 | |
| *** gjayavelu has quit IRC | 06:43 | |
| *** sidx64 has quit IRC | 06:44 | |
| *** masber has joined #openstack-nova | 06:44 | |
| *** gaoyan has joined #openstack-nova | 06:46 | |
| *** psachin has joined #openstack-nova | 06:47 | |
| *** masber has quit IRC | 06:47 | |
| *** masber has joined #openstack-nova | 06:47 | |
| *** Eran_Kuris has quit IRC | 06:48 | |
| *** Eran_Kuris has joined #openstack-nova | 06:50 | |
| *** avolkov has joined #openstack-nova | 06:54 | |
| *** yufei has joined #openstack-nova | 06:56 | |
| *** lpetrut has quit IRC | 06:57 | |
| *** avolkov has quit IRC | 06:57 | |
| *** avolkov has joined #openstack-nova | 06:57 | |
| *** yufei has quit IRC | 06:58 | |
| *** chyka has joined #openstack-nova | 07:00 | |
| *** sidx64 has joined #openstack-nova | 07:00 | |
| *** gjayavelu has joined #openstack-nova | 07:00 | |
| *** dineshbhor__ has quit IRC | 07:00 | |
| *** sidx64 has quit IRC | 07:01 | |
| *** suresh12 has joined #openstack-nova | 07:02 | |
| *** tetsuro has joined #openstack-nova | 07:03 | |
| *** chyka has quit IRC | 07:04 | |
| *** suresh12 has quit IRC | 07:06 | |
| *** sar has joined #openstack-nova | 07:08 | |
| *** abhishekk has joined #openstack-nova | 07:09 | |
| *** gjayavelu has quit IRC | 07:12 | |
| *** sahid has joined #openstack-nova | 07:13 | |
| *** kholkina has joined #openstack-nova | 07:15 | |
| *** rcernin has quit IRC | 07:23 | |
| *** rcernin has joined #openstack-nova | 07:24 | |
| *** rcernin has quit IRC | 07:24 | |
| *** ccamacho has joined #openstack-nova | 07:25 | |
| *** alexchadin has joined #openstack-nova | 07:30 | |
| *** trinaths has quit IRC | 07:31 | |
| *** tovin07 has joined #openstack-nova | 07:31 | |
| *** salv-orl_ has quit IRC | 07:32 | |
| *** salv-orlando has joined #openstack-nova | 07:33 | |
| *** salv-orlando has quit IRC | 07:33 | |
| *** salv-orlando has joined #openstack-nova | 07:33 | |
| *** pcaruana has joined #openstack-nova | 07:34 | |
| *** alexchadin has quit IRC | 07:36 | |
| *** alexchadin has joined #openstack-nova | 07:37 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi disk resizing to privsep. https://review.openstack.org/552242 | 07:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi partition copies to privsep. https://review.openstack.org/553605 | 07:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move image conversion to privsep. https://review.openstack.org/554437 | 07:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | 07:45 |
| openstackgerrit | Michael Still proposed openstack/nova master: We don't need utils.trycmd any more. https://review.openstack.org/554439 | 07:45 |
| *** sidx64 has joined #openstack-nova | 07:52 | |
| *** lajoskatona has joined #openstack-nova | 07:52 | |
| *** sidx64 has quit IRC | 07:58 | |
| *** tesseract has joined #openstack-nova | 08:00 | |
| *** namnh has joined #openstack-nova | 08:01 | |
| *** andreas_s has joined #openstack-nova | 08:02 | |
| *** sidx64 has joined #openstack-nova | 08:03 | |
| *** AlexeyAbashkin has joined #openstack-nova | 08:04 | |
| *** germs has joined #openstack-nova | 08:05 | |
| *** germs has quit IRC | 08:05 | |
| *** germs has joined #openstack-nova | 08:05 | |
| *** sidx64 has quit IRC | 08:06 | |
| *** rmart04 has joined #openstack-nova | 08:07 | |
| *** sidx64 has joined #openstack-nova | 08:07 | |
| *** priteau has joined #openstack-nova | 08:08 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova-specs master: Allow abort live migrations in queued status https://review.openstack.org/536722 | 08:08 |
| *** germs has quit IRC | 08:08 | |
| *** damien_r has joined #openstack-nova | 08:10 | |
| *** _pewp_ has quit IRC | 08:11 | |
| *** jpena|off is now known as jpena | 08:11 | |
| *** _pewp_ has joined #openstack-nova | 08:15 | |
| *** Zames has joined #openstack-nova | 08:19 | |
| *** ragiman has joined #openstack-nova | 08:22 | |
| *** sidx64 has quit IRC | 08:22 | |
| openstackgerrit | sahid proposed openstack/nova master: Revert "[libvirt] Add _get_vcpu_realtime_scheduler()" https://review.openstack.org/554448 | 08:22 |
| openstackgerrit | sahid proposed openstack/nova master: Revert "[libvirt] Add _get_numa_memnode()" https://review.openstack.org/554449 | 08:22 |
| *** trinaths has joined #openstack-nova | 08:24 | |
| *** gaoyan has quit IRC | 08:28 | |
| *** _ix has quit IRC | 08:29 | |
| *** _ix has joined #openstack-nova | 08:29 | |
| *** afaranha has joined #openstack-nova | 08:29 | |
| *** Kevin_Zheng has quit IRC | 08:30 | |
| *** redondo-mk has quit IRC | 08:30 | |
| sahid | jaypipes: can you have a look at https://review.openstack.org/#/c/511188/ | 08:30 |
| openstackgerrit | licanwei proposed openstack/nova master: Make nova-manage capable of syncing all cell databases https://review.openstack.org/519275 | 08:30 |
| *** redondo-mk has joined #openstack-nova | 08:31 | |
| sahid | i mean it would be nice if you cn cut on it we can make some progress, basically you brought that idea and we never see you again :) | 08:31 |
| *** Kevin_Zheng has joined #openstack-nova | 08:31 | |
| *** moshele has joined #openstack-nova | 08:33 | |
| kaisers1 | efried: ping | 08:35 |
| openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run with privileges in Quobyte driver https://review.openstack.org/554195 | 08:37 |
| *** tetsuro has left #openstack-nova | 08:40 | |
| *** _pewp_ has quit IRC | 08:41 | |
| *** Zames has quit IRC | 08:41 | |
| *** amoralej|off is now known as amoralej | 08:45 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Move placement test cases from db to placement https://review.openstack.org/553149 | 08:46 |
| *** _pewp_ has joined #openstack-nova | 08:47 | |
| *** sidx64 has joined #openstack-nova | 08:49 | |
| *** lpetrut has joined #openstack-nova | 08:49 | |
| *** cdent has joined #openstack-nova | 08:50 | |
| *** lpetrut_ has joined #openstack-nova | 08:50 | |
| *** lpetrut has quit IRC | 08:50 | |
| fanzhang | hi, nova team. Can I add some specified parameter to control boot --min-count instances within one request on different hosts? Like forcing instances launched on different hosts? | 08:51 |
| fanzhang | I think there may be not functions like this, right? | 08:52 |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova-specs master: Complex (Anti)-Affinity Policies https://review.openstack.org/546925 | 08:54 |
| *** Zames has joined #openstack-nova | 08:54 | |
| *** sidx64 has quit IRC | 08:58 | |
| *** alexchadin has quit IRC | 08:58 | |
| openstackgerrit | jichenjc proposed openstack/nova master: Avoid raise InstanceNotFound exception https://review.openstack.org/541152 | 08:58 |
| *** Zames has quit IRC | 08:58 | |
| *** alexchadin has joined #openstack-nova | 08:59 | |
| *** lucas-pto is now known as lucasagomes | 09:00 | |
| *** ccamacho has quit IRC | 09:00 | |
| *** andreas_s has quit IRC | 09:01 | |
| *** andreas_s has joined #openstack-nova | 09:01 | |
| bauzas | good morning folks | 09:02 |
| * bauzas is released from the magic kingdom | 09:02 | |
| *** tssurya has joined #openstack-nova | 09:02 | |
| Kevin_Zheng | fanzhang, maybe you should try anti-affinity filter | 09:02 |
| fanzhang | Kevin_Zheng: thanks so much. Reading docs about anti-affinity filter now :) | 09:03 |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add PENDING vm state https://review.openstack.org/554212 | 09:05 |
| *** alexchadin has quit IRC | 09:09 | |
| *** andreas_s has quit IRC | 09:10 | |
| *** alexchadin has joined #openstack-nova | 09:10 | |
| *** andreas_s has joined #openstack-nova | 09:13 | |
| *** baffle has quit IRC | 09:13 | |
| Kevin_Zheng | fanzhang yw | 09:13 |
| *** baffle has joined #openstack-nova | 09:14 | |
| *** ccamacho has joined #openstack-nova | 09:18 | |
| *** andreas_s has quit IRC | 09:22 | |
| *** andreas_s has joined #openstack-nova | 09:22 | |
| *** sidx64 has joined #openstack-nova | 09:24 | |
| *** _pewp_ has quit IRC | 09:24 | |
| *** _pewp_ has joined #openstack-nova | 09:26 | |
| *** mdbooth has joined #openstack-nova | 09:26 | |
| *** _ix has quit IRC | 09:33 | |
| *** _pewp_ has quit IRC | 09:33 | |
| jianghuaw_ | bauzas, good morning:-) | 09:33 |
| bauzas | just catching up emails this morning | 09:34 |
| jianghuaw_ | hope you enjoyed the time in the magic kingdom | 09:34 |
| bauzas | well, my daughters did at least :) | 09:34 |
| jianghuaw_ | good enough:-) | 09:34 |
| *** ralonsoh has joined #openstack-nova | 09:35 | |
| *** abhishekk has quit IRC | 09:35 | |
| *** andreas_s has quit IRC | 09:36 | |
| *** andreas_s has joined #openstack-nova | 09:37 | |
| *** derekh has joined #openstack-nova | 09:40 | |
| *** _pewp_ has joined #openstack-nova | 09:40 | |
| *** _ix has joined #openstack-nova | 09:41 | |
| *** abhishekk has joined #openstack-nova | 09:48 | |
| *** josecastroleon has quit IRC | 09:49 | |
| *** josecastroleon has joined #openstack-nova | 09:50 | |
| *** hiro-kobayashi has quit IRC | 09:50 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Provide framework for setting placement error codes https://review.openstack.org/546177 | 09:53 |
| *** jichen has quit IRC | 09:53 | |
| *** liverpooler has quit IRC | 09:54 | |
| *** mgoddard has joined #openstack-nova | 09:54 | |
| *** andreas_s has quit IRC | 09:56 | |
| *** andreas_s has joined #openstack-nova | 09:57 | |
| gibi | morning nova | 10:02 |
| *** yamamoto has quit IRC | 10:04 | |
| *** alexchadin has quit IRC | 10:04 | |
| *** germs has joined #openstack-nova | 10:05 | |
| *** germs has quit IRC | 10:05 | |
| *** germs has joined #openstack-nova | 10:05 | |
| *** andreas_s has quit IRC | 10:07 | |
| *** andreas_s has joined #openstack-nova | 10:07 | |
| *** Zames has joined #openstack-nova | 10:08 | |
| *** yamamoto has joined #openstack-nova | 10:09 | |
| *** germs has quit IRC | 10:10 | |
| *** namnh has quit IRC | 10:10 | |
| *** chyka has joined #openstack-nova | 10:13 | |
| *** yamamoto has quit IRC | 10:14 | |
| *** Zames has quit IRC | 10:18 | |
| *** chyka has quit IRC | 10:19 | |
| *** Zames has joined #openstack-nova | 10:23 | |
| *** Zames has quit IRC | 10:25 | |
| *** sree has quit IRC | 10:28 | |
| openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run with privileges in Quobyte driver https://review.openstack.org/554195 | 10:29 |
| *** mvk has quit IRC | 10:31 | |
| *** ralonsoh_ has joined #openstack-nova | 10:33 | |
| *** sidx64 has quit IRC | 10:35 | |
| *** sambetts|afk is now known as sambetts | 10:36 | |
| *** ralonsoh has quit IRC | 10:37 | |
| *** jpena is now known as jpena|brb | 10:38 | |
| openstackgerrit | Pranab proposed openstack/os-vif master: Add native implementation OVSDB API https://review.openstack.org/482226 | 10:41 |
| gibi | melwitt: I left a comment and a question in https://etherpad.openstack.org/p/nova-runways-rocky with '[gibi]' prefix. But overall I'm OK with the proposal. | 10:41 |
| gibi | melwitt: I think we have to start doing it to gather real experience and then we can improve the process iteratively | 10:42 |
| *** masber has quit IRC | 10:43 | |
| *** elmaciej has joined #openstack-nova | 10:46 | |
| *** mvk has joined #openstack-nova | 10:46 | |
| *** tbachman has quit IRC | 10:49 | |
| *** sidx64 has joined #openstack-nova | 10:52 | |
| *** tiendc has quit IRC | 10:53 | |
| *** phuongnh has quit IRC | 10:56 | |
| *** vladikr has quit IRC | 11:00 | |
| *** sidx64 has quit IRC | 11:02 | |
| *** rcernin has joined #openstack-nova | 11:02 | |
| *** suresh12 has joined #openstack-nova | 11:02 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled field to CellMapping object https://review.openstack.org/550090 | 11:03 |
| *** ralonsoh__ has joined #openstack-nova | 11:03 | |
| Kevin_Zheng | gibi, Hi, I might need some suggestion on https://review.openstack.org/#/c/553288/ about tests | 11:05 |
| *** josecastroleon has quit IRC | 11:06 | |
| *** rcernin has quit IRC | 11:06 | |
| *** ralonsoh_ has quit IRC | 11:07 | |
| *** suresh12 has quit IRC | 11:07 | |
| *** masber has joined #openstack-nova | 11:08 | |
| *** yamamoto has joined #openstack-nova | 11:10 | |
| *** masuberu has joined #openstack-nova | 11:12 | |
| *** yamamoto has quit IRC | 11:12 | |
| *** yamamoto has joined #openstack-nova | 11:12 | |
| *** masber has quit IRC | 11:14 | |
| *** annp has quit IRC | 11:15 | |
| *** pcaruana has quit IRC | 11:23 | |
| *** psachin has quit IRC | 11:28 | |
| *** psachin has joined #openstack-nova | 11:34 | |
| *** chyka has joined #openstack-nova | 11:35 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled field to CellMapping object https://review.openstack.org/550090 | 11:37 |
| *** jpena|brb is now known as jpena | 11:37 | |
| *** sidx64 has joined #openstack-nova | 11:38 | |
| *** chyka has quit IRC | 11:39 | |
| *** yamamoto has quit IRC | 11:40 | |
| *** yamamoto has joined #openstack-nova | 11:41 | |
| *** abhishekk has quit IRC | 11:44 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add CellMappingList.get_all_enabled() query method https://review.openstack.org/550188 | 11:46 |
| *** vladikr has joined #openstack-nova | 11:47 | |
| *** pchavva has joined #openstack-nova | 11:52 | |
| *** sidx64 has quit IRC | 11:53 | |
| *** sidx64 has joined #openstack-nova | 11:54 | |
| *** pcaruana has joined #openstack-nova | 11:55 | |
| *** sidx64 has quit IRC | 11:57 | |
| *** sidx64 has joined #openstack-nova | 11:58 | |
| *** sidx64 has quit IRC | 11:59 | |
| *** amoralej is now known as amoralej|lunch | 12:04 | |
| *** liuzz has quit IRC | 12:04 | |
| *** sree has joined #openstack-nova | 12:04 | |
| *** liuzz has joined #openstack-nova | 12:04 | |
| *** sree_ has joined #openstack-nova | 12:05 | |
| *** sree_ is now known as Guest61245 | 12:05 | |
| *** lucasagomes is now known as lucas-hungry | 12:06 | |
| *** germs has joined #openstack-nova | 12:06 | |
| *** germs has quit IRC | 12:06 | |
| *** germs has joined #openstack-nova | 12:06 | |
| *** sidx64 has joined #openstack-nova | 12:07 | |
| *** sree has quit IRC | 12:08 | |
| *** jaosorior has quit IRC | 12:09 | |
| *** josecastroleon has joined #openstack-nova | 12:09 | |
| *** germs has quit IRC | 12:10 | |
| *** odyssey4me has quit IRC | 12:11 | |
| *** odyssey4me has joined #openstack-nova | 12:11 | |
| *** sidx64 has quit IRC | 12:11 | |
| gibi | Kevin_Zheng: I shortly checked your test, functionally it looks OK. Do you feel that adding the request_id everywhere in the test is too much? | 12:14 |
| *** artom_ has quit IRC | 12:15 | |
| sean-k-mooney[m] | bauzas: o/ i believe you reviewed this last cycle would you mind taking a look at the re-proposal of the nic feature based scheduling spec when you have time. https://review.openstack.org/#/c/545951/ | 12:16 |
| Kevin_Zheng | gibi, sort of, but I'm not be afraid to add them, just don't know whether we have a better way to do that, in the latest patchset, I did some nodification for test test_create_server_error, but somehow the req id is different, I will dig into it latter | 12:16 |
| *** READ10 has joined #openstack-nova | 12:16 | |
| Kevin_Zheng | gibi for those tests that tested multiple actions in one test, it seems a little bit complicated, so I wonder we have a better way to do it. | 12:17 |
| sean-k-mooney[m] | bauzas: i have 1 or two nits to adress in the corresponding code but should have the series rebased and uploaded by the end of the week. im hoping we can get this all merged before milestone 1 | 12:17 |
| *** edmondsw has joined #openstack-nova | 12:17 | |
| *** sidx64 has joined #openstack-nova | 12:18 | |
| openstackgerrit | sahid proposed openstack/nova master: only increment disk address unit for scsi devices https://review.openstack.org/538310 | 12:19 |
| *** trinaths has quit IRC | 12:20 | |
| *** sidx64 has quit IRC | 12:21 | |
| sean-k-mooney[m] | dansmith: just saw your comment on https://review.openstack.org/#/c/449257/59 you last comment on this topic was before i took this over from rodolfo. ill try and adress this when i do the rebase later this week. | 12:22 |
| openstackgerrit | Jan Zerebecki proposed openstack/os-vif stable/ocata: Check if interface belongs to a Linux Bridge before removing https://review.openstack.org/554523 | 12:22 |
| *** tbachman has joined #openstack-nova | 12:22 | |
| sean-k-mooney[m] | dansmith: looking at your old comment you would like us to add a spec_object field in addtion to the spec field correct? | 12:23 |
| *** jaosorior has joined #openstack-nova | 12:23 | |
| gibi | Kevin_Zheng: thanks for describing your concerns, I have to think about a bit. I will reply in the review | 12:23 |
| Kevin_Zheng | gibi thanks alot | 12:23 |
| openstackgerrit | Jan Zerebecki proposed openstack/os-vif stable/ocata: Check if interface belongs to a Linux Bridge before removing https://review.openstack.org/554523 | 12:24 |
| *** sidx64 has joined #openstack-nova | 12:27 | |
| efried | kaisers1: Howdy | 12:28 |
| efried | kaisers1: /me US Central time :) | 12:28 |
| openstackgerrit | sahid proposed openstack/nova master: libvirt: handle DiskNotFound during update_available_resource https://review.openstack.org/553067 | 12:29 |
| *** _ix has quit IRC | 12:31 | |
| *** bkopilov has quit IRC | 12:32 | |
| *** openstackgerrit has quit IRC | 12:33 | |
| *** sidx64 has quit IRC | 12:34 | |
| *** openstackgerrit has joined #openstack-nova | 12:37 | |
| openstackgerrit | Merged openstack/nova master: Fix message for unexpected external event https://review.openstack.org/554380 | 12:37 |
| *** Zames has joined #openstack-nova | 12:38 | |
| *** _ix has joined #openstack-nova | 12:38 | |
| *** sidx64 has joined #openstack-nova | 12:40 | |
| *** Zames has quit IRC | 12:40 | |
| jaypipes | alex_xu_: still around? not sure what you were asking about preferred_traits... I have no issue with decorating traits in flavors as being preferred. I wouldn't send them to placement, though... just allow the scheduler weighers to use them in their sorting. is that what you were thinking of? | 12:40 |
| *** r-daneel has quit IRC | 12:41 | |
| *** jpena is now known as jpena|lunch | 12:44 | |
| *** AlexeyAbashkin has quit IRC | 12:48 | |
| openstackgerrit | sahid proposed openstack/nova-specs master: virt: allow instances to be booted with trusted VFs https://review.openstack.org/485522 | 12:50 |
| *** sidx64 has quit IRC | 12:52 | |
| jaypipes | efried, cdent: so based on you guys' and tetsuro's feedback on the "nested providers allocation candidates" series, I'm wondering if there's *any* reason to use non-granular request groups when nested providers are present. I spent all day reworking that series yesterday to get things working so that a non-granular request group could work against a tree of providers (but not by summing inventories across the entire tree). Instead, what I did | 12:54 |
| jaypipes | was say that individual providers needed within the tree needed to satisfy each quantitative resource request and then *collectively* the tree needed to satisfy the traits request. do you think that's wrong as well? | 12:54 |
| jaypipes | lemme push what I have.. one se.c | 12:55 |
| efried | jaypipes: That sounds correct to me. | 12:55 |
| efried | Within the request, one resource_class:amount needs to be satisfied by one provider in the tree, or one associated via aggregate with any provider in the tree. And collectively, the RPs satisfying the resource request (which may in fact *exclude* some of the RPs in the tree) must satisfy the traits. | 12:57 |
| efried | Not sure if that last thing is what you did. | 12:57 |
| efried | jaypipes: But I think that's important. I don't think we want to say traits are satisfied by a provider that's not providing any resource to the request. | 12:58 |
| efried | ...and we do need to make sure agg-associated RPs (but only ones providing resource) are included in the traits calculation. | 12:58 |
| jaypipes | efried: what about when a trait is applied to the NUMA node that is the parent of a provider satisfying some part of the resource request? | 12:58 |
| jaypipes | efried: forget the shared stuff for right now... | 12:59 |
| efried | jaypipes: Hmmm, the NUMA thing... I said something yesterday, I think in a spec comment, related to this. I'll dig it up, but I think taking that into account, the answer to the above will be "that will depend on different syntax". | 13:00 |
| efried | jaypipes: The gist was that (what I recall from PTG discussions, early Wednesday) we want to support the NUMA subtree business (and cdent this may also play into the vmware cluster thing) via a syntax that expresses: "Get all the resources in this group from a subtree whose (sub)root is marked with trait X" | 13:02 |
| efried | jaypipes: So if the NUMA node is marked with trait I_AM_A_NUMA_NODE, the syntax would be like GET /a_c?resources4=...&subtree_trait4=I_AM_A_NUMA_NODE | 13:03 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: stop lying to the RT when ironic is down https://review.openstack.org/545479 | 13:03 |
| efried | That would affect the calculation of which providers in the tree are eligible to provide resources - i.e. just the subtree rooted at a provider marked I_AM_A_NUMA_NODE - and that trait gets special treatment such that the NUMA root RP itself doesn't actually need to provide resources. | 13:04 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Address issues raised in adding member_of to GET /a-c https://review.openstack.org/554357 | 13:04 |
| *** jianghuaw has joined #openstack-nova | 13:05 | |
| *** abhishekk has joined #openstack-nova | 13:05 | |
| openstackgerrit | Jay Pipes proposed openstack/nova master: tests for alloc candidates with nested and traits https://review.openstack.org/531899 | 13:05 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: resource requests for nested providers https://review.openstack.org/554529 | 13:05 |
| jaypipes | efried, cdent: please see above (different series, forked from the base of the old one) | 13:06 |
| *** sidx64 has joined #openstack-nova | 13:06 | |
| gibi | efried, jaypipes: I agree that a single resource request need to be satisfied from a single RP but in case of traits I think the trait needs to be satisfied by the RPs on the path from the root (or subroot) to the RPs that are providing resources to that request | 13:08 |
| sean-k-mooney[m] | efried: thanks for your review on https://review.openstack.org/#/c/449257/59 you spotted my **{} sed hack which is fair i was just hopping i would not need to manually convert them all. ill fix it in the next respin. i answered some of your other questions inline | 13:08 |
| gibi | efried, jaypipes: where the path can be defined also by taking the RPs that providing resources and collecting all the ancestors for those RPs and then checking that the traits are satisfied in the set of RPs or not | 13:10 |
| *** AlexeyAbashkin has joined #openstack-nova | 13:10 | |
| *** lucas-hungry is now known as lucasagomes | 13:10 | |
| *** ratailor has quit IRC | 13:11 | |
| mdbooth | Could somebody take a look at this live migration bugfix for me: https://review.openstack.org/#/c/551302/ . It's got a bunch of +1s and I hacked a CI run to ensure coverage. | 13:11 |
| jaypipes | gibi, efried: ok, understood. which gets back to my original question... if nested providers are present, is there really any point in *not* requiring granular request groups? | 13:12 |
| gibi | jaypipes, efried: do I remember correctly that a granular numbered request group means that both the resources and the traits needs to be fulfilled from a single RP? | 13:13 |
| *** Guest61245 has quit IRC | 13:14 | |
| efried | gibi: I don't think I agree that we should collect traits from "tree paths". That seems excessively complicated. What use case does it satisfy? (Hint: I don't think it satisfies the NUMA thing without further semantic work) | 13:14 |
| efried | gibi: Correct. | 13:14 |
| *** sree has joined #openstack-nova | 13:14 | |
| efried | jaypipes: Trying to think through whether it's always possible to express an un-numbered request group as one or more numbered groups. | 13:15 |
| efried | I think there's some cases you can't express - but it's actually a good thing that you can't. | 13:15 |
| gibi | jaypipes, efried: I've started thinking about the same | 13:15 |
| efried | Like getting the a trait from a provider you weren't expecting. | 13:15 |
| gibi | jaypipes, efried: i.e. trait is on the compute RP, resource inventory is on the PF | 13:16 |
| gibi | that would need two separate numbered group | 13:16 |
| efried | gibi: Yeah. It *should* be the case that traits on the one aren't applicable to the other. But who knows? | 13:16 |
| *** fragatina has joined #openstack-nova | 13:17 | |
| openstackgerrit | sahid proposed openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605 | 13:17 |
| *** mriedem has joined #openstack-nova | 13:18 | |
| jaypipes | efried, gibi: yeah, that's been my dilemma :) | 13:18 |
| *** jianghuaw has quit IRC | 13:18 | |
| gibi | jaypipes: you successfully shared your pain :) | 13:18 |
| efried | jaypipes: I'm not opposed to this idea in principle - it makes a couple of things simpler, which is good. | 13:18 |
| jaypipes | gibi: you're welcome. ;) | 13:18 |
| efried | jaypipes: My concern is how we express this to operators. | 13:19 |
| jaypipes | efried: agree with you. I'm hunting for ideas. | 13:19 |
| *** sree has quit IRC | 13:19 | |
| efried | jaypipes: They need to have, what, separate flavors for nested-modeled hosts than for non? | 13:19 |
| efried | I mean, even a non-nested host you can express requests with granular. | 13:19 |
| efried | So the line could just be: start using granular for everything, period. | 13:20 |
| jaypipes | efried: meh, I don't think that will be common. I'm more concerned about how to document the quirks of each "solving algorithm", depending on whether they use granular or not, nested or not, sharing providers or not, etc | 13:20 |
| efried | But that's kind of a dick punch to the traits-in-glance thing. | 13:20 |
| efried | jaypipes: Don't think what will be common? Environments where some are trees and some are not? | 13:20 |
| *** psachin has quit IRC | 13:21 | |
| *** jianghuaw has joined #openstack-nova | 13:21 | |
| jaypipes | efried: no, I mean a need for flavors that request the same resources/traits but "in different ways" (i.e. collectively met vs individually met) | 13:21 |
| openstackgerrit | sahid proposed openstack/nova-specs master: libvirt: add support for virtio-net rx/tx queue sizes https://review.openstack.org/539605 | 13:22 |
| kaisers1 | efried: Hey, ok :) | 13:22 |
| *** sidx64 has quit IRC | 13:22 | |
| efried | kaisers1: What you have looks fine to me, but you'll want to take your lead from mikal since he's engaged at this point. | 13:23 |
| efried | kaisers1: I'm not sure if he wants you to put the non-systemd exec into the privsep lib, or something. | 13:23 |
| cdent | jaypipes, efried: I think we should make the simple cases as simple to express as possible and for some deployments that ought to mean that some hardware doesn't "turn on" numa, so it just reports simple inventory. | 13:23 |
| kaisers1 | ok, i just wanted to ask regarding the kwargs topic, did you read my reply on that? | 13:23 |
| efried | kaisers1: Not yet, looking... | 13:23 |
| cdent | I'd like to think that it is possible to do some nested things without granular but I've not thought it all the way through | 13:24 |
| dansmith | sean-k-mooney[m]: yep, that's what needs to happen | 13:24 |
| cdent | s/it is/ought to be/ | 13:24 |
| gibi | efried, jaypipes: granularity in the flavor needs to express what resource needs to be collocated and what can be spread inside the selected host. This should not depend on the fact that the host provide nested RP tree that allows the spreading | 13:24 |
| *** _ix has quit IRC | 13:24 | |
| kaisers1 | efried: ok. It's just that either i don'r fully grasp it or it doesn't make that much sense to me | 13:24 |
| kaisers1 | *don't | 13:24 |
| efried | kaisers1: Oh, I totally didn't see that response. | 13:25 |
| efried | I'll answer in the patch. Sorry about that. | 13:25 |
| kaisers1 | efried: np, thanks for looking into the patch | 13:25 |
| efried | gibi: I tend to agree. | 13:25 |
| gibi | efried, jaypipes: so If I don't care about to collocate cpu and ram to the same numa then I can create the cpu request in a different group than the memory request and that will work against nested and not nested hosts | 13:25 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add CellMappingList.get_all_enabled() query method https://review.openstack.org/550188 | 13:26 |
| gibi | efried, jaypipes: or does it? | 13:26 |
| jaypipes | gibi: in a non-nested representation, there's no reason to use granular request groups. because every request is a granular request group (it cannot be satisfied by >1 provider non-sharing provider) | 13:26 |
| efried | gibi: Yes. (Though that still doesn't help us collocate e.g. CPU and VF on the same NUMA, cause they'll be members of different providers.) | 13:26 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled field to CellMapping object https://review.openstack.org/550090 | 13:26 |
| efried | gibi: Same numbered request group == same provider. Different numbered request groups == maybe same, maybe different providers | 13:26 |
| gibi | efried: your last point made my above statment false | 13:27 |
| efried | gibi: We definitely have no way (other than unique traits) to express that two requests MUST be from separate providers. | 13:27 |
| *** sree has joined #openstack-nova | 13:27 | |
| gibi | efried: sorry, mixed up | 13:27 |
| gibi | efried: let me try again | 13:28 |
| efried | gibi: Always same tree-or-associated-sharing, though, if it's in one request. | 13:28 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add CellMappingList.get_all_enabled() query method https://review.openstack.org/550188 | 13:28 |
| gibi | efried: so we can instruct the deployer to create flavors always with granular groups if he does not care about numa affinity as two separate group can be satisfied by the same RP | 13:29 |
| *** sidx64 has joined #openstack-nova | 13:30 | |
| gibi | jaypipes: I guess we don't need granular in non-nested deployments, but in a mixed deployment, the granular groups works against both type of hosts, which is good | 13:30 |
| efried | gibi: Even if he cares about NUMA affinity (assuming we're just talking about NUMA_CORE+MEMORY_MB - stuff in the same RP). You would then specify both resources in the same numbered group, meaning they have to come from the same provider. | 13:30 |
| gibi | efried: true | 13:30 |
| *** hongbin has joined #openstack-nova | 13:30 | |
| efried | (still doesn't help the NUMA_CORE+PF case) | 13:30 |
| efried | (we still need new syntax for that) | 13:31 |
| efried | (and the new syntax has to be within granular) | 13:31 |
| *** burt has joined #openstack-nova | 13:31 | |
| gibi | efried: NUMA_CORE + PF case needs a group that is satisfied by a subtree specified by something that identifies the root of the subtree somehow | 13:32 |
| *** r-daneel has joined #openstack-nova | 13:32 | |
| gibi | efried: your I_M_A_NUMA_NODE trait based subtree specification makes sense to me | 13:32 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: [WIP] Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527 | 13:32 |
| cdent | jaypipes: if you're able to go back and respond to the various comments on the reviews, in context, that would be awesome. It's sometimes hard to keep the many thread clear. | 13:33 |
| gibi | efried: I can even forsee that I_AM_A_NETWORKING_RP_WHICH_DEFINES_A_SUBTREE_BELONGIG_TO_THE_SAME_NEUTRON_AGENT trait :) | 13:34 |
| jaypipes | cdent: sure, I will. was just looking to brainstorm. | 13:34 |
| cdent | Yeah, sure, not complaining, definitely want the brainstorming too | 13:35 |
| gibi | efried: i mean I foresee a possible need of such trait based selection | 13:35 |
| cdent | gibi: I think you missed _PLEASE on the end of that trait | 13:35 |
| *** r-daneel has quit IRC | 13:36 | |
| jaypipes | in any case, unfortunately, I now have a dentist appointment I need to get to... and then drop the dogs off at the spa and sit while my car gets serviced (will have my lappie for the last part, though, so will be online later) | 13:36 |
| efried | enjay :) | 13:37 |
| sean-k-mooney[m] | jaypipes: doggy spa days. you really do spoil them :) | 13:37 |
| *** sree has quit IRC | 13:39 | |
| *** esberglu has quit IRC | 13:39 | |
| *** voelzmo has joined #openstack-nova | 13:40 | |
| gibi | cdent: I think there is a limit about the lenght of a trait :) | 13:40 |
| cdent | :) | 13:40 |
| *** voelzmo has quit IRC | 13:41 | |
| gibi | jaypipes: good luck with your errands | 13:41 |
| * gibi needs a spa day | 13:41 | |
| *** esberglu has joined #openstack-nova | 13:42 | |
| *** jpena|lunch is now known as jpena | 13:42 | |
| sean-k-mooney[m] | gibi i belive its set in the db scema probaly 64,128 or256 charaters | 13:42 |
| *** voelzmo has joined #openstack-nova | 13:42 | |
| *** amoralej|lunch is now known as amoralej | 13:43 | |
| sean-k-mooney[m] | gibi: its 255 https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions/041_resource_provider_traits.py#L43 | 13:43 |
| *** Spaz-Home has quit IRC | 13:44 | |
| sean-k-mooney[m] | also aprently we set the charset to latin1 so no unicode traits. is that intentional ? | 13:44 |
| gibi | sean-k-mooney[m]: OK, then I can fit the PLEASE at the end :) | 13:44 |
| gibi | sean-k-mooney[m]: I think that is intentional https://github.com/openstack/nova/blob/f80b4e50093002f84b43ff245a605fbe44d34711/nova/api/openstack/placement/handlers/trait.py#L74 | 13:45 |
| alex_xu_ | jaypipes: For example, I want to a FPGA device with Funciton A, the request is 'resources=RC_FPGA_DEV:1&preferred=FPGA_FUNCTION_A'. Do you mean you only request one RC_FPGA_DEV, then weigh the RPs base on the summary of RPs in the response of allocation_candidates? | 13:47 |
| efried | kaisers1: Responded. Note that I'm not saying any of those things will necessarily ever happen in this case. It's just one of those things that's good practice. | 13:47 |
| sean-k-mooney[m] | ah of url parsing so we dont get sql injections... we still might get requests form people with non latin based alphabets at some point but i guess we can cross that bridge when we come to it | 13:47 |
| efried | sean-k-mooney[m]: The schema only allows A-Z and _ | 13:48 |
| efried | and 0-9 | 13:48 |
| sean-k-mooney[m] | efried: the api schema yes i see that | 13:48 |
| alex_xu_ | oh, jaypipes has a dentist appointment | 13:48 |
| *** tbachman has quit IRC | 13:49 | |
| gibi | cdent: there is a debate about a need of microversion bump in https://review.openstack.org/#/c/502306/17/specs/rocky/approved/bandwidth-resource-provider.rst@142 this might be interesting to you as well | 13:50 |
| * cdent wishes he never got associated with microversions :) | 13:51 | |
| sean-k-mooney[m] | efried: the api schema yes i see that | 13:51 |
| *** bkopilov has joined #openstack-nova | 13:51 | |
| *** links has quit IRC | 13:51 | |
| sean-k-mooney[m] | ed :: | 13:52 |
| gibi | cdent: you can free to forward the ping to other API experts :) | 13:52 |
| gibi | s/can/are/ | 13:52 |
| *** SamYaple has quit IRC | 13:52 | |
| cdent | gibi: hmmm. that is an interesting one. my general rule is capture well here: http://eavesdrop.openstack.org/meetings/nova_scheduler/2018/nova_scheduler.2018-03-19-14.00.log.html#l-178 | 13:53 |
| alex_xu_ | jaypipes: one more question, what is the reason you didn't want to add prefered trait in placement | 13:53 |
| *** tbachman has joined #openstack-nova | 13:53 | |
| * gibi clicks | 13:54 | |
| sean-k-mooney[m] | efried: damit lost the message. any TL:DR CUSTOM_SEÁNS_TRAIT would be invalid which is not much of a deal for me but some operators may want to create custom traits in there own language in teh future that said we should try to converge in a set of standard traits to cover there needs and cross the non english traits bridge when we really need too | 13:54 |
| efried | alex_xu_: Because placement is a placer, not a weigher | 13:56 |
| efried | alex_xu_: That's my interpretation of jaypipes' opinion, not necessarily my own | 13:57 |
| sean-k-mooney[m] | alex_xu_: placement is fitting hard constratins, a prefereed trait is a best effort request. filtering based on a best effort request would potentially result in no valid host when there were hosts that met the required traits but no the prefered hence why jaypipes wanted to handel prefred in a weigher not in placement | 13:57 |
| *** jmlowe has joined #openstack-nova | 13:58 | |
| sean-k-mooney[m] | what efried said :) | 13:58 |
| gibi | cdent: thanks, does this also means that in the old microversion we have to keep the old behavior which means not to fail the request just ignore the requested resources? | 13:58 |
| alex_xu_ | efried: thanks for the high level explain, sean-k-mooney[m] thanks for the detail explain :) | 13:58 |
| efried | heh | 13:59 |
| sean-k-mooney | interesting. i have been testing out riot.im as an irc bouncer/bridge. looks like there is a 5-10 secodn delay when i use riot vs direct irc at least on posting. i get messages almost instanly on both | 13:59 |
| *** lajoskatona has quit IRC | 13:59 | |
| *** gouthamr has joined #openstack-nova | 14:00 | |
| sean-k-mooney | sean-k-mooney[m] is my riot.im client | 14:00 |
| cdent | gibi: strictly speaking, yes, but it depends on the severity of the bad behavior | 14:00 |
| cdent | I'm not generally a fan of the strict interpretation | 14:00 |
| *** voelzmo has quit IRC | 14:01 | |
| *** jianghuaw has quit IRC | 14:01 | |
| gibi | cdent: in this particular case the nova boot request was accepted and the requested resources was ignored. After the proposed change such a request would be rejected with an error message | 14:02 |
| alex_xu_ | sean-k-mooney: Does that mean I require TRAIT_A, and prefer TRAIT_B, there is chance to return nothing when there is no TRAIT_B for any RP? not quite understand | 14:02 |
| cdent | gibi: Again, using the strict interpretation, the older microversions should continue to accept | 14:02 |
| *** tbachman has quit IRC | 14:03 | |
| *** eharney has joined #openstack-nova | 14:03 | |
| gibi | cdent: assume for a second that we don't make it strict and allow failing in the old microversion. This would mean that there would be no difference between the old and the new microversion | 14:04 |
| *** voelzmo has joined #openstack-nova | 14:04 | |
| gibi | cdent: the only difference would be in the documentation | 14:04 |
| gibi | cdent: about the meaning of the accpet | 14:04 |
| edleafe | alex_xu_: "preferred" doesn't mean anything in placement, since it doesn't order anything | 14:04 |
| gibi | cdent: in this case do we need the microversion bump just for the doc? | 14:05 |
| edleafe | "required" implies a filtering, which is what placement does | 14:05 |
| alex_xu_ | edleafe: 'preferred' order the number of traits which the RP has | 14:05 |
| cdent | gibi: assuming that, then I wouldn't think a microverison was required for a doc change, because the behavior is the same everywhered, right? But if there is in fact a behavior change (in the API itself), that seems like a microversion, right? | 14:07 |
| *** germs has joined #openstack-nova | 14:07 | |
| *** germs has quit IRC | 14:07 | |
| *** germs has joined #openstack-nova | 14:07 | |
| *** finucannot is now known as stephenfin | 14:07 | |
| alex_xu_ | or you guys refer to the implement problem | 14:08 |
| *** voelzmo has quit IRC | 14:08 | |
| edleafe | alex_xu_: if I prefer a trait, I will accept an RP that doesn't have that trait if necessary. So Placement will return all RPs that satisfy the requirements. It is then up to the scheduler to sort them based on various things, and preferred traits would be one of those things | 14:09 |
| gibi | cdent: I'm hesitant. The external behaviour will change as nova will reject a previously accepted the request. But even if nova today accept such request nova actually lies and nova does not fulfill the request properly as nova does not consider the QoS bandwidth policy on the port that is included in the request during the scheduling | 14:10 |
| *** SamYaple has joined #openstack-nova | 14:11 | |
| gibi | cdent: so by rejecting such request nova we clean up a lie | 14:11 |
| alex_xu_ | edleafe: oh, the weigher will change the order again | 14:11 |
| cdent | gibi: and you break working code | 14:11 |
| gibi | cdent: true | 14:11 |
| *** germs has quit IRC | 14:11 | |
| *** sidx64 has quit IRC | 14:11 | |
| cdent | gibi: Which is what microversions are supposed to prevent, even though it means broken (in other ways) client code gets to continue existing | 14:11 |
| cdent | I agree that it is a tricky problem and your hesitancy is warranted | 14:12 |
| *** awaugama has joined #openstack-nova | 14:12 | |
| cdent | Do you feel the "lie" is a security problem? | 14:12 |
| *** yamamoto_ has joined #openstack-nova | 14:12 | |
| *** yamamoto has quit IRC | 14:12 | |
| gibi | cdent: no, the lie is not a security issue. It is just a resource allocation issue | 14:13 |
| edleafe | gibi: would you call the "lie" a bug? | 14:13 |
| edleafe | IOW, Nova isn't doing what it's supposed to? | 14:13 |
| gibi | edleafe: Nova missing support for including the port QoS policy in the placement decision | 14:14 |
| * alex_xu_ is suddenly enlightened | 14:14 | |
| edleafe | gibi: understood. What I'm asking is if nova not doing that is a bug | 14:14 |
| edleafe | or is it just a feature that has not yet been implemented | 14:14 |
| gibi | edleafe: it is a missing feature, not a bug. | 14:15 |
| gibi | edleafe: it never worked before | 14:15 |
| edleafe | gibi: ok, then it definitely needs a microversion | 14:15 |
| gibi | edleafe, cdent: thank you for the discussion. I will link this discussion to the spec | 14:16 |
| mriedem | gibi: "The external behaviour will change as nova will reject a previously accepted the request. But even if nova today accept such request nova actually lies and nova does not fulfill the request properly as nova does not consider the QoS bandwidth policy on the port that is included in the request during the scheduling" - same story with rebuilding a volume-backed instance with a new image; we used to accept that and just n | 14:16 |
| mriedem | onor it, but changed it to a 400 in the API in queens | 14:16 |
| alex_xu_ | each microversion discussion is a war | 14:16 |
| edleafe | alex_xu_: https://twitter.com/EdLeafe/status/976100354650791936 | 14:17 |
| mriedem | not really | 14:17 |
| alex_xu_ | edleafe: :) | 14:17 |
| gibi | mriedem: so there is precedence not to bump microversion in this case even if it breaks client code | 14:18 |
| *** chyka has joined #openstack-nova | 14:18 | |
| gibi | edleafe, alex_xu_: :) | 14:19 |
| mriedem | gibi: https://review.openstack.org/#/c/520660/ | 14:20 |
| mriedem | like most things, it's a case by case basis, but in the case of ^ we said it wasn't a microversion because it was a silent failure on the compute side | 14:20 |
| mriedem | you shouldn't have to opt into being not broken | 14:20 |
| mriedem | but i'm sure with enough time and examples we can spin that all ways | 14:21 |
| *** pcaruana has quit IRC | 14:21 | |
| *** mlavalle has joined #openstack-nova | 14:21 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Demo code for microversion parse extraction https://review.openstack.org/550265 | 14:22 |
| *** eharney_ has joined #openstack-nova | 14:23 | |
| edleafe | mriedem: yeah, that was why I was asking if it was a bug or a new feature. You shouldn't have to opt into bug fixes. | 14:23 |
| *** kholkina has quit IRC | 14:23 | |
| *** chyka has quit IRC | 14:23 | |
| *** fragatina has quit IRC | 14:23 | |
| alex_xu_ | mriedem: https://review.openstack.org/#/c/520660/ is a bug, I feel it is different | 14:23 |
| gibi | mriedem, cdent, edleafe: in the bfv rebuild case that rebuild has never worked before | 14:24 |
| openstackgerrit | Silvan Kaiser proposed openstack/nova master: Exec systemd-run with privileges in Quobyte driver https://review.openstack.org/554195 | 14:24 |
| alex_xu_ | gibi: the qos policy works with neutron currently, right? | 14:24 |
| *** eharney has quit IRC | 14:24 | |
| gibi | alex_xu_: the qos policy without placement support cannot really work properly even if neutron implemented some data plane enforcement for that policy | 14:25 |
| mriedem | gibi: depends on what you mean by 'worked', | 14:26 |
| *** sree has joined #openstack-nova | 14:26 | |
| mriedem | the api used to accept that kind of request and return a 202, and nothing would fail, but it was wrong | 14:26 |
| mriedem | wrong in that it didn't actually replace the root disk | 14:26 |
| gibi | alex_xu_: imagine that you promised 10G minimum bandwidth for two VMs on the same host and on the same 10G PF. That will cause inconsistency when both VM start using that minimum bandwidth heavily | 14:26 |
| mriedem | today you can create a port with a qos policy and attach it to an instance, right? nova allows that w/o changing any resource consumption for that qos policy. | 14:26 |
| gibi | mriedem: right | 14:27 |
| *** felipemonteiro has joined #openstack-nova | 14:27 | |
| mriedem | if we followed the volume-backed rebuild + new image path here, | 14:27 |
| mriedem | we'd merge a patch to check if the port (or network) has a qos policy and fail if so | 14:27 |
| gibi | yes | 14:27 |
| mriedem | and then land a microversion that allows supporting those types of ports | 14:27 |
| gibi | mriedem: wait a bit | 14:28 |
| mriedem | which is what https://review.openstack.org/#/c/532407/ is for | 14:28 |
| *** pcaruana has joined #openstack-nova | 14:28 | |
| mriedem | thinking about volume multiattach, we kind of have the same situation - you can't attach a multiattach volume with microversion < 2.60, you have to opt into using those types of volumes | 14:29 |
| gibi | mriedem: we talking about adding microversion bump to the first patch where we start rejecting such requests not just microversion bump when we later start supporting such request | 14:29 |
| *** felipemonteiro_ has joined #openstack-nova | 14:29 | |
| bauzas | sean-k-mooney[m]: looking at https://review.openstack.org/#/c/545951/2/specs/rocky/approved/enable-sriov-nic-features.rst | 14:29 |
| mriedem | gibi: if you do that, anyone that doesn't have that code in their cloud can have users still attaching those kinds of ports | 14:29 |
| mriedem | if we consider it a bug that we don't support ports with qos policies, then we should make that a hard failure (bug fix) and then add a new microversion which adds the support for those types of ports | 14:30 |
| *** sree has quit IRC | 14:30 | |
| gibi | mriedem: so making it a hard failure is a bugfix, I like that | 14:31 |
| alex_xu_ | emm...wait, we don't check the network has qos policy, we check the port has required resource or not. That is something new added in neutron | 14:31 |
| mriedem | gibi: well that's what we did for the bfv rebuild case | 14:31 |
| *** voelzmo has joined #openstack-nova | 14:32 | |
| gibi | alex_xu_: required resources is just another representation of the QoS policy to let neutron do the transformation between QoS policy entity on the API to the resource classes and traits in the Placement | 14:32 |
| mriedem | tbh, we're inconsistent on this type of thing. with the multiattach support in queens, we check if the volume is multiattach and if the microversion is high enough (2.60) and if not, we fail. but i'm pretty sure in pike you could probably attach a multiattach volume to at least one instance at a time without failures. | 14:32 |
| *** felipemonteiro has quit IRC | 14:32 | |
| alex_xu_ | gibi: the neutron side add 'resources' field to the port API by extension? or just add directly? | 14:33 |
| gibi | cdent, edleafe, mriedem: I start to get convinced that making the API hardfail instead of accept and lie is a bugfix | 14:33 |
| *** eharney_ is now known as eharnye | 14:34 | |
| *** eharnye is now known as eharney | 14:34 | |
| mriedem | so the bug is today we are potentially over-subscribing the qos bandwidth right? | 14:35 |
| mriedem | for a given network | 14:35 |
| *** gouthamr has quit IRC | 14:35 | |
| gibi | alex_xu_: that will come from the QoS neutron plugin based on my current understanding | 14:35 |
| gibi | alex_xu_: the neutron spec will describe this in detail (writing is in progress) | 14:35 |
| *** Nil_ has joined #openstack-nova | 14:35 | |
| mriedem | the neutron qos stuff has been around for a long time hasn't it? kind of surprised no one hasn't already reported this as a bug. | 14:35 |
| *** hemna_ has joined #openstack-nova | 14:36 | |
| gibi | mriedem: yes, SRIOV ports with QoS minimum bandwidth policies are not properly enforced and the minimum bandwidth cannot be garanteed | 14:36 |
| gibi | mriedem: only SRIOV ports supporting minimum bandwidth policies | 14:36 |
| *** voelzmo has quit IRC | 14:37 | |
| gibi | mriedem: and there was a release notes that stated the problem | 14:37 |
| gibi | mriedem: https://github.com/openstack/neutron/blob/49d614895f44c44f9e1735210498facf1886c404/releasenotes/notes/qos-min-egress-bw-rule-b1c80f5675a4c1c3.yaml | 14:37 |
| gibi | mriedem: so maybe the deployers read the release notes and understood that the support for the minimum bandwidth rule is incomplete | 14:38 |
| mriedem | added in newton | 14:38 |
| gibi | mriedem: yes | 14:38 |
| mriedem | looks like the policy defaults allow only admins to create these types of policies | 14:39 |
| mriedem | so there is at least that | 14:39 |
| *** sar has quit IRC | 14:40 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add disabled field to CellMapping object https://review.openstack.org/550090 | 14:42 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add CellMappingList.get_all_enabled() query method https://review.openstack.org/550188 | 14:42 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527 | 14:42 |
| bauzas | sean-k-mooney[m]: jaypipes: stephenfin: soft -1 on https://review.openstack.org/#/c/545951/2 but I need to understand why we can't just get the port info in the conductor, and pass the traits to the scheduler | 14:42 |
| * stephenfin looks | 14:43 | |
| bauzas | I could be wrong but AFAIK we haven't said why it wasn't possible to do the above ^ | 14:43 |
| *** liverpooler has joined #openstack-nova | 14:44 | |
| *** _ix has joined #openstack-nova | 14:44 | |
| *** elmaciej has quit IRC | 14:45 | |
| *** artom_ has joined #openstack-nova | 14:46 | |
| *** voelzmo has joined #openstack-nova | 14:47 | |
| *** sree has joined #openstack-nova | 14:47 | |
| *** itlinux has joined #openstack-nova | 14:49 | |
| *** eharney_ has joined #openstack-nova | 14:50 | |
| *** gouthamr has joined #openstack-nova | 14:50 | |
| *** sree has quit IRC | 14:51 | |
| *** eharney has quit IRC | 14:51 | |
| *** r-daneel has joined #openstack-nova | 14:55 | |
| *** artom_ is now known as artom | 14:55 | |
| dansmith | jaypipes: sean-k-mooney[m]: do either of you know if/when we can remove the old vif plugging stuff from the libvirt driver? i.e. everything reachable past the "if not os-vif" line? | 14:57 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Address issues raised in adding member_of to GET /a-c https://review.openstack.org/554357 | 14:57 |
| edleafe | efried: ^^ yay pep8! | 14:57 |
| *** amodi has joined #openstack-nova | 15:00 | |
| *** dtantsur|afk is now known as dtantsur | 15:00 | |
| *** jistr is now known as jistr|mtg | 15:01 | |
| *** voelzmo has quit IRC | 15:02 | |
| *** voelzmo has joined #openstack-nova | 15:03 | |
| *** sree has joined #openstack-nova | 15:10 | |
| *** Eran_Kuris has quit IRC | 15:10 | |
| *** moshele has quit IRC | 15:12 | |
| *** mdnadeem has quit IRC | 15:13 | |
| mriedem | dansmith: presumably once all of the legacy methods in libvirt/vif.py are converted to using os-vif objects | 15:14 |
| dansmith | mriedem: I don't know how to tell that. are those just linux bridge and ovs right now? | 15:15 |
| mriedem | look in nova.network.os_vif_util at the _nova_to_osvif_vif_* methods that raise NotImplementedError | 15:16 |
| *** yassine has quit IRC | 15:16 | |
| mriedem | there are quite a few | 15:16 |
| stephenfin | dansmith: Yeah, once all of those have been converted | 15:16 |
| *** eharney_ is now known as eharney | 15:16 | |
| stephenfin | jaypipes and I have discussed it before. The IVS driver has been converted to an os-vif plugin but I don't know about the rest of them. We might need to jettison them or bring them into os-vif core | 15:17 |
| dansmith | is that actually happening in the background? | 15:17 |
| stephenfin | Just this one so far https://bugs.launchpad.net/bugs/1704129 | 15:17 |
| openstack | Launchpad bug 1704129 in networking-bigswitch "Add an IVS os-vif plugin" [Undecided,New] - Assigned to Aditya Vaja (wolverine-av) | 15:17 |
| dansmith | couldn't we remove the ones from our tree that are converted already? | 15:17 |
| stephenfin | They should already be removed | 15:17 |
| stephenfin | e.g. OVS and linuxbridge | 15:18 |
| dansmith | there are several still that have a method and just pass | 15:18 |
| dansmith | are those converted and "removed" or are those things we just don't do anything for? | 15:18 |
| *** abhishekk has quit IRC | 15:18 | |
| mriedem | can you give an example? | 15:19 |
| stephenfin | For example? | 15:19 |
| dansmith | def plug_802qbg(self, instance, vif): | 15:19 |
| dansmith | pass | 15:19 |
| stephenfin | I went through and removed some others ones a while back. IIRC, everything else was still needed. See commit 1b872996d08b01a1b8a1e82d13d6d7b06bc3aa01 | 15:19 |
| mriedem | ok that's in nova.virt.libvirt.vif | 15:19 |
| dansmith | I was probably reading ivs and ovs and just assumed we had orphaned all this code, but if we're removing them as we can, that's cool | 15:19 |
| dansmith | mriedem: right | 15:19 |
| mriedem | i guess there is no plug for 802qbg but there is a get_config method for it | 15:20 |
| stephenfin | Hmm, I'm not sure about that one. We do getattr magic somewhere in there so it's tricky to figure out what's called and what's dead | 15:20 |
| mriedem | 2.1 is changed to 21 | 15:21 |
| mriedem | so it becomes 802qbg | 15:21 |
| mriedem | the os-vif conversion code looks for _nova_to_osvif_vif_802_1qbg | 15:21 |
| mriedem | because the vif type is "802.1qbg" | 15:21 |
| stephenfin | Which raised NotImplemented? | 15:23 |
| stephenfin | *raises | 15:23 |
| mriedem | yes | 15:23 |
| *** sree has quit IRC | 15:23 | |
| mriedem | i don't know why we don't have to plug/unplug those, but we have to get the config apparently for the guest | 15:23 |
| mriedem | um https://github.com/openstack/os-vif/blob/master/os_vif/objects/vif.py#L263 | 15:24 |
| mriedem | https://github.com/openstack/os-vif/blob/master/os_vif/objects/vif.py#L277 | 15:24 |
| mriedem | seems we could convert two right there | 15:24 |
| dansmith | the reason I'm asking is around removing our linux_net module | 15:25 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs to instance_extra https://review.openstack.org/537897 | 15:25 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | 15:25 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | 15:25 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | 15:25 |
| dansmith | it's really hard to tell if/when it's used right now | 15:25 |
| stephenfin | this is for nova-network deletion, I assume? | 15:26 |
| *** yamamoto_ has quit IRC | 15:27 | |
| dansmith | not really, although that's part of it.. the linux_net module uses a lot of config in conf/network that we can't remove when we nuke nova-net | 15:27 |
| *** sree has joined #openstack-nova | 15:28 | |
| *** dave-mccowan has joined #openstack-nova | 15:29 | |
| dansmith | mriedem: are you cool with backporting this? https://review.openstack.org/#/c/552691/ | 15:31 |
| mriedem | sure | 15:33 |
| *** jistr|mtg is now known as jistr | 15:36 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't log a warning for InstanceNotFound with deleted VIFs https://review.openstack.org/554591 | 15:37 |
| *** sree has quit IRC | 15:37 | |
| *** trinaths has joined #openstack-nova | 15:39 | |
| *** suresh12 has joined #openstack-nova | 15:41 | |
| mriedem | welp, volume multiattach does'nt work with libvirt 4.0.0 in the queens UCA | 15:43 |
| *** trinaths has quit IRC | 15:43 | |
| *** trinaths has joined #openstack-nova | 15:44 | |
| mnaser | mriedem: honest question is there multiattach tempeset jobs | 15:44 |
| mriedem | hells yes | 15:45 |
| mriedem | https://review.openstack.org/#/c/554317/ | 15:45 |
| *** trinaths has quit IRC | 15:45 | |
| mriedem | looks like the shareable flag isn't getting set in the disk config xml for some reason, should be relatively easy to figure out why | 15:45 |
| *** fragatina has joined #openstack-nova | 15:45 | |
| mnaser | mriedem: so does that mean that a certain company is releasing things without possibly running tempest jobs | 15:45 |
| mriedem | i've just been getting distracted with the amount of bullshit warnings in the n-cpu logs | 15:45 |
| *** sree has joined #openstack-nova | 15:46 | |
| mriedem | mnaser: libvirt is open source yeah? but even still, i wouldn't expect them to test openstack stuff against their code. | 15:46 |
| mriedem | or qemu for that matter | 15:46 |
| mnaser | well not libvirt but if uca ships libvirt 4.0.0 | 15:46 |
| mnaser | i'd expect that they tested all of them together | 15:46 |
| mriedem | i don't expect ubuntu / canonical to test all of the permutations of openstack features | 15:47 |
| mriedem | since multiattach is optional too | 15:47 |
| *** Spazmotic has joined #openstack-nova | 15:47 | |
| mriedem | maybe if it were part of the interop guidelines, but it's not | 15:47 |
| *** felipemonteiro_ has quit IRC | 15:48 | |
| *** chyka has joined #openstack-nova | 15:49 | |
| *** felipemonteiro_ has joined #openstack-nova | 15:49 | |
| openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Add --by-service to discover_hosts https://review.openstack.org/554600 | 15:50 |
| *** trinaths has joined #openstack-nova | 15:51 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add CPUWeigher https://review.openstack.org/379525 | 15:52 |
| openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Add --by-service to discover_hosts https://review.openstack.org/554600 | 15:53 |
| *** tbachman has joined #openstack-nova | 15:54 | |
| openstackgerrit | Dan Smith proposed openstack/nova stable/pike: Add --by-service to discover_hosts https://review.openstack.org/554603 | 15:54 |
| *** tbachman_ has joined #openstack-nova | 15:56 | |
| openstackgerrit | Tyler Blakeslee proposed openstack/nova stable/queens: Add method repr() to NovaException https://review.openstack.org/554604 | 15:56 |
| *** salv-orlando has quit IRC | 15:56 | |
| *** salv-orlando has joined #openstack-nova | 15:57 | |
| *** salv-orlando has quit IRC | 15:57 | |
| *** salv-orlando has joined #openstack-nova | 15:57 | |
| *** tbachman has quit IRC | 15:59 | |
| *** tbachman_ is now known as tbachman | 15:59 | |
| *** sree has quit IRC | 16:01 | |
| openstackgerrit | Tyler Blakeslee proposed openstack/nova master: Add method repr() to NovaException https://review.openstack.org/554607 | 16:03 |
| *** zhaochao has quit IRC | 16:05 | |
| *** felipemonteiro_ has quit IRC | 16:06 | |
| *** voelzmo has quit IRC | 16:17 | |
| *** masuberu has quit IRC | 16:18 | |
| *** ragiman has quit IRC | 16:20 | |
| *** lyan has joined #openstack-nova | 16:21 | |
| *** lyan is now known as Guest64768 | 16:22 | |
| *** andreas_s has quit IRC | 16:22 | |
| *** gyee has joined #openstack-nova | 16:26 | |
| *** tbachman has quit IRC | 16:26 | |
| *** andreas_s has joined #openstack-nova | 16:26 | |
| *** yamamoto has joined #openstack-nova | 16:28 | |
| *** salv-orlando has quit IRC | 16:28 | |
| *** salv-orlando has joined #openstack-nova | 16:28 | |
| openstackgerrit | Tyler Blakeslee proposed openstack/nova master: Add method repr() to NovaException https://review.openstack.org/554607 | 16:30 |
| *** salv-orlando has quit IRC | 16:33 | |
| *** wolverineav has joined #openstack-nova | 16:33 | |
| *** yamamoto has quit IRC | 16:34 | |
| *** ameeda has joined #openstack-nova | 16:34 | |
| *** andreas_s has quit IRC | 16:36 | |
| ameeda | Hi, I try to create stack using rest api. current api version is V1 . I POST request with auth key , I got this error message { "explanation": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "code": 400, "error": { "message": "The server could not comply with the request since it is either malformed or otherwise incorrect.", "traceback": null, | 16:36 |
| *** sree has joined #openstack-nova | 16:37 | |
| *** wolverineav has quit IRC | 16:38 | |
| *** wolverineav has joined #openstack-nova | 16:39 | |
| *** tbachman has joined #openstack-nova | 16:40 | |
| kashyap | Hmm, `git fetch origin` again failing for me, with 'origin' == https://git.openstack.org/openstack/nova.git | 16:41 |
| * kashyap wonders if anyone else sees that | 16:42 | |
| *** rmart04 has quit IRC | 16:42 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Clarify log in RT._update_usage_from_migration https://review.openstack.org/554623 | 16:42 |
| *** sree has quit IRC | 16:42 | |
| mriedem | we should log this more often i think: "Defaulting the value of the field 'projects' to None in FlavorPayload due to 'Cannot call _load_projects on orphaned Flavor object'" | 16:44 |
| lyarwood | kashyap: https://git.openstack.org/openstack/nova without the .git as a remote WORKSFORME | 16:44 |
| mriedem | 1K+ times isn't enough | 16:44 |
| kashyap | lyarwood: Err, let me try that | 16:45 |
| lyarwood | kashyap: as listed here - https://git.openstack.org/cgit/openstack/nova | 16:45 |
| kashyap | lyarwood: No luck for me; very bizarre; yesterday it worked briefly, but not anymore | 16:47 |
| dansmith | efried: around? | 16:47 |
| efried | dansmith: yeaux | 16:47 |
| efried | Am I reading backscroll or...? | 16:47 |
| dansmith | efried: trying to write a functional test that talks to placement and do some aggregate stuff | 16:47 |
| kashyap | lyarwood: Both (.git or otherwise) worked for me in the past. Not anymore. /me digs further | 16:47 |
| dansmith | efried: and the reportclient I have is complaining that it doesn't know about a provider, which I think might be a local cache thing and I need to refresh its view or something? | 16:47 |
| efried | dansmith: Which module are you writing your test in? | 16:47 |
| dansmith | but the provider definitely exists | 16:47 |
| efried | dansmith: Wanna post what you've got? | 16:47 |
| dansmith | efried: see if this is enough: https://pastebin.com/ekYNZJWR | 16:48 |
| *** mvk has quit IRC | 16:49 | |
| dansmith | I did a raw get against placement and the provider uuid I'm using is definitely there | 16:49 |
| efried | dansmith: Yeah, set_aggregates_for_provider does require that you've loaded up the provider previously. I think it says so in the docstring. Stand by... | 16:49 |
| *** moshele has joined #openstack-nova | 16:50 | |
| efried | oh, it says the provider must exist. Which isn't the same at all. | 16:50 |
| dansmith | I don't think it does | 16:50 |
| dansmith | yeah | 16:50 |
| *** gjayavelu has joined #openstack-nova | 16:50 | |
| dansmith | presumably I need a .refresh() or some such? | 16:50 |
| efried | dansmith: Well, the PUT ought to be working. | 16:50 |
| dansmith | t'aint | 16:50 |
| efried | cause that's definitely not relying on the cache. | 16:50 |
| efried | yeah, 'tis. It's the cache update that ain't. | 16:51 |
| efried | provider_tree.update_aggregates is what's complaining about the provider not being found. | 16:51 |
| dansmith | you mean the aggregate association completed and it's just trying to refresh its view and failing? | 16:51 |
| efried | Correct. | 16:52 |
| dansmith | okay, what I meant is "this call is failing" | 16:52 |
| dansmith | ;) | 16:52 |
| efried | If you put an _ensure_resource_provider before your set_aggs, it'll fix it. | 16:52 |
| dansmith | okay, but that's a bug yes? | 16:52 |
| efried | Thinking... | 16:52 |
| melwitt | bauzas: runways draft, please add feedback if you have any, want to kick off using it later this week https://etherpad.openstack.org/p/nova-runways-rocky | 16:53 |
| *** AlexeyAbashkin has quit IRC | 16:53 | |
| dansmith | efried: I think the deal here is that this makes report client unable to be generally useful to manage things, as it seems opinionated about only knowing about one provider | 16:53 |
| efried | dansmith: It's not unreasonable for you to expect to be able to use that method as you are. So yeah, I think it's a bug. But tbh I don't know what the right fix would be. | 16:53 |
| dansmith | where it used to work well | 16:53 |
| openstackgerrit | Mathieu Gagné proposed openstack/nova master: Fix rebuild of baremetal instance when vm_state is ERROR https://review.openstack.org/523559 | 16:54 |
| dansmith | efried: I shall stick a #FIXME(efried) in my test | 16:54 |
| efried | dansmith: wfm. Or even open a bug. | 16:54 |
| dansmith | well, just this for the moment | 16:55 |
| efried | I'd like to fix this on top of the upt stack, cause I want to take advantage of the move of the cache refresh timer business. | 16:55 |
| *** felipemonteiro has joined #openstack-nova | 16:55 | |
| dansmith | I can muck with internals in my test, but for other uses it'd be more of a wart | 16:55 |
| *** tbachman has quit IRC | 16:56 | |
| efried | dansmith: I think the answer is going to be to trap such errors and invalidate the cache for that RP. Though after the aforementioned "move of the cache refresh timer business", the cache is already invalidated in this case. So... we'll just wind up ignoring that exception, pretty much. But there's similar code elsewhere. Quite a bit of it, I fear. | 16:57 |
| dansmith | efried: why wouldn't we just _ensure_resource_provider(uuid) in the set_aggregates call? | 16:58 |
| efried | dansmith: That would *usually* be an extraneous REST call. Which I don't like (but cdent does). | 16:58 |
| sean-k-mooney | dansmith: currently we cant remove all the old vif plugging stuff yet. but i think we could get there in rocky | 16:58 |
| dansmith | efried: ah I figured it wouldn't do so if it's already in our known tree | 16:59 |
| dansmith | sean-k-mooney: ack, I got caught up by the others, thanks | 16:59 |
| *** masuberu has joined #openstack-nova | 16:59 | |
| efried | dansmith: That may be true, looking... | 16:59 |
| efried | dansmith: Well, you're correct that it wouldn't retrieve it if it's already in the cache. But it does call _refresh_associations. | 17:00 |
| dansmith | which we're about to do after the put? | 17:01 |
| efried | no, where? | 17:01 |
| sean-k-mooney | dansmith: ok cool, we have been trying to remove the old methods as they get converted to use os-vif. eventully they will all go. | 17:01 |
| efried | dansmith: I mean, if you were gonna do that via some other call after set_aggregates_for_provider, that's different. | 17:01 |
| *** sahid has quit IRC | 17:03 | |
| *** suresh12 has quit IRC | 17:03 | |
| *** lucasagomes is now known as lucas-afk | 17:03 | |
| dansmith | efried: we're refreshing associations after the put, which is failing now, right? or is that something else? | 17:03 |
| dansmith | efried: anyway, it's your call on how to fix it, but I think this needs to be a thing you can do | 17:03 |
| efried | No, that's just updating the cache | 17:03 |
| efried | which is local and fast. | 17:03 |
| *** mvk has joined #openstack-nova | 17:03 | |
| dansmith | okay, from the result? | 17:04 |
| efried | dansmith: Anyway, the reason I wouldn't always want to do that _ensure within this method is because it will go do a potential buttload of placement calls to populate the report client's cache. But if ALL you cared about was updating agg associations, and you were never going to use any of that other stuff, all of that would be wasted. | 17:04 |
| efried | yes | 17:04 |
| dansmith | wouldn't you want the refresh of the associations first anyway so the cache ends up consistent with the server? | 17:04 |
| *** sridharg has quit IRC | 17:04 | |
| *** felipemonteiro_ has joined #openstack-nova | 17:04 | |
| *** suresh12 has joined #openstack-nova | 17:04 | |
| efried | Well, yes, actually, once I get around to it, I'll need to make this guy take generation into account. | 17:04 |
| efried | At which point you'll HAVE to have previously retrieved the RP, cause that's where you'll get the generation from. | 17:05 |
| *** damien_r has quit IRC | 17:06 | |
| efried | But I agree that a public method such as this one ought to be able to be called without you having to know you needed to populate the cache beforehand. | 17:06 |
| efried | So yeah, I'll need to change it (and its brethren) to _ensure/_refresh. | 17:06 |
| efried | Bummer. | 17:07 |
| *** felipemonteiro has quit IRC | 17:08 | |
| dansmith | ack | 17:08 |
| *** felipemonteiro has joined #openstack-nova | 17:08 | |
| melwitt | has anyone seen this error before "EndpointNotFound: Could not find requested endpoint for any of the following interfaces: ['internal', 'public']" when we try to send a notification? | 17:10 |
| gibi | melwitt: I did see this before | 17:11 |
| *** felipemonteiro_ has quit IRC | 17:11 | |
| *** bkero- has quit IRC | 17:11 | |
| mriedem | mnaser: figured it out; it's only a problem for resizing an instance that has multiattach volumes attached; i don't know why the newer libvirt package versions have anything to do with it, because it looks like a latent bug in how we actually handle resize + multiattach | 17:12 |
| *** ameeda has quit IRC | 17:12 | |
| melwitt | gibi: do you know if it's a problem on our end or is it a problem with a deployment? | 17:12 |
| gibi | melwitt: is this EndpointNotFound a keystone exception? | 17:13 |
| melwitt | gibi: here's the full trace https://bugs.launchpad.net/nova/+bug/1753550/comments/3 | 17:13 |
| openstack | Launchpad bug 1753550 in OpenStack Compute (nova) "Status does not update to "Shutoff" when instance shuts down itself" [Undecided,Incomplete] | 17:13 |
| mriedem | melwitt: for the given endpoint, which interfaces are available for it in the service catalog | 17:13 |
| mriedem | the new ksa code will try to use internal and public by default if you don't specify a specific interface in config, like 'admin' | 17:13 |
| mriedem | it's trying to hit glance | 17:14 |
| melwitt | okay. they said they have services listed under 'public' when they do a 'openstack endpoint list' | 17:14 |
| mriedem | so presumably they have glance configured in nova on an admin interface but don't have it configured that way | 17:14 |
| *** mdbooth has quit IRC | 17:14 | |
| melwitt | oh, hm | 17:14 |
| *** suresh12 has quit IRC | 17:15 | |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#glance | 17:15 |
| mriedem | see 'valid_interfaces' | 17:15 |
| melwitt | woo, thanks | 17:15 |
| *** _ix has quit IRC | 17:16 | |
| gibi | mriedem, melwitt: hm, this means that sending a notification causes a REST call to glance. interesting... | 17:16 |
| melwitt | the trace says 'internal' and 'public' though so it seems like they haven't changed that | 17:17 |
| mriedem | File "/opt/stack/nova/nova/notifications/base.py", line 398, in info_from_instance | 17:17 |
| mriedem | context) | 17:17 |
| mriedem | File "/opt/stack/nova/nova/image/api.py", line 65, in generate_image_url | 17:17 |
| melwitt | gibi: yeah, that surprised me | 17:17 |
| *** fragatina has quit IRC | 17:17 | |
| gibi | melwitt: me too :) | 17:17 |
| mriedem | there was also something related to this which we fixed at the end of queens | 17:18 |
| gibi | mriedem: this one? https://review.openstack.org/#/c/511397/14/nova/notifications/base.py | 17:18 |
| mriedem | https://github.com/openstack/nova/commit/62ef6cfcf01d84813f71d1e8252b86c170ee39f0 | 17:18 |
| gibi | mriedem: OK yours is different and seems more relevant | 17:19 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Allow to specify granular CPU feature flags https://review.openstack.org/534384 | 17:20 |
| mriedem | they likely need to update nova, or restack their devstack env, and try again | 17:20 |
| melwitt | the commit id they pulled is from march 4 though, so they should have that fix | 17:22 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_extra_specs_get db api method https://review.openstack.org/539706 | 17:23 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Correct documentation for '[pci] passthrough_whitelist' https://review.openstack.org/552874 | 17:24 |
| mriedem | ildikov: a real hum dinger https://bugs.launchpad.net/nova/+bug/1757190 | 17:26 |
| openstack | Launchpad bug 1757190 in OpenStack Compute (nova) "resize fails with volume multiattach using with libvirt 4.0.0 (and qemu 2.11.1): Failed to get shared "write" lock" [Medium,Triaged] - Assigned to Matt Riedemann (mriedem) | 17:26 |
| mriedem | will fix after lunch | 17:26 |
| mriedem | from what i can tell, it has nothing to do with the actual libvirt/qemu versions, except maybe older libvirt/qemu were masking a bug in nova | 17:26 |
| *** vivsoni__ has joined #openstack-nova | 17:26 | |
| mriedem | during resize we blow away our special 'multiattach' flag in the connection_info here https://github.com/openstack/nova/blob/f80b4e50093002f84b43ff245a605fbe44d34711/nova/virt/block_device.py#L639 | 17:27 |
| *** liverpooler has quit IRC | 17:27 | |
| mriedem | derp | 17:27 |
| ildikov | mriedem: nasty... :/ | 17:28 |
| ildikov | mriedem: I guess it gives us another reason to find a better way to pass the multiattach info | 17:28 |
| *** salv-orlando has joined #openstack-nova | 17:29 | |
| *** lpetrut_ has quit IRC | 17:30 | |
| gibi | melwitt, mriedem: we call glance API from the notification sending due to the else branch here https://github.com/openstack/nova/blob/24379f1822e3ae1d4f7c8398e60af6e52b386c32/nova/image/glance.py#L120 | 17:30 |
| *** priteau has quit IRC | 17:30 | |
| *** yamamoto has joined #openstack-nova | 17:30 | |
| gibi | melwitt,mriedem: s/glance/keystone/ | 17:30 |
| gibi | melwitt, mriedem: and that ksa code was introduced here https://review.openstack.org/#/c/488137/23/nova/image/glance.py | 17:31 |
| melwitt | okay, yeah, and we're failing on L126 on endpoint = utils.get_endpoint(ksa_adap) | 17:33 |
| gibi | melwitt, mriedem: before that ^^ the notification url generation only depened on the config params but after that it can fall back to keystone | 17:33 |
| *** masuberu has quit IRC | 17:33 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add CellMappingList.get_enabled_or_disabled(disabled=False) query method https://review.openstack.org/550188 | 17:33 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Allow scheduling only to enabled cells (Filter Scheduler) https://review.openstack.org/550527 | 17:33 |
| gibi | melwitt: I think so yes | 17:34 |
| *** salv-orlando has quit IRC | 17:34 | |
| melwitt | I meant that's what's in the trace. so that fails before we ever get to the code that will try to strip the version from the url | 17:34 |
| *** ralonsoh__ has quit IRC | 17:35 | |
| gibi | melwitt: this also means that you can workaround the problem by setting CONF.glance.api_servers in the nova.conf | 17:35 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_access_remove db api method https://review.openstack.org/539709 | 17:35 |
| *** yamamoto has quit IRC | 17:36 | |
| melwitt | yeah, true. on the surface, it seems like there must be something wrong with their service catalog, because this is a straight call to keystone, or so it seems | 17:37 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_access_add db api methods https://review.openstack.org/539714 | 17:38 |
| gibi | melwitt: yes, hence my classification of this change as only a workaround and not a real solution | 17:38 |
| gibi | mriedem, melwitt: I have to leave now, but I will read back tomorrow to see if we have to do someting about not talking to keyston from the notification sending codepath | 17:39 |
| melwitt | thanks gibi | 17:39 |
| *** baoli has joined #openstack-nova | 17:39 | |
| *** baoli has quit IRC | 17:40 | |
| *** moshele has quit IRC | 17:48 | |
| edmondsw | melwitt there are a ton of unapproved specs... is there any focus on reviewing those and knocking down that queue? | 17:48 |
| *** suresh12 has joined #openstack-nova | 17:48 | |
| edmondsw | I know the powervm spec has been sitting without comment since Feb | 17:48 |
| *** ccamacho has quit IRC | 17:48 | |
| *** suresh12_ has joined #openstack-nova | 17:50 | |
| *** suresh12 has quit IRC | 17:50 | |
| *** trinaths has quit IRC | 17:51 | |
| *** derekh has quit IRC | 17:52 | |
| *** jackie-truong has joined #openstack-nova | 17:56 | |
| melwitt | edmondsw: I think several people have been reviewing specs. I've been busy with PTG summary writeups and getting the runways proposal going. spec freeze is Apr 19 so I'm going to send email this week to get input on a spec review day date where everyone will focus on spec reviews | 17:57 |
| *** _ix has joined #openstack-nova | 17:58 | |
| edmondsw | melwitt I know a bunch of spec reviews are going on. Just wondering if there is a concerted focus on burning down the queue. Sounds like you're thinking about that | 17:58 |
| edmondsw | kinda a prereq for the runways stuff | 17:59 |
| edmondsw | gotta get the spec approved first :) | 17:59 |
| dansmith | edleafe: around? | 18:00 |
| melwitt | yeah. as usual, the beginning of the cycle people are reviewing specs more because as you said, have to approve things before we can focus on reviewing the implementations. I think people are already doing that. last cycle we had a 79% approved spec/bp completion percentage so I think this cycle we're probably going to approve fewer things to increase that percentage significantly | 18:01 |
| melwitt | like last cycle, we'll have a dedicated spec review day to burn through a lot of them before spec freeze. that's what I'm going to send email about this week so we can pick a date that works for most | 18:02 |
| edleafe | dansmith: yeah - somewhat distracted atm | 18:02 |
| *** wolverineav has quit IRC | 18:03 | |
| *** wolverineav has joined #openstack-nova | 18:03 | |
| dansmith | edleafe: okay, I'm having trouble getting member_of to work and there's kindof a missing case in your tests, which I thought maybe was covering up a bug, but I tweaked it and it still passes, so .. I'm still digging | 18:03 |
| *** moshele has joined #openstack-nova | 18:03 | |
| dansmith | edleafe: but, I've associated a provider with an aggregate, but when I member_of it, I get back no candidates | 18:04 |
| edleafe | dansmith: that's... odd | 18:04 |
| edmondsw | melwitt just a little anxious not seeing reviews on the powervm spec. I'd have thought it was a fairly easy review. I believe the nova team is already committed to this effort and it's just a matter of how much or how little we bite off per release, not whether. I'd like to get the rocky content locked down. | 18:04 |
| edleafe | dansmith: the gabbi tests show that exact case | 18:05 |
| edmondsw | we've been putting up commits for review, but until the spec is approved hard to ask anyone to look at functional commits | 18:05 |
| melwitt | edmondsw: has it been previously approved? I don't know the history about it | 18:05 |
| edmondsw | melwitt yes, and the history is all in the spec at the end | 18:05 |
| dansmith | edleafe: yeah, I know, that's why I'm trying to diff what I'm doing with those | 18:05 |
| edmondsw | melwitt it's a multirelease effort | 18:06 |
| edmondsw | melwitt since it was considered too much to bite off in one release from a reviewer bandwidth perspective | 18:06 |
| dansmith | edleafe: you don't have a case where you only have one thing in member_of that actually returns results, but I shaved one down and it still seems to work (although it's hard to know if it's returning the thing we expect | 18:06 |
| edmondsw | melwitt https://review.openstack.org/#/c/545111/ | 18:06 |
| *** salv-orlando has joined #openstack-nova | 18:07 | |
| artom | So, do we handle *any* NUMA stuff during live migration? I know hugepages and CPU pinning aren't, but what about plain old NUMA topology? | 18:08 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_access_get_by_flavor_id db api method https://review.openstack.org/539720 | 18:09 |
| melwitt | edmondsw: okay, probably just needs some reminding then. looks like mriedem has reviewed it last month so maybe he can take another look at it soon | 18:09 |
| edleafe | dansmith: you could inspect the returned a-c, but that's always too complicated to write in gabbi | 18:09 |
| dansmith | edleafe: yeah, I'm doing that in my real test and getting back [], but I dunno how to do much with the gabbit | 18:09 |
| cdent | edleafe: untrue, either make the test verbose: True or make it fail intentionally in the response_json_paths | 18:09 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_destroy db api method https://review.openstack.org/539729 | 18:09 |
| *** mgoddard has quit IRC | 18:09 | |
| dansmith | but it's checking the length of allocation_requests | 18:10 |
| cdent | and the error message will show the full response | 18:10 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_get_by_flavor_id db api method https://review.openstack.org/539733 | 18:10 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_get_by_name db api method https://review.openstack.org/544379 | 18:10 |
| dansmith | cdent: ah, duh, trying | 18:10 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_get db api method https://review.openstack.org/544621 | 18:10 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_get_all db api method https://review.openstack.org/544688 | 18:10 |
| openstackgerrit | Merged openstack/nova master: Remove old flavor_create db api method https://review.openstack.org/544703 | 18:11 |
| openstackgerrit | Merged openstack/python-novaclient master: Fix local test fails with pypy https://review.openstack.org/553426 | 18:11 |
| dansmith | cdent: I tried just setting the asserted length to something wrong, but it doesn't show me anything else other than 17 != 1 | 18:11 |
| cdent | dansmith: yeah, I meant something more breaking that that: | 18:12 |
| cdent | response_json_paths:\n $: foo | 18:12 |
| cdent | will try to compre the whole json object to foo | 18:12 |
| * dansmith uses larger hammer | 18:12 | |
| *** sambetts is now known as sambetts|afk | 18:13 | |
| *** lpetrut has joined #openstack-nova | 18:13 | |
| *** AlexeyAbashkin has joined #openstack-nova | 18:14 | |
| vivsoni__ | In case of 'nova create' - NO cinder API is called | 18:14 |
| vivsoni__ | In case of 'nova delete' - NO cinder API is called | 18:14 |
| *** AlexeyAbashkin has quit IRC | 18:14 | |
| vivsoni__ | Hi Team, please correct if my understanding is wrong | 18:15 |
| mriedem | if you're booting from volume then of course cinder is called | 18:15 |
| mriedem | if you've attached volumes before you delete the instance, of course cinder is called | 18:15 |
| *** tbachman has joined #openstack-nova | 18:15 | |
| vivsoni__ | mriedem: ok | 18:15 |
| *** AlexeyAbashkin has joined #openstack-nova | 18:15 | |
| *** dtantsur is now known as dtantsur|afk | 18:16 | |
| edleafe | cdent: /me learns new gabbi trick | 18:16 |
| dansmith | cdent: edleafe: yeah, well, that clearly shows it's working as expected | 18:16 |
| dansmith | so I be stumped | 18:16 |
| dansmith | if I don't pass member_of, I get back the things I expect | 18:17 |
| vivsoni__ | mriedem: so if my instance is attached to volume, then if i shutoff the nova instance and restart the instance... then cinder api of detach and attach is called is it ? | 18:17 |
| dansmith | interestingly, the request logging from placement is urlencoded from inside my functional test, but not when the gabbit runs | 18:17 |
| dansmith | so I wonder if it's getting more than the uuid | 18:18 |
| mriedem | vivsoni__: no, if you're just stopping and starting the instance, nova doesn't detach the volume | 18:18 |
| *** jpena is now known as jpena|away | 18:18 | |
| *** suresh12_ has quit IRC | 18:18 | |
| edleafe | dansmith: are you sending a single uuid string, or a 1-element list? | 18:19 |
| *** suresh12 has joined #openstack-nova | 18:19 | |
| dansmith | edleafe: I've tried both, initially just one, then tried in:$uuid | 18:19 |
| vivsoni__ | mriedem: ok... i wanted to understand more on 'nova create/delete/live-migration' w.r.t cinder api call.. do you have some reference link, if yes, please share | 18:20 |
| * edleafe is grasping at straws | 18:20 | |
| mriedem | vivsoni__: not really | 18:21 |
| mriedem | vivsoni__: hemna has some diagrams of those flows though i think | 18:21 |
| mriedem | not sure if they are published | 18:21 |
| vivsoni__ | ok | 18:22 |
| dansmith | edleafe: yeah I know | 18:22 |
| dansmith | edleafe: I printed member_of from inside the normalize function and it is properly [$uuid] | 18:22 |
| *** harlowja has joined #openstack-nova | 18:23 | |
| cdent | dansmith, edleafe: any chance that this is the result of some base database condition that is different between the dan tests and the ed tests? | 18:23 |
| dansmith | cdent: such as what? | 18:23 |
| dansmith | cdent: I can get /resource_providers/$rp/aggregates and I get back $uuid | 18:24 |
| dansmith | so I feel like placement is working and storing the association I made | 18:24 |
| dansmith | ohhhhhhh | 18:25 |
| dansmith | I may be completely stupid | 18:25 |
| dansmith | maybe | 18:25 |
| * edleafe bites tongue | 18:25 | |
| dansmith | edleafe: I deserve whatever was coming before you bit your tongue for this one | 18:27 |
| *** vivsoni__ has quit IRC | 18:28 | |
| dansmith | good news, member_of seems to work.. EOM | 18:29 |
| efried | Well now you gotta tell us what you were doing wrong. | 18:29 |
| efried | Not just so we can make fun of you - although that too. | 18:29 |
| efried | But mainly so we can avoid similar pratfalls ourselves. | 18:29 |
| efried | And therefore avoid being made fun of in turn. | 18:29 |
| edleafe | uh, yeah - for *education* | 18:30 |
| *** gyee has quit IRC | 18:30 | |
| dansmith | I had a method that would create and add hosts to aggregates, both in nova and placement | 18:31 |
| dansmith | and the nova call I was making is additive, where the placement one is declarative | 18:31 |
| dansmith | so I was putting multiple hosts in the nova aggregates, but only in the latest placement one | 18:31 |
| dansmith | and the last placement one I created was "no-hosts" | 18:31 |
| dansmith | so, you know.. | 18:31 |
| dansmith | and the order in which I was printing debug stuff was consistent, but.. not the full story | 18:32 |
| *** dave-mccowan has quit IRC | 18:32 | |
| *** yamamoto has joined #openstack-nova | 18:32 | |
| edleafe | wish I could say I | 18:33 |
| mriedem | https://www.youtube.com/watch?v=AMQ8E3mTgY0 | 18:34 |
| edleafe | I've never done something similar | 18:34 |
| dansmith | mriedem: yeah I know, but thanks for putting into visual form | 18:34 |
| mriedem | ha | 18:34 |
| mriedem | ask smcginnis or jungleboyj, i was bashing my head for about 2 hours one morning over a unit test i was writing and couldn't figure out why it didn't work | 18:35 |
| mriedem | can't remember what it was, but it was really dumb | 18:35 |
| *** fragatina has joined #openstack-nova | 18:35 | |
| edleafe | cdent: ok, given the output using your trick: http://paste.openstack.org/show/706443/, how would I reference the resource provider uuid within the response_json_paths in order to compare it to the cn1uuid? | 18:36 |
| *** yamamoto has quit IRC | 18:37 | |
| *** tssurya has quit IRC | 18:37 | |
| cdent | edleafe: reading | 18:38 |
| smcginnis | mriedem: What what? | 18:38 |
| mriedem | smcginnis: last time i was at claddagh and we were upstairs, i was trying to get a gd unit test working most of the morning | 18:40 |
| mriedem | maybe i was the only one that heard my swearing | 18:40 |
| cdent | edleafe: you can to check that the cn1uuid is in there, which in there? | 18:41 |
| smcginnis | Oh yeah, I remember now. :) | 18:41 |
| cdent | s/can/want/? | 18:41 |
| edleafe | cdent: dunno, just thought that checking that it's in there wasn't definitive enough. Getting allocation_requests[0].keys()[0] or something like that | 18:42 |
| melwitt | I'm able to repro the EndpointNotFound problem in an old devstack I have, and what I get from ksa_adapter.get_endpoint() in both the non-list and interface list cases is: "*** EmptyCatalog: The service catalog is empty." | 18:42 |
| melwitt | even though I have glance in 'openstack endpoint list' | 18:43 |
| cdent | edleafe: if you know the uuid in advance, it's generally easier to check for some key down its path. you can do environ and response expansions within the left hand side of a json path thing | 18:44 |
| *** AlexeyAbashkin has quit IRC | 18:44 | |
| efried | melwitt: Can you show me your openstack endpoint list for glance? | 18:45 |
| efried | or 'show' would be better | 18:45 |
| *** andreas_s has joined #openstack-nova | 18:45 | |
| *** dave-mccowan has joined #openstack-nova | 18:46 | |
| * edleafe tries to translate what cdent just typed | 18:46 | |
| efried | melwitt: The ksa_adapter is going to be narrowed down based on (possibly defaulted) config options. So as Matt was saying earlier, if for example your catalog has the admin endpoint, but your conf has (possibly by defaulting) internal and public, you'll get that EmptyCatalog/EndpointNotFound when you ask the adapter for an endpoint. | 18:46 |
| cdent | edleafe, sorry, I'm doing the usual too many things at once and then writing two thoughts down in one sentence | 18:47 |
| melwitt | efried: okay, sec | 18:47 |
| cdent | edleafe: first example after the heading: https://gabbi.readthedocs.io/en/latest/jsonpath.html#substitution | 18:47 |
| cdent | (well, only example) | 18:47 |
| edleafe | cdent: So given the example paste, what would I put in the left side? IOW, what is "nested.structure"? | 18:49 |
| melwitt | efried: http://paste.openstack.org/show/706465/ | 18:50 |
| *** andreas_s has quit IRC | 18:50 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: PowerVM Driver: vSCSI volume driver https://review.openstack.org/526094 | 18:50 |
| efried | melwitt: And the glance section of the conf? | 18:50 |
| melwitt | no [glance] section, all default | 18:51 |
| efried | melwitt: ...and the version document from the endpoint? (curl http://127.0.0.1/image) | 18:51 |
| melwitt | http://paste.openstack.org/show/706468/ | 18:51 |
| cdent | edleafe: $.allocation_requests[0].allocations.["$ENVIRON['CN1']"].resources.DISK_GB: 100 | 18:52 |
| cdent | I think jay did this somewhere, will find example | 18:52 |
| cdent | edleafe: $.allocation_requests..allocations["$ENVIRON['SS_UUID']"].resources[DISK_GB]: [100, 100] in the allocation-candidates.yaml | 18:53 |
| efried | melwitt: Yup, that ought to work. | 18:53 |
| efried | melwitt: You said you were using and old stack? What version of ksa, and what commit of nova? | 18:54 |
| melwitt | stepping through the code in pdb, when I get to keystoneauth1/access/service_catalog.py(362)endpoint_data_for() the self._catalog is an empty list [], even for the 'public' interface | 18:55 |
| melwitt | that's where it raises exceptions.EmptyCatalog('The service catalog is empty.') | 18:55 |
| melwitt | sec | 18:55 |
| edleafe | cdent: ok, thanks - that helps. | 18:55 |
| melwitt | keystoneauth1==3.4.0 | 18:56 |
| melwitt | commit a5a569d6670c29f995b1e8a2a2013471d57469d7 of nova | 18:56 |
| melwitt | it raises exceptions.EmptyCatalog('The service catalog is empty.') for each of 'internal' and 'public' | 18:57 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Make get_allocation_candidates() honor aggregate restrictions https://review.openstack.org/547990 | 18:57 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Add require_tenant_aggregate request filter https://review.openstack.org/545002 | 18:57 |
| openstackgerrit | Dan Smith proposed openstack/nova master: WIP: Honor availability_zone hint via placement https://review.openstack.org/546282 | 18:57 |
| dansmith | edleafe: your stuff in action: https://review.openstack.org/#/c/545002/12/nova/tests/functional/test_aggregates.py | 18:57 |
| dansmith | L224 specifically | 18:58 |
| efried | melwitt: Is that ocata? | 18:58 |
| melwitt | no? | 18:59 |
| efried | sorry, trying the wrong way to find where that is chronologically :) | 19:00 |
| melwitt | master from Feb 12, the commit I linked is dated Feb 12 | 19:00 |
| melwitt | it might have probably merged later | 19:00 |
| efried | melwitt: Is this code path using a RequestContext? | 19:00 |
| melwitt | but it's from around that time | 19:00 |
| melwitt | yes, it's a RequestContext | 19:01 |
| efried | What's in the RequestContext.service_catalog? | 19:01 |
| melwitt | it's empty ... though I think that's my fault, I created an admin context to call glance.api_servers | 19:03 |
| *** dave-mccowan has quit IRC | 19:03 | |
| *** AlexeyAbashkin has joined #openstack-nova | 19:03 | |
| efried | Well, unless I'm mistaken, the get_endpoint() stuff winds up in whatever context you're using. So that would splain why it's empty in your pdb. | 19:05 |
| melwitt | is the RequestContext.service_catalog supposed to get populated as a result of the get_endpoint() call? or is it supposed to be pre-populated before the get_endpoint call? | 19:06 |
| *** amoralej is now known as amoralej|off | 19:06 | |
| efried | The latter | 19:06 |
| *** avolkov has quit IRC | 19:06 | |
| melwitt | okay, yeah, then I've messed up this attempt to repro the problem | 19:07 |
| efried | get_endpoint gets its information *from* the context. | 19:07 |
| *** maciejjozefczyk has quit IRC | 19:07 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: WIP: Parse placement forbidden traits query string https://review.openstack.org/554665 | 19:08 |
| openstackgerrit | Giridhar Jayavelu proposed openstack/nova-specs master: VMware: place instances on resource pool https://review.openstack.org/549067 | 19:08 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: stop lying to the RT when ironic is down https://review.openstack.org/545479 | 19:13 |
| jroll | whee, that should be good now | 19:13 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | 19:13 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Preserve multiattach flag when refreshing connection_info https://review.openstack.org/554667 | 19:13 |
| *** rmart04 has joined #openstack-nova | 19:15 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Preserve multiattach flag when refreshing connection_info https://review.openstack.org/554667 | 19:16 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use Queens UCA for nova-multiattach job https://review.openstack.org/554317 | 19:16 |
| *** rmart04 has quit IRC | 19:20 | |
| *** suresh12 has quit IRC | 19:21 | |
| melwitt | okay, repro'd again by actually creating an instance, doing a virsh shutdown, then waiting for nova-compute to stop the instance, same error and again RequestContext.service_catalog = [] (I logged it from nova/image/glance.py) | 19:24 |
| melwitt | so somewhere along the way, the context we pass isn't one with a properly populated service_catalog | 19:25 |
| *** psachin has joined #openstack-nova | 19:28 | |
| *** _ix has quit IRC | 19:28 | |
| *** awaugama has quit IRC | 19:29 | |
| melwitt | uh oh, I think I know why. this path is being run through a periodic task which has been given an anonymous get_admin_context(), and when we eventually try to do something with a service_catalog in the RequestContext, there isn't going to be one | 19:31 |
| efried | melwitt: That'd do it. Lemme dig up the patch that did the context-y stuff... | 19:32 |
| efried | melwitt: https://review.openstack.org/#/c/490057/ | 19:32 |
| melwitt | I'm not sure what the answer is here, other than gibi looking into whether we can avoid relying on service catalog stuff for sending notifications | 19:32 |
| efried | melwitt: That patch ought to help us isolate which context is giving us grief here. | 19:33 |
| mriedem | nova doesn't have admin creds to glance like we do for cinder and neutron, so you can't rely on that in a periodic either | 19:33 |
| melwitt | because the context we use in periodic tasks is going to be a mostly empty admin one intended to read the database, etc. it's not going to have service catalog info in it | 19:33 |
| *** suresh12 has joined #openstack-nova | 19:33 | |
| melwitt | oh, okay. so we are ok for cinder and neutron then | 19:33 |
| *** _ix has joined #openstack-nova | 19:33 | |
| mriedem | if properly configured | 19:33 |
| mriedem | you have to configure nova to talk to neutron with an admin role token for port binding, | 19:34 |
| efried | Mm, yeah, it's coming back to me. I don't remember where else we've seen this, but the answer was: if you want this to work, you have to supply creds in the conf so that we can build a proper admin context. | 19:34 |
| mriedem | the cinder <> admin config was added in queens for a related bug with periodic tasks | 19:34 |
| mriedem | doing things with volumes | 19:34 |
| *** yamamoto has joined #openstack-nova | 19:34 | |
| efried | Yeah, what mriedem said. | 19:34 |
| melwitt | gotcha, okay | 19:34 |
| efried | And... is that okay? | 19:34 |
| efried | or do we need to "fix" it? | 19:34 |
| mriedem | for notifications, we really shouldn't have to hit a REST API every time we send a notification, because that's kind of crazy | 19:35 |
| openstackgerrit | sean mooney proposed openstack/nova master: add mtu to libvirt xml for ethernet and bridge types https://review.openstack.org/553072 | 19:35 |
| efried | Yeah, I'll agree with that. Probably the first thing to look into, then - why we need to talk to glance to send a notification. | 19:35 |
| mriedem | i've mentioned this before, | 19:35 |
| melwitt | yeah, gibi said he's going to investigate that tomorrrow | 19:35 |
| mriedem | but whenever we construct a glance "client" object in-tree, it goes thorugh the 'get endpoint url' stuff | 19:36 |
| mriedem | notifications goes through info_from_instance to build a payload, | 19:36 |
| mriedem | which gets an image ref URL | 19:36 |
| efried | mriedem: Yeah, I remember you mentioning it was doing it like 4000 times in a devstack run, or something. | 19:36 |
| *** dave-mccowan has joined #openstack-nova | 19:36 | |
| mriedem | efried: yup | 19:36 |
| efried | s/devstack/tempest/ | 19:36 |
| mriedem | we f'ing love to hit glance | 19:36 |
| *** d34dh0r53 has quit IRC | 19:37 | |
| mriedem | if [glance]/api_servers is set, we just build a static string based on that | 19:37 |
| sean-k-mooney[m] | mriedem: so the api call to glance when sending the notificaiton is from creating the glance client object? | 19:37 |
| *** d34dh0r53 has joined #openstack-nova | 19:37 | |
| mriedem | but if it's not set, we go through the ksa magik | 19:37 |
| efried | Which *should* be using cached values, I thought. | 19:37 |
| efried | i.e. we're not actually hitting the API 4000 times. | 19:37 |
| mriedem | maybe it is | 19:37 |
| mriedem | that would be nice to know | 19:38 |
| efried | mriedem: Yeah, we build the adapter every time, but the session & auth are cached. | 19:38 |
| sean-k-mooney[m] | efried: well if the call to the api is coming from createing the client object perhaps we should be storing that between requests so that we dont have to keep creating it | 19:39 |
| *** yamamoto has quit IRC | 19:39 | |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs to instance_extra https://review.openstack.org/537897 | 19:39 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | 19:39 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | 19:39 |
| openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | 19:39 |
| mriedem | sean-k-mooney[m]: efried is saying we're not hitting the api every time | 19:40 |
| mriedem | just the first time to get the service catalog entry | 19:40 |
| mriedem | and it's the identity api in this case | 19:40 |
| sean-k-mooney | mriedem: yes but we are still creating the client 4000 and hitting the cache 3999 times right? | 19:40 |
| mriedem | alternatively, don't put an image ref url in the notification payload, just the image uuid | 19:40 |
| mriedem | sean-k-mooney[m]: yeah | 19:40 |
| mriedem | the notification payload attempts to mimic the GET /servers/detail API which returns the image id and bookmark link to the image | 19:41 |
| sean-k-mooney | so if we create teh client once and resuit we definetlly dont hit the api and we get rid of 3999 calls to the client constructor? | 19:41 |
| mriedem | using the same thing we're hitting here | 19:41 |
| *** tssurya has joined #openstack-nova | 19:42 | |
| mriedem | sure, but that's lower priority atm | 19:42 |
| mriedem | the thing now is how to fix this periodic | 19:42 |
| efried | It'd be interesting to see if I can cache the adapter and stuff will still work. | 19:42 |
| sean-k-mooney | mriedem: image uuid might be better in general then the url | 19:43 |
| mriedem | sean-k-mooney: at least for the notification yeah i agree | 19:43 |
| mriedem | we can change that with versioned notifications, but this also gets shoved into the legacy notifications and changing those is like breaking an api | 19:43 |
| sean-k-mooney | glance can have multiple image urls correct? is there any guarntee the consumer of the url can reach that backend | 19:43 |
| mriedem | well, it's over rpc so if you've configured nova to hit internal glance api endpoints, your notification consumer probably can too | 19:44 |
| *** psachin has quit IRC | 19:44 | |
| mriedem | if that consumer actually needs to get the image details, idk | 19:44 |
| mriedem | or why the consumer can't just take the image id and form it's own image api request, ... | 19:45 |
| sean-k-mooney | mriedem: sure but with the uuid they can query for which ever one they actully need so that is more generally useful i think | 19:45 |
| mriedem | it's not like we return links to volumes and ports in the notification either | 19:45 |
| mriedem | sean-k-mooney: i agree | 19:45 |
| melwitt | yeah, the only other thing I can think of for fixing the periodic task is if we could somehow seed periodic tasks with the service catalog instead of the empty get_admin_context() one, but I don't know what would be involved there | 19:45 |
| melwitt | I guess it anyway won't be able to get any info from glance even if it had the catalog, it would just fail later, right? | 19:46 |
| mriedem | the notification path here isn't actually doing a GET to glance | 19:46 |
| efried | Yeah, I'm not worried about that bit at all. | 19:46 |
| mriedem | it's getting the service catalog from keystone via ksa | 19:46 |
| melwitt | meaning, it won't do that even if it has the catalog? | 19:46 |
| mriedem | but w/o a token | 19:46 |
| melwitt | okay | 19:47 |
| mriedem | so for now, the easiest thing to do probably do is in this notification code, handle EndpointNotFound and just set the image_ref_url to the image id | 19:47 |
| efried | are context hashable? | 19:47 |
| mriedem | they are serializable | 19:47 |
| dansmith | edleafe: hmm, so member_of with multiple aggregates is doing an "or" of all the ones you pass, right? is that what we really want? | 19:47 |
| openstackgerrit | Merged openstack/nova master: [libvirt] Add _get_XXXpin_cpuset() https://review.openstack.org/527631 | 19:47 |
| efried | so probably | 19:47 |
| mriedem | efried: i'd rather not do crazy cache shit with contexts | 19:48 |
| openstackgerrit | Merged openstack/nova master: api-ref: add a note in DELETE /os-services about deleting computes https://review.openstack.org/553598 | 19:48 |
| *** psachin has joined #openstack-nova | 19:48 | |
| mriedem | we already have some craziness with contexts and periodics today pulling off the local thread storage | 19:48 |
| mriedem | which totally effs with request id log tracing | 19:48 |
| efried | mriedem: I was just going to experiment, cool yer jets. | 19:48 |
| mriedem | i will not cool my jets or hold my horses | 19:48 |
| melwitt | slow your roll | 19:48 |
| mriedem | this is the part that blows up https://github.com/openstack/nova/blob/master/nova/notifications/base.py#L397 | 19:48 |
| * efried leafs back through the logs for assless chaps references | 19:48 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Use nova.db.api directly https://review.openstack.org/543262 | 19:48 |
| sean-k-mooney | mriedem: is instance.image_ref the image id? | 19:49 |
| melwitt | gdi I tried to set the importance at the same time as mriedem and set it differently. will set it back | 19:51 |
| *** AlexeyAbashkin has quit IRC | 19:51 | |
| *** liverpooler has joined #openstack-nova | 19:51 | |
| *** salv-orlando has quit IRC | 19:51 | |
| *** salv-orlando has joined #openstack-nova | 19:52 | |
| openstackgerrit | Eric Young proposed openstack/nova master: Support extending attached ScaleIO volumes https://review.openstack.org/554679 | 19:53 |
| mriedem | sean-k-mooney: yes | 19:54 |
| melwitt | mriedem, efried: are one of you going to propose the patch or shall I? | 19:54 |
| mriedem | i can push a patch | 19:54 |
| melwitt | k | 19:54 |
| mriedem | if LP would not timeout on me | 19:54 |
| *** jackie-truong has quit IRC | 19:54 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Move configurable mkfs to privsep. https://review.openstack.org/551921 | 19:55 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi xenstore_read's to privsep. https://review.openstack.org/552241 | 19:55 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi disk resizing to privsep. https://review.openstack.org/552242 | 19:55 |
| openstackgerrit | Michael Still proposed openstack/nova master: Sync xenapi and libvirt on what flags to pass e2fsck. https://review.openstack.org/554078 | 19:55 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi partition copies to privsep. https://review.openstack.org/553605 | 19:55 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move image conversion to privsep. https://review.openstack.org/554437 | 19:55 |
| openstackgerrit | Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | 19:55 |
| openstackgerrit | Michael Still proposed openstack/nova master: We don't need utils.trycmd any more. https://review.openstack.org/554439 | 19:55 |
| *** tesseract has quit IRC | 19:56 | |
| *** salv-orlando has quit IRC | 19:56 | |
| sean-k-mooney | mriedem: one other tought. you said if you get an endpoint not found you would set image_ref_url to the image id. any reason to not always use the image id? | 19:56 |
| mriedem | sean-k-mooney: i said above, it changes the api | 19:57 |
| mriedem | in this case, meh | 19:57 |
| mriedem | we can change the payload for the versioned notifications later, | 19:57 |
| mriedem | but this also goes in the legacy notifications | 19:57 |
| jaypipes | guh, this day turned into a giant disaster. | 19:57 |
| sean-k-mooney | mriedem: yes but we could have a microverion for that no? oh this is an unversioned notification | 19:57 |
| mriedem | jaypipes: i have a photo that might make your day better | 19:57 |
| efried | jaypipes: Sokay, I still haven't gotten through those two patches yet. | 19:57 |
| jaypipes | efried: no worries, duder. today's pretty much a goner for me. | 19:57 |
| sean-k-mooney | jaypipes: how did the dentist go. | 19:58 |
| jaypipes | efried: suffice to say it involves me cleaning up a giant pile of dog shit in the back of my car in the rain. | 19:58 |
| jaypipes | efried: ^ not related to the dentist | 19:58 |
| mriedem | but at least your teeth are clean | 19:58 |
| efried | jaypipes: Geez, you try to do something nice for the dog... | 19:58 |
| jaypipes | indeed. | 19:58 |
| mriedem | unless there was...splatter | 19:58 |
| sean-k-mooney | mriedem: ew, let hop not for jaypipes sake | 19:59 |
| efried | jaypipes: You want a quick feeling of satisfaction, https://review.openstack.org/#/c/545111/ ought to be eligible for quick-approve :) | 19:59 |
| sean-k-mooney | *hope | 19:59 |
| mriedem | efried: i'll take a look again at that after this | 19:59 |
| jaypipes | my dogs have become a nested provider tree with inventory of DOG_SHIT_KG | 19:59 |
| efried | rofl | 20:00 |
| mriedem | CUSTOM_DOG_SHIT_KG? | 20:00 |
| tssurya | lol | 20:00 |
| jaypipes | yes, sorry. | 20:00 |
| jaypipes | well corrected, mriedem | 20:00 |
| mriedem | :) | 20:00 |
| sean-k-mooney | mriedem: i dont know me might want to standarise it | 20:00 |
| jaypipes | sean-k-mooney: os-dog-poop? | 20:01 |
| jaypipes | though that's a big specific. os-poop would be more generic and future proof. | 20:01 |
| efried | Well, we already have os-brick | 20:01 |
| jaypipes | well played efried | 20:01 |
| sean-k-mooney | jaypipes: you know im surprised you have not come up with an os-pug yet | 20:01 |
| *** sar has joined #openstack-nova | 20:01 | |
| *** masber has joined #openstack-nova | 20:02 | |
| jaypipes | sean-k-mooney: Provider Usage Group. | 20:03 |
| * jaypipes gets to work... | 20:03 | |
| *** dave-mccowan has quit IRC | 20:03 | |
| *** artom has quit IRC | 20:03 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Don't log a warning for InstanceNotFound with deleted VIFs https://review.openstack.org/554591 | 20:03 |
| *** dave-mccowan has joined #openstack-nova | 20:04 | |
| dansmith | jaypipes: loaded question for you in here: https://review.openstack.org/#/c/544694/2/specs/rocky/approved/alloc-candidates-member-of.rst | 20:07 |
| mriedem | efried: edmondsw: question inline about evacuate https://review.openstack.org/#/c/545111/ | 20:07 |
| *** psachin has quit IRC | 20:08 | |
| *** psachin has joined #openstack-nova | 20:09 | |
| *** germs has joined #openstack-nova | 20:10 | |
| *** germs has quit IRC | 20:10 | |
| *** germs has joined #openstack-nova | 20:10 | |
| edmondsw | mriedem looking | 20:10 |
| *** suresh12 has quit IRC | 20:10 | |
| *** suresh12 has joined #openstack-nova | 20:10 | |
| mriedem | also, | 20:10 |
| mriedem | at some point, we should not have powervm specs for 'implement random feature parity stuff in our driver', | 20:10 |
| mriedem | just do what other virt drivers do and have specless feature parity blueprints per feature in questoin | 20:11 |
| sean-k-mooney | mriedem: for https://review.openstack.org/554591 are the network-vif-deleted event generated by neutron port being deleted as part of instance deleteion or are they neutron vif unplugged events form the ports being removed from ovs? | 20:11 |
| *** suresh12 has quit IRC | 20:11 | |
| efried | mriedem: That would be lovely. Are we at that point yet? | 20:11 |
| *** suresh12 has joined #openstack-nova | 20:11 | |
| mriedem | efried: i'm ok with that if melwitt is | 20:11 |
| mriedem | sean-k-mooney: the former | 20:12 |
| melwitt | mriedem, efried: sounds fine to me | 20:12 |
| efried | Cool beans. edmondsw esberglu ^. | 20:12 |
| sean-k-mooney | mriedem: ok then ya that makes sense to me. i just was not sure where the event was comming from | 20:12 |
| efried | mriedem, melwitt: But for this release, since we're already here, use the bp/spec that's proposed, yah? | 20:13 |
| mriedem | shrug, i personally don't care for the wishlist | 20:13 |
| mriedem | because it just seems really random | 20:13 |
| melwitt | is the proposed spec all just feature parity stuff? | 20:13 |
| *** germs has quit IRC | 20:13 | |
| mriedem | and i see adding support for hot plugging vifs was added after PS3 | 20:14 |
| efried | melwitt: yes | 20:14 |
| edmondsw | I'm happy to abandon this and start using specless blueprints this release | 20:14 |
| efried | ++ | 20:14 |
| mriedem | there are no actual design details in the spec, so it's not really useful imo | 20:14 |
| sean-k-mooney | mriedem: so just to follow on from that, does that mean if i delete a neutron port that is bound to an instance that automatically results in a notifcation to nova to call detach interface on the virt driver? | 20:14 |
| *** psachin has quit IRC | 20:14 | |
| mriedem | so specless bp is fine | 20:14 |
| edmondsw | yep | 20:14 |
| melwitt | sounds reasonable to me | 20:14 |
| mriedem | sean-k-mooney: yup | 20:14 |
| efried | Sweet. Swat that paperwork down! | 20:15 |
| sean-k-mooney | mriedem: hum ok is that documented anywhere. it kind of makes sense i just would not have expected neutron to allow you to delete teh port in that case | 20:15 |
| *** awaugama has joined #openstack-nova | 20:15 | |
| edmondsw | mriedem to the evacuate question... I'm honestly not sure if there's anything more to it than a) supporting moving to another host in general and b) updating the support matrix | 20:16 |
| jaypipes | dansmith: will try to answer it first thing in the morning... been a day. having a beer. | 20:16 |
| dansmith | jaypipes: oh you said you were getting to work, so I figured you were around | 20:16 |
| dansmith | mah bad, tomorrow is fine | 20:16 |
| melwitt | getting to work cleaning dog poodoo | 20:17 |
| mriedem | edmondsw: someone on your dev or qa team could actually test it first | 20:18 |
| mriedem | to flush out any obvious problems | 20:18 |
| edmondsw | mriedem oh definitely | 20:18 |
| edmondsw | was just talking from a code perspective | 20:18 |
| mriedem | sean-k-mooney: good question; i didn't trace the requests in the tempest job to see if the port was actually detached before it was deleted, or what the order was there, | 20:18 |
| mriedem | sean-k-mooney: should be pretty easy to find out - create a server from a pre-existing port and try to delete the port | 20:19 |
| mriedem | i don't have a devstack handy | 20:19 |
| mriedem | i know you can't delete an attached volume | 20:19 |
| sean-k-mooney | mriedem: ill give it a try i think i have an env running | 20:19 |
| *** tidwellr has joined #openstack-nova | 20:21 | |
| *** tidwellr has left #openstack-nova | 20:22 | |
| openstackgerrit | Eric Berglund proposed openstack/nova master: DNM: EXPERIMENTAL: Set proc_units_factor to 0.1 in VMBuilder init https://review.openstack.org/554688 | 20:23 |
| sean-k-mooney | mriedem: so ya i can create the port, boot with it then delete it and the interface gets detatched from the running vm | 20:23 |
| sean-k-mooney | mriedem: im just going to check the xml to confirm | 20:23 |
| sean-k-mooney | mriedem: yep the vm no has no nics | 20:24 |
| sean-k-mooney | i wonder if the revers works. can i create a neutron port and then set its owner to a vm and trigger an attache | 20:25 |
| cfriesen | So in the discussion around https://review.openstack.org/#/c/552924/ something interesting has come up...currently vcpus and 4KB pages can be consumed on a compute-node bases (where it floats across the whole compute node) or on a numa-node basis (where it's constrained to a single numa node) depending on whether the instance has a numa topology or not. (https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver. | 20:27 |
| cfriesen | py#L4435) This makes it hard to reason about resource tracking. | 20:27 |
| *** priteau has joined #openstack-nova | 20:29 | |
| mriedem | sean-k-mooney: no the reverse doesn't work, | 20:30 |
| mriedem | not might get a network-vif-plugged event but it won't know what to do with it and just log a warning | 20:31 |
| mriedem | *nova | 20:31 |
| *** esberglu has quit IRC | 20:31 | |
| *** dave-mccowan has quit IRC | 20:31 | |
| sean-k-mooney | mriedem: ya just tried it. | 20:31 |
| mriedem | the fact you can delete a port while it's attached is pretty scary | 20:31 |
| sean-k-mooney | i create another port and did openstack port set --device 55a08a3b-fd27-495f-85da-39eedd5fde9c --device-owner "compute:nova" --host "ubuntu" myport | 20:31 |
| *** eharney has quit IRC | 20:31 | |
| sean-k-mooney | it ended up binding the port on the host correctly | 20:32 |
| sean-k-mooney | im going to do a hard reboot but i think it will then update the xml correctly | 20:32 |
| mriedem | in the compute api we check instance states when detaching a port, | 20:32 |
| edleafe | dansmith: sorry, was away. Yes, the idea for member_of was "member of at least one of these". The typical use case would be only 1 agg. | 20:32 |
| mriedem | but the networking api wouldn't have that | 20:32 |
| mriedem | so you could detach ports by deleting them while an instance is migrating | 20:32 |
| *** masber has quit IRC | 20:32 | |
| mriedem | which seems, not good | 20:33 |
| *** yassine has joined #openstack-nova | 20:33 | |
| *** openstackgerrit has quit IRC | 20:33 | |
| cfriesen | does it mess up our accounting? | 20:34 |
| *** esberglu has joined #openstack-nova | 20:34 | |
| cfriesen | why is "while it's migrating" different than "while it's running"? | 20:34 |
| *** yassine has quit IRC | 20:35 | |
| sean-k-mooney | mriedem: strange the hard reboot did not cause the interface to show up either. i must be missing something that the attach/detach whould have done | 20:35 |
| *** yamamoto has joined #openstack-nova | 20:35 | |
| dansmith | edleafe: yeah, so that's kinda what I was thinking at first, but then I tried to apply it to two things at once, so: | 20:36 |
| dansmith | edleafe: imagine I have a tenant-restricted aggregate and the user boots with a requested AZ | 20:36 |
| dansmith | edleafe: if I just concatenate the tenant and az list, then I won't restrict them properly | 20:37 |
| efried | cfriesen: Are you suggesting that somewhere in the stack there will be code responsible for translating the allocation_request for [PCPU:2 from NUMA_RP_1] to [PCPU:2 from NUMA_RP_1 + VCPU:2 from compute node RP] ?? | 20:37 |
| mriedem | cfriesen: it was an example of a bad time to detach a port | 20:37 |
| cfriesen | efried: I'm suggesting that our logic right now is not rigorous. :) | 20:37 |
| dansmith | edleafe: I can restrict the tenant aggs to only the one that matches the az, but they might not be the same size (and thus are different) | 20:37 |
| sean-k-mooney | right we could have already caluated the new xml and miss the port is being removed | 20:38 |
| mriedem | we do refresh the nw info cache when we get network-vif-deleted events | 20:38 |
| dansmith | edleafe: I think placement _could_ know whether one is a subset of the other, but I can't really | 20:38 |
| efried | dansmith, edleafe: Do we need member_of=all:... ? | 20:38 |
| edleafe | dansmith: yeah, that really isn't how placement aggs should work | 20:38 |
| dansmith | edleafe: so, just trying to figure out what to do there.. maybe we can have a hangout with jaypipes (et al) tomorrow after I have stewed on it a bit and make sure we're happy with what we've got | 20:38 |
| cfriesen | efried: but other than that, I think we had been talking about tracking PCPU entirely separately from VCPU (where PCPU is the number of "dedicated" pcpus we have, and VCPU is the number of VCPUs we can support) | 20:39 |
| edleafe | dansmith: we could add an 'and' case by leaving off the 'in:' operator | 20:39 |
| efried | I guess if pressed, I would have said we were implementing in: for parity with GET /rps, but wouldn't actually be using it. | 20:39 |
| dansmith | efried: well, I thought of that, but I'm not sure that's right either | 20:39 |
| cfriesen | efried: PCPUs are always consumed on a per-numa-node basis | 20:39 |
| cfriesen | efried: VCPUs can be consumed either per-numa-node or per-compute node, which is a problem for resource tracking. | 20:39 |
| efried | cfriesen: Just so. Which is why I was of the opinion that we should only be presenting that inventory from within the NUMA node RP. | 20:40 |
| dansmith | efried: if you have two groups of computes and both are legit for a tenant, then you don't want all | 20:40 |
| efried | dansmith: Yeah, I wasn't really up on the use cases for GET /a_c?member_of | 20:40 |
| dansmith | efried: almost want member_of=one:$agg1,$agg2&member_of=$agg3 | 20:41 |
| cfriesen | efried: but that doesn't line up with the current logic, where you can consume a VCPU (and 4KB pages) from the entire compute node | 20:41 |
| sean-k-mooney | mriedem: fyi http://paste.openstack.org/show/706642/ any guess why the "attach" by recreating the port and hard rebooting did not work? the port look identical to me bar the fact the status is down on the new port | 20:41 |
| efried | dansmith: But I knew we would want in:[any] for parity. | 20:41 |
| *** yamamoto has quit IRC | 20:41 | |
| dansmith | efried: edleafe: I'll try to come up with a few different hard examples to talk about and we can decide whether we care (or care right now) about them | 20:41 |
| efried | Yeah, sounds good. I mean, the bp is approved and completed, but heck, let's throw another one after it. | 20:42 |
| edleafe | dansmith: ok. If it's needed, it shouldn't be too big of a change | 20:42 |
| cfriesen | efried: and for various reasons it's really hard to track 4KB pages, since the host can consume them from either compute node at will. (Unless you run host stuff in a separate cgroup or container to provide hard limits.) | 20:42 |
| dansmith | edleafe: okay | 20:42 |
| efried | cfriesen: s/either compute node/either NUMA node/ ? | 20:42 |
| mriedem | sean-k-mooney: not really | 20:42 |
| cfriesen | efried: whoops,yes | 20:43 |
| cfriesen | efried: we try to account for memory fairly tightly on our compute nodes, and had to make 4KB pages use "preferred" rather than "strict" numa mempolicy because we were hitting the oom killer | 20:43 |
| efried | cfriesen: You could have your memory expressed in terms of resource class MEMORY_4K_PAGE. | 20:44 |
| efried | cfriesen: But I guess that's not the issue you're stuck on. | 20:44 |
| efried | cfriesen: You're concerned about being allowed to get those pages from separate NUMA nodes in cases where a) NUMA affinity is not strictly required; and b) you can't get 'em all from one. | 20:44 |
| efried | cfriesen: So what you can actually do here is express your resource requests as separate numbered groups. | 20:45 |
| cfriesen | efried: sort of. currently if you have an instance numa_topology nova will strictly constrain your instance to a single host numa node. | 20:45 |
| efried | cfriesen: It *sort of* requires knowing how many NUMA nodes are possible on a host. What's the max for that, anyway? 2? 4? | 20:46 |
| cfriesen | efried: and if you don't have an instance numa_topology then nova will let you float over the whole compute node | 20:46 |
| efried | cfriesen: I'm talking about in the idyllic future where we're doing all this with NRP and granular resource requests. | 20:46 |
| cfriesen | efried: common hardware is typically 2, some is 4. exotic hardware has many more. | 20:46 |
| efried | Okay, so if you care about strict affinity, you always ask for all your memory in one numbered request group. If you don't - if you want to allow spread - you ask for blocks in separate numbered request groups. | 20:47 |
| cfriesen | efried: first we need to figure out how we want to handle host NUMA affinity for instances with no numa_topology. | 20:47 |
| cfriesen | efried: because if we continue to allow it to consume VCPUs and 4KB pages from the whole compute node, then we can't track those per numa node | 20:48 |
| efried | Bear with me, then | 20:48 |
| efried | cfriesen: So let's say you want 4096 4K pages and 4 VCPUs. | 20:49 |
| efried | cfriesen: If you want strict affinity, you say resources1=VCPU:4,MEMORY_4K_PAGE:4096 | 20:50 |
| *** tssurya has quit IRC | 20:50 | |
| efried | cfriesen: But if you don't care about strict affinity... | 20:50 |
| efried | let's say you know your cloud doesn't have any hosts that support more than 4 NUMA nodes. | 20:50 |
| *** edmondsw has quit IRC | 20:51 | |
| *** tssurya has joined #openstack-nova | 20:51 | |
| *** felipemonteiro_ has joined #openstack-nova | 20:51 | |
| efried | Your flavor that doesn't need strict affinity could say resources1=MEMORY_4K_PAGE:1024&resources2=MEMORY_4K_PAGE:1024&resources3=MEMORY_4K_PAGE:1024&resources4=MEMORY_4K_PAGE:1024&resources5=VCPU:1&resources6=VCPU:1&resources7=VCPU:1&resources8=VCPU:1 | 20:52 |
| efried | Now, the results you get back can still include permutations where all of those resources come from the same RP. | 20:52 |
| efried | And if you're using a weigher, you can choose those first. | 20:52 |
| *** salv-orlando has joined #openstack-nova | 20:52 | |
| efried | But it'll also allow for permutations where the resources come from different RPs. | 20:52 |
| cfriesen | efried: the problem arises when we actually start up the qemu process. The memory is actually consumed on whatever numa node happens to request it...and now nova/placement no longer knows how much memory is available on each host numa node. | 20:52 |
| sean-k-mooney | efried: there is nothing in the api preventing you form creating a vm with 4 virtual numa nodes on a host with on 2 phyical numanodes | 20:52 |
| efried | cfriesen: That would be a libvirt fix. | 20:53 |
| efried | sean-k-mooney: Cool, that's useful. | 20:53 |
| sean-k-mooney | efried: the libvirt dirver does not allow that but the only guarentee that the api provides in this case is that if i request a vm with 1 numa node it will not span 2+ host numa nodes | 20:53 |
| cfriesen | efried: it's not a libvirt problem, it's how linux works. If you haven't overridden the numa affinity, then by default memory allocations occur on the numa node that you're on. | 20:54 |
| sean-k-mooney | cfriesen: not quite | 20:54 |
| efried | cfriesen: Then I don't understand how we ever wind up with memory coming from disparate NUMA nodes. | 20:54 |
| sean-k-mooney | the linux kernel prefers to but numactl makes the desision and it may allocated form a remote numa node in some cases | 20:54 |
| *** felipemonteiro has quit IRC | 20:54 | |
| *** tssurya has quit IRC | 20:55 | |
| sean-k-mooney | efried: a vm without hugepages can have its memory provided by any numa node | 20:55 |
| *** tssurya has joined #openstack-nova | 20:55 | |
| cfriesen | efried: if there is no instance numa topology, then nova doesn't specify any affinity. this means the qemu threads are free to float across the whole compute node. when they do a memory allocation they will by default be allocated memory from the numa node they're currently running on. | 20:55 |
| efried | But it's all provided by the same NUMA node? | 20:55 |
| cfriesen | efried: no | 20:55 |
| sean-k-mooney | efried: infact today we do not have a facility to enforce that it comes form the same numa node in nova | 20:55 |
| cfriesen | we explicitly let the host decide....we did for a while restrict it to a single numa node, then removed that for increased density | 20:56 |
| cfriesen | (we being nova) | 20:56 |
| efried | And just so I'm clear, you want that kind of VM to be able to run on the same node as the strictly-affinitized one? | 20:56 |
| efried | s/node/host/ | 20:56 |
| sean-k-mooney | cfriesen: did that ever land upstream. i dont think we ever did a release with that behavior for 4k pages | 20:57 |
| sean-k-mooney | efried: yes i dont see why not | 20:57 |
| *** salv-orlando has quit IRC | 20:57 | |
| *** kaisers has joined #openstack-nova | 20:57 | |
| efried | Well, y'all seem to be splainin why not. | 20:57 |
| sean-k-mooney | efried: the only way to request numa affined memory in openstack today is via hugepages | 20:57 |
| cfriesen | efried: same compute node, yes. there's a spec under review right now to support shared and dedicated vcpus on the same compute node, and we already support 4KB and 2MB page backing on the same compute node | 20:57 |
| cfriesen | it's just that right now the resource tracking is kind of messed up for 4KB pages | 20:58 |
| *** kaisers has quit IRC | 20:58 | |
| sean-k-mooney | cfriesen: yes 4k pages are not tracked in the numa topology blob so we cant numa afine them | 20:59 |
| sean-k-mooney | cfriesen: you can request 4k pages specifically but the is special case code the skips the numa suff for them | 20:59 |
| cfriesen | sean-k-mooney: aren't they part of mempages? | 20:59 |
| *** openstackgerrit has joined #openstack-nova | 21:00 | |
| openstackgerrit | Merged openstack/nova master: Always pass 'NUMACell.siblings' to _pack_instance_onto_cores' https://review.openstack.org/537364 | 21:00 |
| *** dave-mccowan has joined #openstack-nova | 21:00 | |
| sean-k-mooney | mempages? | 21:00 |
| cfriesen | NUMACell.mempages | 21:00 |
| sean-k-mooney | you can set hw:mem_page_size=4k but i dont think they are stored in NUMACell.mempages | 21:01 |
| sean-k-mooney | i guess i can check the db one seck | 21:01 |
| *** pchavva has quit IRC | 21:02 | |
| sean-k-mooney | that is what the numatopology blob looks like http://paste.openstack.org/show/706682/ | 21:04 |
| sean-k-mooney | cfriesen: so yes "nova_object.data": {"used": 0, "total": 2043576, "reserved": 0, "size_kb": 4} they are there | 21:05 |
| sean-k-mooney | cfriesen: they are actully tracked per numa node but we cant tie teh per numa node values back to the host memory_mb value simply today | 21:06 |
| *** suresh12 has quit IRC | 21:06 | |
| *** suresh12 has joined #openstack-nova | 21:06 | |
| cfriesen | I think we can, for ones that are strictly pinned. you just subtract the same value from both | 21:06 |
| cfriesen | but for floating ones we don't know the actual per-host-numa-node consumption | 21:07 |
| sean-k-mooney | i mean technically host memory_mb is the sum of all the cells 4k pages but im not sure if they will always agreee | 21:07 |
| *** jmlowe has quit IRC | 21:08 | |
| sean-k-mooney | cfriesen: the reall issue is the host reserved memory option. that is host wide and we jsut subtract it form the memory_mb value in the code but not sure how to translate that to per numa reserved values | 21:08 |
| mriedem | efried: melwitt: i reckon an upgrade release note will be in order for this endpoint not found thing with the legacy notification payload image_ref_url being an image id rather than a url | 21:09 |
| sean-k-mooney | cfriesen: that said we really need to deprecate it and replace it with a per numa version at some point | 21:09 |
| cfriesen | sean-k-mooney: agreed | 21:09 |
| efried | mriedem: Yeah, sounds like a plan. Though I expect it to be a while before folks quit using api_servers IRL. | 21:09 |
| sean-k-mooney | anyway its 9 so im going to go home and have dinner o/ | 21:10 |
| cfriesen | later | 21:10 |
| mriedem | efried: same | 21:10 |
| efried | ...which (I think) makes the issue moot. | 21:10 |
| efried | This bug came out of a devstack? | 21:10 |
| efried | Cause that's the one place I know for sure we got rid of api_servers. | 21:10 |
| mriedem | yes it did | 21:10 |
| mriedem | "This on devstack with commit id: 5d2add74534719c5670b29152964a60e8f23b42b" | 21:10 |
| melwitt | yeah, release note is always helpful. I do wonder if the field is nullable and if sending nothing would be better than breaking the contract, but meh, not sure | 21:11 |
| efried | But this is good - flushing out these bugs/corner cases in devstack before they hit the proverbial fan in production. | 21:11 |
| *** jmlowe has joined #openstack-nova | 21:11 | |
| mriedem | melwitt: nullable is also breaking the contract a bit | 21:11 |
| mriedem | fwiw, this is already dumb if you're using bfv, | 21:12 |
| mriedem | because instance.image_ref is '' for bfv | 21:12 |
| mriedem | so we're sending an image ref url with no image id in it | 21:12 |
| melwitt | heh, okay | 21:12 |
| cfriesen | sean-k-mooney: prior to 1231c469d (circa 2014) we did pin instances without a numa_topology to a single host numa node | 21:12 |
| melwitt | efried: yeah, we need to check when this broke so we know how far back to backport. and also if it's in ocata then people are probably hitting this in production | 21:13 |
| efried | melwitt: https://review.openstack.org/#/c/490057/ | 21:14 |
| melwitt | it's maybe not "as noticeable" since it's likely only this sync power states periodic that results in notifications maybe | 21:14 |
| melwitt | efried: okay, cool. | 21:14 |
| melwitt | so only need to backport to queens | 21:15 |
| efried | melwitt: And here's the change that removed api_servers from devstack: https://review.openstack.org/#/c/490031/ | 21:15 |
| *** suresh12 has quit IRC | 21:15 | |
| efried | melwitt: Well, I'll be a little surprised if that's the only place we're using a tokenless auth context to look up a glance endpoint. | 21:16 |
| efried | given the scope of 490057 | 21:16 |
| *** AlexeyAbashkin has joined #openstack-nova | 21:19 | |
| melwitt | efried: yeah ... probably. but since it's so recent, I think you're right this is being caught a lot sooner than most people upgrade to queens. I was worried it was going to go back to ocata or something crazy. I know you linked the patch earlier but I didn't notice it was pretty recent | 21:19 |
| *** suresh12 has joined #openstack-nova | 21:21 | |
| melwitt | mriedem: what I was thinking with the nullable thing, would be in the worst case if someone had automation parsing image_ref_url and we make it not a url, but empty string isn't one either. not sure if they're equally bad. just thinking out loud | 21:22 |
| mriedem | if someone blindly takes the image_ref_url and makes a GET curl request with it, it's going to blow up for all volume-backed instances | 21:22 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle EndpointNotFound when building image_ref_url in notifications https://review.openstack.org/554703 | 21:23 |
| mriedem | i think in the long ago, you could also create servers by passing an image URL to the imageRef parameter | 21:23 |
| melwitt | yeah. I was thinking before that as far as the format of it. probably overthinking it | 21:23 |
| mriedem | and nova would parse it | 21:23 |
| *** AlexeyAbashkin has quit IRC | 21:23 | |
| mriedem | multiattach is fixed with the queens UCA | 21:29 |
| mriedem | https://review.openstack.org/#/c/554667/ | 21:29 |
| mriedem | that's the only thing blocking us from using the queens UCA in devstack now | 21:29 |
| imacdonn | ooh | 21:30 |
| imacdonn | is there any actually need to use the UCA? ZFSSA CI seems to be doing OK with out .. but that's cinder | 21:31 |
| mriedem | well, a few things, | 21:31 |
| *** itlinux has quit IRC | 21:31 | |
| mriedem | we get newer libvirt and qemu, | 21:31 |
| mriedem | testing rocky against pike UCA seems weird, we should at least use queens if it's there | 21:31 |
| mriedem | and it allows us to remove these weird workarounds in devstack for the multiattach job to *not* use the UCA b/c the pike UCA didn't have the right package versions, but queens UCA does https://review.openstack.org/#/c/554317/ | 21:32 |
| mriedem | https://review.openstack.org/#/c/554314/ | 21:32 |
| imacdonn | yeah, I have one of those workarounds (per your recommendation) | 21:32 |
| *** moshele has quit IRC | 21:32 | |
| imacdonn | I suppose it does make sense to test nova stuff with the latest virt stuff | 21:32 |
| mriedem | s/nova/openstack/ | 21:33 |
| imacdonn | seems it'd matter more for nova... but yeah | 21:33 |
| mriedem | sure, for libvirt and qemu yes, but you also get newer things like rados and tgtd | 21:33 |
| mriedem | ovs | 21:33 |
| mriedem | etc | 21:33 |
| imacdonn | OK. Has devstack master already been updated to use the Queens UCA? | 21:35 |
| mriedem | https://review.openstack.org/#/c/554314/ | 21:35 |
| mriedem | depends on fixing this bug in nova first | 21:35 |
| *** esberglu has quit IRC | 21:35 | |
| imacdonn | k... I'll subscribe to the bug | 21:36 |
| mriedem | get stvnoyes on the oracle phone | 21:36 |
| imacdonn | not sure if he's still in EU, but even if not, he's probably gone for the day (east coast) | 21:37 |
| melwitt | mriedem: +2 | 21:37 |
| *** yamamoto has joined #openstack-nova | 21:37 | |
| mriedem | thanks | 21:37 |
| openstackgerrit | Merged openstack/nova stable/queens: Revert "Refine waiting for vif plug events during _hard_reboot" https://review.openstack.org/553817 | 21:39 |
| imacdonn | Something completely unrelated to bounce off you, mriedem (or anyone else) | 21:39 |
| imacdonn | per Queens release notes, I tried to remove neutron.url from my nova.conf, but it seems to be unable to get the neutron endpoint from the service catalog | 21:40 |
| mriedem | efried: replied to all comments in https://review.openstack.org/#/c/554703/ and i don't think any of them are worth changing | 21:40 |
| imacdonn | I had someone in #openstack buddy-check my config, and it seems sane | 21:40 |
| mriedem | imacdonn: there is a bug fix that you need, | 21:40 |
| mriedem | sec | 21:40 |
| *** esberglu_ has joined #openstack-nova | 21:40 | |
| imacdonn | k ;) | 21:40 |
| mriedem | imacdonn: https://github.com/openstack/nova/commit/3a3b0f09db318faf1a1ea711a73bb365cab8b233 | 21:40 |
| imacdonn | mriedem: Interesting, Looks pertinent. Will try it. Thanks! | 21:41 |
| *** yamamoto has quit IRC | 21:43 | |
| efried | mriedem: Brain fart, sorry 'bout that. +1. | 21:44 |
| *** lpetrut has quit IRC | 21:44 | |
| mriedem | efried: np, thanks for the quick review | 21:45 |
| *** esberglu_ has quit IRC | 21:45 | |
| *** josecastroleon has quit IRC | 21:47 | |
| mriedem | hongbin: yikun: Kevin_Zheng: easy bug https://bugs.launchpad.net/nova/+bug/1757273 | 21:47 |
| openstack | Launchpad bug 1757273 in OpenStack Compute (nova) "nova-compute fails to start even if [placement]/region_name is set" [Medium,Triaged] | 21:47 |
| *** gouthamr has quit IRC | 21:49 | |
| *** salv-orlando has joined #openstack-nova | 21:53 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove RequestContext.instance_lock_checked https://review.openstack.org/554378 | 21:53 |
| *** READ10 has quit IRC | 21:56 | |
| *** tssurya has quit IRC | 22:00 | |
| *** gjayavelu has quit IRC | 22:02 | |
| *** sar has quit IRC | 22:09 | |
| *** germs has joined #openstack-nova | 22:10 | |
| *** germs has quit IRC | 22:10 | |
| *** germs has joined #openstack-nova | 22:10 | |
| *** rcernin has joined #openstack-nova | 22:13 | |
| *** Guest64768 has quit IRC | 22:13 | |
| *** germs has quit IRC | 22:14 | |
| *** priteau has quit IRC | 22:15 | |
| *** priteau has joined #openstack-nova | 22:15 | |
| *** slagle has quit IRC | 22:16 | |
| *** dave-mccowan has quit IRC | 22:17 | |
| *** slagle has joined #openstack-nova | 22:17 | |
| *** felipemonteiro_ has quit IRC | 22:18 | |
| *** oomichi has joined #openstack-nova | 22:18 | |
| *** AlexeyAbashkin has joined #openstack-nova | 22:19 | |
| *** priteau has quit IRC | 22:20 | |
| *** awaugama has quit IRC | 22:20 | |
| *** liverpooler has quit IRC | 22:23 | |
| *** AlexeyAbashkin has quit IRC | 22:23 | |
| *** dave-mccowan has joined #openstack-nova | 22:28 | |
| mriedem | i wonder if anyone that uses DVR also uses shared ephemeral storage (rbd imagebackend) and live migration (HP cloud anyone?) | 22:29 |
| mriedem | because i'm pretty sure we don't cleanup on failed live migration properly in that case | 22:29 |
| mriedem | https://github.com/openstack/nova/blob/3fd863d8bf2fa1fc09acd08d976689462cffd2e3/nova/compute/manager.py#L6506 will also cleanup some stuff we put in the port's binding profile for DVR during live migration, | 22:30 |
| mriedem | but if do_cleanup is False, which it is if you're using shared ephemeral storage, then we don't clean that up | 22:30 |
| *** yamamoto has joined #openstack-nova | 22:39 | |
| efried | mriedem: Are you talking specifically libvirt? | 22:40 |
| efried | Cause that sounds like something we either support already or will support shortly in PowerVM (out of tree). | 22:41 |
| mriedem | for shared local disk? | 22:41 |
| mriedem | https://github.com/openstack/nova/blob/3fd863d8bf2fa1fc09acd08d976689462cffd2e3/nova/compute/manager.py#L6185 | 22:42 |
| mriedem | looks like that is also checked for xen and hyperv | 22:42 |
| mriedem | anyway, https://github.com/openstack/nova/blob/3fd863d8bf2fa1fc09acd08d976689462cffd2e3/nova/compute/manager.py#L6524 is totally doing more than just cleaning up local disk created on the dest host for non-shared storage | 22:43 |
| mriedem | it's also cleaning up network stuff on the dest host | 22:43 |
| *** hongbin has quit IRC | 22:44 | |
| mriedem | added back in mitaka https://review.openstack.org/#/c/227897/ | 22:45 |
| mriedem | oh nvm, that was the error handling | 22:45 |
| *** yamamoto has quit IRC | 22:45 | |
| *** andreas_s has joined #openstack-nova | 22:47 | |
| mriedem | wow added long ago https://review.openstack.org/#/c/4646/ | 22:47 |
| mriedem | essex | 22:47 |
| efried | Gotta run | 22:48 |
| *** andreas_s has quit IRC | 22:51 | |
| *** gjayavelu has joined #openstack-nova | 22:56 | |
| *** harlowja has quit IRC | 22:57 | |
| *** _ix has quit IRC | 22:57 | |
| *** chyka has quit IRC | 23:00 | |
| *** chyka has joined #openstack-nova | 23:01 | |
| *** vladikr has quit IRC | 23:01 | |
| *** vladikr has joined #openstack-nova | 23:01 | |
| *** masber has joined #openstack-nova | 23:03 | |
| *** masuberu has joined #openstack-nova | 23:04 | |
| *** masber has quit IRC | 23:08 | |
| *** AlexeyAbashkin has joined #openstack-nova | 23:19 | |
| *** jpena|away is now known as jpena|off | 23:21 | |
| *** AlexeyAbashkin has quit IRC | 23:23 | |
| *** Zames has joined #openstack-nova | 23:25 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Move configurable mkfs to privsep. https://review.openstack.org/551921 | 23:27 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi xenstore_read's to privsep. https://review.openstack.org/552241 | 23:27 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi disk resizing to privsep. https://review.openstack.org/552242 | 23:27 |
| openstackgerrit | Michael Still proposed openstack/nova master: Sync xenapi and libvirt on what flags to pass e2fsck. https://review.openstack.org/554078 | 23:27 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move xenapi partition copies to privsep. https://review.openstack.org/553605 | 23:27 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move image conversion to privsep. https://review.openstack.org/554437 | 23:27 |
| openstackgerrit | Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | 23:27 |
| openstackgerrit | Michael Still proposed openstack/nova master: We don't need utils.trycmd any more. https://review.openstack.org/554439 | 23:27 |
| *** Anticime1 is now known as Anticimex | 23:29 | |
| *** Zames has quit IRC | 23:30 | |
| *** mriedem has quit IRC | 23:34 | |
| *** mlavalle has quit IRC | 23:36 | |
| *** yamamoto has joined #openstack-nova | 23:41 | |
| *** chyka has quit IRC | 23:46 | |
| *** yamamoto has quit IRC | 23:46 | |
| *** amodi has quit IRC | 23:46 | |
| *** harlowja has joined #openstack-nova | 23:49 | |
| *** cdent has quit IRC | 23:50 | |
| *** mriedem has joined #openstack-nova | 23:52 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Remove useless run_periodic_tasks call in ClientRouter https://review.openstack.org/554381 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!