Tuesday, 2018-08-07

*** tetsuro_ has quit IRC00:01
*** tbachman has joined #openstack-nova00:02
*** gyee has quit IRC00:06
*** Nel1x has joined #openstack-nova00:09
openstackgerritzhufl proposed openstack/nova master: Fix none-ascii char in doc  https://review.openstack.org/58842200:13
*** frankwang has joined #openstack-nova00:17
*** gbarros has quit IRC00:19
*** zhurong has joined #openstack-nova00:22
*** harlowja has quit IRC00:26
*** mvkr has quit IRC00:26
*** mvkr has joined #openstack-nova00:27
*** mvkr has quit IRC00:39
*** mvkr has joined #openstack-nova00:40
*** gbarros has joined #openstack-nova00:40
*** frankwang has quit IRC00:42
*** tetsuro_ has joined #openstack-nova00:44
openstackgerritMerged openstack/nova master: Refactor AllocationFixture in placement test  https://review.openstack.org/58815900:46
openstackgerritMerged openstack/nova master: Adds a test for getting allocations API  https://review.openstack.org/58888600:50
*** mvkr has quit IRC00:50
*** mvkr has joined #openstack-nova00:51
*** gbarros has quit IRC00:59
*** tetsuro_ has quit IRC01:01
*** mvkr has quit IRC01:01
*** mvkr has joined #openstack-nova01:02
*** hongbin has joined #openstack-nova01:03
*** erlon has joined #openstack-nova01:06
*** tetsuro_ has joined #openstack-nova01:09
openstackgerritMerged openstack/nova master: Not use project table for user table  https://review.openstack.org/58888701:11
*** erlon has quit IRC01:13
mriedemgibi: fyi i can't attend the notification meeting this week01:15
*** mriedem has quit IRC01:15
*** erlon has joined #openstack-nova01:26
*** zhurong has quit IRC01:26
*** tetsuro_ has quit IRC01:27
*** tetsuro_ has joined #openstack-nova01:28
*** tetsuro_ has quit IRC01:34
openstackgerritzhufl proposed openstack/nova master: xx_instance_type_id in list_migrations should be integer  https://review.openstack.org/58848101:40
*** tetsuro_ has joined #openstack-nova01:41
*** tetsuro_ has quit IRC01:44
*** gbarros has joined #openstack-nova01:49
*** tetsuro_ has joined #openstack-nova01:53
*** Dinesh_Bhor has joined #openstack-nova02:01
openstackgerritTetsuro Nakamura proposed openstack/nova master: Use common functions in granular fixture  https://review.openstack.org/58811302:04
*** naichuans has joined #openstack-nova02:12
*** lbragstad has quit IRC02:18
*** tetsuro_ has quit IRC02:19
*** tetsuro_ has joined #openstack-nova02:23
*** tetsuro_ has quit IRC02:26
*** donghm has joined #openstack-nova02:26
*** BrinZhang has joined #openstack-nova02:38
*** psachin has joined #openstack-nova02:40
*** erlon has quit IRC02:45
*** tetsuro_ has joined #openstack-nova02:50
openstackgerritmelanie witt proposed openstack/nova-specs master: Add a script for counting blueprints  https://review.openstack.org/58191402:52
*** tetsuro__ has joined #openstack-nova02:53
*** tetsuro_ has quit IRC02:54
openstackgerritMerged openstack/nova master: Define irrelevant-files for tempest-full-py3 job  https://review.openstack.org/58903902:58
*** tetsuro__ has quit IRC02:59
*** bbbbzhao_ has quit IRC03:08
*** jhesketh_ is now known as jhesketh03:14
*** vivsoni_ has quit IRC03:19
*** Nel1x has quit IRC03:22
*** gbarros has quit IRC03:26
*** dave-mccowan has quit IRC03:31
*** liuyulong has joined #openstack-nova03:34
*** tetsuro_ has joined #openstack-nova03:39
*** janki has joined #openstack-nova03:41
*** tetsuro_ has quit IRC03:47
*** tetsuro_ has joined #openstack-nova03:48
*** udesale has joined #openstack-nova03:48
*** tetsuro_ has quit IRC03:51
*** jaypipes has quit IRC04:02
*** jaypipes has joined #openstack-nova04:02
*** Dinesh_Bhor has quit IRC04:04
*** tetsuro_ has joined #openstack-nova04:12
*** Bhujay has joined #openstack-nova04:13
*** vivsoni has joined #openstack-nova04:14
*** hongbin has quit IRC04:15
*** Bhujay has quit IRC04:30
openstackgerritMerged openstack/python-novaclient master: Refactor the getid method in novaclient/base.py  https://review.openstack.org/58898304:56
*** links has joined #openstack-nova04:57
*** Dinesh_Bhor has joined #openstack-nova05:01
openstackgerritMerged openstack/nova master: Use common functions in NonSharedStorageFixture  https://review.openstack.org/58811405:01
*** stakeda has joined #openstack-nova05:04
*** ratailor has joined #openstack-nova05:11
*** tetsuro_ has quit IRC05:15
*** psachin has quit IRC05:18
*** tetsuro_ has joined #openstack-nova05:22
*** psachin has joined #openstack-nova05:25
*** jaosorior has quit IRC05:26
*** tetsuro_ has quit IRC05:27
*** nicolasbock has joined #openstack-nova05:46
*** Luzi has joined #openstack-nova05:47
*** moshele has joined #openstack-nova05:56
*** tetsuro_ has joined #openstack-nova05:59
*** tetsuro_ has quit IRC06:02
*** tetsuro_ has joined #openstack-nova06:02
*** jaosorior has joined #openstack-nova06:05
openstackgerritChen proposed openstack/nova master: Trivial fix on migration doc  https://review.openstack.org/58902806:24
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Fix server strings in reboot operation  https://review.openstack.org/58898106:30
openstackgerritTakashi NATSUME proposed openstack/python-novaclient master: Fix server strings in reboot operation  https://review.openstack.org/58898106:31
*** hoonetorg has quit IRC06:33
*** pcaruana has joined #openstack-nova06:36
*** Dinesh_Bhor has quit IRC06:36
*** tetsuro_ has quit IRC06:36
*** Dinesh_Bhor has joined #openstack-nova06:37
*** Bhujay has joined #openstack-nova06:42
*** whoami-rajat has joined #openstack-nova06:45
*** hoonetorg has joined #openstack-nova06:49
*** Dinesh_Bhor has quit IRC06:52
*** luksky has joined #openstack-nova06:54
*** adrianc has joined #openstack-nova06:55
*** amarao has joined #openstack-nova07:01
*** dpawlik has joined #openstack-nova07:23
*** wznoinsk has quit IRC07:27
*** wznoinsk has joined #openstack-nova07:27
*** avolkov has joined #openstack-nova07:28
*** adrianc_ has joined #openstack-nova07:30
*** adrianc has quit IRC07:34
*** jpena|off is now known as jpena07:35
*** sahid has joined #openstack-nova07:36
*** luksky has quit IRC07:41
*** rmart04 has joined #openstack-nova07:44
*** jaosorior has quit IRC07:49
*** maciejjozefczyk has quit IRC07:57
*** slunkad has quit IRC07:58
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] Add version directives in the history doc  https://review.openstack.org/58939207:59
*** maciejjozefczyk has joined #openstack-nova07:59
*** maciejjozefczyk has joined #openstack-nova08:00
*** jaosorior has joined #openstack-nova08:02
*** jarod_ has joined #openstack-nova08:07
*** rcernin has quit IRC08:11
*** cdent has joined #openstack-nova08:14
openstackgerritChris Dent proposed openstack/nova master: [placement] Add /reshaper handler for POST  https://review.openstack.org/57692708:27
openstackgerritChris Dent proposed openstack/nova master: reshaper: Look up provider if not in inventories  https://review.openstack.org/58503308:27
openstackgerritChris Dent proposed openstack/nova master: Make get_allocations_for_resource_provider sane  https://review.openstack.org/58459808:27
openstackgerritChris Dent proposed openstack/nova master: Report client: Real get_allocs_for_consumer  https://review.openstack.org/58459908:27
openstackgerritChris Dent proposed openstack/nova master: Report client: get_allocations_for_provider_tree  https://review.openstack.org/58464808:27
openstackgerritChris Dent proposed openstack/nova master: Report client: _reshape helper, placement min bump  https://review.openstack.org/58503408:27
openstackgerritChris Dent proposed openstack/nova master: Report client: update_from_provider_tree w/reshape  https://review.openstack.org/58504908:27
openstackgerritChris Dent proposed openstack/nova master: Compute: Handle reshaped provider trees  https://review.openstack.org/57623608:27
*** sayalilunkad has joined #openstack-nova08:31
*** owalsh_ is now known as owalsh08:33
*** panda|rover-ish is now known as panda|rover08:33
*** mhen has joined #openstack-nova08:37
mhenhello everybody! can you tell me the reason for using binascii.hexlify() to transform secrets into a hex representation before passing them to cryptsetup? example: https://github.com/openstack/nova/blob/25fa2470e220a83ce632fed70ad41e55aabda0da/nova/privsep/libvirt.py#L5308:38
openstackgerritzhufl proposed openstack/nova master: xx_instance_type_id in list_migrations should be integer  https://review.openstack.org/58848108:39
openstackgerritLee Yarwood proposed openstack/nova master: fixtures: Track volume attachments within CinderFixtureNewAttachFlow  https://review.openstack.org/58701308:42
openstackgerritLee Yarwood proposed openstack/nova master: Add regression test for bug#1784353  https://review.openstack.org/58701408:42
openstackgerritLee Yarwood proposed openstack/nova master: conductor: Recreate volume attachments during a reschedule  https://review.openstack.org/58707108:42
lyarwoodmhen: so at present we are getting binary key from barbican that we need to turn into an ascii string to provide to cryptsetup / libvirt08:43
lyarwoodmhen: the binascii calls are legacy code that we've not changed in order to not break existing users08:44
lyarwoodmhen: really we need barbican to provide actual passphrases instead of binary keys to avoid using this in the future08:44
lyarwoodmdbooth: ^ respun my changes above, thanks for catching the off by one mistake!08:45
lyarwoodhopefully the py35-functional tests should also be fixed now08:46
*** derekh has joined #openstack-nova08:46
mhenlyarwood, ok thanks I figured as much - is there any reason for using a hex encoding specifically? When talking about passphrases, you effectively double the string length since hex can only represent half as much in a single byte/character iirc.08:46
mdboothlyarwood: :) Do you know where the attachment delete code is, btw?08:46
mdboothlyarwood: Any reason it doesn't remove the attachment from the BDM.08:46
lyarwoodmdbooth: yeah I left a comment, https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L2503-L2505, is bdm.attachment_id is used all over the place to determine if we are using v308:47
mdboothlyarwood: Eurgh08:48
lyarwoodbut we'd still need this code to recreate the attachments even if we do remove it08:48
lyarwoodas we don't get back up to the API where the initial attachments are created08:48
mdboothlyarwood: Right, it's just that by getting out of whack we're opening ourselves to potential errors, and we also have to ping cinder to see if that's what we've done08:48
mdboothThat's not a problem in your patch, but it's unfortunate08:49
lyarwoodmhen: I don't know of any reason, it's just a legacy choice that I've wanted to remove for ages but can't with existing users08:49
lyarwoodmdbooth: yeah, I guess without the attachment_id we would need to do the v2 / v3 checks again in the conductor etc08:49
mdboothlyarwood: Would you, though? Why wouldn't you just use v3 if conn_info and attachment_id are both unset?08:50
*** luksky has joined #openstack-nova08:51
lyarwoodmdbooth: iirc the API has a series of compute version checks it goes through before creating the attachments08:51
*** adrianc__ has joined #openstack-nova08:51
*** Kevin_Zheng has joined #openstack-nova08:51
*** tetsuro_ has joined #openstack-nova08:53
*** donghm has quit IRC08:54
mdboothlyarwood: Is it possible to have a reschedule *without* the original 'reservation' attachments having been deleted?08:54
mdboothe.g. by an early failure which doesn't cause cleanup to go through _shutdown_instance?08:55
*** adrianc_ has quit IRC08:55
*** priteau has joined #openstack-nova08:55
lyarwoodmdbooth: yeah but that should be fine, the destination compute should just UPDATE the existing attachment at that point and an updated connection_info dict in return from cinder08:56
lyarwoodmdbooth: nova/virt/block_device.py -> _volume_attach08:58
gibidansmith: double checked the legacy allocation handling and we are lucky as the legacy codepath does the right thing for revert. Anyhow filled a bug https://bugs.launchpad.net/nova/+bug/178577608:59
openstackLaunchpad bug 1785776 in OpenStack Compute (nova) "resize revert still hitting the legacy allocation handling " [Undecided,New]08:59
mhenlyarwood, I assume that the hex representation was chosen to have a predictable and limited set of string characters in order to avoid any problems related to special characters?09:01
lyarwoodmhen: yeah that could very well be the case09:01
*** jaosorior has quit IRC09:02
*** josecastroleon has joined #openstack-nova09:14
openstackgerritTakashi NATSUME proposed openstack/nova master: [placement] api-ref: add description for 1.29  https://review.openstack.org/58940709:15
kosamaraefried: I studied your device-passthrough spec for nova-powervm. Do you plan to propose this or part of it in nova?09:18
*** finucannot is now known as stephenfin09:30
*** maciejjozefczyk has quit IRC09:32
*** sahid has quit IRC09:39
openstackgerritRajesh Tailor proposed openstack/nova master: Fix host validity check for live-migration  https://review.openstack.org/40100909:42
*** tetsuro_ has quit IRC09:45
*** tetsuro_ has joined #openstack-nova09:45
mdboothlyarwood: I still think you're missing a test, btw. https://review.openstack.org/#/c/587071/8/nova/tests/unit/conductor/test_conductor.py09:50
lyarwoodmdbooth: kk, where it exists?09:50
lyarwoodmdbooth: either way the functional tests still aren't happy so I'll respin later today09:51
mdboothlyarwood: yeah. I just checked the code and there's a reasonably wide window for errors which don't result in attachment deletion.09:51
*** rmart04 has quit IRC09:53
*** tetsuro_ has quit IRC09:59
*** tetsuro has joined #openstack-nova09:59
*** adrianc_ has joined #openstack-nova10:01
*** Dinesh_Bhor has joined #openstack-nova10:02
*** adrianc__ has quit IRC10:05
*** sahid has joined #openstack-nova10:05
*** jamesdenton has quit IRC10:18
*** tetsuro has quit IRC10:21
*** sahid has quit IRC10:27
*** links has quit IRC10:27
*** tetsuro has joined #openstack-nova10:28
*** tetsuro has quit IRC10:28
*** Dinesh_Bhor has quit IRC10:30
*** jamesdenton has joined #openstack-nova10:37
*** sahid has joined #openstack-nova10:38
*** BrinZhang has quit IRC10:45
*** liuyulong has quit IRC10:45
*** jaosorior has joined #openstack-nova10:46
*** dave-mccowan has joined #openstack-nova10:51
*** avolkov has quit IRC10:55
*** Dinesh_Bhor has joined #openstack-nova10:56
openstackgerritBalazs Gibizer proposed openstack/nova master: Fix resize revert to use non-legacy alloc handling  https://review.openstack.org/58942511:03
openstackgerritBalazs Gibizer proposed openstack/nova master: Fix resize revert to use non-legacy alloc handling  https://review.openstack.org/58942511:04
openstackgerritMerged openstack/nova master: Grease some more tests hitting RetryDecorator  https://review.openstack.org/58839111:09
openstackgerritMerged openstack/nova master: Grease test_try_deallocate_network_retry_direct  https://review.openstack.org/58836411:09
*** artom has joined #openstack-nova11:11
*** Dinesh_Bhor has quit IRC11:14
*** udesale has quit IRC11:15
*** Dinesh_Bhor has joined #openstack-nova11:15
*** Bhujay has quit IRC11:18
*** panda|rover is now known as panda|rover|lunc11:18
*** ltomasbo has left #openstack-nova11:22
*** vivsoni has quit IRC11:24
*** stakeda has quit IRC11:26
*** vivsoni has joined #openstack-nova11:31
*** Dinesh_Bhor has quit IRC11:36
*** slagle has joined #openstack-nova11:39
*** erlon has joined #openstack-nova11:56
*** s10 has joined #openstack-nova11:57
*** jaosorior has quit IRC11:59
*** edmondsw has joined #openstack-nova12:04
*** ratailor has quit IRC12:05
*** s10 has quit IRC12:09
openstackgerritZhenyu Zheng proposed openstack/nova master: Update installation guide to be more clear about cellsv2  https://review.openstack.org/58424412:09
*** s10 has joined #openstack-nova12:10
*** s10 has quit IRC12:10
*** jpena is now known as jpena|lunch12:15
*** panda|rover|lunc is now known as panda|rover12:16
*** jaosorior has joined #openstack-nova12:19
*** Bhujay has joined #openstack-nova12:24
*** zhangbailin_ has joined #openstack-nova12:26
*** zhangbailin_ has quit IRC12:27
*** jaosorior has quit IRC12:27
*** BrinZhang has joined #openstack-nova12:27
*** BrinZhang has quit IRC12:27
*** BrinZhang has joined #openstack-nova12:28
*** BrinZhang has quit IRC12:29
*** jaosorior has joined #openstack-nova12:42
*** sapcc-bot3 has joined #openstack-nova12:46
*** d063130_ has quit IRC12:50
*** sapcc-bot has quit IRC12:50
*** thomasem has quit IRC12:50
*** weshay has quit IRC12:50
*** gbarros has joined #openstack-nova12:56
*** ccamacho has quit IRC12:58
*** dklyle has quit IRC12:59
*** bigdogstl has joined #openstack-nova13:00
mdboothlyarwood: FYI, I accidentally started looking at your functional test failure, btw13:02
mdboothlyarwood: Haven't fixed it yet, but I can continue or not as you like.13:02
*** janki has quit IRC13:03
*** janki has joined #openstack-nova13:04
lyarwoodmdbooth: I've not got back around to it yet but I assume there are multiple attachments in the self.attachments[instance_uuid] list when I've used [0] to delete things from self.volume_to_attachment right?13:05
lyarwoodmdbooth: if you already have something feel free to continue and push it up when you're done13:05
mdboothlyarwood: Ok, will do.13:05
*** tssurya has joined #openstack-nova13:07
mdboothlyarwood: What's the thinking here, btw:13:07
mdbooth         def fake_get(self_api, context, volume_id, microversion=None):13:07
mdbooth+            attachment_id = self.volume_to_attachment.get(volume_id, volume_id)13:07
mdboothWhy would you want fake attachment_id to default to volume_id if not present?13:07
mdboothAh... that's what it did before13:09
* mdbooth wonders if that's the bug...13:09
lyarwoodhmm this was working with a previous PS with that in place13:09
*** jamesdenton has quit IRC13:10
mdboothlyarwood: I'm speculating. Issue is that we end up trying to fetch an attachment id not present. If the attachment id is a volume id ,that might be it.13:10
mdboothI'm going to make it unique and stash it before returning.13:11
*** lbragstad has joined #openstack-nova13:11
*** jamesdenton has joined #openstack-nova13:11
lyarwoodmdbooth: https://review.openstack.org/#/c/587013/6..7/nova/tests/fixtures.py@1708 - I bet it's that line13:12
mdboothMight be in more than 1 place. This is attachment_update13:13
mdboothBut yeah, that looks like another candidate13:13
*** s10 has joined #openstack-nova13:15
*** mriedem has joined #openstack-nova13:16
*** mriedem has joined #openstack-nova13:16
efriedkosamara: Yes, in Stein or the T release. Is this something that interests you?13:18
*** eharney has joined #openstack-nova13:22
*** tbachman has quit IRC13:23
kosamaraefried: yes. I'm working at CERN and I think you had a relevant discussion with Belmiro late June.13:26
efriedkosamara: I would be happy to talk through it further. I assume you're interested in doing this with libvirt, or do you have Power systems in your deployment?13:27
kosamaraefried: We would like at least a simplified version of what you're developing. With the prospect of working on it, I was looking for any relevant specs, but only came across yours recently.13:27
*** namnh has joined #openstack-nova13:27
kosamaraefried: libvirt13:27
efriedkosamara: Are you aware of the cyborg project?13:28
*** ccamacho has joined #openstack-nova13:28
kosamaraefried: yes, but it appears to be doing much more than what we need, so a solution within nova to use GPUs seems better at this point.13:29
efriedkosamara: How much have you done with the existing PCI passthrough framework?13:30
*** jpena|lunch is now known as jpena13:30
kosamaraefried: we are already using it in testing and slowly moving to production.13:30
kosamaraefried: and only for GPUs13:32
s10Hello. I've found, that function update_available_resource(), https://github.com/openstack/nova/blob/stable/pike/nova/compute/resource_tracker.py#L694 , driver.get_available_resource(nodename) with Libvirt driver takes 30 seconds to execute on the host with 150 instances on local storage.13:32
s10Is this behaviour normal or is it a regression introduced by https://github.com/openstack/nova/commit/d88b75e81eabfbd463007f6a4f27e6966a466530 and following commits? Before this commit it was 1-2 seconds for all of them.13:32
efriedkosamara: Okay. That's going to be your best bet for the near future. And it should do pretty much everything you need if all you're trying to do is pass through whole GPUs.13:32
efrieds10: You've specifically nailed it down to that commit? If you revert it, your performance goes back to normal?13:33
kosamaraefried: Whole GPUs is our main use case ATM. "that"? Our main problems with the current pci passthrough in nova are quotas and scheduling without an extra filter. So basically, implementing RPs/RCs for GPUs.13:34
s10Yes, if I return this function to be like before this commit, performance of this function goes back to normal. But then we will lose all fixes, introduced by this commits and following, like https://github.com/openstack/nova/commit/938c0a745325fa73d098c6d5ddd20b2a599f962413:35
*** bigdogstl has quit IRC13:36
s10efried: if i turn on logging for oslo_concurrency, I see, that a lot of time takes for 300 calls of "qemu-img info" for /var/lib/nova/instances/UUID/disk and disk.config13:37
s10efried: at least 0.052 for every call. 300 calls - 15 seconds.13:38
*** moshele has quit IRC13:39
*** adrianc__ has joined #openstack-nova13:40
efriedkosamara: Not sure if we've really started making plans to implement quotas around placement artifacts yet. alex_xu, were you working on that?13:40
efrieds10: Let me take a look at what's piled on top of that commit. Trying to figure out what the effect would be of *just* reverting that one fix.13:41
s10efried: so for 150 instances qemu_img info is being called 600 times. 2 times for dk_size = disk_api.get_allocated_disk_size(path) and 2 times for virt_size = disk_api.get_disk_size(path).13:41
dansmiths10: maybe talk to lyarwood about it13:42
efrieds10: Note that we have some work ongoing to cut that in half at least: https://review.openstack.org/#/c/520024/13:42
efrieds10: But that still wouldn't make it okay as performance regressions go.13:43
sean-k-mooneys10: we could proably cache it but is it an apricalble portion of the total boot time13:43
*** adrianc_ has quit IRC13:43
sean-k-mooneythere are other cases where we could chache thing in the boot process but dont because it was felt the caching introduced complexity that did not result in a significant perfomace increase when taken as a propotion of the total boot time13:45
s10efried: I added timer before driver.get_available_resource(nodename) and stopped it after and logged it, so only one call of this function takes 30 seconds.13:46
efrieds10: I think your best course of action right now is to open a bug and make sure lyarwood sees it.13:46
efrieds10: Do you have the ability to apply a single patch on the fly in your env and reproduce the flow?13:47
*** tbachman has joined #openstack-nova13:47
s10efried: yes, I have such ability13:47
efrieds10: I can try throwing out a quick revert just to confirm that it does the trick. Stand by.13:48
*** psachin has quit IRC13:51
openstackgerritEric Fried proposed openstack/nova stable/pike: DNM: Revert d88b75e  https://review.openstack.org/58947913:51
efrieds10: ^13:51
*** jamesdenton has quit IRC13:52
* lyarwood reads up13:52
lyarwoods10: and this is with an images_type of raw?13:52
efrieds10: Note: a real revert would be tied to a bug and introduced in master and backported. This is just to confirm.13:52
kosamaraefried: alex_xu had proposed a spec for that. The first step is to have pci passthrough GPUs in placement of course.13:53
mriedemyay https://review.openstack.org/#/q/If642e51a4e186833349a8e30b04224a3687f559413:53
s10efried: Before: Took 20.89 seconds to get available resources for nodename. update_available_resource /usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py:70513:53
*** jamesdenton has joined #openstack-nova13:53
s10efried: After: Took 11.04 seconds to get available resources for nodename13:53
sean-k-mooneykosamara: the first step to having pcie gpus in placement is having pci devices in placement13:53
s10If I change  virt_size = disk_api.get_disk_size(path) same way, this time reduces to 2 seconds.13:53
efriedOkay s10, I'll leave you in lyarwood's capable hands at this point.13:54
s10lyarwood: yes, image_type is raw13:54
mriedemit's going to be slower b/c it's exec'ing qemu-img info13:55
lyarwoods10: unfortunatley I'm just between calls, could you write this up in a bug and I'll get back to you in ~60mins or so13:55
s10lyarwood: and preallocate_images=space. I will fill bug report.13:55
mriedemdansmith: want to send https://review.openstack.org/#/q/topic:bug/1784705+(status:open+OR+status:merged)+branch:stable/queens to their maker?13:58
dansmithmriedem: yeah13:58
*** maciejjozefczyk has joined #openstack-nova13:59
maciejjozefczykmriedem: efried hey, about https://review.openstack.org/#/c/520024; we have it on production and it works properly; release is newton14:00
efriedmaciejjozefczyk: Sweet, thanks for the info.  mriedem cdent Ima +2 that sucker. Shall we backport it too?14:00
maciejjozefczykonly one thing is that, as I remember correctly, on nova master was issue with https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L866 def_resource_change()14:01
maciejjozefczykefried: so could you please take me a moment to confirm that its fiexed on master, or not?14:01
cdentefried: assume maciejjozefczyk's concerns there are okay, I think a backport would be nice but not critical?14:01
mriedemmaciejjozefczyk: you said you think https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L866 should be removed yes?14:02
maciejjozefczykmriedem:  afailr I know it says all the time 'False'; so the update is not send to the DB14:03
maciejjozefczykand it ends with old timestamp for compute row14:03
maciejjozefczykso it could be deleted14:04
maciejjozefczykIt was running all the time because in memory some resources were changed (because of this double-update) so it was always returning 'True'14:04
mriedemis it always False because we've already updated the compute node before we get here?14:04
*** awaugama has joined #openstack-nova14:05
maciejjozefczyknope, if you apply my patch - compute is def _update() is called only once. but inside def _update() there is a check if somethings change about ram,disk etc in memory14:06
*** _ix has quit IRC14:06
maciejjozefczykso without any change - it ends with False and the DB update is not called14:06
openstackLaunchpad bug 1785827 in OpenStack Compute (nova) "Performance regression in libvirt get_available_resource()" [Undecided,New]14:06
mriedemis there any reason to update the compute node record in the db if nothing changed?14:07
s10lyarwood: ^ i've written a bug report14:07
maciejjozefczykwithout my change - it was called basically all the time, because first update was without for ex: shutdown instances, and the second was with (so the amout of ram, disk, etc was always changing - the def _resource_change() was saying 'True')14:07
lyarwoods10: ack thanks14:07
maciejjozefczykmriedem: exept timestamp, no14:07
kosamarasean-k-mooney: yes, which is what efried 's spec does.14:07
maciejjozefczykmriedem:  but for me if 'nova compute-show' shows that compute is UP and timestamp is new - it says, yea, compute is working14:08
maciejjozefczykmriedem:  maybe somebody use it as source for some scripts14:08
maciejjozefczykmriedem: but ye, service-list should be used anyway for that pourpose14:09
mriedemi want to say i think there is something in the scheduler that cares about the compute node updated_at time and uses it for some refresh threshold14:09
maciejjozefczykmriedem: no idea14:09
mriedemif (self.updated and compute.updated_at14:10
mriedem                and self.updated > compute.updated_at):14:10
mriedem            return14:10
maciejjozefczykyep, so thats the issue14:10
maciejjozefczykmriedem: the code is from placement?14:11
mriedemno that's in the nova scheduler HostManager14:11
maciejjozefczykmriedem: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L92914:11
sean-k-mooneykosamara: oh i did not know efried had a pci device in placement spec up for stien cool il should review that14:12
maciejjozefczykmriedem: ah, so it bases on the db, right?14:12
kosamaraefried: I would like to contribute. I think I could start with making the libvirt driver report PCI RPs, similar to your spec. I can propose a small spec just for that for Stein.14:12
kosamarasean-k-mooney: not in nova, in the nova-powervm fork: https://review.openstack.org/#/c/579359/10/doc/source/specs/rocky/device-passthrough.rst14:12
maciejjozefczykok, so I vote for dropping resource_change logic, but first let me see how we use it in our deployment14:13
efriedkosamara: That would be cool, please add me as a reviewer.14:13
sean-k-mooneykosamara: if your are doing that can you also make sure to pull the pci feature flags from the pci_devices table in the nova db and add them as traits to the RP14:13
maciejjozefczykmriedem: anyway my RUN team doesn't scream about this issue anymore14:13
mriedemmaciejjozefczyk: since I40c17ed88f50ecbdedc4daf368fff10e90e7be11 i'm not sure this check in the HostState object even matters14:13
mriedemwe don't cache HostStates in the scheduler anymore14:14
*** Bhujay has quit IRC14:18
*** nicolasbock has quit IRC14:18
efriedkosamara: One main aspect of the nova-powervm spec that I expect to be carried through to nova proper is a YAML configuration file allowing the operator to:14:21
efried  - Identify devices to be permitted for passthrough14:21
efried  - Specify a resource class14:21
efried  - Specify traits14:21
kosamarasean-k-mooney: I wasn't aware of this info in the pci_devices table. What exactly is it?14:22
maciejjozefczykmriedem:  anyway I need to go, I'll check this resource_updated() logic with my patch once again, I'll leave comment tonight14:22
sean-k-mooneyefried: um im not sure about that... given how we did the numa aware vswitch spec i would have assumed we would have used dynamic config insead of a yaml file14:22
efriedmaciejjozefczyk: Thanks!14:22
efriedsean-k-mooney: -1 to dynamic config.14:23
kosamaraefried: I may propose something more basic, with the existing passthrough_whitelist conf.14:23
efriedsean-k-mooney: I think the only reason we did that instead of yaml is to minimize the effort.14:23
sean-k-mooneykosamara: for network devices we use ethtool ioctls via libvirt to get the nic feature flags such as tcp checksume offload14:23
efriedkosamara: Oh dear gods please no.14:23
sean-k-mooneykosamara: adding more stuff to passthough_whitelist is basically an automatic -314:24
*** tbachman has quit IRC14:24
*** maciejjozefczyk has quit IRC14:25
efriedsean-k-mooney: We talked about using yaml in Denver. It really makes the most sense for this kind of thing, because trying to manage nested hierarchical data via oslo_config is a major PITA.14:25
*** tbachman has joined #openstack-nova14:25
sean-k-mooneyefried: well there is an argument to be made that today we dont use yaml for any other configs so we should not intoduce it for this feature14:25
efriedkosamara: ...Also, automatically generated traits. In my spec we've "namespaced" the generated traits with _POWERVM_ but some of them will potentially overlap on any platform (e.g. vendor & product IDs)14:25
sean-k-mooneyefried: that said im not really against it either14:26
kosamaraefried: I had it like that in my mind following a previous discussion with sean-k-mooney and gibi. I'll rethink it.14:26
efriedsean-k-mooney: IIRC jaypipes was a proponent and even dansmith was in agreement.14:26
efriedkosamara: In case the prospect of yaml schema/parsing is intimidating, here's code: https://review.openstack.org/#/c/579289/14:27
sean-k-mooneyefried: well jaypipes didnt want more semantics to the whitelist and having a dedicated config not in nova was cleaner as nova was not using dynamic config at the time. as i said im not against it but its adding another dependicy to nova e.g. yaml parsing14:28
sean-k-mooneyefried: are you going to propose that spec to nova propper for stien?14:29
sean-k-mooneyefried: if so is the scope just pci devices or generic device passthouhg. we had talked about expanding it to usb/sata devices in denver too but not sure if that is a different spec14:29
sean-k-mooney* should be a different sepc14:30
efriedsean-k-mooney: I hadn't yet decided whether to propose a nova spec for Stein or wait until T, but it sounds like kosamara may be interested in doing it for Stein.14:31
efriedsean-k-mooney: The way I've written the nova-powervm spec, the schema would be easily extensible to support non-PCI.14:31
jaypipesefried: vendor and product IDs should not be traits.14:31
jaypipesjust sayin.14:31
mdboothlyarwood: Hey, this is interesting14:32
sean-k-mooneyefried: i abandonded my nic feature based schduling work in favour of using this in the furutre but not sure ill be working on that now14:32
mdboothlyarwood: Still investigating your functional failures, came across _terminate_volume_connections in ComputeManager, which does exactly what I proposed14:32
sean-k-mooneyjaypipes: im assuming you would prefer a resouce_class to track that14:33
jaypipessean-k-mooney: no.14:33
sean-k-mooneyjaypipes: no?14:33
mdboothlyarwood: Specifically create a blank attachment, delete the old attachment, update the bdm to point to the blank.14:33
*** dklyle has joined #openstack-nova14:33
jaypipessean-k-mooney: I'm just saying traits are capabilities. they aren't key/value metadata items.14:33
efriedsean-k-mooney: I think jaypipes wants traits for the *capabilities* associated with a vendor/product14:33
sean-k-mooneyah ok14:33
openstackgerritVlad Gusev proposed openstack/nova stable/pike: Fix message for unexpected external event  https://review.openstack.org/58950314:33
jaypipesefried: bingo.14:33
sean-k-mooneyyep i agree that they are capablityes not metadata14:33
efriedwhich irl will entail maintaining a matrix of vendor/product to capabilities14:34
openstackgerritVlad Gusev proposed openstack/nova stable/queens: Fix message for unexpected external event  https://review.openstack.org/58950514:34
jaypipesefried: correct.14:34
sean-k-mooneyi was implying that vendor/product id is metadata that should be associated with the resouce_class14:34
efriedwhich means whenever a new device is introduced that we want to support, we need to change code.14:34
jaypipesefried: which... *gasp* the friggin vendors should be responsible for.14:34
efriedjaypipes: The vendors should be responsible for proposing nova patches to support their devices?14:34
sean-k-mooneyi had asked previously about intodusing resouce class metadata at some point but did not really push the point in the past14:34
*** amarao has quit IRC14:35
jaypipesefried: no. the vendors should be responsible for keeping the matrix of capabilities up to date with their product lines.14:35
efriedjaypipes: And that matrix of capabilities should be discoverable by querying the device somehow14:35
*** josecastroleon has quit IRC14:35
jaypipesefried: in the same way they are responsible for ensuring the pciids database is kept up to date with all their vendor, subvendor/reseller and product information.14:35
sean-k-mooneyjaypipes: well the vendor id/product id is important for other reasons such as knowing what driver is required for the device14:35
*** josecastroleon has joined #openstack-nova14:36
efriedIf that were the case, and if vendors could agree on names (IDs?) for capabilities across the board, I could get behind it.14:36
sean-k-mooneyefried: well intel has been pushing to try and stardise some of them in etsi and dmtf(redfish)14:36
efriedBut I think we need to have a realistic fallback plan so that, if such a nirvana does not come into being, operators will still be able to ask for a GPU by product ID.14:37
sean-k-mooneyfrom an openstack point of view that what the standard traits in os-traits are for14:37
jaypipesefried: well, if vendors want to enable their technology in OpenStack, they should work with us. Instead, nova needs to jump through a bunch of hacky hoops to work with vendors. ala https://review.openstack.org/#/c/579897/. which is complete bullshit, IMHO.14:37
efriedyes, I noticed you getting excited about that yesterday :)14:38
*** hongbin has joined #openstack-nova14:38
sean-k-mooneyjaypipes: ya.. well that is working around a limitation that nvidia put in there driver to prevent you using there gpus in vm unless you baught there datacenter gpus14:39
jaypipessean-k-mooney: I'm fully aware of that, yes.14:39
jaypipessean-k-mooney: please see my comment on whether we are OpenStack or WorkaroundForClosedStack14:40
jaypipessean-k-mooney: I think the answer has already been answered on that, actually. we have been VendorStack for about 5 years now.14:40
sean-k-mooneyoh i see it i have it open now. i agree that we should not need to do this but its a usecase they have14:40
sean-k-mooneyi think there are usecase beyond the gpu case for wanting to hide the hypervios signture14:41
jaypipesname one.14:42
openstackgerritBalazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client  https://review.openstack.org/58366714:42
sean-k-mooneylinux kernel dev14:42
*** maciejjozefczyk has joined #openstack-nova14:42
*** _ix has joined #openstack-nova14:42
sean-k-mooneyanyway i know that is a reach but for ovs-dpdk i dev i have used trick in the past to allow me to use openstack as a dev env14:42
*** READ10 has joined #openstack-nova14:43
sean-k-mooneysuch as setting the nic model to e1000 to i could test the non viutalised nic binding workflow14:43
jaypipessean-k-mooney: you're performing use case gymnastics here.14:44
sean-k-mooneyyes and no. in the early days of dpdk support i had to do thing like set the cpu-mode to host-passthough becasue i did not have the cpu feature reqiure to even compile it otherwise14:45
dansmithjaypipes: it's a common case for kernel development to use virtualization to simulate environments for that dev.. like fake numa nodes, pci devices, etc14:46
sean-k-mooneyas i said i was reaching a bit but if i want to do driver dev and i want to do that in a vm on openstack then i might need to hide the hypervisor signiture. that said i also recognise that  openstack is not a virtualisation stack14:46
dansmithjaypipes: that's not to say that I think IaaS really needs to worry about such things14:46
*** maciejjozefczyk has quit IRC14:47
dansmithjaypipes: given that developers can use bare hypervisors on their laptops to do that with much better control14:47
*** tbachman has quit IRC14:47
sean-k-mooneydansmith: yep that said for ci of that code openstack is a much more compelling option14:47
dansmithyup, CI is another reasonable use case imho14:48
*** maciejjozefczyk has joined #openstack-nova14:48
*** maciejjozefczyk has quit IRC14:48
sean-k-mooneythe alternitive without hypervior hiding is ironic but that is arguable more complex. the difference is the complexity is on the enduer not the technical debt nova has to maintain14:49
dansmithsean-k-mooney: note that I'm defending the dev case for needing finer grained control, but *not* defending the hypervisor-hiding case14:49
dansmiththe latter is mostly total closed-source proprietary BS14:49
jaypipessean-k-mooney, dansmith: in that case, the use case would be to be able to *set* the vendor ID to a particular value, not hardcode it to "1234567890ab" just for the HyperV emulator just to avoid licensing issues with Nvidia: https://review.openstack.org/#/c/579897/4/nova/virt/libvirt/config.py@224114:50
*** priteau has quit IRC14:50
lyarwoods10: hey, thanks for the report, so what's the actual knock of impact of update_available_resource taking longer here? Scheduling accuracy?14:50
sean-k-mooneyjaypipes: well i had originally asked shoudl we just remove the hypervier section entirly if hiding was asked for to fully hide the fact we were in a vm but that has performance impacts14:51
sean-k-mooneybut yes making this more generic to allow setting a spefic vendor id would be just as vaild14:52
s10lyarwood: It looks like, when I try to live migrate instances from this host, all operations should wait for this update_available_resource every minute. So I have a small live migration window: 30 seconds every minute, only after update_available_resource ends and before it starts.14:53
sean-k-mooneys10: why?14:54
jaypipess10: why are you running update_available_resource every minute?14:55
*** josecastroleon has quit IRC14:55
s10sean-k-mooney: I don't know, that's what I see in logs. No live migration could be performed during update_available_resources. If change libvirt/driver.py to like before first commits, migrations goes without pause.14:55
cdentjaypipes: that's the periodic job default14:56
sean-k-mooneys10: well that makes more sense14:56
jaypipesquoi? wtf is it that often? :(14:56
sean-k-mooneywe are running the periodic job on a green thread14:56
cdentjaypipes: dunno, but that's what it is14:56
jaypipessean-k-mooney: there is a semaphore lock around it.14:56
*** hoonetorg has quit IRC14:57
*** josecastroleon has joined #openstack-nova14:57
sean-k-mooneyso if update_available_resources is taking 30sec on a node with 150 instance then the compute agent would be tied up for 30secs every minute executing it14:57
s10jaypipes: because update_resources_interval run with default periodic interval with default value update_resources_interval=0, and periodic_task_interval=60 by default.14:58
sean-k-mooneyjaypipes: the point im trying to make is the periodic jobs are time sharing with the rest of the compute agent14:58
stephenfinefried: Does POWER expose NUMA to the OS?14:59
sean-k-mooneystephenfin: as in powerpc ?14:59
stephenfinsean-k-mooney: up14:59
dansmithsean-k-mooney: but most of that is waiting for IO so it's not keeping the process busy14:59
mdboothlyarwood: I have another meeting now, so I've chucked a brain dump in the functional failure review.14:59
efriedstephenfin: My understanding is that POWER handles NUMA under the covers, and does it well enough that the deployer doesn't need control.14:59
sean-k-mooneystephenfin: numa is exposed to linux via the bios so it should would the same on powerpc15:00
sean-k-mooneythe quest is does the hypervior expose it or not15:00
stephenfinefried: Is that the hardware doing the work or the hypervisor?15:00
stephenfinsean-k-mooney: yeah ^15:00
stephenfin(the work of abstracting NUMA'ness)15:00
*** jaosorior has quit IRC15:01
sean-k-mooneydansmith: good point it shoudl yeild exection15:01
*** panda|rover is now known as panda|backin2h15:01
jaypipescdent, dansmith: is there any reason now that placement claims are doing most of the resource consumption work (in an atomic manner) that we can't set the update_available_resource default to something sensible like 15 minutes?15:01
* cdent thinks15:02
sean-k-mooneyjaypipes: just trying to think is there any late claim still dont in the compute node. i think not15:02
cdentthere's not15:02
dansmithjaypipes: I would have to look, because I think we have other things hung off that process15:02
cdentjaypipes: do we do anything with host states and Filters where that info needs to be up to date?15:03
cdentnon-placement-related filtering15:03
sean-k-mooneystephenfin: im pretty sure that hyperv hides the numa info from nova15:03
jrolljaypipes: one catch would be picking up new/changed ironic nodes, but maybe we can just doc that caveat15:04
sean-k-mooneystephenfin: i know they have made numa affinity of instance memory a hypervir config option15:04
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Reduce calls to qemu-img during update_available_resource  https://review.openstack.org/58951315:04
jroll"changed" e.g. ironic node goes to / comes out of maintenance mode15:05
lyarwoods10: ^ we can remove half of the qemu-img calls with that15:05
*** mriedem is now known as mriedem_afk15:06
stephenfinsean-k-mooney: Fair enough. I was trying to write a high-level overview of NUMA but needed to figure out what platforms would be affected. I'll just leave that piece out :)15:07
*** nicolasbock has joined #openstack-nova15:07
*** Luzi has quit IRC15:07
sean-k-mooneystephenfin: i would recommend putting it in the context of which hypervior rather then which plathform e.g. achitecuter if you do15:08
sean-k-mooneyi would expect power-kvm via libvirt to work the same as kvm on x86 but i would expect powervm direct driver to work very different15:09
*** dpawlik has quit IRC15:10
stephenfinsean-k-mooney: Good call. I'll do that15:11
*** pcaruana has quit IRC15:11
cdentjaypipes: we could consider have a different periodic job for the placement calls? There's a bazillion periodic jobs in the compute manager already, aren't there? What's one more :()15:11
sean-k-mooneycdent: they mainly all share the same config option today  however. we dont have a interval time option for each15:12
*** hemna_ has joined #openstack-nova15:12
*** s10 has quit IRC15:12
sean-k-mooneythat said i dont see an issue with adding a new config option and default it to the current periodic job interval15:12
sean-k-mooneyor if you want the behavior change to be implicit on upgrade then default to 15 mins or whatever makes sense15:13
cdentsean-k-mooney: there are several *_internal conf settings in nova/conf/compute.py and they all fall back to the default for any periodic in an oslo_service based service15:13
*** s10 has joined #openstack-nova15:15
efriedstephenfin, sean-k-mooney: Sorry, had to step away for a sec. From your point of view, the "hardware" and the "hypervisor" are the same thing.15:15
sean-k-mooneyefried: meaning we cant see the hardware so we should only care about what the hypervior reporst15:16
efriedsean-k-mooney: That's probably a sane way to think about it. I don't think the hypervisor reports NUMA cells at all. You see procs and memory.15:17
efriedI'm not an expert here, for sure.15:17
sean-k-mooneyefried: that is how the host is reported form hyperv also as far as i know15:17
sean-k-mooneyjust one big pool for ram and cpus15:18
sean-k-mooneythey have a hypervior config option (not in openstack) to turn on numa affinity at the host level15:18
efriedyeah, see, I think NUMA affinity Just Happens (tm) on Power.15:19
jaypipescdent: perhaps, yes15:19
sean-k-mooneyefried: i dont think vsphere exposes it either. its just libvirt that exposes it  as far as i can tell15:19
efriedfrickin libvirt <rolls eyes>15:19
sean-k-mooneyhehe well libvirt exposes it because libvirt is not a hypervior its an hypervior abstraction layer and isnce qemu does not do this by default it fell to nova to fill in the gapps15:20
jaypipessean-k-mooney: WRONG! libvirt is an XML file management system.15:22
* cdent uses xpath and xslt to launch vms15:22
sean-k-mooneyjaypipes: lol15:22
sean-k-mooneyjaypipes: it does a little bit more then that but more or less15:22
* jaypipes crafts artisanal VMs from unicorns and rainbows15:23
*** Bhujay has joined #openstack-nova15:23
sean-k-mooneyyou know it could be argured that a new service could be insrted between nova and libvirt that did all the nfv stuff and the libvirt driver could be made way simpeler15:24
cdentI sure hope those unicorns are free range15:25
*** priteau has joined #openstack-nova15:26
*** tbachman has joined #openstack-nova15:27
*** adrianc__ has quit IRC15:28
*** Sigyn2 has joined #openstack-nova15:29
*** Sigyn2 has joined #openstack-nova15:29
*** Sigyn2 has joined #openstack-nova15:30
*** Sigyn2 has joined #openstack-nova15:30
*** Sigyn2 has joined #openstack-nova15:31
*** Sigyn2 has joined #openstack-nova15:31
*** gyee has joined #openstack-nova15:32
s10lyarwood: with https://review.openstack.org/589513 update_available_resource() lasts 10 seconds for 100 instances instead of 20 seconds without.15:35
*** dpawlik has joined #openstack-nova15:38
lyarwoods10: kk, that's a single qemu-img info call per disk to avoid the other issues fixed by the original changes15:39
*** s10 has quit IRC15:39
dansmiths10: this sounds like a silly question, but what does it matter how long that takes?15:39
dansmithit's not blocking other work right?15:39
efriedstephenfin: I think https://review.openstack.org/#/c/588422/ is ready for your +A now15:40
*** s10 has joined #openstack-nova15:40
lyarwooddansmith: I did ask above and it's causing issues with LM apparently15:40
dansmithlyarwood: oh sorry I missed that.. maybe because it's holding the RT semaphore?15:41
lyarwooddansmith: yes I think so15:41
dansmithokay makes sense I guess15:41
dansmithlyarwood: the new call checks the allocated value, which doesn't change over the lifecycle of the image right?15:42
sean-k-mooneydansmith: correct it should not15:42
*** dpawlik has quit IRC15:42
sean-k-mooneydansmith: unless the call is checking the actul used space on the host and we are not preallocting ?15:43
lyarwooddansmith: allocated can, virtual shouldn't15:43
lyarwoodyeah pretty much15:43
dansmithlyarwood: but which are we looking at now?15:43
lyarwooddansmith: now it's just a single qemu-img info call that grabs both15:44
sean-k-mooneylyarwood: the resouce track should be tracking the virtual amont right not the currently allcoated amount?15:44
dansmithlyarwood: ah, both so one of them is dynamic and we can't really cache it yeah?15:44
lyarwoodsean-k-mooney: the allocated amount is used to work out over commit etc15:44
sean-k-mooneylyarwood: that seams wrong the maxium it could use should be used for that15:44
dansmithsean-k-mooney: IIRC, this is not for reporting to placement but for some of the legacy values (right lyarwood ?)15:45
jaypipesdansmith: what are your thoughts on https://bugs.launchpad.net/nova/+bug/1784826? I can't tell what the expected behaviour there should be...15:45
openstackLaunchpad bug 1784826 in OpenStack Compute (nova) "Guest remain in origin host after evacuate and unset force-down nova-compute" [Undecided,In progress] - Assigned to huanhongda (hongda)15:45
dansmithbecause auditing on the compute node doesn't change what placement is counting15:45
lyarwooddansmith: yeah correct, this doesn't make it to placement AFAIK15:46
dansmithlyarwood: so we really only need to be collecting this info for old RT, which is ignored if you don't have DiskFilter enabled, and only useful for people using the deprecated CachingScheduler yeah?15:47
sean-k-mooneylyarwood: what advantage is there to using allocated though? since placement would already be using its allocation ratio to filter the hosts before the compute ever need to check its over_subsciption ratio setting15:47
lyarwooddansmith: in master but this series was backported all the way back to ocata15:47
lyarwooddansmith: wasn't it still used back then?15:48
dansmithlyarwood: yeah I know, but diskfilter hasn't been needed since claims in the scheduler15:48
dansmith*maybe* ocata, but not pike or later IIRC15:48
*** maciejjozefczyk has joined #openstack-nova15:50
lyarwoodsean-k-mooney: right, I think the only reason this came up before is that it's visable from the CLI15:50
dansmithlyarwood: anyway, just wondering if maybe we should add a workaround config tweak to disable this extra inspection so you can turn it off if you're not using DiskFilter and/or are on a new enough version15:50
lyarwooddansmith: yeah that sounds like the way to go tbh15:51
dansmithlyarwood: ah, well, that makes it even more pain than gain if it was just "make the numbers line up" :)15:51
dansmithlyarwood: probably best to get some sign-offs from the PTL(s) of the affected releases, but that's kinda where I'm thinking15:52
sean-k-mooneylyarwood: so if its visable via the cli i think thats even more reason to use the virtual not allocated disk size to have it corralte with placement15:52
sean-k-mooneylyarwood: that said that would be a behavior change i guess.15:52
sean-k-mooneylyarwood: i assume this is visable via the hyperviors api?15:53
lyarwoodsean-k-mooney: yeah via a hypervisor-show - https://bugs.launchpad.net/nova/+bug/176448915:54
openstackLaunchpad bug 1764489 in OpenStack Compute (nova) queens "Preallocated disks are deducted twice from disk_available_least when using preallocated_images = space" [Medium,Fix committed] - Assigned to Lee Yarwood (lyarwood)15:54
*** maciejjozefczyk has quit IRC15:55
sean-k-mooneyright so its messing up  disk_available_least not local_gb_used...15:56
*** adrianc__ has joined #openstack-nova15:58
*** Bhujay has quit IRC16:00
openstackgerritBalazs Gibizer proposed openstack/nova master: Use placement 1.28 in scheduler report client  https://review.openstack.org/58366716:01
*** luksky has quit IRC16:02
*** tbachman has quit IRC16:08
*** namnh has quit IRC16:10
mdboothlyarwood: I'm partially through the functional failure. Got past the shelve/unshelve failure, error is now in _test_attach_volume_error16:11
*** mriedem_afk has quit IRC16:11
mdboothlyarwood: However, I'm going to leave shortly, it doesn't pass yet, and there are still print statements all over it16:11
mdboothlyarwood: So I'm not inclined to push it this evening unless that would be especially helpful to you16:12
lyarwoodmdbooth: yeah np leave it for now16:15
*** jaypipes_ has joined #openstack-nova16:16
*** jaypipes has quit IRC16:17
*** jaypipes_ has quit IRC16:17
*** burt has quit IRC16:19
*** jpena is now known as jpena|off16:21
*** burt has joined #openstack-nova16:22
*** lbragstad[m] has quit IRC16:32
*** jaypipes has joined #openstack-nova16:33
*** tbachman has joined #openstack-nova16:36
*** s10 has quit IRC16:38
*** imacdonn has quit IRC16:38
*** imacdonn has joined #openstack-nova16:38
*** gbarros has quit IRC16:39
*** tbachman_ has joined #openstack-nova16:40
*** takedakn has joined #openstack-nova16:42
*** tbachman has quit IRC16:42
*** tbachman_ is now known as tbachman16:42
*** moshele has joined #openstack-nova16:44
*** gbarros has joined #openstack-nova16:44
*** takedakn has left #openstack-nova16:45
*** tssurya has quit IRC16:45
*** janki has quit IRC16:47
openstackgerritSergii Golovatiuk proposed openstack/nova master: Fix URI for IPv6  https://review.openstack.org/58954816:50
melwittdansmith: this review looks in your wheelhouse https://review.openstack.org/58942516:57
dansmithyeah, gibi and I have been discussing it16:58
dansmithI'll hit it in a few16:58
*** sahid has quit IRC16:58
*** derekh has quit IRC17:00
*** panda|backin2h is now known as panda|rover17:08
efriedstephenfin, sean-k-mooney: I found out a little more about NUMA affinity on Power. It is indeed done dynamically; you can't ask for it to be done a certain way. However, the NUMA cell and affinity information *is* exposed to the guest, so if the guest has the savvy and ability to use such information for clever scheduling of threads/processes, it can do so.17:10
efriedstephenfin, sean-k-mooney: Also of note, the system may swizzle affinities dynamically for various reasons (including, apparently, at the behest of a generic "improve my affinity" background job you can run) so the guest needs to be able to hang with that.17:11
*** ThomasWhite has quit IRC17:15
openstackgerritDan Smith proposed openstack/nova master: Fix resize revert to use non-legacy alloc handling  https://review.openstack.org/58942517:26
*** harlowja has joined #openstack-nova17:31
*** adrianc__ has quit IRC17:35
sean-k-mooneyefried: stephenfin so am i right in saying i can request a partcalar virtual numa topology for the guest but powervm will dynmically map that as it sees fit and adjust as needed17:38
*** adrianc__ has joined #openstack-nova17:41
*** harlowja has quit IRC17:43
*** markvoelker_ has quit IRC17:45
*** moshele has quit IRC17:50
*** gbarros has quit IRC18:06
*** Swami has joined #openstack-nova18:12
*** luksky has joined #openstack-nova18:14
efriedsean-k-mooney: You can *not* request a particular topology for a guest. PowerVM will set it up dynamically and adjust as needed.18:25
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Reduce calls to qemu-img during update_available_resource  https://review.openstack.org/58951318:29
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT  https://review.openstack.org/58956718:29
lyarwooddansmith: ^ was that what you had in mind for the qemu-img workaround? If it is I'll clean it up and ping the ML to see if others agree.18:29
dansmithlyarwood: kinda, but we weren't just returning zero before right? what were we doing?18:30
dansmithI was thinking just make the workaround trigger "do the old thing"18:31
lyarwooddansmith: well before we were calling os.path.getsize18:31
dansmithah right, so .. I would just make the workaround revert to that I think18:32
sean-k-mooneyefried: does that imply that from within the guest i can see that the topoloyg is chaning?18:45
efriedsean-k-mooney: Yeah. Is that crazy?18:45
sean-k-mooneyefried: yes18:45
efriedsean-k-mooney: It's possible that the topo only changes if you run that job I mentioned.18:45
sean-k-mooneywindows and linux are really not going to do the right thing18:46
sean-k-mooneyim pretry sure that neither will check the numa topology on an ongoing basis18:46
sean-k-mooneyso the linux schduler is going to detect the numa topology when it start up and then contiue to use the same mappings for the liftime of the vm18:47
*** openstackgerrit has quit IRC18:49
*** moshele has joined #openstack-nova18:50
sean-k-mooneyif the hyperviors hide the phyical topology from the guest but dynamically rempas the ram across the host numa nodes then atleast the info present to the guest is consitent and it can try to make sane decisions but if the virtual tolopogy the guest sees is dynamic things like numactl are going to not work correctly in the guest18:51
efriedsean-k-mooney: Yeah, I don't know to this level of detail. I imagine it'd be in response to other LPARs going away or whatever, and thus freeing up the system to shuffle my stuff to make it more efficient. Or maybe I hot plug a device and now it's better to move my procs/mem to the cell that's affined to that device. But I don't know how that actually appears to the guest.18:53
*** adrianc__ has quit IRC18:53
sean-k-mooneyefried: well the guest view is what i was asking about when i said can you request a virtual numa toplogy. as long as the geust view does not change the hypervior can do what it like underneath without worriying it will break the guest18:55
sean-k-mooneythe guest can still make bad desisions if the hyperviour dicides to split a virutal numa node across host phyical numa nodes but it wont break anything18:56
sean-k-mooneyif the gust view is changing things like hugepages within the guest would break18:57
efriedsean-k-mooney: When you say "request a virtual numa topology" do you mean "ask what the topology is" or do you mean "request that the virtual topology look like XYZ" ?18:57
efrieds/look like/get set up as/18:57
sean-k-mooneyi mean "power vm please create a vm that to the guest looks like it has 2 numa node, do whatever you like on the host"18:58
sean-k-mooneythat is effectivly what hw:numa_nodes=2 means18:59
efriedsean-k-mooney: Right, okay, so I'm saying you definitely can't do that in PowerVM. I'm not sure whether you can do it in PowerKVM.18:59
sean-k-mooneyok cool18:59
sean-k-mooneyyou can do that with hyperv i think but i dont think i guarntees they are mapped to 2 phyical host numa nodes19:00
sean-k-mooneyor rather if they are mapped/affinitised to host numa nodes is governed by a hypervior config option as far as i know19:01
*** tbachman has quit IRC19:14
sean-k-mooneyefried: its burried in the docs but ya it looks like you can do numa affinity with powerkvm https://www.ibm.com/support/knowledgecenter/SSZJY4_3.1.0/liabp/liabphotplugcpunuma.htm19:14
sean-k-mooneyyou can also do cpu pinning19:15
*** maciejjozefczyk has joined #openstack-nova19:15
sean-k-mooneyyep and discribe the virual cpu topoloyg in terms of socket,cores and thread19:15
efriedsean-k-mooney: This appears to be talking about just virtual topology, yah?19:16
sean-k-mooneyyes unlsee you can use the numa tune elements19:16
*** s10 has joined #openstack-nova19:17
sean-k-mooneyefried: so ya this all apears to be virtual thought the docs are so hard to navigate one could almost belive ibm did not want you to use those features :P19:18
*** moshele has quit IRC19:18
efriedsean-k-mooney: Oh, don't worry, that's not specific to NUMA.19:18
sean-k-mooneyhaha well i partcalarly like how the numa stuff is not linked too for the "manage processor and memory" section https://www.ibm.com/support/knowledgecenter/en/SSZJY4_3.1.0/liabp/liabpmanageprocessors.htm19:20
efriedsean-k-mooney: I think I can now officially say you've spent more time on the IBM "knowledge" center in the past twenty minutes than I have in the past five years.19:22
sean-k-mooneyhaha well from what i can tell if you used the libvirt kvm driver an pointed it at a power-kvm host i think it would "just work" with all of the nuam and epa stuff we do for normal kvm19:23
*** mriedem has joined #openstack-nova19:24
efriedsean-k-mooney: I have found out that there is some kind of notification the OS can get when affinities change.19:24
sean-k-mooneyproably some ahci interupt or something19:24
sean-k-mooneyi know there are intrupts that can be sent for memory and cpu hotplug which argubly could change numa topology19:25
sean-k-mooneyso it not out of the question that linux or widnows could handel a dynmaic topology if you were to notify it. not sure i would like a numa node to go away but adding one might be ok19:26
*** openstackgerrit has joined #openstack-nova19:27
openstackgerritLee Yarwood proposed openstack/nova master: WIP libvirt: Add workaround to stop use of qemu-img by the RT  https://review.openstack.org/58956719:27
efriedsean-k-mooney: I think it's more likely to be like, "You thought procs 0 and 2 were next to each other, and 1 and 3 were next to each other. Well, now it's 0-1 and 2-3. Deal."19:27
sean-k-mooneyhaha ya maybe. if you ever test it let me know19:28
efriedsean-k-mooney: I feel confident in saying that will never happen.19:28
efried...for some value of "never".19:28
sean-k-mooneyefried: you obvioulsy are not spending enough time talking to telcos19:29
efriedIf it's anything like talking to the AT&T help desk, I assure you I don't want to spend any more.19:30
*** READ10 has quit IRC19:40
*** cdent has quit IRC19:42
openstackgerritMatt Riedemann proposed openstack/nova stable/pike: Make ResourceTracker.stats node-specific  https://review.openstack.org/58803719:46
mriedemwas there a zuul reset or something such that we need to recheck things?19:49
dansmith513 backlog,19:51
dansmithso I imagine it's catching up19:51
dansmithbacklog has been up and down all day so I figured something was going on19:51
sean-k-mooneynothing on the openstack-infra twitter for the last 4 days. then normally post somthing if they need to do a reset19:53
sean-k-mooneywell it auto posts when then set a status message on the infra channel but in either case the gate is proably just busy/backed up19:54
openstackgerritMatt Riedemann proposed openstack/nova stable/ocata: Make ResourceTracker.stats node-specific  https://review.openstack.org/58807720:00
*** tbachman has joined #openstack-nova20:16
openstackgerritMatt Riedemann proposed openstack/nova master: doc: mark the max microversion for rocky  https://review.openstack.org/58959820:20
*** hoonetorg has joined #openstack-nova20:30
maciejjozefczykmriedem:  efried please check my response: https://review.openstack.org/#/c/52002420:31
efriedThanks maciejjozefczyk20:32
maciejjozefczykefried:  u2 for pinging me, I forgot about this topic20:32
mriedemmelwitt: comments in the reno prelude https://review.openstack.org/#/c/589303/20:37
*** maciejjozefczyk has quit IRC20:42
-openstackstatus- NOTICE: Due to a bug, Zuul has been unable to report on cherry-picked changes over the last 24 hours. This has now been fixed; if you encounter a cherry-picked change missing its results (or was unable to merge), please recheck now.20:43
mriedemjroll: you might want to check the rebalance question in here https://review.openstack.org/#/c/520024/20:43
*** dpawlik has joined #openstack-nova20:45
*** nicolasbock has quit IRC20:45
jroll"want" is a weird word to use :P20:49
mriedemi'm just asking for a witness that i asked for an ironic person to look before we break the rebalance stuff20:53
jrollso I'm thinking it's fine to remove that, but... I don't know everything that _update does and I'm not sure things like _update_usages_from_instances() and such will do the right thing, if we haven't updated which compute hosts we moved active ironic instances to20:54
jrollI feel like I'm nervous to remove it20:55
sean-k-mooneymriedem: is there an ironic job we can trigger maybe via the experimental pipelien to test that patch20:56
jrollsean-k-mooney: there isn't one that actively kills compute hosts and such to trigger rebalances20:57
sean-k-mooneyjroll: it looks like the only ironic job that did run on this skipp 22 out of the 23 test ... http://logs.openstack.org/24/520024/9/check/ironic-tempest-dsvm-ipa-wholedisk-bios-agent_ipmitool-tinyipa/1fe9001/testr_results.html.gz20:58
*** slaweq has quit IRC20:58
*** tbachman has quit IRC21:06
jrollsean-k-mooney: yes, we run a very small set of tests on nova changes because our tests take a long time21:07
jrollsean-k-mooney: that doesn't change the fact that there aren't any full-stack tests that exercise this code21:08
jrollbecause it's really hard to do - would involve multiple nova-compute instances and orchestrating bringing them up and down and such21:08
efriedjust mock everything21:08
* jroll thinks that's probably sarcasm21:09
jrollanyway, I'm gonna post my comments on the change and then not computer anymore today. see y'all later21:09
sean-k-mooneyjroll: im not saying there would be test for this just taking note of the fact that the ironic testing is much lighter then i would have expected21:11
sean-k-mooneyjroll: that is not a slight againt ironic i just would have expect more tests even if they only ran in the gate pipeline and not check21:12
jrollsean-k-mooney: figure out how to get nested virt in the gate and we can run them fast enough to test more :)21:12
jrollthat one instance boot takes like 20 minutes or something iirc21:13
sean-k-mooneywhell i honestly dont know why we dont use nested virt in the gate other then the fact that rackspace provides xen based instance and most of our test would like kvm21:14
jrollthere's a long history of why that I don't have time to get into21:14
* jroll really out now21:14
sean-k-mooneyjroll: i have been trying to get nested virt in the gate since 2013 im well awaree of the history but no worries21:14
* sean-k-mooney sometimes i hate dpdk ...21:17
melwittonly sometimes?21:18
sean-k-mooneyi just spent 6 hours debuging network connectivit issue with ovs-dpdk because it has buggy numa detection logic21:18
sean-k-mooneymelwitt: normally it works fine but when it doesnt its a pain in the ass to figure out why21:19
*** awaugama has quit IRC21:21
*** slaweq has joined #openstack-nova21:23
*** tbachman has joined #openstack-nova21:26
*** edmondsw has quit IRC21:29
mriedemdansmith: do you have any idea why we have a Service.availability_zone field on the versioned object?21:31
mriedemi see that we try to lazy-load it for notifications21:31
mriedem    2018-08-07 17:24:53,165 DEBUG [nova.objects.service] Lazy-loading 'availability_zone' on Service id 221:31
mriedem    2018-08-07 17:24:53,166 DEBUG [nova.notifications.objects.base] Defaulting the value of the field 'availability_zone' to None in ServiceStatusPayload due to 'Object action obj_load_attr failed because: attribute availability_zone not lazy-loadable'21:31
mriedemi guess for https://github.com/openstack/nova/commit/a2c6838ff5ff095940a76ebd4d578e24575c30d8#diff-b9be5fa188b7efd457da79e9c543344bR11021:34
sean-k-mooneyjaypipes: are you  around?21:39
*** tbachman has quit IRC21:42
sean-k-mooneyi just realised that we missed21:43
sean-k-mooney* i just realised that https://review.openstack.org/#/c/587378/3/vif_plug_ovs/ovs.py misses passing the ovsdb_connection on on of the vhost-user code paths. should i just submit a patch for the missing fucntion or revert and submit an updated versions21:45
sean-k-mooneyim thinking just add a patch on top the get the missing function call but just said i would ask21:46
*** tbachman has joined #openstack-nova21:47
*** bitskrie1 has quit IRC21:54
*** slagle has quit IRC21:56
sean-k-mooneymriedem: by the way im assuming you did not have time to test livemigrating between different neutron backends as part of the multiple port binding blueprint?22:01
mriedemsean-k-mooney: mlavalle did it between ovs and linuxbridge using neutron directly, not via nova22:02
mriedemi don't have a mixed vif type env setup no22:02
sean-k-mooneymriedem: now that i have figured out why ovs-dpdk was not working from me ill try an test it out later this week. ill also try it via linux bridge if i get a chance22:02
sean-k-mooneycool no worries22:02
sean-k-mooneyi normaly have ovs, ovs-dpdk and linux bride deployed concurrnetly or at least i did before i move. ill test the matirx of all 3 setups and let you know how it goes22:04
*** dpawlik has quit IRC22:09
*** owalsh_ has joined #openstack-nova22:09
*** priteau has quit IRC22:09
*** owalsh has quit IRC22:12
*** owalsh- has joined #openstack-nova22:12
*** owalsh- is now known as owalsh22:13
*** beagles has quit IRC22:13
*** owalsh_ has quit IRC22:14
*** beagles has joined #openstack-nova22:20
*** rcernin has joined #openstack-nova22:20
mriedemKevin_Zheng: comments all over https://review.openstack.org/#/q/topic:bug/1781880+(status:open+OR+status:merged) so it should be clear to update now22:30
mriedemand abandon the functional test since it will never work22:30
*** luksky has quit IRC22:37
openstackgerritMatt Riedemann proposed openstack/nova master: Update really old comments about vmware hosts managing multiple nodes  https://review.openstack.org/58966622:39
*** hongbin has quit IRC22:39
openstackgerritMatt Riedemann proposed openstack/nova master: Add microversion info in the os-server-groups API samples  https://review.openstack.org/58900622:47
melwittmriedem: I'm not sure what to make of this comment after trying to make the changes to make things fail if a cell raises an exception https://review.openstack.org/#/c/540258/10/nova/scheduler/utils.py@84322:50
melwittit's not clear to me if a cell raising an exception will be part of our "down cell" handling down the line or if we should expect to potentially fail a server create with a 500 if a cell raises an exception when we query it22:51
melwittpre-down-cell handling22:51
melwittinitially I was thinking yes, it makes sense to fail the create if a cell raises an exception, but that would propagate up to the user as a 500 because there's something wrong with a cell, and then I became unsure if that falls under the "down cell" stuff or if it's just something we should do now22:53
*** edmondsw has joined #openstack-nova22:54
efriedmriedem: I assume that +1 is so we wait until stein to land it22:58
efried(the "Update resources once" patch)22:58
mriedemefried: yeah22:58
efriedmriedem: I'm going to -2 it just in case, but I'm with ya.22:59
mriedemefried: don't think you need to -222:59
*** edmondsw has quit IRC22:59
openstackgerritMerged openstack/os-traits master: Update reno for stable/rocky  https://review.openstack.org/58610322:59
mriedemefried: i -W'ed it23:00
efriedmriedem: You mean because no other cores are going to "accidentally" merge it? :)23:00
mriedemmelwitt: how does a cell raise an exception?23:00
mriedemif the called function does?23:00
mriedemlike the DB API query explodes or something?23:00
melwittmriedem: yeah, exactly. if whatever is called under target_cell raises23:00
efriedmriedem: ack, +2ed23:00
melwittit makes sense to fail the boot over that if someone is trying to boot with affinity and a cell somehow raises an exception (gibi pointed it out on the review)23:01
mriedemmelwitt: it wouldn't be a 500 to the user for server create23:01
mriedemb/c we've already cast from api to conductor which is what calls setup_instance_group23:01
melwittI'm just getting mixed up about whether that is in the scope of the bug fix I'm working on right now, or if that's going to be later on with the handling of a down cell work23:02
mriedemcold/live migrate + evacuate + unshelve might return a 500....23:02
mriedemwell gibi said it was ok to add a TODO and deal with it later yeah?23:02
melwittoh, okay. my bad, I was thinking setup_instance_group was called from compute/api but was mistaken23:02
mriedemit's called from conductor, but whether or not we've already returned 202 to the user depends on the operation23:03
melwitthe said the TODO for the "did not respond" but for the raised exception case, suggeste failing the boot23:03
mriedemi would lump that into tssurya's bp in stein23:03
mriedemor as a separate bug fix23:03
mriedemit's not the issue for this patch23:03
melwittokay. thanks. that's what I was thinking as I went to add it, it's adding a lot to the scope23:04
mriedemmelwitt: i also looked at https://review.openstack.org/#/c/582332/ and i'm not sure what it changes,23:04
mriedembut i did notice we're logging in a greenthread in one place there and i thought that was a real no-no23:05
mriedemdansmith: yeah? ^23:05
*** _ix has quit IRC23:05
melwittmriedem: I'm not 100% sure where the methods that were changed get called during the scheduler run, so I have to look at that to see if the result is visible during a gate run23:07
melwittbasically, any time those methods run, they will replace the thread local context that oslo.context stores underneath, and oslo.log pulls from that to log request-ids23:07
melwittso what would happen is a request-id for a thread would change midway if one of the methods that created RequestContext without overwrite=False ran during it23:08
mriedemboth run on startup23:08
melwittokay, I think that's why it wouldn't show up23:08
mriedemwell, one does23:08
mriedemthe cells refresh one23:08
mriedemthe other doesn't, presumably b/c it's in a greenthread23:08
mriedemor maybe b/c we have'nt discovered any hosts yet23:09
*** itlinux_ has joined #openstack-nova23:13
melwittyeah, looks like the greenthread is spawned during startup so it's effectively during startup too23:15
openstackgerritMerged openstack/nova master: [placement] Add version directives in the history doc  https://review.openstack.org/58939223:15
openstackgerritMerged openstack/nova master: Avoid joins in _server_group_count_members_by_user  https://review.openstack.org/58076423:16
openstackgerritMerged openstack/nova master: Use common functions in granular fixture  https://review.openstack.org/58811323:16
*** priteau has joined #openstack-nova23:17
melwittbut I see now, it doesn't run at all, don't find it in the log23:17
melwittoh, because logging in the greenthread is expected not to work?23:17
melwittoh, it's because [filter_scheduler]/track_instance_changes = False23:19
mriedemmelwitt: no i think it's because we disable CONF.filter_scheduler.track_instance_changes in superconductor mode in devstack23:19
mriedemgrenade runs in singleconductor mode so it's logged there23:20
melwittso let's see if there's a difference in that job before the patch23:20
melwittlooks like it, new request-id as of async_init_instance_info http://logs.openstack.org/58/540258/10/check/neutron-grenade/54f96c0/logs/screen-n-sch.txt.gz#_Jul_24_06_24_31_35306823:22
melwittwait, but I still see the request-id prior to that being logged too23:22
melwittoh, bc it has its own greenthread. duh. yeah so that one wouldn't show anything anyway23:28
melwittit can't cause a change in any other greenthread's local context23:29
* melwitt will bbl23:34
*** s10 has quit IRC23:37
*** s10 has joined #openstack-nova23:38
*** s10 has quit IRC23:38
jaypipessean-k-mooney: sure, just add a patch on top.23:38
*** s10 has joined #openstack-nova23:39
*** s10 has quit IRC23:39
*** s10 has joined #openstack-nova23:39
*** s10 has quit IRC23:40
*** s10 has joined #openstack-nova23:40
*** s10 has quit IRC23:41
*** s10 has joined #openstack-nova23:41
*** s10 has quit IRC23:41
*** Swami has quit IRC23:46
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: fix min_version for parent_provider_uuid in responses  https://review.openstack.org/57957723:47
openstackgerritMatt Riedemann proposed openstack/nova master: api-ref: Add descriptions for rebuild  https://review.openstack.org/58893123:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!