| mnaser | well, here goes nothing | 00:00 |
|---|---|---|
| mnaser | ok, conductor backup with patch, gonna try a live migration | 00:01 |
| mnaser | status=MIGRATING | 00:01 |
| mnaser | let me watch nova-compute to check everything is ok | 00:01 |
| mnaser | live migrated successfully *without* forcing host, let me try forcing host | 00:02 |
| *** moshele has quit IRC | 00:02 | |
| *** huanxie has quit IRC | 00:02 | |
| mnaser | mriedem: fixed for both forced host and unspecified host for live migrations | 00:03 |
| mriedem | sweet | 00:04 |
| mnaser | thank you so much, moving forwards i guess we'll have to find a way to clean this up | 00:04 |
| mriedem | about done with tests for the workaround | 00:04 |
| *** salv-orlando has quit IRC | 00:05 | |
| *** salv-orlando has joined #openstack-nova | 00:05 | |
| openstackgerrit | Merged openstack/nova master: Remove the objects which related to the old v2 API implementation https://review.openstack.org/519997 | 00:06 |
| openstackgerrit | Merged openstack/nova master: Remove the objects for describing the extension for v2.1 API https://review.openstack.org/519998 | 00:06 |
| *** huanxie has joined #openstack-nova | 00:06 | |
| mnaser | found some other VMs in error state which failed to delete ... "{"message": "Field `disabled' cannot be None", "code": 500, "details": " File \"/usr/lib/python2.7/site-packages/nova/compute/manager.py\", line 205," | 00:06 |
| mnaser | investigation round #2 | 00:07 |
| openstackgerrit | Merged openstack/nova master: libvirt: throw NotImplementedError if qga is not responsive when setting password https://review.openstack.org/523828 | 00:07 |
| openstackgerrit | Merged openstack/nova master: Update and complete volume attachments during resize https://review.openstack.org/527228 | 00:07 |
| *** kumarmn has joined #openstack-nova | 00:07 | |
| mriedem | mnaser: service record maybe? | 00:08 |
| mriedem | nova.services table | 00:08 |
| *** edmondsw has joined #openstack-nova | 00:08 | |
| *** tetsuro_ has joined #openstack-nova | 00:09 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround missing RequestSpec.project_id when moving an instance https://review.openstack.org/529185 | 00:09 |
| *** Brin has joined #openstack-nova | 00:09 | |
| mriedem | here is the workaround patch with tests ^ | 00:09 |
| mnaser | http://paste.openstack.org/show/629406/ | 00:09 |
| mnaser | got the full trace back, ill start digging into why | 00:09 |
| mnaser | i guess its a flavor where disabled is none | 00:10 |
| mriedem | yeah... | 00:10 |
| mnaser | instance created at 2015-03-24T09:14:09Z | 00:10 |
| mriedem | so i think in that case, the flavor we're using is pulled out of the embedded flavor stored with the instance | 00:10 |
| mriedem | from the instance_extra table | 00:10 |
| *** salv-orlando has quit IRC | 00:11 | |
| mnaser | most very likely | 00:11 |
| mriedem | not sure why disabled would be null | 00:11 |
| mnaser | let me double check the db | 00:11 |
| mriedem | the disabled column in the db defaults to False | 00:12 |
| mnaser | this is from instance_extra? | 00:13 |
| *** edmondsw has quit IRC | 00:13 | |
| *** jdurgin has joined #openstack-nova | 00:14 | |
| mriedem | yeah, self.flavor = flavor_payload.FlavorPayload(flavor=instance.flavor) | 00:14 |
| mnaser | http://paste.openstack.org/show/629410/ | 00:14 |
| mriedem | instance.flavor pulls the embedded flavor out of the instance_extra table | 00:14 |
| mnaser | if i recall, the embedded flavor didnt exist before | 00:15 |
| mnaser | could it be possible that when whatever migration ran that did that, it didnt add it? | 00:16 |
| mriedem | we started embedding the flavor in kilo | 00:16 |
| mriedem | flavors were the first online data migration in nova i think | 00:16 |
| mriedem | moving the flavor from the instance_system_metadata table | 00:16 |
| mnaser | i remember something along these lines | 00:16 |
| mriedem | so the serialized form of this flavor says it's version 1.1 of the object which has the disabled field, but maybe it just wasn't set when the flavor was serialized with the instance... | 00:16 |
| mriedem | if you look at a more recent instance, is it set there? | 00:17 |
| mnaser | i think thats what happened, the disabled field seems to be part of when the object was first created | 00:17 |
| mnaser | mriedem: willing to be it is, but let me check to be sure | 00:17 |
| mnaser | mriedem: yup | 00:18 |
| mriedem | ok | 00:18 |
| mnaser | https://github.com/openstack/nova/commit/b4f07f42763489ddf459e4e795477530bdcbd994 | 00:23 |
| mriedem | this is where we set the flavor on the instance during create https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1450 | 00:26 |
| mnaser | im not sure how we ended up in that situation honestly | 00:26 |
| *** threestrands has joined #openstack-nova | 00:26 | |
| *** threestrands has quit IRC | 00:26 | |
| *** threestrands has joined #openstack-nova | 00:26 | |
| *** yangyapeng has quit IRC | 00:27 | |
| mnaser | actually, instance was created on '2015-03-24T09:14:09Z' and instance_extra record was on '2015-03-24 09:14:09' which means that it wasnt part of the migration (sorry if you knew this but it just clicked for me) | 00:27 |
| *** yangyapeng has joined #openstack-nova | 00:28 | |
| mnaser | updated on 2017-10-04 22:30:33 hmm | 00:28 |
| mnaser | ok that was the date the instance was attempted to be deleted | 00:29 |
| *** mlavalle has quit IRC | 00:30 | |
| *** takashin has joined #openstack-nova | 00:31 | |
| mriedem | ok hacking something up | 00:32 |
| *** yangyapeng has quit IRC | 00:32 | |
| *** dave-mccowan has quit IRC | 00:35 | |
| *** esberglu has quit IRC | 00:38 | |
| *** Dinesh_Bhor has joined #openstack-nova | 00:40 | |
| *** kumarmn has quit IRC | 00:41 | |
| mnaser | i feel really bad just finding all these issues all at once lol | 00:45 |
| mnaser | i just found a keyerror bug :x | 00:45 |
| mriedem | i don't have a great solution for the flavor thing in the notification. you could disable versioned notifications for now, not sure if you are relying on those. | 00:46 |
| mnaser | mriedem: we're not atm, but i didnt see a config option to disable it afaik | 00:46 |
| mriedem | CONF.notifications.notification_format | 00:47 |
| mriedem | set that to 'unversioned' | 00:47 |
| mriedem | you'll still get the old unversioned notifications which things like ceilometer consume | 00:48 |
| mnaser | mriedem: https://bugs.launchpad.net/nova/+bug/1739323 not sure how far back this goes | 00:50 |
| openstack | Launchpad bug 1739323 in OpenStack Compute (nova) "KeyError in host_manager for _get_host_states" [Undecided,New] | 00:50 |
| mnaser | looks like its different in stable/newton but ocata+ are probably affected | 00:51 |
| mnaser | i dont understand why its iterating it back.. is it trying to return a set? | 00:51 |
| mriedem | can you put a stacktrace in the bug? | 00:52 |
| mnaser | sure | 00:52 |
| mriedem | this is all really old i think | 00:52 |
| mriedem | https://github.com/openstack/nova/commit/b0582603a401b2e71348e9a862b83acc426097df | 00:53 |
| openstackgerrit | Merged openstack/nova master: Improve error message on invalid BDM fields https://review.openstack.org/527915 | 00:53 |
| mriedem | been around since grizzly | 00:53 |
| mnaser | mriedem: https://github.com/openstack/nova/commit/4660333d0d97d8e00cf290ea1d4ed932f5edc1dc#diff-978b9f8734365934eaf8fbb01f11a7d7L624 this is what broke it though | 00:54 |
| *** tuanla____ has joined #openstack-nova | 00:54 | |
| mnaser | before it would return it as is.. which means that the KeyError didnt matter.. now it loops over it for some reason | 00:54 |
| mriedem | oh nice catch, that's ocata | 00:55 |
| mnaser | ok so six.itervalues => Returns an iterator over dictionary‘s values. | 00:55 |
| mriedem | i thought you were saying the KeyError was this | 00:55 |
| mriedem | del self.host_state_map[state_key] | 00:55 |
| mnaser | no sorry | 00:55 |
| mnaser | return (self.host_state_map[host] for host in seen_nodes) | 00:55 |
| mnaser | why not just "return self.host_state_map[host]" i dont know | 00:56 |
| mnaser | err i mean "return self.host_state_map" | 00:56 |
| mriedem | not sure, i don't remember why that change was necessary, would have to ask bauzas | 00:56 |
| mriedem | but it's clearly wrong | 00:56 |
| mnaser | this only took 41 patchsets to merge | 00:57 |
| mnaser | lol | 00:57 |
| mnaser | https://review.openstack.org/#/c/417961/ | 00:57 |
| mriedem | yeah man, and i was in i think cancun at the time on vacation | 00:58 |
| *** itlinux has joined #openstack-nova | 00:59 | |
| mriedem | https://review.openstack.org/#/c/417961/41/nova/scheduler/filter_scheduler.py@100 was also wrong | 01:00 |
| *** Dinesh_Bhor has quit IRC | 01:01 | |
| mnaser | now let me disable versioned notification and see if that gets things moving | 01:01 |
| *** phuongnh has joined #openstack-nova | 01:06 | |
| *** salv-orlando has joined #openstack-nova | 01:06 | |
| *** huanxie has quit IRC | 01:06 | |
| *** huanxie has joined #openstack-nova | 01:07 | |
| mnaser | unversioned notifications did the trick mriedem. that lets me delete the weird vms and live migrate the others. thanks so much for your time, ill track those bugs and hopefully we can get them merged/backported | 01:10 |
| *** salv-orlando has quit IRC | 01:11 | |
| *** Dinesh_Bhor has joined #openstack-nova | 01:11 | |
| mriedem | mnaser: can you report a bug for the flavor versioned notification thing? | 01:12 |
| mnaser | mriedem: oh sure | 01:12 |
| *** dave-mccowan has joined #openstack-nova | 01:12 | |
| *** yangyapeng has joined #openstack-nova | 01:12 | |
| jaypipes | mriedem, mnaser: just getting back to the house after dinner... have you figured out what was up? | 01:15 |
| mriedem | jaypipes: https://review.openstack.org/#/q/topic:bug/1739318+(status:open+OR+status:merged) | 01:15 |
| mriedem | latent bug from newton bites us in the ass starting in pike | 01:16 |
| mnaser | mriedem: fyi https://bugs.launchpad.net/nova/+bug/1739325 | 01:18 |
| openstack | Launchpad bug 1739325 in OpenStack Compute (nova) "Server operations fail to complete with versioned notifications" [Undecided,New] | 01:18 |
| mnaser | also thanks for asking jaypipes but yeah i think we're good .. for now, going to wait to get that backported so we can upgrade to latest stable/pike | 01:18 |
| mriedem | i can feel my wife getting ready to yell at me for still being in my office | 01:19 |
| *** asettle has quit IRC | 01:19 | |
| mnaser | lol | 01:20 |
| mnaser | i still have some gift shopping to do.. | 01:20 |
| mnaser | but i guess that's not happening tonight | 01:20 |
| mnaser | though not looking forward to how much snow is dumped on my car since 8am today and cleaning all that | 01:20 |
| *** yamamoto has joined #openstack-nova | 01:21 | |
| *** yamamoto has quit IRC | 01:21 | |
| mriedem | we're virtually snow free in SE minnesota | 01:21 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194 | 01:21 |
| mriedem | mnaser: so this is a start on that versioned notification bug ^ | 01:21 |
| mriedem | super hacky though i think, | 01:21 |
| mriedem | but would need other input, likely from dansmith on that one | 01:21 |
| mriedem | no idea how we could get in that situation to begin with | 01:22 |
| *** asettle has joined #openstack-nova | 01:22 | |
| mnaser | yeah i tried looking everywhere and i couldnt identify a source of how it ended up happening unfortunately | 01:22 |
| *** takashin has quit IRC | 01:22 | |
| mnaser | it was def not converted because the creation date matched the vm launch date | 01:22 |
| *** annp has joined #openstack-nova | 01:23 | |
| *** psachin has joined #openstack-nova | 01:24 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Default some flavor notification payload fields https://review.openstack.org/529194 | 01:25 |
| *** Apoorva_ has joined #openstack-nova | 01:26 | |
| mriedem | as for https://bugs.launchpad.net/nova/+bug/1739323 i don't think ill be able to crack that tonight | 01:27 |
| openstack | Launchpad bug 1739323 in OpenStack Compute (nova) pike "KeyError in host_manager for _get_host_states" [High,Confirmed] | 01:27 |
| mriedem | should be a simple 'in' check or .get() to handle the key not being in there | 01:28 |
| *** psachin has quit IRC | 01:28 | |
| *** Apoorva has quit IRC | 01:29 | |
| mnaser | mriedem: i would have done it but it seemed weird why it was changed so i figured maybe there was a... purpose to the decision | 01:29 |
| mriedem | not sure, kind of why i wanted bauzas to look at it | 01:30 |
| *** Apoorva_ has quit IRC | 01:31 | |
| *** psachin has joined #openstack-nova | 01:32 | |
| *** yamahata has quit IRC | 01:38 | |
| *** huanxie has quit IRC | 01:38 | |
| *** yamahata has joined #openstack-nova | 01:39 | |
| *** huanxie has joined #openstack-nova | 01:41 | |
| *** bhagyashri_s has joined #openstack-nova | 01:42 | |
| *** andreykurilin has quit IRC | 01:43 | |
| *** bhagyashris has quit IRC | 01:46 | |
| *** andreykurilin has joined #openstack-nova | 01:46 | |
| *** claudiub|2 has quit IRC | 01:53 | |
| *** edleafe- has joined #openstack-nova | 01:57 | |
| *** edleafe has quit IRC | 01:57 | |
| *** yamamoto has joined #openstack-nova | 01:57 | |
| *** felipemonteiro has quit IRC | 01:57 | |
| *** salv-orlando has joined #openstack-nova | 02:07 | |
| *** takashin has joined #openstack-nova | 02:07 | |
| *** salv-orlando has quit IRC | 02:11 | |
| *** huanxie has quit IRC | 02:11 | |
| *** karthiks has joined #openstack-nova | 02:16 | |
| *** Dinesh_Bhor has quit IRC | 02:17 | |
| *** huanxie has joined #openstack-nova | 02:20 | |
| *** r-daneel has quit IRC | 02:20 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:28 | |
| *** Dinesh_Bhor has quit IRC | 02:32 | |
| openstackgerrit | Jay Pipes proposed openstack/nova master: allow compute nodes to be associated with host agg https://review.openstack.org/526753 | 02:32 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Remove server group sched filter support caching https://review.openstack.org/529200 | 02:32 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters https://review.openstack.org/529201 | 02:32 |
| *** Dinesh_Bhor has joined #openstack-nova | 02:32 | |
| *** catintheroof has joined #openstack-nova | 02:38 | |
| *** Dinesh_Bhor has quit IRC | 02:38 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:40 | |
| *** Tom-Tom has joined #openstack-nova | 02:41 | |
| *** Dinesh_Bhor has quit IRC | 02:48 | |
| *** calebb has quit IRC | 02:49 | |
| *** huanxie has quit IRC | 02:50 | |
| *** Dinesh_Bhor has joined #openstack-nova | 02:50 | |
| *** catintheroof has quit IRC | 02:53 | |
| *** gmann has quit IRC | 02:55 | |
| *** calebb has joined #openstack-nova | 02:56 | |
| *** huanxie has joined #openstack-nova | 02:56 | |
| *** takashin has left #openstack-nova | 03:00 | |
| *** Dinesh_Bhor has quit IRC | 03:01 | |
| openstackgerrit | Chen Hanxiao proposed openstack/nova master: libvirt: add Linux distribution guest only description for inject_xxx options https://review.openstack.org/528441 | 03:01 |
| *** hshiina has joined #openstack-nova | 03:02 | |
| *** threestrands has quit IRC | 03:03 | |
| *** threestrands has joined #openstack-nova | 03:04 | |
| *** threestrands has quit IRC | 03:04 | |
| *** threestrands has joined #openstack-nova | 03:04 | |
| *** threestrands has quit IRC | 03:05 | |
| *** threestrands has joined #openstack-nova | 03:06 | |
| *** threestrands has quit IRC | 03:06 | |
| *** threestrands has joined #openstack-nova | 03:06 | |
| *** threestrands has quit IRC | 03:07 | |
| *** threestrands has joined #openstack-nova | 03:07 | |
| *** salv-orlando has joined #openstack-nova | 03:07 | |
| *** yamahata has quit IRC | 03:11 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: Remove extensions module https://review.openstack.org/529206 | 03:12 |
| *** salv-orlando has quit IRC | 03:12 | |
| alex_xu | oomichi: ^ remove extensions module entirely | 03:13 |
| *** dave-mccowan has quit IRC | 03:13 | |
| *** vivsoni has quit IRC | 03:15 | |
| *** abhishekk has joined #openstack-nova | 03:23 | |
| *** karthiks has quit IRC | 03:23 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: Remove the LoadedExtensionInfo object https://review.openstack.org/529207 | 03:24 |
| *** Tom-Tom has quit IRC | 03:25 | |
| *** Tom-Tom has joined #openstack-nova | 03:25 | |
| *** huanxie has quit IRC | 03:26 | |
| *** Tom-Tom has quit IRC | 03:29 | |
| *** andreas_s has joined #openstack-nova | 03:29 | |
| *** tinwood has quit IRC | 03:30 | |
| *** tinwood has joined #openstack-nova | 03:31 | |
| *** artom has quit IRC | 03:31 | |
| *** artom has joined #openstack-nova | 03:31 | |
| *** huanxie has joined #openstack-nova | 03:32 | |
| *** andreas_s has quit IRC | 03:33 | |
| *** edmondsw has joined #openstack-nova | 03:44 | |
| *** Apoorva has joined #openstack-nova | 03:44 | |
| *** gongysh has joined #openstack-nova | 03:48 | |
| *** edmondsw has quit IRC | 03:49 | |
| *** gyee has quit IRC | 03:51 | |
| *** owalsh_ has joined #openstack-nova | 03:55 | |
| openstackgerrit | blkart proposed openstack/nova master: Force delete an instance in any vm_state/task_state https://review.openstack.org/529212 | 03:56 |
| *** udesale has joined #openstack-nova | 03:57 | |
| *** threestrands_ has joined #openstack-nova | 03:57 | |
| *** threestrands has quit IRC | 03:57 | |
| *** threestrands_ has quit IRC | 03:58 | |
| *** zhurong has joined #openstack-nova | 03:58 | |
| *** threestrands_ has joined #openstack-nova | 03:59 | |
| *** owalsh has quit IRC | 03:59 | |
| *** crushil has joined #openstack-nova | 04:02 | |
| *** penick has joined #openstack-nova | 04:02 | |
| *** huanxie has quit IRC | 04:02 | |
| *** diga has joined #openstack-nova | 04:03 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:06 | |
| *** penick has quit IRC | 04:07 | |
| *** penick_ has joined #openstack-nova | 04:07 | |
| *** huanxie has joined #openstack-nova | 04:08 | |
| *** yamamoto_ has joined #openstack-nova | 04:08 | |
| *** salv-orlando has joined #openstack-nova | 04:08 | |
| *** yamamoto has quit IRC | 04:11 | |
| *** salv-orlando has quit IRC | 04:13 | |
| *** namnh has joined #openstack-nova | 04:14 | |
| *** liverpooler has quit IRC | 04:14 | |
| openstackgerrit | Alex Xu proposed openstack/nova master: Merge ResourceV21 obj into Resource obj https://review.openstack.org/529217 | 04:22 |
| openstackgerrit | Alex Xu proposed openstack/nova master: Remove the inherits parameter for the Resource object https://review.openstack.org/529218 | 04:22 |
| *** Tom-Tom has joined #openstack-nova | 04:30 | |
| *** kumarmn has joined #openstack-nova | 04:33 | |
| *** mlavalle has joined #openstack-nova | 04:35 | |
| *** Tom-Tom has quit IRC | 04:35 | |
| *** Dinesh_Bhor has quit IRC | 04:37 | |
| *** mlavalle has quit IRC | 04:38 | |
| *** huanxie has quit IRC | 04:38 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:42 | |
| *** huanxie has joined #openstack-nova | 04:42 | |
| *** liuzz has joined #openstack-nova | 04:44 | |
| *** armax has quit IRC | 04:44 | |
| *** Dinesh_Bhor has quit IRC | 04:46 | |
| *** armax has joined #openstack-nova | 04:51 | |
| *** armax has quit IRC | 04:51 | |
| *** Dinesh_Bhor has joined #openstack-nova | 04:53 | |
| *** Dinesh_Bhor has quit IRC | 04:56 | |
| *** ianw is now known as ianw_pto | 04:57 | |
| *** links has joined #openstack-nova | 04:57 | |
| *** Dinesh_Bhor has joined #openstack-nova | 05:01 | |
| *** kumarmn has quit IRC | 05:02 | |
| *** felipemonteiro has joined #openstack-nova | 05:03 | |
| *** Dinesh_Bhor has quit IRC | 05:06 | |
| *** Tom-Tom has joined #openstack-nova | 05:08 | |
| *** salv-orlando has joined #openstack-nova | 05:09 | |
| *** yamamoto_ has quit IRC | 05:10 | |
| *** huanxie has quit IRC | 05:12 | |
| *** salv-orlando has quit IRC | 05:15 | |
| *** moshele has joined #openstack-nova | 05:15 | |
| *** huanxie has joined #openstack-nova | 05:16 | |
| *** penick has joined #openstack-nova | 05:18 | |
| *** namnh has quit IRC | 05:19 | |
| *** penick_ has quit IRC | 05:19 | |
| *** Dinesh_Bhor has joined #openstack-nova | 05:27 | |
| *** Dinesh_Bhor has quit IRC | 05:31 | |
| *** edmondsw has joined #openstack-nova | 05:32 | |
| *** yamamoto has joined #openstack-nova | 05:32 | |
| *** edmondsw has quit IRC | 05:37 | |
| *** AlexeyAbashkin has joined #openstack-nova | 05:37 | |
| *** sridharg has joined #openstack-nova | 05:38 | |
| *** penick_ has joined #openstack-nova | 05:38 | |
| *** yangyapeng has quit IRC | 05:40 | |
| *** yangyapeng has joined #openstack-nova | 05:41 | |
| *** penick has quit IRC | 05:41 | |
| *** AlexeyAbashkin has quit IRC | 05:42 | |
| *** jichen has joined #openstack-nova | 05:42 | |
| jichen | mriedem: hi, canyou please help to take a look at https://review.openstack.org/#/c/523387/ ? | 05:43 |
| *** yangyapeng has quit IRC | 05:45 | |
| *** huanxie has quit IRC | 05:46 | |
| *** crushil has quit IRC | 05:47 | |
| *** kumarmn has joined #openstack-nova | 05:50 | |
| *** Apoorva has quit IRC | 05:50 | |
| *** huanxie has joined #openstack-nova | 05:53 | |
| *** kumarmn has quit IRC | 05:55 | |
| *** Dinesh_Bhor has joined #openstack-nova | 05:56 | |
| *** gouthamr has quit IRC | 05:57 | |
| *** Dinesh_Bhor has quit IRC | 06:00 | |
| *** yangyapeng has joined #openstack-nova | 06:03 | |
| *** penick_ has quit IRC | 06:04 | |
| *** penick has joined #openstack-nova | 06:05 | |
| *** Dinesh_Bhor has joined #openstack-nova | 06:05 | |
| *** salv-orlando has joined #openstack-nova | 06:10 | |
| openstackgerrit | Chen Hanxiao proposed openstack/nova master: trivial: conf: libvirt: fix a typo https://review.openstack.org/529229 | 06:12 |
| *** Dinesh_Bhor has quit IRC | 06:15 | |
| *** salv-orlando has quit IRC | 06:15 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Imported Translations from Zanata https://review.openstack.org/524795 | 06:16 |
| *** Dinesh_Bhor has joined #openstack-nova | 06:20 | |
| *** Tom-Tom has quit IRC | 06:21 | |
| *** Tom-Tom has joined #openstack-nova | 06:24 | |
| *** salv-orlando has joined #openstack-nova | 06:28 | |
| *** huanxie has quit IRC | 06:29 | |
| *** Dinesh_Bhor has quit IRC | 06:30 | |
| *** diga has quit IRC | 06:31 | |
| *** huanxie has joined #openstack-nova | 06:32 | |
| *** Dinesh_Bhor has joined #openstack-nova | 06:34 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add migration db and object pagination support. https://review.openstack.org/514904 | 06:38 |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add cross cell sort support for get_migrations https://review.openstack.org/517273 | 06:38 |
| *** felipemonteiro has quit IRC | 06:38 | |
| *** felipemonteiro has joined #openstack-nova | 06:39 | |
| *** trungnv has joined #openstack-nova | 06:39 | |
| openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Add pagination and Changes-since filter support for os-migrations. https://review.openstack.org/330406 | 06:42 |
| openstackgerrit | Minho Ban proposed openstack/nova master: prevent removing numa_topology usage info from DB https://review.openstack.org/529236 | 06:47 |
| *** yamamoto has quit IRC | 06:49 | |
| *** karthiks has joined #openstack-nova | 06:49 | |
| *** janki has joined #openstack-nova | 06:49 | |
| *** felipemonteiro has quit IRC | 06:50 | |
| *** kumarmn has joined #openstack-nova | 06:51 | |
| *** karthiks has quit IRC | 06:54 | |
| *** kumarmn has quit IRC | 06:55 | |
| *** andreas_s has joined #openstack-nova | 06:56 | |
| *** threestrands_ has quit IRC | 06:57 | |
| *** huanxie has quit IRC | 07:02 | |
| *** yamamoto has joined #openstack-nova | 07:04 | |
| *** karthiks has joined #openstack-nova | 07:06 | |
| *** rcernin has quit IRC | 07:08 | |
| *** huanxie has joined #openstack-nova | 07:08 | |
| *** yamamoto has quit IRC | 07:09 | |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | 07:09 |
| *** Dinesh_Bhor has quit IRC | 07:09 | |
| *** Dinesh_Bhor has joined #openstack-nova | 07:13 | |
| *** penick has quit IRC | 07:14 | |
| *** Dinesh_Bhor has quit IRC | 07:14 | |
| *** penick has joined #openstack-nova | 07:16 | |
| *** gongysh has quit IRC | 07:17 | |
| *** claudiub|2 has joined #openstack-nova | 07:19 | |
| *** yamamoto has joined #openstack-nova | 07:20 | |
| *** edmondsw has joined #openstack-nova | 07:21 | |
| *** Brin has quit IRC | 07:23 | |
| *** liuzz has quit IRC | 07:23 | |
| *** yamamoto has quit IRC | 07:24 | |
| *** edmondsw has quit IRC | 07:25 | |
| openstackgerrit | TommyLike proposed openstack/nova master: Remove redundant try/except block when authorize https://review.openstack.org/528991 | 07:25 |
| openstackgerrit | 龚肖 proposed openstack/nova master: compute: Catch binding failed exception while init host https://review.openstack.org/529249 | 07:32 |
| *** huanxie has quit IRC | 07:38 | |
| *** sapd has quit IRC | 07:40 | |
| *** sapd has joined #openstack-nova | 07:41 | |
| *** gongysh has joined #openstack-nova | 07:42 | |
| *** huanxie has joined #openstack-nova | 07:44 | |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | 07:44 |
| openstackgerrit | guanzuoyu proposed openstack/nova-specs master: Fix an spelling error of driver https://review.openstack.org/529251 | 07:47 |
| *** vladikr has joined #openstack-nova | 07:49 | |
| openstackgerrit | Chen Hanxiao proposed openstack/nova master: InstanceInfo: store more runtime information of instance https://review.openstack.org/526348 | 07:50 |
| *** kumarmn has joined #openstack-nova | 07:51 | |
| openstackgerrit | Marc Koderer proposed openstack/nova master: Enhance error handling for InvalidBDMVolume https://review.openstack.org/529254 | 07:53 |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: Fix duplicate allocation candidates https://review.openstack.org/528591 | 07:56 |
| *** kumarmn has quit IRC | 07:57 | |
| *** mkoderer__ has joined #openstack-nova | 08:01 | |
| *** yamamoto has joined #openstack-nova | 08:04 | |
| *** sahid has joined #openstack-nova | 08:08 | |
| *** yangyapeng has quit IRC | 08:09 | |
| *** yamamoto has quit IRC | 08:09 | |
| *** yangyapeng has joined #openstack-nova | 08:11 | |
| *** hoangcx has quit IRC | 08:12 | |
| *** hoangcx has joined #openstack-nova | 08:13 | |
| *** Tom-Tom has quit IRC | 08:14 | |
| *** Tom-Tom has joined #openstack-nova | 08:15 | |
| *** huanxie has quit IRC | 08:16 | |
| openstackgerrit | Minho Ban proposed openstack/nova master: prevent removing numa_topology usage info from DB https://review.openstack.org/529236 | 08:16 |
| *** zhurong has quit IRC | 08:17 | |
| maciejjozefczyk | mriedem: jaypipes thanks for https://review.openstack.org/#/c/520024/ | 08:17 |
| *** tuanla____ has quit IRC | 08:18 | |
| *** hieulq has quit IRC | 08:18 | |
| *** daidv has quit IRC | 08:18 | |
| *** daidv has joined #openstack-nova | 08:19 | |
| *** tuanla____ has joined #openstack-nova | 08:19 | |
| *** AlexeyAbashkin has joined #openstack-nova | 08:19 | |
| *** hieulq has joined #openstack-nova | 08:19 | |
| *** hoonetorg has quit IRC | 08:19 | |
| *** Tom-Tom has quit IRC | 08:19 | |
| *** Tom-Tom has joined #openstack-nova | 08:19 | |
| *** huanxie has joined #openstack-nova | 08:19 | |
| *** eric_yang has joined #openstack-nova | 08:20 | |
| openstackgerrit | 龚肖 proposed openstack/nova master: compute: Catch binding failed exception while init host https://review.openstack.org/529249 | 08:22 |
| *** salv-orlando has quit IRC | 08:22 | |
| *** alexchadin has joined #openstack-nova | 08:26 | |
| *** ralonsoh has joined #openstack-nova | 08:27 | |
| *** sapd has quit IRC | 08:33 | |
| *** hoonetorg has joined #openstack-nova | 08:33 | |
| *** yamamoto has joined #openstack-nova | 08:34 | |
| *** sapd has joined #openstack-nova | 08:35 | |
| *** yamamoto has quit IRC | 08:39 | |
| *** kmalloc has quit IRC | 08:41 | |
| *** damien_r has joined #openstack-nova | 08:43 | |
| *** damien_r has quit IRC | 08:43 | |
| *** damien_r has joined #openstack-nova | 08:43 | |
| *** jpena|off is now known as jpena | 08:44 | |
| *** pprokop has joined #openstack-nova | 08:49 | |
| *** claudiub has joined #openstack-nova | 08:49 | |
| *** yamamoto has joined #openstack-nova | 08:50 | |
| *** mdnadeem has joined #openstack-nova | 08:51 | |
| *** huanxie has quit IRC | 08:52 | |
| *** claudiub|2 has quit IRC | 08:52 | |
| *** kumarmn has joined #openstack-nova | 08:52 | |
| *** yamamoto has quit IRC | 08:54 | |
| *** ragiman has joined #openstack-nova | 08:56 | |
| *** ameeda has joined #openstack-nova | 08:56 | |
| *** yamamoto has joined #openstack-nova | 08:56 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** huanxie has joined #openstack-nova | 08:56 | |
| ameeda | can you please review my code here ? https://review.openstack.org/#/c/528069/ also here https://review.openstack.org/#/c/528385/ | 08:57 |
| *** kumarmn has quit IRC | 08:57 | |
| *** salv-orlando has joined #openstack-nova | 08:57 | |
| *** claudiub|2 has joined #openstack-nova | 08:58 | |
| *** claudiub has quit IRC | 09:01 | |
| *** sbezverk has quit IRC | 09:02 | |
| *** alexchadin has quit IRC | 09:04 | |
| *** alexchadin has joined #openstack-nova | 09:04 | |
| *** yamamoto has joined #openstack-nova | 09:05 | |
| *** vivsoni has joined #openstack-nova | 09:06 | |
| *** sahid_ has joined #openstack-nova | 09:09 | |
| *** hshiina has quit IRC | 09:11 | |
| *** sahid has quit IRC | 09:11 | |
| *** tetsuro_ has quit IRC | 09:15 | |
| *** owalsh_ is now known as owalsh | 09:17 | |
| *** s1061123 has quit IRC | 09:19 | |
| *** ralonsoh has quit IRC | 09:19 | |
| *** s1061123 has joined #openstack-nova | 09:21 | |
| *** s1061123 has quit IRC | 09:22 | |
| *** s1061123 has joined #openstack-nova | 09:23 | |
| *** sahid_ has quit IRC | 09:23 | |
| *** damien_r has quit IRC | 09:25 | |
| *** huanxie has quit IRC | 09:26 | |
| *** huanxie has joined #openstack-nova | 09:30 | |
| *** fragatina has joined #openstack-nova | 09:30 | |
| *** lpetrut has joined #openstack-nova | 09:30 | |
| *** fragatina has quit IRC | 09:31 | |
| *** fragatina has joined #openstack-nova | 09:31 | |
| *** lucas-afk is now known as lucasagomes | 09:34 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Refactor encryptor attach and detach calls https://review.openstack.org/460243 | 09:36 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Introduce disk encryption config classes https://review.openstack.org/464008 | 09:36 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: QEMU native LUKS decryption for volumes https://review.openstack.org/523958 | 09:36 |
| *** sshwarts has joined #openstack-nova | 09:36 | |
| lyarwood | mdbooth: ^ updated the config class change, also if you have time reviews of the WIP change would be great, some of the code in there is horrid but required to keep prior behaviour | 09:37 |
| mdbooth | Did gerrit just die? | 09:38 |
| mdbooth | Was working a second ago | 09:38 |
| lyarwood | yeah I think so | 09:38 |
| lyarwood | Didn't think my change was _that_ bad | 09:38 |
| mdbooth | Hehe | 09:39 |
| mdbooth | It's really not happy about something, though | 09:39 |
| openstackgerrit | Chen Hanxiao proposed openstack/nova master: libvirt: implement NotImplementedError for _set_quiesced https://review.openstack.org/529278 | 09:41 |
| *** bhagyashri_s is now known as bhagyashris | 09:43 | |
| *** bhagyashris is now known as bhagyashri_s | 09:43 | |
| *** bhagyashri_s is now known as bhagyashris | 09:43 | |
| *** derekh has joined #openstack-nova | 09:44 | |
| *** ralonsoh has joined #openstack-nova | 09:45 | |
| *** hui_ has joined #openstack-nova | 09:51 | |
| *** claudiub has joined #openstack-nova | 09:54 | |
| *** pprokop has quit IRC | 09:54 | |
| *** hui has quit IRC | 09:54 | |
| danpawlik | Hi, I have small question: I try to run unit tests on newton, and it raises me an error described here: https://bugs.launchpad.net/nova/+bug/1630420 . My question is: should I cherry-pick patch: https://review.openstack.org/#/c/383524/ to newton or add genisoimage to bindeps? | 09:55 |
| openstack | Launchpad bug 1630420 in OpenStack Compute (nova) "config_drive unit tests (libvirt driver) aren't mocking genisoimage" [Low,Fix released] - Assigned to Diana Clarke (diana-clarke) | 09:55 |
| *** claudiub|2 has quit IRC | 09:56 | |
| openstackgerrit | Daniel Pawlik proposed openstack/nova stable/newton: Patch mkisofs calls https://review.openstack.org/529284 | 09:58 |
| *** huanxie has quit IRC | 10:00 | |
| *** psachin has quit IRC | 10:01 | |
| ebbex | mdbooth: I've created an instance with swap, where both root and swap are on rbd, yet there's a big swap file under nova/instances/_base on the compute-node, and according to the nova-compute logs it get's touched about once every minute (which would prevent _remove_old_enough_file from acting upon it). Is any of this intended behaviour? | 10:01 |
| mdbooth | It's intended behaviour that it exists, although it could be done much better. | 10:02 |
| mdbooth | We initialise a swap device on the compute host, then copy it to rbd iirc | 10:03 |
| mdbooth | It's in _base because that's the cache | 10:03 |
| mdbooth | Caching swap and ephemeral disks is a bug imho | 10:03 |
| mdbooth | Not sure why it's being touched every minute, though | 10:04 |
| mdbooth | Any idea what's touching it? | 10:04 |
| ebbex | 2017-12-20T10:06:22.143079+00:00 oscompute01 sudo: nova : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/opt/openstack/nova/venv/bin/nova-rootwrap /opt/openstack/nova/etc/nova/rootwrap.conf touch -c /opt/openstack/nova/instances/_base/swap_16384 | 10:07 |
| *** huanxie has joined #openstack-nova | 10:08 | |
| *** sahid has joined #openstack-nova | 10:08 | |
| *** sahid_ has joined #openstack-nova | 10:09 | |
| *** sahid_ has quit IRC | 10:09 | |
| mdbooth | ebbex: Ah... IIRC there's a periodic task which touches cache files of active instances to prevent them from being deleted. | 10:09 |
| mdbooth | But of course that's irrelevant for rbd. | 10:09 |
| mdbooth | Another reason that rbd should have its own cache... | 10:09 |
| mdbooth | Also LVM. | 10:10 |
| mdbooth | And flat | 10:10 |
| mdbooth | Everything except qcow2, in fact | 10:10 |
| *** psachin has joined #openstack-nova | 10:10 | |
| ebbex | Super, cause the compute-node has about 120gb storage, while ceph has a couple TBs. And I can see this clog up the disk on compute-node after 5-6 created vms. | 10:12 |
| ebbex | Any idea for a quick fix? Disable the periodic touching perhaps? | 10:13 |
| mdbooth | ebbex: If you found the relevant periodic task you could disable it if images_type != qcow2 | 10:14 |
| mdbooth | As long as you accompanied it with a sufficiently explanatory comment | 10:14 |
| *** jeblair has quit IRC | 10:15 | |
| *** sahid has quit IRC | 10:15 | |
| *** sahid has joined #openstack-nova | 10:15 | |
| * mdbooth hates all this scattered cruft | 10:15 | |
| mdbooth | We should have refactored imagebackend first time we implemented a backend other than qcow2 | 10:15 |
| *** jeblair has joined #openstack-nova | 10:16 | |
| mdbooth | Unfortunately getting sufficient review attention for large refactors is hard, which discourages doing them. | 10:19 |
| *** psachin has quit IRC | 10:20 | |
| ebbex | Yeah, I have yet to understand how everything hangs together, so can't really contribute much there yet. | 10:24 |
| *** Tom-Tom has quit IRC | 10:24 | |
| *** sambetts|afk is now known as sambetts | 10:27 | |
| *** sahid has quit IRC | 10:29 | |
| *** aarefiev has joined #openstack-nova | 10:30 | |
| *** sahid has joined #openstack-nova | 10:31 | |
| *** damien_r has joined #openstack-nova | 10:31 | |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | 10:32 |
| *** jaianshu has joined #openstack-nova | 10:37 | |
| *** huanxie has quit IRC | 10:38 | |
| *** alexchadin has quit IRC | 10:41 | |
| *** alexchadin has joined #openstack-nova | 10:42 | |
| *** gongysh has quit IRC | 10:42 | |
| *** huanxie has joined #openstack-nova | 10:43 | |
| *** sahid has quit IRC | 10:44 | |
| *** sahid has joined #openstack-nova | 10:45 | |
| *** sahid_ has joined #openstack-nova | 10:52 | |
| *** annp has quit IRC | 10:53 | |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | 10:53 |
| *** krtaylor has quit IRC | 10:53 | |
| *** sahid has quit IRC | 10:54 | |
| *** kumarmn has joined #openstack-nova | 10:54 | |
| *** edmondsw has joined #openstack-nova | 10:57 | |
| *** kumarmn has quit IRC | 10:59 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: doc: update supported drivers for cpu topology https://review.openstack.org/529294 | 11:00 |
| *** sahid_ has quit IRC | 11:00 | |
| *** edmondsw has quit IRC | 11:01 | |
| *** dtantsur|afk is now known as dtantsur | 11:02 | |
| lyarwood | mdbooth: FYI I'm dropping the code to handle rbd volumes from the native LUKS change | 11:06 |
| *** krtaylor has joined #openstack-nova | 11:06 | |
| lyarwood | mdbooth: turns out that c-vol will always attempt to attach the os-brick encryptors and thus fail for rbd when creating a volume from an image. | 11:07 |
| *** yamamoto_ has joined #openstack-nova | 11:08 | |
| *** AlexeyAbashkin has quit IRC | 11:08 | |
| *** yamamoto has quit IRC | 11:11 | |
| openstackgerrit | Tetsuro Nakamura proposed openstack/nova master: doc: update supported drivers for cpu topology https://review.openstack.org/529294 | 11:11 |
| *** gcb has quit IRC | 11:12 | |
| *** tuanla____ has quit IRC | 11:14 | |
| *** jichen has quit IRC | 11:14 | |
| *** huanxie has quit IRC | 11:14 | |
| *** huanxie has joined #openstack-nova | 11:16 | |
| *** abhishekk has quit IRC | 11:21 | |
| *** moshele has quit IRC | 11:22 | |
| *** priteau has joined #openstack-nova | 11:24 | |
| *** gszasz has joined #openstack-nova | 11:25 | |
| *** andreas__ has joined #openstack-nova | 11:30 | |
| *** vivsoni has quit IRC | 11:31 | |
| *** andreas__ has quit IRC | 11:31 | |
| *** andreas_s has quit IRC | 11:33 | |
| *** alexchadin has quit IRC | 11:33 | |
| *** szaher has quit IRC | 11:36 | |
| *** alexchadin has joined #openstack-nova | 11:39 | |
| *** psachin has joined #openstack-nova | 11:40 | |
| *** nore_rabel has joined #openstack-nova | 11:46 | |
| *** huanxie has quit IRC | 11:46 | |
| *** alexchadin has quit IRC | 11:49 | |
| *** sahid has joined #openstack-nova | 11:51 | |
| *** AlexeyAbashkin has joined #openstack-nova | 11:52 | |
| *** kumarmn has joined #openstack-nova | 11:55 | |
| *** huanxie has joined #openstack-nova | 11:56 | |
| *** phuongnh has quit IRC | 11:56 | |
| *** cdent has joined #openstack-nova | 11:57 | |
| *** mingyu has joined #openstack-nova | 11:58 | |
| *** eric_yang has quit IRC | 11:58 | |
| *** slunkad has quit IRC | 11:59 | |
| *** salv-orl_ has joined #openstack-nova | 11:59 | |
| *** kumarmn has quit IRC | 12:00 | |
| *** smatzek has joined #openstack-nova | 12:01 | |
| *** slunkad has joined #openstack-nova | 12:02 | |
| *** kumarmn has joined #openstack-nova | 12:02 | |
| *** priteau has quit IRC | 12:03 | |
| *** salv-orlando has quit IRC | 12:03 | |
| *** priteau has joined #openstack-nova | 12:03 | |
| *** claudiub has quit IRC | 12:03 | |
| *** andreas_s has joined #openstack-nova | 12:06 | |
| *** kumarmn has quit IRC | 12:07 | |
| *** priteau has quit IRC | 12:08 | |
| *** andreas_s has quit IRC | 12:11 | |
| *** andreas_s has joined #openstack-nova | 12:11 | |
| *** dave-mccowan has joined #openstack-nova | 12:13 | |
| *** alexchadin has joined #openstack-nova | 12:15 | |
| jianghuaw_ | bauzas, jaypipes: Hi you may be interested on the new *test-vgpu* CI job added in "Citrix XenServer CI": https://review.openstack.org/#/c/516899/ | 12:15 |
| jianghuaw_ | basically it's monitoring changes on some files in openstack/nova and openstack/os-xenapi and run vGPU test in devstack env. | 12:16 |
| jianghuaw_ | bauzas, do you have time to review the above patch sometime today? Once it's merged, we can make the above test-vgpu CI job to be a formal test for XenAPI. | 12:18 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Refactor encryptor attach and detach calls https://review.openstack.org/460243 | 12:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Introduce disk encryption config classes https://review.openstack.org/464008 | 12:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP libvirt: QEMU native LUKS decryption for volumes https://review.openstack.org/523958 | 12:22 |
| *** moshele has joined #openstack-nova | 12:23 | |
| *** huanxie has quit IRC | 12:26 | |
| *** lucasagomes is now known as lucas-hungry | 12:26 | |
| *** huanxie has joined #openstack-nova | 12:28 | |
| *** aarefiev has quit IRC | 12:33 | |
| *** crushil has joined #openstack-nova | 12:35 | |
| *** gcb has joined #openstack-nova | 12:39 | |
| jaypipes | jianghuaw_: excellent. :) | 12:39 |
| *** crushil has quit IRC | 12:42 | |
| cdent | jaypipes: can you confirm that gerrit is being a big sow in the wallow down back the holler? | 12:43 |
| *** liuyulong has joined #openstack-nova | 12:43 | |
| *** edmondsw has joined #openstack-nova | 12:45 | |
| *** felipemonteiro has joined #openstack-nova | 12:46 | |
| *** gcb has quit IRC | 12:47 | |
| jaypipes | cdent: confirmed. | 12:47 |
| jaypipes | cdent: it doesn't just hate you. | 12:47 |
| cdent | thanks | 12:48 |
| *** edmondsw has quit IRC | 12:49 | |
| *** felipemonteiro has quit IRC | 12:50 | |
| *** jpena is now known as jpena|lunch | 12:58 | |
| *** kumarmn has joined #openstack-nova | 13:03 | |
| *** kumarmn has quit IRC | 13:08 | |
| *** yangyapeng has quit IRC | 13:08 | |
| *** yangyapeng has joined #openstack-nova | 13:09 | |
| mnaser | and i think someone just emailed the ml about gerrit issues ^ | 13:09 |
| *** yangyapeng has quit IRC | 13:13 | |
| *** openstackgerrit has quit IRC | 13:13 | |
| -openstackstatus- NOTICE: gerrit is being restarted due to extreme slowness | 13:14 | |
| jaypipes | gibi: https://review.openstack.org/#/c/528591/ is ready to go, IMHO. | 13:20 |
| jaypipes | jianghuaw_: I'm not seeing in the test-vgpu logs where a flavor is being modified to include a request for resources:VGPU=1? | 13:23 |
| jaypipes | jianghuaw_: erp, never mind. found it :) | 13:24 |
| jaypipes | | flavor:extra_specs | {"resources:VGPU": "1"} | 13:24 |
| jianghuaw_ | jaypipes, yeah. it's set at here "+ ./test_vgpu.sh:main:175 : nova flavor-key 1 set resources:VGPU=1" | 13:25 |
| *** vladikr has quit IRC | 13:26 | |
| jaypipes | jianghuaw_: yep, found it eventually :) | 13:26 |
| jianghuaw_ | :-) | 13:26 |
| *** vladikr has joined #openstack-nova | 13:26 | |
| jaypipes | jianghuaw_: I'm wondering... is there any way to functionally test that the VM (once started) has the vGPU resources properly allocated from the hypervisor? | 13:27 |
| *** alexchadin has quit IRC | 13:27 | |
| jianghuaw_ | the test script will query xapi to verify if the VGPU is created and attached to the VM. | 13:28 |
| *** lucas-hungry is now known as lucasagomes | 13:28 | |
| *** huanxie has quit IRC | 13:29 | |
| jianghuaw_ | jaypipes, + ./test_vgpu.sh:main:214 : result='other-config (MRW): nova-instance-uuid: 64d20778-055a-4b58-a615-87a467caf7a2' | 13:30 |
| jaypipes | jianghuaw_: excellent! | 13:30 |
| jaypipes | jianghuaw_: nice work on this. :) | 13:30 |
| jianghuaw_ | jaypipes, :-) thanks. | 13:31 |
| *** BryanS68 has joined #openstack-nova | 13:32 | |
| jianghuaw_ | jaypipes, what we need is to get another core review on the patch. Hope bauzas will have time to review it. I really don't want to leave it to the new year:-( | 13:33 |
| jianghuaw_ | I mean this patch: https://review.openstack.org/#/c/516899/ | 13:34 |
| *** jaianshu has quit IRC | 13:34 | |
| jaypipes | gibi: ^^ | 13:34 |
| *** yamamoto_ has quit IRC | 13:34 | |
| jianghuaw_ | thanks. | 13:35 |
| gibi | jaypipes: looking... | 13:35 |
| gibi | jaypipes: I was about to +2 https://review.openstack.org/#/c/528591/ when gerrit disappeared | 13:36 |
| *** openstackgerrit has joined #openstack-nova | 13:37 | |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Remove server group sched filter support caching https://review.openstack.org/529200 | 13:37 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters https://review.openstack.org/529201 | 13:37 |
| maciejjozefczyk | Guys, maybe you know if there is similiar thing implemented like https://blueprints.launchpad.net/nova/+spec/boot-order-for-instance | 13:37 |
| maciejjozefczyk | Boot order for instance | 13:38 |
| maciejjozefczyk | Or this idea died in Kilo? | 13:38 |
| jianghuaw_ | gibi, Thanks. Gerrit service was restarted to resolve the slowness issue. it's recovered now. | 13:38 |
| *** pchavva has joined #openstack-nova | 13:39 | |
| gibi | bauzas: is there a chance that you can check https://review.openstack.org/#/c/516899/ today as you already familiar with that patch? If not then I can dig | 13:43 |
| *** rmart04 has joined #openstack-nova | 13:43 | |
| *** edleafe- is now known as edleafe | 13:45 | |
| *** gongysh has joined #openstack-nova | 13:45 | |
| *** yangyapeng has joined #openstack-nova | 13:45 | |
| *** cdent has quit IRC | 13:47 | |
| *** gongysh has quit IRC | 13:49 | |
| *** kumarmn has joined #openstack-nova | 13:49 | |
| *** psachin has quit IRC | 13:49 | |
| *** eharney has joined #openstack-nova | 13:49 | |
| *** mriedem1 has joined #openstack-nova | 13:52 | |
| *** catintheroof has joined #openstack-nova | 13:52 | |
| mriedem1 | bauzas: can you take a look at https://bugs.launchpad.net/nova/+bug/1739323 ? | 13:53 |
| openstack | Launchpad bug 1739323 in OpenStack Compute (nova) pike "KeyError in host_manager for _get_host_states" [High,Confirmed] | 13:53 |
| *** cdent has joined #openstack-nova | 13:53 | |
| mriedem1 | i can't remember why we changed from an iterator there in ocata, but you must have had a reason | 13:53 |
| *** kumarmn has quit IRC | 13:54 | |
| *** jpena|lunch is now known as jpena | 13:54 | |
| *** mriedem has quit IRC | 13:55 | |
| *** claudiub has joined #openstack-nova | 13:55 | |
| *** mriedem1 is now known as mriedem | 13:56 | |
| jaypipes | bauzas, kashyap, mriedem: regarding https://review.openstack.org/#/c/529294/ ... do I remember some patch recently that disabled either CPU pinning or NUMA topology (or both?) for any hypervisor other than KVM? I have this strange feeling that we recently restricted NUMA/CPU pinning to just KVM... | 13:58 |
| gibi | mriedem: I left some suggestion in https://review.openstack.org/#/c/529194/ | 13:59 |
| gibi | mriedem: if you want to discuss then i'm still here for couple of hours today | 13:59 |
| kashyap | jaypipes: Hey, just going to get a piece of bread for lunch | 13:59 |
| mriedem | jaypipes: yes i can dig up the link | 13:59 |
| jaypipes | kashyap: I think you can probably afford more than just a piece of bread for lunch. | 14:00 |
| kashyap | Should be back soon | 14:00 |
| mriedem | jaypipes: https://review.openstack.org/#/c/465160/ | 14:00 |
| jaypipes | kashyap: clearly, Holland has become too expensive. | 14:00 |
| kashyap | jaypipes: Haha, I have the spreads at home; it's the Italian Ciabatta bread :P | 14:00 |
| jaypipes | :) | 14:00 |
| mriedem | "the spreads" | 14:00 |
| mriedem | sounds like "the runs" | 14:00 |
| jaypipes | ah, mriedem that's the ticket. thanks! | 14:00 |
| kashyap | jaypipes: Gentleman, it's called Gent -- http://wikitravel.org/en/Gent | 14:00 |
| kashyap | s/men/man/ | 14:00 |
| jaypipes | heh | 14:00 |
| kashyap | (It's in Flanders, Belgium :P) | 14:00 |
| kashyap | It's late lunch; was debugging the Gate migration bug from hell | 14:01 |
| kashyap | While I'm here; I'll update lyarwood too | 14:02 |
| kashyap | lyarwood: On that migration bug we were talking about | 14:02 |
| kashyap | I spent past 2 hours creating a reproducer -- | 14:02 |
| kashyap | Migrate a guest with a `dd if=/dev/zero bs=1M | md5sum` workload in it, AND throttle the bandwitdh to 1MiB/s | 14:02 |
| stephenfin | jaypipes: Why do you ask? I was waiting on sean-k-mooney to wade in on that again before doing anything, personally | 14:02 |
| kashyap | And also set the clock back 20 minutes on the source | 14:02 |
| *** catintheroof has quit IRC | 14:02 | |
| kashyap | To see if those keepalives will kick in | 14:03 |
| kashyap | s/keepalives/keep-alive timeout/ | 14:03 |
| jaypipes | stephenfin: mostly I was just looking for a link to that other patch :) I couldn't remember what the commit summary was and couldn't find it on gerrit. | 14:03 |
| kashyap | I couldn't reproduce it so far; the guest migrated slowly after a long while. So, the investigation to be conitnued (assuming it's still a problem for the stable gate) | 14:03 |
| * kashyap now really bbiab | 14:04 | |
| *** catintheroof has joined #openstack-nova | 14:04 | |
| *** liverpooler has joined #openstack-nova | 14:07 | |
| *** logan- has quit IRC | 14:08 | |
| *** yangyapeng has quit IRC | 14:09 | |
| *** yangyapeng has joined #openstack-nova | 14:09 | |
| *** cleong has joined #openstack-nova | 14:10 | |
| *** logan- has joined #openstack-nova | 14:10 | |
| *** yamamoto has joined #openstack-nova | 14:10 | |
| danpawlik | mriedem: thanks for review the patch for mkisofs calls | 14:11 |
| mdbooth | Incoming | 14:11 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | 14:12 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | 14:12 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | 14:12 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | 14:12 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests https://review.openstack.org/527916 | 14:12 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info https://review.openstack.org/529328 | 14:12 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info https://review.openstack.org/529329 | 14:12 |
| *** andreas_s has quit IRC | 14:12 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: console: Provide an RFB security proxy implementation https://review.openstack.org/345399 | 14:13 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544 | 14:13 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529184 | 14:16 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Workaround missing RequestSpec.project_id when moving an instance https://review.openstack.org/529185 | 14:16 |
| mriedem | jaypipes: i had to fix the test for py35 ^ otherwise unchanged | 14:16 |
| *** andreas_s has joined #openstack-nova | 14:17 | |
| mriedem | stephenfin: gibi: ^ contains a fix for a regression introduced in newton which breaks us later in pike, | 14:17 |
| mriedem | so i'm hoping to get that backported | 14:17 |
| mriedem | before we eol newton | 14:17 |
| stephenfin | Looking | 14:17 |
| mriedem | the workaround patch will go back to pike, | 14:18 |
| mriedem | the other one can go to newton | 14:18 |
| ameeda | Hello, Can you please review my code here ? https://review.openstack.org/#/c/528069/ | 14:20 |
| *** catinthe_ has joined #openstack-nova | 14:22 | |
| *** yamamoto has quit IRC | 14:23 | |
| stephenfin | ameeda: If you could add a little more context to that, it would make it much easier to review :) | 14:23 |
| *** catintheroof has quit IRC | 14:26 | |
| *** andreas_s has quit IRC | 14:26 | |
| *** jmlowe has quit IRC | 14:28 | |
| *** smatzek has quit IRC | 14:28 | |
| stephenfin | mriedem: Two questions left on https://review.openstack.org/#/c/529185/. I can ignore them if you really want to get it in now (or I've misunderstood something) | 14:29 |
| stephenfin | The first one is a-ok | 14:29 |
| *** smatzek has joined #openstack-nova | 14:29 | |
| *** smatzek has quit IRC | 14:29 | |
| ameeda | stephenfin: thanks for comment, I change the commit and I will submit it now | 14:31 |
| *** andreas_s has joined #openstack-nova | 14:31 | |
| mriedem | stephenfin: replied | 14:32 |
| *** openstackgerrit has quit IRC | 14:33 | |
| stephenfin | Cool. +2 to that too, in that case | 14:34 |
| mriedem | stephenfin: thanks | 14:39 |
| mriedem | mnaser: i'm at a loss for how you're hittin this https://bugs.launchpad.net/nova/+bug/1739323 | 14:39 |
| openstack | Launchpad bug 1739323 in OpenStack Compute (nova) pike "KeyError in host_manager for _get_host_states" [High,Confirmed] | 14:39 |
| mriedem | mnaser: dug through it again and the entries in seen_nodes should all also be in host_state_map because of https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/scheduler/host_manager.py#L698 | 14:40 |
| mriedem | and i don't see anything else in the HostStateManager that modifies that dict, so no locking issues that i can see, unless you have a multiple workers patch or something | 14:40 |
| mriedem | mdbooth: maybe your keen eye can spot an obvious issue in here ^ | 14:41 |
| *** andreas_s has quit IRC | 14:41 | |
| * mdbooth looks | 14:42 | |
| *** kumarmn has joined #openstack-nova | 14:44 | |
| *** andreas_s has joined #openstack-nova | 14:45 | |
| *** openstackgerrit has joined #openstack-nova | 14:50 | |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: detaching volumes when deleted VM is in error state https://review.openstack.org/528069 | 14:50 |
| openstackgerrit | Ameed Ashour proposed openstack/nova master: detach instance volumes when VM creation fails https://review.openstack.org/528385 | 14:50 |
| *** gouthamr has joined #openstack-nova | 14:51 | |
| *** burt has joined #openstack-nova | 14:53 | |
| ameeda | stephenfin: please check the comment and let me know, Thanks | 14:55 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Cleanups for the scheduler code https://review.openstack.org/529343 | 14:57 |
| *** mriedem is now known as mriedem_away | 14:57 | |
| *** Tom-Tom has joined #openstack-nova | 14:59 | |
| mdbooth | mriedem_away: I'm a little lacking in scheduler context. It seems to me that we would hit this if the scheduler was capable of evaluating filters 'concurrently'. | 15:00 |
| mdbooth | For some pythonic/eventlet definition of concurrently, obviously. | 15:01 |
| stephenfin | lyarwood, mdbooth: Think you could take a look at this at some point? BDM'y things https://review.openstack.org/#/c/528069 | 15:01 |
| mdbooth | Does the scheduler serialize all scheduling requests? | 15:01 |
| *** catinthe_ has quit IRC | 15:02 | |
| *** catintheroof has joined #openstack-nova | 15:03 | |
| jaypipes | mdbooth: no | 15:04 |
| *** awaugama has joined #openstack-nova | 15:05 | |
| mdbooth | jaypipes: Thanks. That looks like a bug, then. | 15:05 |
| jaypipes | mdbooth: what's that? | 15:05 |
| mdbooth | https://bugs.launchpad.net/nova/+bug/1739323/ | 15:06 |
| openstack | Launchpad bug 1739323 in OpenStack Compute (nova) "KeyError in host_manager for _get_host_states" [High,Incomplete] | 15:06 |
| jaypipes | ah | 15:06 |
| jaypipes | lemme looksie | 15:06 |
| mdbooth | _get_host_states is returning an iterator over global state | 15:06 |
| *** marst has joined #openstack-nova | 15:06 | |
| mdbooth | So if any 2 'threads' can be calling it simultaneously, it's potentially corrupt | 15:06 |
| openstackgerrit | wes hayutin proposed openstack/nova master: DNM, testing only https://review.openstack.org/529349 | 15:07 |
| mdbooth | Especially as it's returning an iterator, so the evaluation period is extended | 15:07 |
| *** catintheroof has quit IRC | 15:08 | |
| mdbooth | I guess you'd expect to see this if you delete a compute node on a busy system? | 15:08 |
| mdbooth | mnaser: ^^^ ? | 15:08 |
| jaypipes | mdbooth: _get_host_states is returning a tuple, not an iterator. | 15:09 |
| mdbooth | jaypipes: This line: | 15:09 |
| mdbooth | return (self.host_state_map[host] for host in seen_nodes) | 15:09 |
| jaypipes | is a tuple. | 15:09 |
| mdbooth | That's a generator expression, no? | 15:09 |
| jaypipes | I don't believe so... | 15:09 |
| edleafe | jaypipes: it's a generator expression | 15:10 |
| jaypipes | a generator would yield or return a function that yield'd, no? | 15:10 |
| *** Tom-Tom has quit IRC | 15:10 | |
| *** Tom-Tom has joined #openstack-nova | 15:10 | |
| mdbooth | The expression itself is a generator | 15:10 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Add PCI NUMA policy fields https://review.openstack.org/527470 | 15:11 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCI NUMA policies https://review.openstack.org/527472 | 15:11 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: trivial: Modify signature of _filter_non_requested_pfs https://review.openstack.org/527473 | 15:11 |
| edleafe | jaypipes: https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L139-L143 | 15:11 |
| edleafe | it calls it an iterator, which is technically correct, but it's a generator | 15:11 |
| mdbooth | jaypipes: So [foo for foo in foos] is a list comprehension | 15:11 |
| stephenfin | cfriesen, bauzas: Fancy taking a look at ^^^ again? Has been changed quite a lot (hopefully for the better) | 15:11 |
| mdbooth | And (foo for foo in foos) is a generator expression which iterates over the same thing | 15:12 |
| *** mlavalle has joined #openstack-nova | 15:12 | |
| jaypipes | hmm, interesting. | 15:13 |
| mdbooth | So if that last line is returning a generator, it's essentially a closure over seen_nodes, which is local, and self.host_state_map, which is global. | 15:14 |
| *** liuyulong has quit IRC | 15:14 | |
| jaypipes | yeah, I understand that now. Why don't we just list() that instead of returning a generator? | 15:15 |
| mdbooth | Well that would still be a race, just a shorter one | 15:15 |
| * mdbooth has a patch | 15:15 | |
| dtantsur | hey folks! I've filed https://bugs.launchpad.net/nova/+bug/1739440 to track ironic API version negotiation in the virt driver. | 15:15 |
| openstack | Launchpad bug 1739440 in OpenStack Compute (nova) "Ironic virt driver is hardcoding Ironic API microversion" [Undecided,New] | 15:15 |
| dtantsur | do you think it makes sense as a bug or a blueprint or a spec or ...? | 15:16 |
| jaypipes | cool, thanks dtantsur | 15:16 |
| edleafe | jaypipes: in a large deployment, it was feared that the number of hosts returned would be huge | 15:16 |
| jaypipes | dtantsur: up to mriedem_away | 15:16 |
| jaypipes | edleafe: so premature optimization, then. just like 90% of the code in the scheduler.' | 15:16 |
| edleafe | yeah, pretty much | 15:16 |
| dtantsur | jaypipes: how away is mriedem_away? :) | 15:16 |
| jaypipes | dtantsur: no idea :) | 15:16 |
| mdbooth | dtantsur: He wasn't away within the last 30 minutes. | 15:17 |
| edleafe | I don't know if that was a result of real-world problems, or just premature, TBH | 15:17 |
| dtantsur | okay, so not like away-for-the-reminder-of-the-year state of being away :) | 15:17 |
| mdbooth | dtantsur: mriedem_away isn't allowed PTO | 15:17 |
| dtantsur | I see, makes sense :D | 15:18 |
| *** esberglu has joined #openstack-nova | 15:20 | |
| jaypipes | mdbooth: bauzas changed the return of _get_host_states() from an iterator over host_state_map's values to the generator expression that included the local seen_nodes variable here: https://github.com/openstack/nova/commit/4660333d0d97d8e00cf290ea1d4ed932f5edc1dc#diff-978b9f8734365934eaf8fbb01f11a7d7L624 | 15:22 |
| *** jaypipes has left #openstack-nova | 15:23 | |
| *** jaypipes has joined #openstack-nova | 15:23 | |
| *** yamamoto has joined #openstack-nova | 15:23 | |
| jaypipes | mdbooth: might be worth trying to reproduce this race before and after that patch to see if it makes a diff | 15:23 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352 | 15:24 |
| jaypipes | note that I'm not saying anything bad about bauzas' patch! :) I'm just pointing out that's the patch that changed from returning an iterator over the global state to a generator expression over local state | 15:24 |
| mdbooth | jaypipes: Looking | 15:25 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Provide example for placement last-modified header of now https://review.openstack.org/529354 | 15:25 |
| jaypipes | mdbooth: of course, trying to reproduce this reliably is virtually impossible... | 15:25 |
| mdbooth | jaypipes: I've written a couple of unit tests now which play with execution order | 15:25 |
| mdbooth | They're always really hard to read, though | 15:26 |
| jaypipes | yta | 15:26 |
| jaypipes | ya | 15:27 |
| mdbooth | Although as this is a generator it might be pretty easy | 15:28 |
| * mdbooth knocks one up | 15:28 | |
| *** Apoorva has joined #openstack-nova | 15:29 | |
| *** karthiks has quit IRC | 15:32 | |
| *** yamamoto has quit IRC | 15:35 | |
| *** fragatina has quit IRC | 15:37 | |
| mdbooth | jaypipes: mriedem_away Reproduced it | 15:37 |
| *** fragatina has joined #openstack-nova | 15:37 | |
| mdbooth | Not sure if the unit test is worth anything, though, as it's irrelevant with my patch in place | 15:37 |
| *** moshele has quit IRC | 15:38 | |
| mdbooth | You have to create an iterator from the returned list in order to trigger a bug evaluating the iterator, which really doesn't make sense if it doesn't return an iterator | 15:38 |
| mdbooth | http://paste.openstack.org/show/629469/ | 15:40 |
| *** trozet has quit IRC | 15:40 | |
| mdbooth | ^^^ There's the test and its execution, anyway. But I don't think it makes sense to include it if we're not returning an iterator any more. | 15:41 |
| *** trozet has joined #openstack-nova | 15:43 | |
| *** armax has joined #openstack-nova | 15:44 | |
| *** ameeda has quit IRC | 15:47 | |
| *** lyan has joined #openstack-nova | 15:47 | |
| *** eharney has quit IRC | 15:49 | |
| *** pcaruana has joined #openstack-nova | 15:51 | |
| *** janki has quit IRC | 15:53 | |
| openstackgerrit | Jay Pipes proposed openstack/nova master: Remove server group sched filter support caching https://review.openstack.org/529200 | 15:53 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: WIP Support aggregate affinity filters https://review.openstack.org/529201 | 15:53 |
| *** felipemonteiro has joined #openstack-nova | 15:54 | |
| *** smatzek has joined #openstack-nova | 15:55 | |
| *** moshele has joined #openstack-nova | 16:01 | |
| *** ameeda has joined #openstack-nova | 16:02 | |
| *** BryanS68 has quit IRC | 16:07 | |
| *** mriedem_away is now known as mriedem | 16:09 | |
| mriedem | just read the scrollback | 16:12 |
| mriedem | https://review.openstack.org/529343 | 16:12 |
| mriedem | i thought it was a tuple of HostState objects too | 16:13 |
| mriedem | because of the change jaypipes pointed out where we stopped using six.itervalues | 16:13 |
| *** ttsiouts has quit IRC | 16:14 | |
| mriedem | mdbooth: so with your change, we lose any yield behavior on the generator for the 10s of thousands of nodes optimization? | 16:14 |
| *** andreas_s has quit IRC | 16:18 | |
| mnaser | ok reading scrollback | 16:19 |
| mriedem | ah v | 16:19 |
| mriedem | http://intermediatepythonista.com/python-generators#generator-expressions_1 | 16:19 |
| mriedem | ok i'm sorely lacking in this area then | 16:19 |
| *** ragiman has quit IRC | 16:20 | |
| mnaser | ok but forgive me if i'm being unclear but i think the problem is a lot simpler than that | 16:20 |
| mnaser | seen_nodes contains a tuple list of (host, node) of all hosts in the system | 16:20 |
| mnaser | dead_notes contains a tuple list of (host, node) of all dead hosts in the system | 16:20 |
| *** edmondsw has joined #openstack-nova | 16:21 | |
| mnaser | as part of the for loop right before the generator, all dead_nodes are removed from self.host_state_map | 16:21 |
| mnaser | which leaves self.host_state_map with keys of the alive hosts *only* | 16:21 |
| mnaser | the generator loops over all of self.host_state_map but uses seen_nodes as the key (which is all hosts) | 16:21 |
| mnaser | which includes dead nodes, that were removed, in the loop just prior | 16:22 |
| mriedem | mnaser: dead_nodes is the set of entries in host_state_map that aren't also in seen_nodes | 16:22 |
| mriedem | so it's removing existing dead entries from host_state_map | 16:22 |
| mriedem | that's what the set is for | 16:22 |
| *** kumarmn has quit IRC | 16:22 | |
| mnaser | OH | 16:22 |
| mnaser | i see what you mean | 16:22 |
| *** felipemonteiro_ has joined #openstack-nova | 16:22 | |
| *** andreas_s has joined #openstack-nova | 16:22 | |
| mriedem | so i think mdbooth's analysis in the commit message here is correct https://review.openstack.org/#/c/529352/ | 16:22 |
| mriedem | because since we're using a generator (which i didn't realize), | 16:23 |
| mriedem | the host_state_map results could change while processing a request, | 16:23 |
| mriedem | if a compute node is deleted in between, in a busy cloud, such as someone's public cloud | 16:23 |
| *** rmart04 has quit IRC | 16:24 | |
| mnaser | ok but im pretty sure this compute node wasnt deleted, it still exists but it is dead | 16:24 |
| mnaser | let me please double check | 16:24 |
| mriedem | mdbooth: rathre than change it from returning a generator to a list, why not just check if the key is still in the host_state_map before returning the next item? | 16:24 |
| *** felipemonteiro_ has quit IRC | 16:24 | |
| *** rmart04 has joined #openstack-nova | 16:24 | |
| *** felipemonteiro_ has joined #openstack-nova | 16:24 | |
| mriedem | mnaser: maybe the service was deleted? | 16:25 |
| mriedem | you can't delete a compute node through the REST API, but you can delete a service | 16:25 |
| mriedem | https://review.openstack.org/#/c/529352/1/nova/scheduler/host_manager.py@686 | 16:25 |
| *** edmondsw has quit IRC | 16:26 | |
| mnaser | on one conductor i see 40 instances of this with the most recent happening on the 20th | 16:26 |
| *** felipemonteiro has quit IRC | 16:26 | |
| mriedem | so i think a simple test could be you run _get_host_states once with 2 computes and 2 services, then delete one of the services and run _get_host_states again, and iterate the results and it should blow up | 16:26 |
| mnaser | # grep KeyError /var/log/nova/nova-scheduler.log | grep req | wc -l => 20 | 16:27 |
| mnaser | one one of the schedulers | 16:27 |
| mnaser | happened with 9 different compute nodes in the KeyError | 16:29 |
| *** mingyu has quit IRC | 16:29 | |
| mnaser | and i know for a fact some of thoser havent been deleted, especially those a few days ago too | 16:29 |
| mriedem | do those computes have corresponding services table entries? | 16:29 |
| *** moshele has quit IRC | 16:29 | |
| mnaser | mriedem: i mean i see them in 'nova service-list' | 16:29 |
| mnaser | and their updated_at is recent so they're reporting in | 16:29 |
| mnaser | afaik the only way for that service record to go away is... if someone deleted it? | 16:30 |
| mriedem | correct | 16:30 |
| mriedem | we do start auto-disabling computes that fail to build 10 times in a row | 16:31 |
| mriedem | but in the scheduler code here, we include disabled services | 16:31 |
| mnaser | mriedem: i think we disabled that because scheduling bugs would shut down our entire cloud lol | 16:31 |
| mnaser | ex: failing to place numa on server will result in a build fail count being increased | 16:32 |
| *** andreas_s has quit IRC | 16:32 | |
| *** lyan has quit IRC | 16:32 | |
| *** sridharg has quit IRC | 16:37 | |
| *** mingyu has joined #openstack-nova | 16:37 | |
| *** AlexeyAbashkin has quit IRC | 16:37 | |
| mriedem | mdbooth: ok i think we can still use part of your test and maintain the generator | 16:38 |
| mriedem | i'm going to update your patch quick | 16:39 |
| mdbooth | mriedem: on a call, but that's cool | 16:39 |
| *** kumarmn has joined #openstack-nova | 16:39 | |
| *** mingyu has quit IRC | 16:41 | |
| cdent | jaypipes, mriedem: if you're both still around, this is a good and relatively straightforward win for placement behaving in a version discovery world: https://review.openstack.org/#/c/522002/ | 16:42 |
| mdbooth | mriedem: Did you see my unit test, btw? | 16:43 |
| mnaser | mriedem: im not sure if this is relevant or not but do you think the fact we run 3x nova-schedulers could play a role in this .. somehow | 16:43 |
| mriedem | mdbooth: yes i pulled part of the test and applied it to an existing test to recreate the bug | 16:44 |
| mriedem | mnaser: separate processes? | 16:44 |
| mnaser | yes, on seperate machines too | 16:44 |
| mriedem | mnaser: should be ok if it's separate processes because the host_state_map is global to the single process | 16:44 |
| mnaser | (i dont think it would but i figure i would mention it0 | 16:44 |
| *** links has quit IRC | 16:44 | |
| mnaser | oh you know what | 16:44 |
| mnaser | let me see if i can grep the logs for "Removing dead compute node" | 16:44 |
| mnaser | if that helps at all.. | 16:45 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix an error in _get_host_states when deleting a compute node https://review.openstack.org/529352 | 16:46 |
| mriedem | mnaser: btw with placement making claims via the filter scheduler in pike, we actually expect you to be able to run multile scheduler processes safely | 16:46 |
| mriedem | i have a todo to update one of our CI jobs to do that (run with 2 schedulers) | 16:46 |
| mriedem | mdbooth: jaypipes: ^ updated | 16:47 |
| mnaser | mriedem: yeah before we have a little hacky trick of letting things reschedule more often | 16:47 |
| mnaser | because in large volume schedulers all take the same decisions (before at least) | 16:47 |
| mriedem | mnaser: yup, exactly why we're doing claims in the scheduler rather than rely on the compute to fail and reschedule | 16:47 |
| mriedem | mnaser: so in pike, you should be good to remove the core/ram/disk filters, but you'll still need to rely on numa claims in the compute | 16:47 |
| *** lucasagomes is now known as lucas-afk | 16:48 | |
| *** moshele has joined #openstack-nova | 16:48 | |
| openstackgerrit | Ilya Shakhat proposed openstack/nova master: Initialize osprofiler in uWSGI application https://review.openstack.org/519664 | 16:48 |
| mnaser | mriedem: ok i just found a really weird amount of compute nodes being marked as dead in logs | 16:49 |
| *** gibi is now known as gibi_away | 16:50 | |
| mnaser | and i guess they're flapping for some reason and the state of self.host_state_map is always changing with the generator | 16:50 |
| mriedem | mnaser: my guess would be the auto-disable thing if you don't have that disabled globally? | 16:50 |
| mriedem | it's a per-compute config | 16:50 |
| mnaser | or ntp | 16:50 |
| *** andreas_s has joined #openstack-nova | 16:50 | |
| gibi_away | I'm starting my vacation now. I will be back officially on 8th of Jan | 16:50 |
| *** chyka has joined #openstack-nova | 16:50 | |
| mriedem | yeah i guess service timeouts | 16:50 |
| mriedem | gibi_away: NOOOO | 16:50 |
| mnaser | gibi_away: have fun :-) | 16:51 |
| mriedem | gibi_away: enjoy, see you in a few weeks | 16:51 |
| gibi_away | mriedem: I'm trying to look at gerrit time to time but no promises | 16:51 |
| mriedem | don't | 16:51 |
| mnaser | mriedem: so logs from august till now in scheduler ... grep 'Removing dead compute node' /var/log/nova/nova-scheduler.log | wc -l => 102490. | 16:51 |
| mnaser | so um | 16:51 |
| gibi_away | mriedem: see you | 16:51 |
| mnaser | something is wrong there, i guess with all those services going dead, it was creating the race condition | 16:51 |
| mnaser | now why they are dead thats an exercise for me | 16:51 |
| *** moshele has quit IRC | 16:51 | |
| openstackgerrit | rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922 | 16:52 |
| mnaser | some being removed 4 minutes ago.. i think this is ntp related | 16:52 |
| *** rmart04 has quit IRC | 16:52 | |
| mnaser | the conductor is the one that updates the service record.. or is it the compute directly? | 16:52 |
| mnaser | (as in, who decides what time goes in updated_at) | 16:53 |
| mriedem | that's the servicegroup api i think | 16:54 |
| mriedem | https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L56 | 16:55 |
| *** andreas_s has quit IRC | 16:55 | |
| mnaser | so one of the compute nodes that went down 4 minutes ago is 4 seconds ahead | 16:55 |
| mriedem | are you seeing any of the logging messages in here? https://github.com/openstack/nova/blob/3ebc358002d3f62a9ff0a3b0f50fea4f2a5c2c91/nova/servicegroup/drivers/db.py#L86 | 16:55 |
| mnaser | mriedem: no | 16:57 |
| melwitt | mnaser: I agree that it sounds ntp related if you're seeing some computes ahead in time of others | 16:57 |
| mnaser | melwitt: this one that went 'down' was actually 4 seconds ahead though so if anything it would have updated with a time that is more ahead | 16:58 |
| mriedem | oopsy doodle http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-n-super-cond.txt.gz#_Dec_20_15_15_20_744636 | 16:58 |
| mriedem | cdent: ^ | 16:58 |
| *** stephenfin has left #openstack-nova | 17:00 | |
| *** stephenfin has joined #openstack-nova | 17:00 | |
| mriedem | http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-placement-api.txt.gz#_Dec_20_15_15_20_666337 | 17:00 |
| mriedem | File "/opt/stack/new/nova/nova/objects/resource_provider.py", line 2029, in _set_allocations | 17:01 |
| mriedem | nova.exception.ObjectActionError: Object action create failed because: already created | 17:01 |
| *** gyee has joined #openstack-nova | 17:01 | |
| mriedem | https://bugs.launchpad.net/nova/+bug/1739453 | 17:03 |
| openstack | Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] | 17:03 |
| *** priteau has joined #openstack-nova | 17:04 | |
| mriedem | not sure if using POST /allocations would fix this or not | 17:06 |
| melwitt | mnaser, mriedem: it's the Service object that joins the servicegroup, so I think updated_at has to be the time of the host the service is running on, in this case the compute host | 17:06 |
| openstackgerrit | Merged openstack/nova master: [placement] Separate API schemas (allocation_candidate) https://review.openstack.org/528635 | 17:07 |
| *** priteau has quit IRC | 17:08 | |
| *** Tom-Tom has quit IRC | 17:10 | |
| cdent | mriedem: that's rather novel | 17:10 |
| *** imacdonn has quit IRC | 17:11 | |
| *** imacdonn has joined #openstack-nova | 17:12 | |
| mriedem | yeah 2 hits in 7 days | 17:14 |
| *** awaugama has quit IRC | 17:15 | |
| cdent | the code is claiming that that error should only happen when the incoming allocation objects already have an 'id' field | 17:16 |
| *** smatzek has quit IRC | 17:16 | |
| *** mdnadeem has quit IRC | 17:17 | |
| *** sahid has quit IRC | 17:18 | |
| cdent | which will take some digging to figure out how is even possible | 17:18 |
| *** udesale has quit IRC | 17:18 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Move aggregates from report client to ProviderTree https://review.openstack.org/521685 | 17:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Track provider traits in report client https://review.openstack.org/521686 | 17:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Track associated sharing RPs in report client https://review.openstack.org/526539 | 17:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Raise on API errors getting aggregates/traits https://review.openstack.org/526540 | 17:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: ProviderTree.populate_from_iterable https://review.openstack.org/520756 | 17:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Track tree-associated providers in report client https://review.openstack.org/526541 | 17:19 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Add nested resources to server moving tests https://review.openstack.org/527728 | 17:20 |
| *** damien_r has quit IRC | 17:20 | |
| stephenfin | jaypipes: Think there might be misunderstanding going on here https://review.openstack.org/#/c/527472/ | 17:20 |
| stephenfin | Not saying I don't do overly clever things. Just that this isn't one of them ;) | 17:21 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Scheduler[Report]Client.get_provider_tree https://review.openstack.org/521098 | 17:21 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: ComputeDriver.update_provider_tree() https://review.openstack.org/521187 | 17:21 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Use update_provider_tree from resource tracker https://review.openstack.org/520246 | 17:22 |
| *** andreas_s has joined #openstack-nova | 17:22 | |
| *** sbezverk has joined #openstack-nova | 17:25 | |
| cdent | mriedem: I think I figured out what's up, I left a comment on the bug, assigned myself, and will try a fix | 17:26 |
| mriedem | cdent: ah yeah didn't see the retry decorator | 17:27 |
| mriedem | nice catch | 17:27 |
| cdent | I can't decide what to shake it at, my my tiny fist is shaking | 17:28 |
| cdent | s/my my/but my/ | 17:28 |
| mriedem | yeah not sure how to fix that | 17:28 |
| mdbooth | Sorry in advance, had a merge conflict half way up :/ | 17:29 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add uuid column to BlockDeviceMapping https://review.openstack.org/242602 | 17:29 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Make BlockDeviceMapping object support uuid https://review.openstack.org/242603 | 17:29 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add an online migration for BDM.uuid https://review.openstack.org/525599 | 17:29 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: DriverBlockDevice: make subclasses inherit _proxy_as_attr https://review.openstack.org/524167 | 17:29 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Expose BDM uuid to drivers https://review.openstack.org/529037 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Give volume DriverBlockDevice classes a common prefix https://review.openstack.org/526346 | 17:30 |
| *** Apoorva has quit IRC | 17:30 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add DriverLocalImageBlockDevice https://review.openstack.org/526347 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Rename block_device_info_get_root https://review.openstack.org/529028 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add local_root to block_device_info https://review.openstack.org/529029 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Expose driver_block_device fields as attributes https://review.openstack.org/528362 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Pass DriverBlockDevice to driver.attach_volume https://review.openstack.org/528363 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Use real block_device_info data in libvirt tests https://review.openstack.org/527916 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Fix libvirt volume tests passing invalid disk_info https://review.openstack.org/529328 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Pass disk_info dict to libvirt_info https://review.openstack.org/529329 | 17:30 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Local disk serial numbers for the libvirt driver https://review.openstack.org/529380 | 17:30 |
| *** diga has joined #openstack-nova | 17:33 | |
| *** dtantsur is now known as dtantsur|afk | 17:34 | |
| cdent | jaypipes: can you recall why/if it is important that AllocationList.create_all() is side effecty on the objects that it passes in? https://github.com/openstack/nova/blob/master/nova/tests/unit/objects/test_resource_provider.py#L355-L357 and https://github.com/openstack/nova/blob/609ddc2244bf9ba84d66cced52e7d4e2987ec669/nova/objects/resource_provider.py#L2061-L2086 and https://bugs.launchpad.net/nova/+bug/1739453 | 17:36 |
| openstack | Launchpad bug 1739453 in OpenStack Compute (nova) "MigrationsAdminTest fails with NoValidHost because resource claim swap in placement fails with 500" [High,Confirmed] - Assigned to Chris Dent (cdent) | 17:36 |
| *** andreas_s has quit IRC | 17:40 | |
| melwitt | mriedem: FYI the consoles series is ready to go, complete with devstack patch sandwich around it for the tempest vnc tests. stephenfin has gone through it already https://review.openstack.org/#/q/topic:bp/convert-consoles-to-objects+(status:open+OR+status:merged) | 17:41 |
| mriedem | devstack sandwich worked? | 17:41 |
| melwitt | yes. I owe you a cream soda | 17:42 |
| mriedem | awesome | 17:42 |
| *** lpetrut has quit IRC | 17:43 | |
| edleafe | cdent: if the allocation had an id, then it already exists, no? | 17:43 |
| cdent | no, it's being assigned the .id, but then db deadlock happens and the transaction is reverted and started again, but the list of allocs now contains that alloc with an id | 17:44 |
| mdbooth | artom: Do I recall you being involved in device tagging? | 17:44 |
| edleafe | cdent: ah, so it should change the object, but it's missing the object revert? | 17:44 |
| *** andreas_s has joined #openstack-nova | 17:45 | |
| cdent | edleafe: that's one way to look at it. The other way is that we should be side effecty on on method parameters that are list or dict when we have an option to re-enter. So I'm trying to figure out how to change that in the best way | 17:46 |
| cdent | dammit, should _not_ be side effecty | 17:46 |
| mriedem | cdent: right if you copy the allocs list, then retry, you'll likely then hit a duplicate entry error | 17:47 |
| edleafe | ok, I see what you're saying. It's modifying a mutable param | 17:47 |
| cdent | i need to poke at what the db retry handler is actually doing to be sure | 17:48 |
| *** andreas_s has quit IRC | 17:49 | |
| mriedem | maybe we need to move the insert portion to another method and that method has the retry on deadlock handler | 17:50 |
| mriedem | apart from the _set_allocations code that's checking the object id | 17:50 |
| mriedem | but, | 17:51 |
| mriedem | we could still then hit duplicate entry errors i'd think | 17:51 |
| mriedem | any time you're retrying to insert over a list... | 17:51 |
| mriedem | so loop over the objects, then insert separately per object in another method with the deadlock retry on that method? would that work? | 17:52 |
| *** derekh has quit IRC | 17:52 | |
| *** smatzek has joined #openstack-nova | 17:53 | |
| *** sambetts is now known as sambetts|afk | 17:53 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529384 | 17:54 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Workaround missing RequestSpec.project_id when moving an instance https://review.openstack.org/529385 | 17:54 |
| cdent | mriedem: unclear. still trying to get all the pieces to line up. one thing that's breaking my brain is that the retry wrapper is not explicitly saying anything about the current transaction. will keep looking | 17:55 |
| *** pchavva has quit IRC | 17:55 | |
| *** hemna_ has quit IRC | 17:56 | |
| cdent | mriedem: i think you're right though, extracting a method for the individual insert seems right | 17:56 |
| *** sshwarts has quit IRC | 17:57 | |
| mriedem | seems easiest anyway | 17:57 |
| *** sapd_ has joined #openstack-nova | 18:01 | |
| *** sapd has quit IRC | 18:01 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/ocata: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529387 | 18:03 |
| *** gszasz has quit IRC | 18:04 | |
| *** Apoorva has joined #openstack-nova | 18:06 | |
| *** Apoorva has quit IRC | 18:06 | |
| *** Apoorva has joined #openstack-nova | 18:07 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/newton: Use instance.project_id when creating request specs for old instances https://review.openstack.org/529389 | 18:08 |
| *** AlexeyAbashkin has joined #openstack-nova | 18:08 | |
| cdent | mriedem: hmmm, jay's comment from when he added the retry is potentially derailing, he's got a diferent theory of what could be causing the issues: https://github.com/openstack/nova/commit/e001b0538a9d413f864 | 18:08 |
| *** edmondsw has joined #openstack-nova | 18:09 | |
| *** claudiub has quit IRC | 18:09 | |
| mriedem | that likely makes more sense, the inventory thing | 18:10 |
| mriedem | but in that case, the deadlock isn't on the insert, it's on the generation update yeah? https://github.com/openstack/nova/commit/e001b0538a9d413f864#diff-a040eef16b56cc7862fb2a334c757deaR1855 | 18:10 |
| cdent | right, which I'm not sure we have any way of knowing | 18:12 |
| cdent | the easiest fix is just simply not doing alloc.id = lastrowid | 18:12 |
| *** AlexeyAbashkin has quit IRC | 18:13 | |
| mriedem | http://logs.openstack.org/84/529184/2/check/legacy-tempest-dsvm-py35/888d647/logs/screen-placement-api.txt.gz#_Dec_20_15_15_19_616413 | 18:13 |
| mriedem | Dec 20 15:15:19.616413 ubuntu-xenial-citycloud-lon1-0001533915 devstack@placement-api.service[15195]: DEBUG oslo_db.api [None req-d117a878-c8e0-4c3b-a073-848d73575036 service placement] Performing DB retry for function nova.objects.resource_provider.AllocationList._set_allocations {{(pid=15198) wrapper /usr/local/lib/python3.5/dist-packages/oslo_db/api.py:149}} | 18:13 |
| mriedem | at least it gets logged :) | 18:13 |
| *** edmondsw has quit IRC | 18:13 | |
| *** Apoorva has quit IRC | 18:13 | |
| cdent | oh, hmmm | 18:13 |
| cdent | that's saying it is on the insert | 18:13 |
| mriedem | where does it say that? | 18:14 |
| cdent | previous line? | 18:14 |
| mdbooth | artom: Hey, I just added a patch which makes it meaningful to tag a local disk | 18:14 |
| mdbooth | As local disks can now have serial numbers | 18:14 |
| artom | mdbooth, https://review.openstack.org/#/c/529380/? | 18:15 |
| mdbooth | The way disk tagging works, though, is that when deleting disk metadata it 'matches' based on device.serial, which it assumes to be volume_id | 18:15 |
| mdbooth | Although.......... | 18:15 |
| mdbooth | Wait a sec, I might be trying to solve a problem which doesn't exist | 18:15 |
| mdbooth | This is only relevant in detach_volume | 18:15 |
| mdbooth | And as you can't detach a local disk.... | 18:15 |
| mdbooth | There's no problem here | 18:15 |
| mdbooth | artom: That's the one, yeah. | 18:16 |
| mdbooth | artom: Ok, stand down. I don't think this is a thing. | 18:16 |
| mdbooth | However, if you want to review that series ;) | 18:16 |
| artom | Also, if I read your patch correctly, it still uses volume UUID as the serial for volumes | 18:16 |
| artom | So nothing changes there | 18:16 |
| mdbooth | artom: Right, that's correct. | 18:17 |
| artom | Then we should be fine | 18:17 |
| mdbooth | So this will continue to work for detach_volume. | 18:17 |
| mdbooth | But it wouldn't work for detach_ephemeral | 18:17 |
| artom | Let me check whether I included this in the tempest tests | 18:17 |
| mdbooth | ...which doesn't exist. | 18:17 |
| artom | ... | 18:17 |
| artom | which haven't merged yet | 18:17 |
| *** yamahata has joined #openstack-nova | 18:17 | |
| mdbooth | orly? | 18:17 |
| mdbooth | I need to write tempest tests for my stuff. | 18:17 |
| mdbooth | Can you ping me the reviews? | 18:17 |
| artom | mdbooth, they'll merge in time for brexit | 18:17 |
| artom | mdbooth, https://review.openstack.org/#/c/391947/ | 18:18 |
| mdbooth | artom: You mean they're a complete trainwreck with inexplicable momentum marching inevitably towards their doom? | 18:18 |
| artom | Yeah, they only test the attach case | 18:18 |
| artom | I should add the detach case | 18:18 |
| *** gszasz has joined #openstack-nova | 18:19 | |
| artom | mdbooth, isn't that all of openstack? ;) | 18:19 |
| *** gszasz has quit IRC | 18:19 | |
| mdbooth | Ooh, harsh! :) | 18:19 |
| *** Apoorva has joined #openstack-nova | 18:19 | |
| artom | I'm an arse because I care? | 18:19 |
| mdbooth | artom: Ok, looks like that needs a rebase. | 18:21 |
| mdbooth | I'll take a look at that tomorrow, as I also need to write a test which boots with a tagged root/ephemeral/swap disk. | 18:21 |
| openstackgerrit | rahul bardia proposed openstack/python-novaclient master: nova limits ERROR (Exception): Field names must be unique https://review.openstack.org/527922 | 18:23 |
| *** Apoorva has quit IRC | 18:24 | |
| *** awaugama has joined #openstack-nova | 18:27 | |
| artom | mdbooth, sure, thanks :) | 18:29 |
| artom | I'll try to get to yours as well | 18:29 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-status check for ironic flavor migration https://review.openstack.org/527541 | 18:29 |
| *** Apoorva has joined #openstack-nova | 18:34 | |
| *** lyan has joined #openstack-nova | 18:34 | |
| *** pchavva has joined #openstack-nova | 18:39 | |
| *** diga has quit IRC | 18:43 | |
| rybridges | Hey guys I have a quick question. Is there a way to list all instances for a particular user across all projects? | 18:45 |
| rybridges | Is that what openstack server list --user <username> does? | 18:46 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Do not set allocation.id in AllocationList.create_all() https://review.openstack.org/529397 | 18:49 |
| cdent | mriedem: there's ^ a first stab, I went for the simplest thing, for reasons listed in the commit message, gonna dine now. | 18:49 |
| melwitt | rybridges: it appears so, it might require user uuid. and you might also need to pass --all-projects | 18:52 |
| *** Tom-Tom has joined #openstack-nova | 18:52 | |
| *** Tom-Tom has quit IRC | 18:52 | |
| rybridges | ok | 18:53 |
| rybridges | Thanks i will give it a shot! | 18:53 |
| melwitt | let me know if it doesn't work. not that familiar with openstackclient but a glance at the code looks like it should do what you want | 18:55 |
| rybridges | looks like this works: openstack server list --user rybridges --all-projects | 18:56 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass RequestSpec to ConductorTaskAPI.build_instances https://review.openstack.org/515495 | 18:57 |
| rybridges | but this does not: openstack server list --user rybridges | 18:57 |
| rybridges | so ya you need --all-projects | 18:57 |
| melwitt | rybridges: okay, thanks for confirming. without --all-projects it will default to filtering that user under the project making the request | 18:58 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass RequestSpec to ConductorTaskAPI.build_instances https://review.openstack.org/515495 | 18:58 |
| rybridges | that is correct melwitt | 18:59 |
| edleafe | cdent: huh, that was way simpler than I thought it would be. | 19:04 |
| *** salv-orl_ has quit IRC | 19:14 | |
| *** salv-orlando has joined #openstack-nova | 19:14 | |
| *** salv-orlando has quit IRC | 19:19 | |
| cfriesen | maybe odd python style question, but why does nova code use super(ClassName, self).func() rather than super(type(self), self).func()? | 19:25 |
| melwitt | I dunno. is the latter a more common pattern elsewhere? | 19:25 |
| cfriesen | melwitt: well, in python 3 you can just use super(). the nice thing about the second one is that you don't need to update it if you subclass it and copy/paste a snippet of code. | 19:26 |
| *** felipemonteiro__ has joined #openstack-nova | 19:28 | |
| cfriesen | melwitt: heh...some additional googling answered my question. turns out in python2 it needs to be explicit to allow for multiple levels of inheritance. | 19:29 |
| *** pcaruana has quit IRC | 19:29 | |
| melwitt | ah, okay. now I know | 19:29 |
| *** felipemonteiro_ has quit IRC | 19:32 | |
| *** jpena is now known as jpena|off | 19:37 | |
| cdent | edleafe: I could have made it way more complex, but figured, what's the point? | 19:39 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Do not set allocation.id in AllocationList.create_all() https://review.openstack.org/529397 | 19:45 |
| edleafe | cdent: I'm not disagreeing | 19:46 |
| cdent | I know | 19:47 |
| *** pcaruana has joined #openstack-nova | 19:48 | |
| mriedem | edleafe: i think we're pretty close on https://review.openstack.org/#/c/511358/ but i don't know why you had to modify the FilterScheduler in there | 19:50 |
| *** jobewan has joined #openstack-nova | 19:51 | |
| mriedem | i think you might be working around something in the claims code in the conductor build_instances method that i pointed out | 19:52 |
| mriedem | also, i think claim_resources might be the wrong method to call from conductor, seems put_allocations is what we want instead | 19:53 |
| *** pcaruana has quit IRC | 19:54 | |
| cdent | I got things done today, but none of the things I mean to to do, so I'm giving up | 19:55 |
| * cdent waves | 19:55 | |
| *** moshele has joined #openstack-nova | 19:55 | |
| mriedem | that's most of my days | 19:55 |
| cdent | true | 19:55 |
| cdent | but today seemed especially bad. or maybe I just feel guilty for not doing the stuff I said I'd do for efried_cya_jan | 19:56 |
| cdent | anyway | 19:56 |
| cdent | goodnight | 19:56 |
| SamYaple | /win/win 20 | 19:56 |
| *** cdent has quit IRC | 19:56 | |
| *** edmondsw has joined #openstack-nova | 19:57 | |
| *** mkoderer__ has quit IRC | 20:00 | |
| *** burt has quit IRC | 20:01 | |
| *** edmondsw has quit IRC | 20:02 | |
| *** burt has joined #openstack-nova | 20:02 | |
| edleafe | mriedem: the filter scheduler change was because the functional tests turned up an issue: that the Selection objects in the alternates had no allocation_request in them. Since the Selection objects weren't used until this patch, it never showed up | 20:03 |
| mriedem | ummm | 20:03 |
| mriedem | edleafe: then what is this? https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L380 | 20:04 |
| mriedem | i realize the first selected host doesn't have an allocation_request in it https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L348 | 20:04 |
| mriedem | which you did by design because you said conductor wouldn't need it, because we claim on that host during scheduling | 20:04 |
| *** eharney has joined #openstack-nova | 20:04 | |
| mriedem | but the alternates should have the allocation requests | 20:05 |
| *** pcaruana has joined #openstack-nova | 20:06 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: [placement] Enable limiting GET /allocation_candidates https://review.openstack.org/513526 | 20:08 |
| edleafe | mriedem: it looks like when I rebased on your patch, it pulled an older version. I'm not sure how | 20:09 |
| mriedem | also realized just now that the claim logic in conductor won't work for the caching scheduler, which does have alternates but doesn't do claims | 20:09 |
| mriedem | edleafe: ok, it seems like this is easily fixed | 20:09 |
| edleafe | well, maybe no | 20:09 |
| edleafe | yeah, I'll fix it, but I'm curious how that happened | 20:09 |
| mriedem | my guess would just be like you said, rebase wonk | 20:09 |
| *** tbachman has quit IRC | 20:09 | |
| mriedem | when there were lots of patches | 20:10 |
| mriedem | note my latest comment in the conductor manager about the caching scheduler | 20:10 |
| mriedem | we should have a test for that - where we have alternates, but they don't have allocation_request entries | 20:10 |
| mriedem | we still reschedule, but we don't claim | 20:10 |
| edleafe | ok, let me switch gears and start working on that | 20:11 |
| *** smatzek has quit IRC | 20:12 | |
| *** pcaruana has quit IRC | 20:13 | |
| mriedem | i think we should probably run one of our CI jobs with the caching scheduler, but not sure which one | 20:20 |
| mriedem | could be the nova-next job | 20:20 |
| mriedem | if we threw it in an experimental queue job we'd likely break it and not realize | 20:20 |
| *** yangyape_ has joined #openstack-nova | 20:25 | |
| *** yangyapeng has quit IRC | 20:26 | |
| mriedem | https://review.openstack.org/529410 | 20:33 |
| *** lyan has quit IRC | 20:36 | |
| *** links has joined #openstack-nova | 20:41 | |
| *** links has quit IRC | 20:47 | |
| *** tbachman has joined #openstack-nova | 20:47 | |
| *** chyka has quit IRC | 20:48 | |
| edleafe | mriedem: so for the caching scheduler case, I should just assume that if there is no allocaiton_request, that it's fine to build on the host? | 20:48 |
| mriedem | edleafe: i think so yeah | 20:50 |
| mriedem | b/c we still get an alternates list for the caching scheduler | 20:50 |
| mriedem | it just won't have anything to do with placement | 20:50 |
| edleafe | do we get alternates for caching? | 20:50 |
| edleafe | or is it just a single-element list per instance? | 20:51 |
| edleafe | oh wait | 20:51 |
| edleafe | yeah, it still uses the filter scheduler underneath | 20:52 |
| *** felipemonteiro_ has joined #openstack-nova | 20:53 | |
| *** felipemonteiro__ has quit IRC | 20:53 | |
| *** catintheroof has joined #openstack-nova | 20:54 | |
| edleafe | so calling put_allocations() instead of claim_resources() would mean having to hack into the allocation_request to extract the resource provider | 21:02 |
| edleafe | One of the goals of allocation_request was it should be opaque | 21:03 |
| edleafe | Just because we've violated that once doesn't mean we should continue to do so in other cases. | 21:03 |
| edleafe | So what if I added a "don't check for moves" param to claim_resources? | 21:04 |
| *** jmlowe has joined #openstack-nova | 21:04 | |
| *** priteau has joined #openstack-nova | 21:04 | |
| *** smatzek has joined #openstack-nova | 21:07 | |
| mriedem | let me look at what you're talking about | 21:08 |
| *** priteau has quit IRC | 21:09 | |
| *** smatzek has quit IRC | 21:09 | |
| mriedem | edleafe: oh yeah you're right, | 21:10 |
| mriedem | didn't think about the rp_uuid parameter | 21:10 |
| mriedem | nevermind then | 21:10 |
| mriedem | we have the move thing fixed by deleting the allocatoins in compute before casting to the cell conductor to do the reschedule | 21:10 |
| mriedem | good call | 21:10 |
| mriedem | mtreinish: thanks for hitting those backports | 21:11 |
| *** salv-orlando has joined #openstack-nova | 21:16 | |
| mtreinish | mriedem: sure, np | 21:16 |
| mtreinish | I just want to eol newton :) | 21:17 |
| *** salv-orlando has quit IRC | 21:20 | |
| *** catinthe_ has joined #openstack-nova | 21:23 | |
| mriedem | i just stepped in a pile of zaqar in the ML | 21:23 |
| mriedem | did everyone else know something about user-space notifications using zaqar and how nova core is "dead set" against integrating with anything like this? | 21:23 |
| *** vladikr has quit IRC | 21:24 | |
| *** catintheroof has quit IRC | 21:25 | |
| edleafe | mriedem: I know that there is a general understanding among non-Nova teams that Nova won't cooperate unless it helps Nova | 21:26 |
| mriedem | huh | 21:26 |
| mriedem | could it be that we, god forbid, actually have to prioritize some things based on how many people we have to work on and review stuff? | 21:27 |
| mriedem | pretty sure we cooperated on the py35 goal, and the wsgi goal | 21:27 |
| jroll | mriedem: there was a discussion years and years ago about that thing | 21:28 |
| jroll | I assume the answer was "not right now" and that means never | 21:29 |
| mriedem | oh ok | 21:29 |
| mriedem | and by years and years ago, it means the current nova cores are the fuckers that said 'not now' | 21:29 |
| jroll | heh, of course | 21:29 |
| jroll | also I think that depended on the creds thing aka the kevin fox thing | 21:29 |
| mriedem | oh | 21:30 |
| mriedem | ok anything to do with the creds fiasco is over my head | 21:30 |
| jroll | :P | 21:30 |
| *** kumarmn has quit IRC | 21:35 | |
| *** catinthe_ has quit IRC | 21:35 | |
| *** catintheroof has joined #openstack-nova | 21:35 | |
| *** bnemec has quit IRC | 21:36 | |
| *** threestrands_ has joined #openstack-nova | 21:36 | |
| *** cleong has quit IRC | 21:38 | |
| *** catintheroof has quit IRC | 21:40 | |
| *** priteau has joined #openstack-nova | 21:51 | |
| *** kumarmn has joined #openstack-nova | 21:52 | |
| *** priteau has quit IRC | 21:56 | |
| *** takashin has joined #openstack-nova | 21:56 | |
| *** wind has quit IRC | 21:56 | |
| *** wind has joined #openstack-nova | 21:57 | |
| *** kumarmn has quit IRC | 21:57 | |
| *** kumarmn has joined #openstack-nova | 21:57 | |
| *** ralonsoh has quit IRC | 21:58 | |
| *** itlinux_ has joined #openstack-nova | 22:00 | |
| *** Apoorva_ has joined #openstack-nova | 22:01 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete https://review.openstack.org/525231 | 22:04 |
| *** Apoorva has quit IRC | 22:04 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (2) https://review.openstack.org/526263 | 22:04 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in delete (3) https://review.openstack.org/526557 | 22:05 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add sending global request ID in post https://review.openstack.org/526823 | 22:05 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Add functional tests for traits API https://review.openstack.org/524094 | 22:05 |
| *** gouthamr has quit IRC | 22:06 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Fix 500 error while passing 4-byte unicode data https://review.openstack.org/407514 | 22:06 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [placement] Fix getting placement request ID https://review.openstack.org/523606 | 22:06 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: [cellv2] Improve getting BDMs in multiple cells https://review.openstack.org/521400 | 22:06 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Do not set allocation.id in AllocationList.create_all() https://review.openstack.org/529397 | 22:14 |
| *** salv-orlando has joined #openstack-nova | 22:17 | |
| *** chyka has joined #openstack-nova | 22:17 | |
| *** itlinux_ has quit IRC | 22:18 | |
| *** felipemonteiro__ has joined #openstack-nova | 22:20 | |
| *** salv-orlando has quit IRC | 22:22 | |
| *** chyka has quit IRC | 22:22 | |
| *** felipemonteiro_ has quit IRC | 22:24 | |
| *** david-lyle has quit IRC | 22:24 | |
| *** pooja_jadhav has quit IRC | 22:26 | |
| *** awaugama has quit IRC | 22:30 | |
| *** itlinux_ has joined #openstack-nova | 22:30 | |
| *** alee is now known as alee_afk | 22:31 | |
| *** mgariepy has quit IRC | 22:31 | |
| *** rcernin has joined #openstack-nova | 22:32 | |
| *** trozet has quit IRC | 22:32 | |
| *** jappleii__ has joined #openstack-nova | 22:35 | |
| *** mgariepy has joined #openstack-nova | 22:36 | |
| *** burt has quit IRC | 22:36 | |
| *** threestrands_ has quit IRC | 22:36 | |
| oomichi | alex_xu: yeah, I reviewed that. Thanks for your work. The other https://review.openstack.org/#/c/529207 needs to be updated in your patch sequence | 22:40 |
| *** pchavva has quit IRC | 22:46 | |
| *** itlinux_ has quit IRC | 22:54 | |
| *** david-lyle has joined #openstack-nova | 22:56 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Don't try to delete build request during a reschedule https://review.openstack.org/528835 | 23:07 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Make conductor pass and use host_lists https://review.openstack.org/511358 | 23:07 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | 23:07 |
| edleafe | mriedem: ^^ addressed your comments | 23:07 |
| edleafe | and rebased your bottom patch | 23:07 |
| edleafe | sorry | 23:07 |
| mriedem | as long as i don't lose the +2 it's fine :) | 23:09 |
| *** itlinux_ has joined #openstack-nova | 23:09 | |
| edleafe | you're still good :) | 23:13 |
| *** dave-mccowan has quit IRC | 23:17 | |
| *** salv-orlando has joined #openstack-nova | 23:17 | |
| *** salv-orlando has quit IRC | 23:21 | |
| *** mlavalle has quit IRC | 23:23 | |
| *** Apoorva_ has quit IRC | 23:24 | |
| *** felipemonteiro__ has quit IRC | 23:24 | |
| *** felipemonteiro__ has joined #openstack-nova | 23:24 | |
| *** Apoorva has joined #openstack-nova | 23:25 | |
| *** gouthamr has joined #openstack-nova | 23:30 | |
| mriedem | edleafe: so can we split the unnecessary-for-this-patch scheduler thing out of this change and out of the series and just have it on it's own? | 23:32 |
| *** edmondsw has joined #openstack-nova | 23:33 | |
| *** edmondsw has quit IRC | 23:38 | |
| *** felipemonteiro__ has quit IRC | 23:41 | |
| edleafe | mriedem: if I did that, I'd have to also undo the test cleanups that assumed that host_list[0] had an empty allocation_request | 23:41 |
| *** itlinux_ has quit IRC | 23:42 | |
| mriedem | in test_filter_scheduler? | 23:42 |
| mriedem | or test_conductor? | 23:42 |
| edleafe | both | 23:43 |
| edleafe | the test_filter_scheduler stuff could be broken out easily enough, but the conductor tests are more tangled | 23:44 |
| mriedem | https://www.amazon.com/Orangutangled-Sudipta-Bardhan-Quallen/dp/1477847170 | 23:44 |
| mriedem | you said the word of the day | 23:44 |
| mriedem | ahhhhhhhhhhhhhh | 23:44 |
| mriedem | sorry, it's a book i've had to read several times | 23:44 |
| mriedem | alright nevermind then | 23:45 |
| mriedem | just makes it hard to review big changes with unrelated stuff mixed into them | 23:45 |
| mriedem | especially if it's not pointed out in the commit message | 23:45 |
| edleafe | I understand, but it is kinda related :) | 23:46 |
| *** kumarmn has quit IRC | 23:46 | |
| *** felipemonteiro__ has joined #openstack-nova | 23:46 | |
| *** itlinux__ has joined #openstack-nova | 23:46 | |
| mriedem | it's certainly orangutangled | 23:49 |
| *** moshele has quit IRC | 23:56 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!